Audio looping and sequencing

[riban]

We have a MIDI step sequencer based on patterns with the ability to create loops and single-shot sequences. This is quite flexible and there are ways we can improve the workflows to give better access to some of its more complex features.

An often discussed feature that is missing is the ability to loop audio clips in time with the sequencer / transport. This is kind of / almost possible by using an audio player engine and triggering it from the sequencer but there are limitations and flaws and the workflow is sub-optimal. I have been considering how we might provide such a feature in a more integrated way and would love to get our wonderful community’s ideas on the subject.

The concept I have in mind is to allow a user to add an audio clip to ZynPad so that you may trigger either MIDI sequences or Audio clips. A ZynPad pad would trigger an audio clip to play and loop (if configured) in sync with MIDI sequences. There are several technical challenges but also a lot of user workflow considerations. I would like to hear ideas and suggestions on how best to integrate this. Here are some of my ideas:

• Associate each zynpad with a chain
• A pad associated with a MIDI synth chain would present as a MIDI sequence
• A pad associated with an audio player would present as an audio clip launcher
• An audio clip launcher could select which track in an audio file to play / loop
• It would be possible to add more tracks to an audio file (maybe up to a maximum of 8)
• The audio file would be stretched to fit the duration (in beats / bars) configured for the launcher (without changing its pitch)
• The audio player would allow editing of audio
• It might be nice to integrate with online sample / clip providers - if there are any

These ideas include some technical details of implementation. Don’t worry too much about that. The idea of having several clips in one file is based on the current implementation that allows multi-track audio files to be loaded and the tracks to be dynamically selected. We could also try to do the same with separate audio files.

[tunagenes]

I think it would be useful to allow a pad to switch between, and perhaps even combine, MIDI and audio. The pad would store a midi sequence, and a function would be available to ‘play MIDI and record audio’. For purposes of discussion lets call this ‘rendering’. If it supported the use of an external synth, all the better.

So the musician could tweak a sound to their liking and use it, then go back to MIDI and change some parameters-effects-[even which synth is used]-etc and ‘re-render’ the audio to change it.

This isn’t intended to replace the basic function of playing MIDI and audio, just a (hopefully) useful-interesting-fun way to combine and use them.

[riban]

In testing branch we have some enhancements to the audio player that you might want to pay with. It may give some insight into some options for implementation.

BTW join Zynth Club tonight to join in the discussion on this subject.

[riban]

These are some of the ideas for design and user workflow that were discussed at Zynth Club:

• Keep it simple for users, having default config / workflow that provides core functionality
• Avoid artificial limitations and aim for flexibility of technical or workflow constraints
• Assign pads to chains (currently assigned to MIDI channels) so that they continue to behave as MIDI sequences for MIDI instruments but act as audio clip players for audio player chains
• Clip players can be monophonic and offer mutually exclusive trigger and playback from pads assigned to the same player / chain
• Provide better visual / navigational linking between pads and mixer strips (chains)
• Allow selection of audio files from UI, preferably with some form of filtering / categorisation
• Allow upload and assignment of audio files from webconf
• Use existing (and possibly additional) ZynPad play modes to define behaviour of clip playback
• Automatically adjust raw audio to fit tempo without changing pitch
• It may be advantageous to have a playback mode that adjusts pitch / playback speed to match tempo - similar to vari-speed
• Try to automatically define the beats in an audio clip and allow user to manually override
• Allow user to change groove of a clip by moving the beats around, e.g. quantise, swing, trigger from sequence, etc.
• It is likely that users will expect (and certainly deserve) to have some audio clips available in the box which may require effort (from you lovely community?) to create them because many (most / all?) publicly available clips have constraints on redistribution
• The existing audio player (which is far more feature rich than initially envisage) provides many of these features and could be expanded / modified to act as the playback engine for audio clips
• It feels like a fair constraint that each player works at one time signature, i.e. that all clips within a player have a length with a common division, e.g. 4 beats, 8 beats, 12 beats, etc.

We also discussed possible technical implementation that might look something like:

• Single multi-track audio file containing all the clips for an audio player
• Edit decision list (EDL) file describing how the raw audio is interpreted, e.g. how each pad is mapped to what part of which track and where loops occur, etc.
• User may not need to know about such a file and can be presented with a method of adding an audio file / clip to a pad which may trigger the addition of data to the player’s multitrack file
• User may be presented with a view of the track associated with the pad they want to edit
• Multitrack view may be available for those who find it useful / interesting / pretty

[Aethermind]

Thanks @riban for raising the subject.

Looking at the list of features that you are planning to possibly develop, I would say that having a quick back-and-forth route, from a Midi synth chain to its immediate audio rendering (with beat-locked time-stretch synchronization), would be a strong upgrade of the currently available tools.

It would allow for using the Zyn sequencer as a means for relieving the processors from excessive computational workload, in the context of complex synthesiser orchestrations.

Personally, I would also suggest to expose a delay-advance option, in beats and/or measures, for the content of a given pad (be it Midi or audio), in order to avoid the rigidity of a block-based approach to music writing.

For how I appreciate the Berlin School and 1970s-80s electronica, using building blocks in a strictly sequential fashion does not lend itself well to my method of composition.

I plan to delve in the Zynthian sequencer subject, in the coming future, to see if it can be stretched to serve a musical language different from techno, soundscapes or grid-based songwriting.

[tmu]

I agree that IMO it would be best for the majority of users that ZynPad would imitate the Ableton Live + Launchpad/Push combination’s user experience as closely as possible. As far as I understand, that combo pretty much revolutionised the workflow to make and perform sequenced music, and many other manufacturers have since taken note and followed the same UX principles. I own a Launchpad Pro MK3 which I would like to use with my Zynthian V4, ideally reaching similar usability as with Live+Launchpad or Akai MPC devices.

Many thanks for your efforts jofemodo, riban et al, Zynthian is a wonderful project!

[Aethermind]

Hi tmu, and welcome to this lively music tech community!

As for the geometrical orientation of tracks in the Zynseq screen arrangement, I am afraid that I disagree with the Ableton’s vertical approach.

There are two reasons, which I deem founded, behind my perplexity regarding a sequencer with a vertical layout.

First: primordial acts of musicality became encoded as recognisable perceptual phenomena when primitive men started thumping their feet on a plane, following recurring patterns with rudimentary instruments in their hands. Therefore, at a deep cognitive level - hardwired in our interpretation of the surroundings -, music is a metaphor of life expressed through horizontal movement, that is (given the inherent vertical direction of gravity) a spatial progression on the Earth’s surface.

Second: within such psychological premises, while horizontal notation has historically concerned the representation of monodic melody and rhythm, the ensuing superimposition of more voices and instruments has implied that simultaneity, between concurrent sonic gestures, must be appreciated in the vertical direction. This is particularly true for people who have undertaken a formal training in music theory, because they are used to check the coherence between simultaneous acoustic facts, in printed scores, evaluating their relationships vertically (in harmony) and diagonally (in counterpoint).

I do understand that music-makers from different backgrounds, maybe inclined towards layering sounds and drums more instinctively (by direct listening), may find equivalent, or even more natural, the dashboard-like arrangement of the Ableton’s interface. Nevertheless, for those like me with a composer’s mindset, a Zynthian sequencer page definitely arranged in columns would be the final tombstone, on the actual chance to be involved in its active usage!

After all, the front panels of 1970s analog sequencers were (mostly) organized in rows of horizontal switches, and this be obviously said with all respect for musicians who instead prefer the latest trend, of vertically arrayed groove-boxes.

Best

[Nicolaz]

I think that the vertical Zynpad layout is perfect for a live use, expecially using an external controller like Lauchpad. On the contrary, chaining sequences in a sort of song mode can be perfect only in horizontal lanes. Both these schemes are intuitive to me, Just because they serve different purposes.

[Aethermind]

Right, it makes much sense put in this perspective. The optimal layout is clearly down to what a sequencer is meant for, and to the best related workflow. I take it from your words that, within a live scenario a-la-Launchpad+Ableton, the vertical columns work better.

Best

[riban]

I moved some posts here that are kinda more relevant to this topic.

Zynthian currently allows free allocation of pads in a grid layout, i.e. you can lay them out as rows, columns or any other (random) positions. By default the grouping is in vertical columns, i.e. mutually exclusive groups are assigned to each column of pads but a user may change this at will. We have discussed the idea of imposing this as a constraint, similar to how Ableton Live does but as we have seen from this recent discussion, this is not necessarily desirable for all users.

Having a user default (either pre-selected or calculated from recent behaviour) may be beneficial, e.g. a user might decide that the default is to have vertical columns or horizontal rows, etc. and for new scenes to be created with that default layout. Currently each new scene is created as a grid of 4x4 with mutual groups as columns. The default grid size should also be similarly configurable / deduced.

Note that there is a different view called the Arranger that provides an interface for linear song type workflows. We are also considering how a third view might offer a kind of track view: lists of sequences behind each pad.

There are many people who have used Ableton Live and are very familiar and / or content with its workflow. (I am not one of them!) These users may expect Zynthian to behave in the same way and it may be advantageous for us to provide a mode in which it does but there are other users who have a different workflow and indeed, Zynthian’s sequencer is more flexible and maybe should not be constrained to someone else’s workflows, so we may also want modes that provide other workflows and allow more abstract or detailed configuration (for the brave). Of course implementing such flexibility comes at a cost which may be unachievable, hence we should discuss what we want and also what we need. (Yes - there is a difference!!!)

Integration with hardware controllers such as the Novation Launchpad has been added recently with further work to do. (Actually, integration with Akai APC was included a long time ago. The recent work has been to make the integration more configurable to provide wider support.) I don’t want us to slavishly follow Live’s integration though. Just because a device like the LaunchPad is designed to work with Live does not mean it cannot be used in a different way to better integrate with Zynthian’s workflows. Indeed, the current implementation uses the scene buttons to select Zynthian scenes rather than to launch salvos of sequences (Novation scenes). This may not be the final implementation but it is an example of how the hardware may be used differently.

And to loop this discussion back into this topic, we want to update Zynthian to support audio clips and loops within the ZynPad view so want to consider how such workflows are best implemented.

Some recent suggestions from you lovely people include:

• Vertical columns of related clips / sequences
• Non-square grid layouts
• Horizontal song type layout (see Arranger)
• Option to layout based on user’s preference (within reasonable limits)
• Different people use sequencers in different ways to achieve different results so there should be some degree of flexibility that supports different workflows

[jofemodo]

Hi @zynthianers!

I’m really enjoying this discussion and it’s becoming more and more interesting.
Thanks @riban for resuming so nicely.

I would like to add my 2 cents:

First, i must say that i’m specially interested in zynthian as a device for live performance and personally, i would prefer to put the resources in this area instead of trying to create a full-featured DAW that can be used for composing / arranging songs.

Also, i see the ZynPad as a tool for “live” performance.I don’t think it’s thought for arranging songs in a linear way and we should focus in having the best tool for live performance. Arranging is a very different task and it should be done using a different tool (the arranger!).

Regarding the ZynPad itself, i think the more serious limitation we have is the limited display size, that limits the grid size we can use in a comfortable way:

• v4 => max 4x4 (3x3?)
• v5 => max 5x5
• 7 inch display => max 6x6

We should consider having an “scrollable” grid, as this would enable to overcome the current limitation.

Regarding Pattern Editor, we also need fluid scroll & zoom in both axis.

(@riban , the current “redraw all” refresh algorithm (sloooow) must be improved. We should use a bigger-than-display draw area with a moving display-window, so we avoid redrawing when scrolling, etc. I’ve made some tests and this would work like a charm)

The best!

[riban]

I did some initial tests on improving redraw and she this needs resolution. I also agree that zoom & pan in pattern editor could be useful.

I am less convinced of zoom & pan in zynpad. I understand it may provide a more comfortable interface for editing larger layouts that are subsequently driven by hardware controllers but someone using the Zynthian touchscreen to launch sequences / clips probably wants to see all the pads and not have a risk of the screen scrolling, e.g. with drag scroll. I wonder whether these are different workflows or maybe a need to lock the screen or maybe this is self-healing by stint of users configuring a performance grid of the right size, e.g. 4x4 for Zynpad launching.

We may also want to abstract hardware control from UI, i.e. allow the hardware controller to drive scenes that are not displayed in the Zynthian UI. Maybe a user wants to use ZynPad as another launcher as well as external hardware controller or use the mixer or control views whilst using a hardware launcher to navigate different scenes. (The later doesn’t actually require such an abstraction.)

[jofemodo]

@riban

I am less convinced of zoom & pan in zynpad

• Perhaps zoom could auto-back to full display after a while.
• Perhaps zoom could be fixed (no zoom!) so you always see the right grid-size for your display resolution, so you only scroll X-Y

If we go forward with “ableton-like” workflow, we certainly need scroll in zynpad. Each time i think about i’m more and more convinced of following the ableton “way”. It works fucking nicely for live performance!! Also, having a pre-fixed structure for the grid would help to sort ideas when creating a piece.

Anyway, we have to think deeper before deciding …

the best

[Nicolaz]

In my opinion, a simple chaining feature would be very useful playing live, expecially combined with pad playing and as a kind of basetrack when connecting a guitar or another instrument.

[riban]

Hi @Nicolaz! What do you mean by “simple chaining feature”?

[Nicolaz]

Hi @riban. Simply being able to put in a specific order the sequences of every instrument. I thing of a grid with cells to fill. Every cell is a sequence/pattern i choose to put in a position. The cell could also be empty, of course. Every horizontal lane (a chain) is for a single channel. I could interact live entering Zynpad and play with non chained channels and/or just play guitar over the chains. This means that we should have a global play button. Everything should be synced, of course.

[Nicolaz]

This is, more or less, what i mean. The app is Caustic for Android.
Every cell is a pattern, with different colour for every channel. Chains are developed horizontally.

[riban]

Do you mean something like this?

[Nicolaz]

Yes, but this is confusing for me. Why there are four Acoustic piano lanes and four Angrysweep?

[riban]

Because there are four sequences assigned to Acoustic Piano as shown in Zynpad. You can set the quantity of sequences in the bank, add / remove tracks from each sequence, add / remove / move patterns in each track, assign different MIDI channels, etc.

Here is an example of four sequences, each with a single track assigned to different MIDI channels with each sequence individually triggerable:

And here is a similar configuration but with four tracks in a single, triggerable sequence:

I am not saying this is necessarily intutive or optimal. We have already identified that the Arranger tries to provide a lot of functionality and can be confusing.