Latency in Oram Bookworm: fresh measurements!

jofemodo · May 30, 2024, 5:38pm

I’ve been measuring latency in the new Bookworm Oram SD image. The audio was configured at 48KHz with 2 x buffers of 256 frames, what gives an internal latency of 10.66 ms.

EDIT: this is the detailed jackd config:

/usr/bin/jackd -P 70 -t 2000 -s -d alsa -d hw:sndrpihifiberry -S -r 48000 -p 256 -n 2 -X raw

I’ve measured the real audio latency. I mean, the delay a signal suffers between the zynthian’s audio input and audio output connectors. In other words, i’m recording simultaneously the signal reaching the audio input and the signal getting out the audio output connector.

Here some plots for different cases:

Chain Bypass (no audio plugins): 17ms

Pre-fader gain in chain: 17ms

Pre-fader gain in chain => 2 x gains (pre-fader + postfader) in main bus: 17ms

Pre-fader gain in chain1 => prefader gain in chain2 => 2 x gains (pre-fader + postfader) in main bus: 22ms

Pre-fader gain in chain1 => prefader gain in chain2 => prefader gain in chain3 => 2 x gains (pre-fader + postfader) in main bus: 28ms

As you can see, the latency is 17ms for all cases, except when routing from chain to chain, what adds an extra period (5.33ms) each.

From the 17ms “straight” total latency, we have:

Total latency = 17 ms = audio input HW + audio input SYS + internal (jack) latency + audio output SYS + audio output HW

As jack latency is fixed by the frequency and buffer parameters:

internal (jack) latency = 10.66 ms

We could deduce:

HW + SYS latency = 6.34 ms

But it seems to high, so i’m inclined to think that zynthian processing (mixer??) is using an extra period.
So, yes, @ricard, it seems you are right and we need to take a deep look in the zynthian audio processing. We don’t want this extra period of latency!!

Cheers!

riban · May 30, 2024, 6:31pm

There should not be any latency added by zynmixer if no loopbacks exist. Can you show a screenshot from patchage or the result of jack_lsp -c?

jofemodo · May 30, 2024, 8:13pm

6.5 ms of latency excess is too much, @riban. In older versions (stable) I measured, at 44kHz, around 13ms, what is a reasonable number for 2 periods. Now we have a period more, at 48KHz: 12+5=17ms. It should be the same 2 periods than before, but it’s 3.

Note that curiously, in the 3rd test, adding plugins pre and postfader doesn’t add latency. Suspicious …

But there is no loops in zynmixer connections (i checked for them carefully) so no extra latency should be added. We are missing something.

Regards

tunagenes · May 30, 2024, 8:43pm

Please explain what you mean by “loopbacks”. I realize it may be a very elemental question. A link would suffice. I did search this forum and the Z wiki with results that seem unrelated or “There were no results matching the query”.

and-or @jofemodo what do you mean by “loops”?

@jofemodo - would it be easy for you to send this for the config where the fresh measurements were taken, or @riban same for an example of what will cause the additional latency?

ricard · May 30, 2024, 9:40pm

I did some quick latency tests tonight, from MIDI in to audio out, using a single MiMi-d instance. I’ve done the measurements by splitting the MIDI signal going from my master keyboard to the Zynthian, and feeding it into the left channel of my sound card, while the audio output from Zynthian goes into the right channel. I then measure the distance from the end of the ‘noise’ in the left channel that represents the MIDI note on message, to the onset of the audio in the right channel.

First, stable. The latency is about 818 samples @44.1 kHz, or 18.5 ms.

Then, oram. The latency is higher, 994 samples @44.1 kHz, or about 22.5 ms

In each recording I played four notes and measured the latency for each one, but there was not a large variation (less than a millisecond).

One thing that I find mildly odd about this is that the extra latency (4 ms) is consistently less than a buffer period (256 samples = 5.8 ms). Sure, it could be down to measurement errors, but the 18.5 and 22.5 ms were very consistent across a couple of note ons.

EDIT: Following the above tests, I bypassed zynmidirouter (in oram) (see the partial patchage screen dump)

which brought the latency down to a much more reasonable 758 samples = 17.2 ms (again, fairly consistent across several notes).

A while ago (when I measured the latency the first time a couple of months ago or so) @riban and I concluded that the excess latency seemed to be in zynmidirouter, and indeed that still seems to be the case. (That of course doesn’t explain the latency from audio in to audio out that @jofemodo observed at the start of this thread).

riban · May 31, 2024, 6:24am

@tunagenes we are refering to a signal that loops back to an earlier part of the chain, e.g. exits a zynmixer output and enters a zynmixer input.

Jack processes its whole graph in one process cycle, processing n frames of audio. (n is the number of frames in a cycle as defined by the jack parameter -p and a “frame” is a set of concurrent samples, e.g. 2 x 32-bit floating point words for left and right channels. The quantity of frames per second is defined by the jack parameter -r and is often refered to as the samplerate.) But if there is a signal that re-enters the graph, this may cause an infinite process time, so jack stops processing that signal and it gets processed in the next cycle.

This means that such a loop would add a processing cycle period of latency which at 48000 fps, 256 frames per period adds 5.3ms (48000*256=12288000 frames) of latency.

Zynthian avoids such loops where possible but to allow post-fader effects in chains, it uses a loop from the mixer chain direct output, through the effect to the main mixbus input which would add this latency. The same is true of pre-fader effects in the main mixbus. But the report from @jofemodo states that this is not the case (no chain post-fader or main mixbus pre-fader effects) hence the internal normalisation of mixer strips to main mixbus is used which should not add any latency. (I will check the code for that to ensure it is doing what it should.)

Whilst writing that description I considered how we might split zynmixer into 2 clients so as to allow those effects to be inserted without the latency but that can be done later - after we resolve the current, unexpected latency issues.

As @ricard describes, some earlier investigation (as detailed in issue 858) showed an extra period of latency introduced by zynmidirouter which we also need to investigate. If both issues are resolved we should see internal latency reduced to one period, e.g. 5.3ms at 48000,256. This would give an overall latency of approx 11ms (2 buffers are required by jack and there is some other latency introduced to support the hardware).

The other related issue is jitter. ticket 858 deduced that zynmidirouter was likely to be the culprit for introducing jitter of up to a jack period (5.3ms at 48000,256) which is why we may see results varying so much.

@jofemodo I completely agree with you that this latency is too high and needs to be fixed. We should be aiming at an internal latency of 2 buffer periods (approx. 11ms by default) and there are issues with our modules that are currently not permitting this. So the areas to look at are:

zynmidirouter latency
zynmidirouter jitter
zynmixer latency
stretch goal: solve additional zynmixer latency under certain routing conditions

One extra snippet of info: We have the ability to feed sidechain inputs on some effects. This is likely to always be delayed by a jack period. I doubt we will fix that and frankly, I don’t think it matters. Sidechaining seldom requires sample accuracy and a small latency will generally be acceptable. Also, sidechaining in Zynthian is a very new feature that is not well known and probably (almost) never used so not a high priority, yet!

jofemodo · May 31, 2024, 7:14am

I just used jack_iodelay to measure roundtrip latency, using a loopback cable connecting audio output to audio input physically. Zynthian was stopped, so the only jack client was jack_iodelay. This is the result:

new playback latency: [512, 512]
   820.889 frames     17.102 ms total roundtrip latency
	extra loopback latency: 52 frames
	use 26 for the backend arguments -I and -O
   820.888 frames     17.102 ms total roundtrip latency
	extra loopback latency: 52 frames
	use 26 for the backend arguments -I and -O
   820.888 frames     17.102 ms total roundtrip latency
	extra loopback latency: 52 frames
	use 26 for the backend arguments -I and -O
   820.888 frames     17.102 ms total roundtrip latency
	extra loopback latency: 52 frames
	use 26 for the backend arguments -I and -O
   820.888 frames     17.102 ms total roundtrip latency
	extra loopback latency: 52 frames
	use 26 for the backend arguments -I and -O
   820.888 frames     17.102 ms total roundtrip latency
	extra loopback latency: 52 frames
	use 26 for the backend arguments -I and -O
   820.888 frames     17.102 ms total roundtrip latency
	extra loopback latency: 52 frames
	use 26 for the backend arguments -I and -O
   820.888 frames     17.102 ms total roundtrip latency
	extra loopback latency: 52 frames
	use 26 for the backend arguments -I and -O
   820.888 frames     17.102 ms total roundtrip latency
	extra loopback latency: 52 frames
	use 26 for the backend arguments -I and -O
   820.888 frames     17.102 ms total roundtrip latency
	extra loopback latency: 52 frames
	use 26 for the backend arguments -I and -O

This is 100% consistent with my previous measurements.

I’ve repeated the test with zynthian in the middle and this is the result:

   820.888 frames     17.102 ms total roundtrip latency
	extra loopback latency: 564 frames
	use 282 for the backend arguments -I and -O
   820.888 frames     17.102 ms total roundtrip latency
	extra loopback latency: 564 frames
	use 282 for the backend arguments -I and -O
   820.888 frames     17.102 ms total roundtrip latency
	extra loopback latency: 564 frames
	use 282 for the backend arguments -I and -O
   820.888 frames     17.102 ms total roundtrip latency
	extra loopback latency: 564 frames
	use 282 for the backend arguments -I and -O
   820.888 frames     17.102 ms total roundtrip latency
	extra loopback latency: 564 frames
	use 282 for the backend arguments -I and -O
   820.888 frames     17.102 ms total roundtrip latency
	extra loopback latency: 564 frames
	use 282 for the backend arguments -I and -O
   820.888 frames     17.102 ms total roundtrip latency
	extra loopback latency: 564 frames
	use 282 for the backend arguments -I and -O

So zynthian MIDI router or mixer seem to not be the cause of the extra period latency.
It seems something related with jackd itself and the way it’s configured???

Regards,

jofemodo · May 31, 2024, 7:31am

More tests. I decided to measure the ALSA latency (without jack). For this, used the same loop back cable connecting audio output to audio input and installed this package:

apt -y install zita-alsa-pcmi-utils

The stopped zynthian and jack:

systemctl stop zynthian
systemctl stop jack2

And do the tests. First with 2 x 256 buffers:

alsa_delay hw:3 hw:3 48000 256 2 1 1 
playback :
  nchan  : 2
  fsamp  : 48000
  fsize  : 256
  nfrag  : 2
  format : S32_LE
capture  :
  nchan  : 2
  fsamp  : 48000
  fsize  : 256
  nfrag  : 2
  format : S32_LE
synced
   564.891 frames     11.769 ms
   564.892 frames     11.769 ms
   564.892 frames     11.769 ms
   564.892 frames     11.769 ms
   564.892 frames     11.769 ms
   564.892 frames     11.769 ms
   564.892 frames     11.769 ms
   564.892 frames     11.769 ms
   564.892 frames     11.769 ms
   564.892 frames     11.769 ms

And then with 3 x 256 buffers:

alsa_delay hw:3 hw:3 48000 256 3 1 1 
playback :
  nchan  : 2
  fsamp  : 48000
  fsize  : 256
  nfrag  : 3
  format : S32_LE
capture  :
  nchan  : 2
  fsamp  : 48000
  fsize  : 256
  nfrag  : 3
  format : S32_LE
synced
   820.888 frames     17.102 ms
   820.888 frames     17.102 ms
   820.888 frames     17.102 ms
   820.888 frames     17.102 ms
   820.888 frames     17.102 ms
   820.888 frames     17.102 ms
   820.888 frames     17.102 ms
   820.888 frames     17.102 ms

So, it’s quite clear that jack is adding the extra latency period.

Googling a little bit i read in a reddit thread:

Async JACK2 (the default) has 1 extra period of latency compared to raw ALSA,

https://www.reddit.com/r/linuxaudio/comments/vgy4zv/latency_too_high_with_jack/?rdt=58436

What would explain the results. The question is, have we changed the way we run jack from stable to oram? AFAIK, no, but it seems yes. Curious!

Regards,

riban · May 31, 2024, 7:34am

From jack_iodelay man page:

Note that JACK2 will add an implicit additional period when using the default asynchronous mode, so for  JACK1  or JACK2  in  synchronous mode, the buffer size is n*p, but for JACK2 in asynchronous mode the buffer size is (n+1)*p.

There are questions about whether synchronous mode is still configurable. Indeed the man page for jackd does not mention it and the -S parameter has been repurposed.

[Edit] Correction: The -S parameter for jackd (not its plugins) is not mentioned in the man page but does infact enable synchronous mode. This does remove the extra internal jack period but may increase the risk of xruns. We need to investigate further.

jofemodo · May 31, 2024, 8:29am

Well, we solve it!!

It seems i changed the order or jackd parameters in some moment and didn’t realize that i was changing from synchronous to asynchronous mode. The jackd man page is not right and the “-S” global option is not documented at all!!

This is the current “bad” jackd config:

-P 70 -t 2000 -s -d alsa -d hw:sndrpihifiberry -S -r 48000 -p 256 -n 2 -X raw

and this is the good one, that i’m going to restore:

-P 70 -t 2000 -s -S -d alsa -d hw:sndrpihifiberry -r 48000 -p 256 -n 2 -X raw

BTW, we have decided to increase the sample resolution from 16 bits to 32 bits, what shouldn’t have an impact in I2S audio interfaces, but it could be in USB audio interfaces due to the increased data transfer. USB audio users should check this carefully and find the configuration that better fits their needs.

Now, the plot:

We return to the 11ms latency

Enjoy!

jofemodo · May 31, 2024, 8:40am

And if you reduce buffer size to 128, like this:

-P 70 -t 2000 -s -S -d alsa -d hw:sndrpihifiberry -r 48000 -p 128 -n 2 -X raw

Then you get this:

An low round-trip latency of just 6 ms! Does somebody need less than this?

Cheers!

riban · May 31, 2024, 9:00am

How low can you go?

I have had jack running with very low (2.7ms) which is super cool for live audio processing but it does push the CPU hard and in Zynthian we have some other overhead that would block that. 6ms can be noticed by some people under some situations but frankly, we are not really aiming for many of those. It is mainly when processing voice and monitoring post-processor that it can be a concern or when playing virtuoso piano but wearing headphones you are close to acoustic latency anyway. Most of our userbase is unlikely to be significantly impacted by 11ms which should be our current target. (Maybe higher on a RPi3.)

Klangschmied · May 31, 2024, 9:24am

@jofemodo great, immediately tried the “low latency” mode (6ms) with Obxd, feels and plays very nicely !

jofemodo · May 31, 2024, 9:45am

I never tried going under this. I doubt nobody can perceive the difference under this value. Anyway, with Pi4 you probably won’t get a very usable system with 2.7ms. With Pi5 it could be different and it must be tested, just for curiosity

Regards,

wyleu · May 31, 2024, 9:49am

Good work Guys.

riban · May 31, 2024, 9:53am

As ever, @wyleu is late to the show. Now that is what I call, “Latency”!

jofemodo · May 31, 2024, 9:53am

I just pushed to oram branch the changes in jackd configuration, so after updating, those using I2S audio interfaces (non-USB ones) should have the latency reduced by one period.
If your audio inteface is an USB one, you still could try to add the “-S” parameter before the “-d alsa” part and see if it works.

Enjoy!

ricard · May 31, 2024, 8:53pm

I updated my oram installation, bringing in @jofemodo 's latest jackd command line fix.
Measuring the latency from end of MIDI note on message to start of MiMi-d output now consistently is in the 758-762 samples range at 44.1 kHz, corresponding to 17.2-17.3 ms.

So I can confirm that the change fixes the MIDI-to-audio-out latency issue reported in the latter half of Unexpectedly high latency · Issue #858 · zynthian/zynthian-issue-tracking · GitHub ,

(BTW, after upgrading, I had to explicitly reboot the system to bring the change, which I think is to be expected, just mentioning it for reference).

Great work, especially figuring out the not exactly clear options for jackd!

gitnob · May 31, 2024, 9:47pm

but it could be in USB audio interfaces due to the increased data transfer. USB audio users should check this carefully and find the configuration that better fits their needs

How could I set this back to 16 if necessary?

riban · June 1, 2024, 7:46am

To set preferred word size to 16-bit, add -S option to end of parameter list.

Note that the -S parameter has different meaning of it is before or after the -d alsa option.