Network Audio

riban · October 9, 2020, 12:38pm

If I have several audio devices connected to an IP network I would like to be able to route audio between them without the need to plug audio cables or connect via a mixing desk. An example might be that I have two Zynthians and want to take the audio from one and route it to the other. Maybe I want to use a common reverb for all my Zynthians or I may be using one as the main mixer without an outboard mixing console. It would be good if there was little (or zero) configuration required and that Zynthian detected other audio devices automatically.

There are a few ways to do this which include:

netJack1: This supports low-bitrate codecs but requires source and destination to be defined at start-up (no discovery option). I don’t this this is a good match for this requirement.
netJack2: This allows an arbitary quantity of clients to connect to a main device with automatic detection (within same LAN segment). Each client may have the quantity of network inputs and outputs (audio and MIDI) defined as well as its client name (defaults to hostname). This looks appealing but it loses JACK ports when disconnected from main device which will lose routing within current Zynthian implementation - this could be resolved in Zynthian. There must be exactly only one main device on the network which may be limiting, i.e. this is server / client, not peer-to-peer. A client is already easy to define by selecting “Custom Device” soundcard and using JACK parameters -d net -C2 -P2 -i1 -o1. To enable the server side we need to add a feature to load / unload the netmanager internal client (which is fairly simple) and then offer ability to route from JACK client ports called, “from_slave_1” and “from_slave_2” which is a fairly trivial tweak in Zynthian Audio Router. This looks quite a good match for this requirement, requiring only small changes to Zynthian.
jack.trip: This requires destination to be defined at start-up (no discovery). I don’t this this is a good match for this requirement.
zita-j2n: This requires destination to be defined at start-up (no discovery). I don’t this this is a good match for this requirement.

There may be other protocols that we could look at, e.g. AES67, Dante, etc. These may improve interoperability. There are some opensource AES67 implementations like this one.

This may be a niche feature but I do see some wider uses, e.g. connecting to digital infrastructure (probably requires standard protocol like AES67) or Internet connectivity for rehearsal / performance (may require data reduction using current Internet technology).

wyleu · October 9, 2020, 2:48pm

Certainly this is an issue that has had me musing for some considerable time. My main use case would be transporting audio from an input card(s) an one stage microphone or guitar ,to a remote zynth that could treat this source as an incoming layer.

Quite how much we contain this is probably a useful set of limits worth imposing.
I would tend to support transportation of control signals around such a network with audio rendering taking place at the output device if at all possible. Processing power will limit that so another element would be a render farm of output zynths that might combine a multitude of sources down to one (or more) output structures. There is a pure approach where all such work should be conducted entirely in the digital domain whilst the pragmatist in me might well favour multiple analogue outputs feeding something like the Behringer 1820, with a zynth managing it.

I’ve run some fairly complicated MIDI over ethernet rigs and it’s impressive what can be achieved but when it goes wrong it goes VERY WRONG ! Die midi howl around die !!!. I’ve built reboot facilities into the pedal board for such an occasion and I’d probably like to see some declared fallback position that would at least allow critical components like front of stage microphones to allow an apology to be delivered without frantic re patching, or menu flapping or possibly worse the provision of a completely separate analogue chain just in case ( I did that on the pedal board too…)

riban · October 9, 2020, 4:42pm

If you load the audioadapter module on the client then you get access to its soundcard which allows the workflow that @wyleu mentions, i.e. routing audio from one Zynthian’s audio inputs to another’s audio outputs (or any other Jack destination).

This definitely needs a good, solid network connection. I just heard some interesting effects when on WiFi with the pitch shifting all over the place!

We would probably want better / different audio routing control if we go down this path. I think this may fit nicely with the zynmixer. I can imagine having a main Zynthian acting as my mixer and synth with other Zynthians feeding into it. Those may be played by me or other people. The Zynthian can become a digital stagebox. With a bit of thought you could have input and output on each Zynthian allowing personal monitoring, etc. The latency can be very low on a good network. There is a configuration that ensures all processing is done within the same jack frame so there is not added latency.

Also… it passes MIDI .

riban · October 10, 2020, 7:54am

Okay - I couldn’t wait for others to reply so I have added netJack2 hub as an option in admin menu on a new “netJack2” branch. I use the word “hub” to represent the one (and only) Zynthian that should act as the main / hub device to which all client Zynthians (and other netJack2 enabled devices) can connect so only enable this on one device on your network.

Because JACK2 service does not wait for network to be up before starting, setting a Zynthian as a netJack2 client (by setting soundcard jack parameters to -d net -C2 -P2 -i1 -o1 may cause it to stutter during start-up. I haven’t tested the client extensively but it seems to work okay on my Pi3. What doesn’t work on the Pi3 is enabling the soundcard whilst in netJack2 client mode. This Zynthian only has headphone soundcard enabled and I haven’t tested with other soundcards - that needs some further investigation. This would only be an issue in @wyleu’s scenario where you want to use the Zynthian as a netJack2 client and use its physical audio inputs / outputs.

With this branch you can enable netJack2 hub on one Zynthian, configure another Zynthian as a netJack2 client and pass audio between them using each engine’s context menu for audio routing. Note: On the client Zynthian the netJack2 connections (to the hub) appear as its default audio ports so it does not need specific routing. Just use it as a normal Zynthian with its input and output appearing on ports on the hub Zynthian. Also note that sending audio from the hub to a client only passes the full stereo pair, i.e. you can’t currently route individual mono channels in that direction. This is similar to a stereo plugin but maybe should be investigated.

I have observed that increasing the quantity of netJack2 channels does increase load on JACK and there is a point at which it starts to xrun. I have not observed that with stereo connections as described in this post but more esoteric configurations may start to vex it. I have not tested with Pi3 as the hub so am not sure how it will behave. (My Pi3 and Pi4 are at opposites ends of the house on different floors and there are only so many simultaneous remote session bounces that my brain can cope with!)

There are other applications and devices that could utilise netJack2, e.g. you could connect your Zynthian over the network to an instance of Ardour or run a rack of plugins on a powerful desktop with send and return in Zynthian audio chain or vice versa.

If you have network devices that run jack2 and want to join them to your Zynthian you can give this branch a bash and report back. If it proves useful / desirable / easy-to-support it might make it to the main branch (but hopefully only after we have an actual stable release - we can all dream ).

[Edit] I have not added any code to handle MIDI routing yet. If you want to play with that you will need to roll up your sleeves and delve into the innards. I will add MIDI routing when I get a moment.

riban · October 10, 2020, 11:28am

For zero latency*, add the parameter, -m fast to the client JACK configuration. The hub will wait for the client’s data each cycle which increases the risk of xrun on hub if client struggles or there is network latency. I recommend only considering using netJack2 on a dedicated hardwired Zynthian LAN. Although netJack2 supports jumbo frames, the Pi NIC driver does not so there is no use changing the network frame size.

*Zero latency between client and hub. There remains the latency within JACK, ALSA driver, etc.

Oops! Old docs… Mode has been replaced by network latency setting which defines the quantity of cycles to delay so -l 0 gives zero latency. Default is 5 cycles.

wyleu · October 11, 2020, 10:13am

Not had a lot of success with this.

After changing the config, the PI wouldn’t start until I removed the audio-injector card, and the custom setup wont start.

Not sure I can configure an audio injector input and a network output…
Still attempting to configure a pure network output fro zynthian-pedal.local

riban · October 11, 2020, 5:17pm

That’s a shame. It worked okay in my (limited) tests. I mostly tested from a laptop running JACK but did tests between my Pi3 and Pi4 Zynthians.

What hardware are you using and what is your test setup? If you can describe exactly what you did with what I could take a look at what is going wrong. I can’t see why a Pi with JACK configured to start with net driver would care about other soundcards - indeed I tested this on my laptop with built-in sound and a USB soundcard plugged in and it still worked. I take it that you did reboot after configuring, etc.

wyleu · October 12, 2020, 5:12am

Il have another try this evening. It’s pi4 with an audio injector card (zynthian-pedal.local)linked over to a pi 4 with a hifiberry dac/adc (zynthian-steel)

jofemodo · October 12, 2020, 11:16am

I was working on something related with this some time ago:

… the code is not finished and would need some update for working with latest changes.

The idea is running engines in several nodes (a simple RBPi with network connection). Audio & MIDI is sent using netjack and returned to the master node. When creating a layer, you choose the node from the available ones, etc. The UI is running on the master node, that control all layers: local & remote.

I can’t say the exact status of the development, but i remember having tested with a master + slave, creating remote layers and receiving the resulting audio on the master.

I would like to work on this again, but it’s not a priority and have no time for it … anyway, it’s a really funny development, jejeje!!

Regards,

riban · October 13, 2020, 8:29pm

It looks like Jack does not create audio input or output ports until a client has connected to a server so the client running -d net will have trouble if it can’t connect to the hub, either due to network issue or the hub is not yet running.

shayneoneill · October 16, 2020, 3:15pm

If anyones interested, heres a project I was working on a while back, “BORTPort”, that basically multiplexes multiple midi ports over USB so you can build midi controllers that talk over ethernet.
It includes a protocol buffers spec so bindings for pretty much any language is possible. The current ‘driver’ relies on QT (QT has an amazingly intuitive network stack) but it wouldn’t be hard tor rewrite for native linux apis.

C0d3man · December 5, 2020, 3:59pm

Hi @ll,

update for an old topic: @lars.pelz and I tested zita-n2j/zita-j2n for connecting our studios via Internet. We used a minimal Raspian with three created (simple) systemd scripts: Works really good with 16bit stereo. We used the hardware of our -v4, jackd with buffer 256 and 10ms zita-buffer. Yes - there is a little bit latency, so we will try next with smaller buffers.

Is there interest in the scripts?

Regards, Holger

jofemodo · December 5, 2020, 4:10pm

Of course!! Share please…

C0d3man · December 6, 2020, 7:53am

Ok, here we go:

This currently does not connect any of the -layers to the network-audio-out! Only external sound sources are routed to the network!
You need to install the following packages:
sudo apt-get install zita-njbridge jackd2
You need to install the following three systemd units to /etc/systemd/system:

jackd.service (Only needed , if jackd isn’t currently installed! This also adds a monitor output, so you can connect your external sound source to the input of the Hifiberry and monitor speakers to the output):

[Unit]
Description=JACK server
Documentation=man:jackd(1)
After=sound.target local-fs.target

[Service]
Type=simple
Environment=JACK_NO_AUDIO_RESERVATION=1
ExecStart=/usr/bin/jackd -dalsa -dhw:1 -S -r48000 -p128 -n2
ExecStartPost=/bin/sleep 2
ExecStartPost=/usr/bin/jack_connect system:capture_1 system:playback_1
ExecStartPost=/usr/bin/jack_connect system:capture_2 system:playback_2
LimitRTPRIO=95
LimitRTTIME=infinity
LimitMEMLOCK=infinity
Restart=on-failure
RestartSec=15

[Install]
WantedBy=default.target

zita-j2n.service (this is for routing the sound to the network):

[Unit]
Description=Zita Jack->Network
After=network.target
After=jackd

[Service]
Type=exec
ExecStartPre=/bin/bash -c "/bin/systemctl set-environment dest_ip=$(grep ^DEST_IP /etc/aoip.conf | he
ad -1 | cut -d"=" -f2)"
ExecStartPre=/bin/bash -c "/bin/systemctl set-environment dest_port=$(grep ^DEST_PORT /etc/aoip.conf
| head -1 | cut -d"=" -f2)"
ExecStartPre=/bin/bash -c "/bin/systemctl set-environment mtu=$(grep ^MTU /etc/aoip.conf | head -1 |
cut -d"=" -f2)"
ExecStart=/usr/bin/zita-j2n --16bit --mtu ${mtu} ${dest_ip} ${dest_port}
ExecStartPost=/bin/sleep 2
ExecStartPost=/usr/bin/jack_connect system:capture_1 zita-j2n:in_1
ExecStartPost=/usr/bin/jack_connect system:capture_2 zita-j2n:in_2
RestartSec=15
Restart=on-failure

[Install]
WantedBy=default.target

zita-n2j.service (needed for routing network audio to jack):

[Unit]
Description=Zita Network->Jack
After=network.target
After=jackd

[Service]
Type=exec
ExecStartPre=/bin/bash -c "/bin/systemctl set-environment ip=$(/sbin/ip -o -4 addr list eth0 | awk '{
print $4}' | cut -d/ -f1)"
ExecStartPre=/bin/bash -c "/bin/systemctl set-environment buff=$(grep ^RECV_BUFFER /etc/aoip.conf | h
ead -1 | cut -d"=" -f2)"
ExecStartPre=/bin/bash -c "/bin/systemctl set-environment src_port=$(grep ^SRC_PORT /etc/aoip.conf |
head -1 | cut -d"=" -f2)"
ExecStart=/usr/bin/zita-n2j --buff ${buff} ${ip} ${src_port}
ExecStartPost=/bin/sleep 2
ExecStartPost=/usr/bin/jack_connect zita-n2j:out_1 system:playback_1
ExecStartPost=/usr/bin/jack_connect zita-n2j:out_2 system:playback_2
RestartSec=15
Restart=on-failure

[Install]
WantedBy=default.target

Now you only need to install a configuration file as /etc/aoip.conf:

MTU=1492
DEST_IP=your.target.server.here
DEST_PORT=12345
SRC_PORT=12345
RECV_BUFFER=5

Both sides have to enter their matching DEST_IP and DEST_PORT. Also you may need to add a port forwarding for this port to each (DSL-)router (and I also additionally added a realtime flag for avoiding that my kids are using all the bandwith for youtube).

You may want to change the RECV_BUFFER size for avoiding latency problems. Also the jackd parameters are adding latency, so best using -p 128 and -n2 (=7.75ms latency). With the current setting you have a latency of 12.75ms plus the latency of the WAN line (for our test about 5ms, we both have the same Provider and 100/40 DSL lines).

A (nearly) ready to use image (1GB) can be found here (until 20.12.2020). You only have to change DEST_IP and DEST_PORT in /etc/aoip.conf.

Have fun!

Regards, Holger

wyleu · December 6, 2020, 9:28am

This is exactly what some of us ( Oh ok probably just me) want!
Cos If you are doing a zynthian/zynthian pair you can generate the sound at either end of the chain. But the reduction in clutter from taking audio from external inputs throu’ your single Ethernet connection, into another zynth to simply move the audio out somewhere convenient.
The nice thing is you can also just send MIDI from the stage zynth & render at the stage-side.
It would need a considered routing model, and do we make it plug & play?

Think of all the too’ing and fro’ing as you make and unmake ethernet cables !! :-D.

How would it fail? ( Drops back to smooth jazz background music … ? )

Elsewhere, I’m still not entirely sure I’ve got a working audio in in one channel so it’s all held together with ravens and crows . . .

From the implementation perhaps we have another selection option
Network audio transports

1/ the entire output, including audio in, of the zynth … or
2/ ONLY the audio in info ( post effects on that layer …? )

I think they are different uses cases.

riban · December 6, 2020, 9:50am

Defo want to see network audio integrated as audio inputs and outputs for Zynthian routing. I did similar but chose NetJack for my experiments. I disregarded Zita for a reason that currently eludes me. It might have been something to doe with the amount of configuration, e.g. needing to configure all nodes. I was looking for something that would allow connection to available nodes. Something similar to Dante.

Regarding Internet connectivity, I worry that the current Internet infrastructure is insufficient to support baseband audio without significant disruption. The common approach currently is to use data reduction to transport audio between endpoints. I certainly see a future where high bandwidth, QoS protected routes are available to us mere mortals but am not sure we are there yet. For controlled environments like @wyleu describes (directly wired endpoints on closed or controlled network segments) this is very desirable. I am of course very interested to hear how Internet connected nodes behave, not just latency but also dropouts (xruns), latency variation (jitter / wow / flutter), physical distance between participants, etc. The laws of physics are there to be challenged but it may take a brain bigger than mine to bend or rewrite them! (I do try occasionally.)

Maybe we can add a hierarchical configuration for network audio which allows enabling of different protocols - but first we need it working.

@wyleu mentions failure mode. I found with NetJack that it could fail to a rather undesirable state which might be worked-around by monitoring and service control. I didn’t pursue that. I also had interesting effects with variable latency where the audio actually pitch shifted. Funny but not fun.

I think we currently sit in the lab testing stage but it would be great to progress this. I am not confident we can get an international Christmas jamming session working but maybe next year…

C0d3man · December 13, 2020, 7:57am

Yesterday we made another test with looping back the audio signal (cable loop). The ping RTT was 15ms and we used a Jack buffer size (-p) of 128 (-n2) and an icoming buffer for zita-network-to-jack of 3ms (that was the lowest value where the audio was not destroyed):

If you have good eyes you can see that the audio-RTT was 31ms, so the one way latency was 15.5ms. That’s exactly what I have calculated before:

latency=samplerate/(jack_buffer_size*(number_of_periods+1))+ping_rtt/2
latency=(128*(2+1)/48000)+0.015/2
latency=0.0155s=15.5ms

No audio dropouts. For me (as a hobby musician) it’s ok.

I thought about integration into Zynthian, but this may cause some problems:

My Zynthian has a buffer size of 256 because I want to avoid audio dropouts for bigger arrangaments. But using 256 instead of 128 adds latency of 8ms to the network-audio-send => not good.
For the tests we used RPi4 with no additional software running. Only the minimal base system with jack and zita-send/receive. On a Zynthian there is much more software running. Maybe this may cause problems with audio sending or receiving.
As @wyleu wrote: Currently we habe no simple zynthian-output (audio-sum of all layers) for jack to connect it to the network-output stream. My current sollution would be adding an additional connection for each layer-output to the network-send. That’s not really nice.

So, our solution will be to use a dedicated RPi (perhaps not the 4, but a 3B) with a Hifiberry DAC/ADC. May be we will build a small web frontend and use the WiFi-AP mode for connecting a tablet to configure the parameters. Also a simple LED on a GPIO pin may help to display the state of the connection.

Regards, Holger