Ok, so I changed in /etc/systemd/system/jack2.service
ExecStart=/usr/bin/jackd -P 70 -t 2000 -s -d alsa -d hw:0 -r 44100 -p 256 -n 2 -X raw
to:
ExecStart=/usr/bin/jackd -P 70 -dalsa -dhw:0 -r48000 -p128 -n2 -X raw
And rebooted.
Now I get ~8.5ms delay. Under 10ms!