looks like a very nice project
First I must say, that all my experience is more or less a rule of thumb, got with trial and error
My experiences with USB-audio (with Linux, but also Windows) showing the same problems like your current one: massive latency and drop-outs. I have several USB-audio devices - the most are really cheap. Only with my Roland UA-25EX I get really good latency and no xruns - but only with Windows and the special USB2 driver from Roland.
The standard USB1.x audio stack seems to be very poor for realtime audio usage. It seems to be designed for chatting/phoning or only playback audio. For more expensive equipment only specialized USB drivers seem to help. But those are normaly not available for Linux
The Hifiberry-DAC+ has a big advantage: It uses Raspberries I2S audio interface, which seems to be fast enough and has not such a big and overloaded stack than USB. But is has only audio-outputs and Hifiberry told us, that they currently not developing cards with audio inputs. So maybe a Cirrus-Logic or a Wolfsen card are the sollution - they have audio inputs (no idea if they sound as good as the Hifiberry-DAC+).
IMHO the only sollution for your project is to use high quality/high speed audio hardware without USB.
Why do we disable swapping? For the Raspi swapping means to write to a swap partition or file on SD card. This is a problem for maximum performance. Using the SD card while playing can result in xruns due to the driver and very slow speeds when writing.
We currently are not using a speciallized RT-Kernel. Until now it was not really necessary. The bottleneck I found is, when you are using a big sample-set which is not loaded fully into RAM and the Raspi needs to load samples while playing…
Hope that helps,