Hi @hannesmenzel! A warm welcome.
I would avoid using pins: 12, 13, 15, 16, 18, 22, 35, 37, 38 & 40 which are the I2S (audio interface) pins on RPi5. You may not be using all of the I2S ports if you use the HifiBerry 2 in, 2 out device but if you want more audio inputs/outputs you might want to use these pins. Of course, if you are sure you only want 2x2 and cannot find sufficient other GPIO then you will need to use these. (There are only 12 spare GPIO that will all be required for 4 encoders plus switches.)
No, they need to go between the encoder/switch pins and ground.