The reason you are seeing this behaviour is because the rotary encoder units have 10k resistors between each encoder switch / button and +. The Raspberry Pi has very weak pull-up resistors ~100k. There is an easier path to ground via the encoder pull-up resistors (2 x 10k resisitors). There are some solutions:
- Remove the resistors from the encoder modules
- Cut the tracks near each resistor
- Connect 3.3V to the + pin of the encoder module
The last option is probably the simplest. You can daisy-chain or star wire from a single 3.3V pin on the Raspberry Pi. You should not use 5V as Raspberry Pi GPIO are not 5V tolerant.