You need to avoid serial pins for midi, i2s pins and audio protocol and their in and out data pins (audio in data pin is only occupied if audio card has adc capability - only few have it).
If you use dsi screen you do not need to worry about pins occupied by screen.
Do not forget that you will need debouncing 100n capacitor for each encoder’s switch and 10n debouncing capacitor for each encoder A and B lines (two per encoder).
I have design before minimalistic zynthian rpi HUT that provides just MIDI IN/OUT, audio out with cheap i2s card and 4 connectors for encoders. Three is also MIDI Through here but only available through connector because there is no enough space to put all PCB mounted MIDI connectors on such a small HUT. Also audio and all MIDI do have connector pins if you want to connect panel connectors to it.
That minimalistic zynthian approach is doing what you want - connect encoders directly to rpi gpios. Here is the rpi connector from this schematics where you can see where encoders are connected.
If you are interested I have few PCB left from this old project. I can sell them to you for 10£ plus shipping. They also come with encoder PCBs where you need to solder capacitors . Here is how all 5 PCBs looks like.
If you buy them I will send you bill of materials to assemble this (even though everything is labeled on PCB) .
If there is interest from other people I will create wiki page for all information about this very minimalistic zynthian hardware and share kicad project on zynthian-hw.
I have used this PCB on my zynthian pedal build explained here Pedal Board from Aeolus discussion