I recently made a table for my drill and went with a low fence
It was fine for what I wanted to do at the time ie drill stopped holes. I've since needed to drill through holes with a backing piece to prevent drilling into the table and then the fence becomes too low so I will be adding a taller fence soon....
One thing to consider with a tall fence is that it might foul the handle of the drill.
Yes you're right about a high fence fouling the handle. I bought a table from Axminster and often need to remove one or even two handles,
Have a look at the Axi table - route a square out of the top piece of ply and make removable/replaceable inserts. With the insert removed you will be able to use sanding drums.
This is my second table. I made a note of the mistakes and seek to improve upon it. I had a few low fences, one for the Wagner safety plane, another for drum sanding plinth cutaways. I have a range of removable inserts for drilling and sanding with my collection of drum sanders. A drawer that collects the dust from the sanders. I decided against below table dust extraction. I have 3 T tracks inlaid flush with the table surface. Clamp downs and extractor clamp. I will recess the other T-track in the face of the fence for stop blocks. I would love a flip fence. I could attach a featherboard. The table is raised so the crank handle does not impede movement. There is some storage at the front of the dust drawer. Just not sure about the fence. Probably a low one.