Derek, the problem with pure wax is that it offers little or no protection against water ingress and dirt. The water ingress might not be too much of a problem on a staircase handrail, but you never know, but the constant rubbing of grimy sweaty hands will probably lead to a dirty grubby look.
People often erroneously blame varnish for a plasticky thick look. It can look that way, but it's usually the fault of the person applying the varnish than the fault of the varnish itself. If you don't prepare the wood surface properly and then slap it on thick, leave sags, fat edges, runs and curtains, over-brush and leave a trail of bubbles, yes varnish does look a mess. Apply it correctly and it looks very fine.
Either water based varnish or short oil based interior varnish will provide a good durable protective film over banister rails. Apply a couple of coats of gloss followed up by a coat of the sheen you require to reduce the the tendency of a cloudy finish to form. You can create custom sheens by adding a portion of a matte finish of the same type to your gloss. For instance, I find making a mix around the numbers 5-15% matte varnish to 85- 95% gloss varnish makes an attractive semi-gloss.
I have a preference for oil based varnishes because I've never yet found a water based polish (including varnish) that I've found completely satisfactory on all the counts important to me. Slainte.