[caution: long, probably boring musing on why some router cutters work better than others for some tasks. Don't read after a heavy meal...]
It seems to be that the finish you get with a router depends (among other things) on the speed of the cutter with respect to the stock. Below a certain cutter speed it cuts roughly and tears. Above a certain speed you get burning as it cuts faster than the feed speed and then rubs on the fibres, scorching them and blunting the cutter.
So for big cutters (panel raisers, big moulders and mitre-lock cutters, etc.) you need generally low RPM, but I think the real issue comes with the smaller ones: they need to rotate really fast to get a decent finish, BUT if you think about how they're used, that causes other problems.
I've got some small cutters, around 4mm-6mm with 1/4" shanks, and they're really hard to use. OK, my small Bosch can get up to about 20,000 rpm (allegedly), but invariably the reason I want a small cutter is slotting of some sort, and that means two problems - proportionately more force sideways on the cutter as more of the rotational circumference is in contact with the wood, and chip clearance.
If I move it fast enough to not burn things, the sideways force on the cutter, where the end of the knife meets the shank, is huge, so they can snap.
If I don't, it easily burns the slot it's cutting. One reason is that the chips inevitably clog any narrow deep slot, no matter how much it's relieved by multiple passes and shallow cuts. That doesn't allow the cutter, or the wood, to cool properly.
I'm sure that's why I get far better results with a 4mm groover* than a single- or twin-flute 4mm straight cutter. The knife-to-wood speed is as fast as it needs to be, and the chips are pulled clear of the work. It can slot with the grain, whereas a straight always cuts with grain reversal. It's the same reason you get better results with a spindle.
Finally, there are two other factors in play: the small cutter has less inertia for a given cut thickness and it is likely to vibrate more (these two are related). That means a poorer finish.
You'd get round a lot of these issues with a spiral cutter (upcut, I guess, for chip clearance) - a slicing cut and more continuous contact with the material, meaning it won't be encouraged to vibrate. The big disadvantage is cost. I simply can't afford them at the moment, but I may one day buy one to experiment with.
So, to come back to bass speaker apertures, using a bigger-diameter cutter will be more like the groover than the small straight. It will cut better, altogether, and you should also get better control doing it.
Sorry - a bit long-winded, but I woke up this morning wondering about why I'd do it with 1/2" cutters and not smaller ones. I hadn't really analysed it before.
E.
*groover - otherwise sold as biscuit joining cutters. They're a flat wide cutter, 4mm thick. I've had disappointing results using them for actual biscuits and I've now got a proper biscuit-jointer instead, but they cut really nice grooves on the router table. You can use my biscuit joiner for grooves too, and the results are even cleaner than the router grooving cutter, as long as you don't wobble the machine!