My first drill press was a machine mart clarke, not the cheapest one they had by far, and i thought it was the mutts proverbials.
However then i got way more into woodworking and realised it was pretty much cheap rubbish, if you dom this as a hobby and not just once a year.
It started having problems as i got into heavier stuff, and then gave up the ghost, so i got a proper 2.5 hp floor standing one, and wow, i can start fires with a forstner, nothing stops this beast... Anyway, i digress..
My new one has min size of 5mm bit, and i needed to do a load of 3mm holes. So i decided to dig it out, as it must be able to handle that, and found i had the exact same issue you are describing.
Took it apart, (had to disable safety so i could run whilst looking under the bonnet), and found it was really simple. The locking nut on the chuck pulley was misthreaded (was at about 10deg angle!) and was loose. I could not straighten it, however i managed to tighten it and now works fine.
I have the two side by side for multiple set ups for repeated operations. However i would check that, sounds similar and is something that can come loose after a few hours running, and would have that effect.