Well, the right tool for the job is a piloted counterbore. I machine up my own, but then, I have my own small machine shop.
If all you have are hand tools, and you aren't looking for perfection, here's what I recommend: Start with a countersink. Gently cut a countersink on the existing hole, checking with a ruler to bring the OD of the countersink right to 11/16". Do the same thing on the back side of the hole. This gives you a guideline circle to work to, which is centered on the original hole. The countersink also helps to prevent chipping out paint or wood at the surface.
Now trim out the hole to that circle, watching front and back. The simplest way is with a big round file, but you can use a Dremel tool if you like. Work carefully, filing right to the line, and keep checking the fit of the bushing.