Tonight I cut my next pushrod to the correct length, and then spent time modifying my tools to build a better clamp for the drill press.
I trimmed some material off my pipe clamp, and then mounted it on a block of wood. I found a bushing that I can use to keep the drill bit centered, and then clamped everything down firmly onto the drill press tray. I tested with some scrap, and the hole was straight and true. I always end up coming back to the same rule; clamp your work.