Ed, a lot depends on the geometry of the part but yes, you can do light milling as you describe. One thing to be wary of is the drill chuck can come loose in the quill from side load and from vibration. When that happens things can get very exciting!
"I am enclosing two tickets to the first night of my new play; bring a
friend...if you have one."
--George Bernard Shaw to Winston Churchill
"Cannot possibly attend first night, will attend second..if there is
--Winston Churchill, in response