I wouldn't do less than a 3 hp with 6" feeder lines dropping down to 4" at the tool blastgate. I know that sounds big but with my experience of setting them up I wouldn't consider anything less. I have a 3hp on 6" feeders set up with 4" on the blast gates and it is taking care of a lot more machinery than you have stated but mine is open at the discharge dumping into a pile outside the shop. If you bag it all it really holds things back.
I once heard that cats and women will do darn well what they please and that men and dogs would do well to accept it and just go on.