My suggestion is based on what you already have for inventory.
If don't have many router bits I suggest you go out and buy one of those cheap sets that has 30 router bits or whatever for under a fifty bucks. Go ahead and use it, and soon you will see that some router bits you use all the time, some you never use. The great thing is, for little money you have those odd-ball bits on hand when you really need them. For the ones that get worn out quick, well when you replace them buy top notch carbide bits since you obviously will be getting your money's worth out of them.
I have no intention of traveling from birth to the grave in a manicured and well preserved body; but rather I will skid in sideways, totally beat up, completely worn out, utterly exhausted and jump off my tractor and loudly yell, "Wow, this is what it took to feed a nation!"