Vaughn, skp has a "library" of all the components you use - not just the ones you see, but any that you used at all at any time and I believe they are stored in the model.
Go to Window > Components - it'll show the components panel. In the dropdown use "in model" and see what's all there. I'll bet you'll find a few of the things you thought you deleted in there.
I'm pretty sure there's a ruby script out there somewhere that runs through and cleans things up, but I'm not sure it does this. In the panel, again, right next to the dropdown is a menu button (the one looking kinda like an arrow pointing right), click that and the menu has a 'purge unused' item. That should get you closer to a smaller file size.