Thanks guys, I do appreciate the reality check.
I'd certainly not tear into this without taking some additional safety precautions of adding A LOT of cross bracing (which it honestly has very little of), shoring up areas being worked, and working in in small sections at a time.
The structure itself is not the "Timber" frame I'd hoped for. To be quite honest it's a stick, "balloon" framed, structure put together with nails. It just happens to be mostly oak which probably still pretty green at the time of building it.
I'm actually surprised the roof weight hasn't spread the center out more than it has, given there really aren't any collar ties or much cross bracing other than in the corners to prevent racking. The only ties are the one the hay track is mounted on, and probably for that purpose. The taller center posts are actually supporting an L shaped strong back, and each rafter is two pieces lapped/nailed over it.
My wife and I have had the discussion that this may need to come down, it's not the direction she wants to take, but understands if it must.