Well, the acetone did soften the CA glue but not enough to get the magnet out. I tried a punch with the hopes of cracking it and maybe digging it out but I didn't have any luck. Then I tried drilling it with the plan of screwing a screw in and pulling it out. No luck, apparently neodymium...