Instead of CA glue, how about 5 minute epoxy. It just requires 5 minutes of patience to hold the magnet in position, but being a steel arbor, the magnet should hold itself there, but it might want to slide a little off center if you don't hold it in exactly the right place. Or maybe 5 minute epoxy surrounding a dot of CA glue where the CA glue would hold it in place while the epoxy sets up. Being on the center of a rotating member, I doubt it will even want to come off and disappear into the saw dust, even without any CA glue or epoxy. It's own magnetic field and it being centered on the arbor should give it no centrifugal forces, so it should want to stay there just fine without glue, if you don't bump into it with something. That's when the CA and epoxy should help to keep it there.
I didn't get to the shop yet, but will be doing this first thing when I do get there. I haven't yet decided on the retainment method that I'll be using, but do like the idea.