Yeah, getting as much off and sanding down the rest might be your best bet. There’s not really an easy way to do it I’m afraid.
A dremmel or other rotary tool would be useful, but I’d imagine you’d get some workout trying to grind it all down by hand. Y’know, gotta keep that bicep game up!