I don't remove the clasps unless I have to. However that being said if there is any sign of rust going on at all, I will remove them. Quite honestly, I don't think it makes that significant difference in an off of card, played with condition pony. If you had your hands on a rare, mint or on card pony, then I could see a reason to leave them unaltered as possible. But loose, what difference does it make? Its not like someone is going to come check out your pony butts unless they are a buyer in which case they may be better served if they don't have to worry about rust... Just some thoughts.
I should add that most of my ponies do have the bracket and a lot of them still have the washer. But I am the original owner of more than half of them and I took very good care of them when I was a little girl.