I generally put them back in. However, if it was leaving a dark mark (I am assuming the weight is rusty?) then I would leave it out. If the sea pony stands in her shell fine, I would not put anything in her. If she needed a weight to stand properly, I would substitute something else.