I was having this discussion with my other half just the other day.
I like cleaning, restoring and displaying mine re arranging them from time to time.
They mostly just sit there looking nice waiting for me to look at them.
Although I do display them they are in my bedroom and I am very private about my collecting, very few people in my life know about it, I think because I am worried that most people just wouldn't understand and I had asked my husband how weird he thought it was.
We eventually came up with this. Firstly it doesn't matter how weird anyone else thinks it is as long as it makes you happy.(you're not hurting anyone right?). Secondly ponies or no ponies lots of people collect lots of different things (I saw a guy on the tv the other day who collected bricks, not judging just saying it's different!).
Also just for your information from what I have read the psychology behind collecting anything (apart from it being enjoyable) is supposededly to do with us attempting to create a sense of security and order in our lives. Perhaps this is why your therapist was asking you about your ponies, maybe they weren't actually judging the ponies themselves if that makes sense