There was a movie I watched a lot when I was a kid called Racing Stripes. It was about a zebra who wanted to be a racehorse xD So I've definitely though about it! There are actually horse-zebra and pony-zebra hybrids, although as with any domestic + wild hybrid, I question the ethicality behind it.
However, I have been riding and owning horses for well over a decade, so I can offer some input on the pony vs horse situation.
Ponies tend to have BIG attitudes. I have never ridden a pony that didn't. That's not to say they're terrible to ride, but basically if they decide they don't want to do something, they will not do it. I had been riding big ol' horses for about five years and my first actual fall while riding was from a pony named Stella. She just decided she didn't want me on her anymore and dumped me in the dirt.
Horses don't usually have this same natural predilection towards being difficult, although of course they can be ornery depending on their natural personality and breed. I generally find the bigger the horse, the more gentle and friendly they are.