Having a horse is definitely expensive, but even more, it takes a lot of learning, to be able to ride one properly. The movies give a very distorted idea of what riding is like, since the rider just jumps on and gallops off, and you never see any of the preliminaries that go into getting the horse ready to be ridden.
The horse there never needs to be fed, watered, groomed, to have shoes put on, or to be wormed; all things that need to be done when you have a horse.
Even when you start out with a trained horse, if you are a novice rider, you need to take lessons to learn how to control the horse, and to communicate with him. It is rather like taking Drivers Ed in school to learn how to drive a car.
Your car can sit in the driveway for a week, and be fine, but your horse has to be fed and watered every day, and usually brushed , and have the hooves cleaned out. If he is in a barn, then the stall has to be cleaned, and the manure hauled off and dumped.
So, even though having a horse, and going riding is wonderful, it takes a lot of commitment to decide to have one of your own.