It really depends on what your definition of exotic is.
If you mean any unusual pets beyond the most popular, then I do not see anything wrong with it. All pets were taken from the wild at some point.
It might not be as easy to get quality pet care for unusual pets. It is really about supply and demand. Since so many people are having dogs and cats, most vets only take care of dogs and cats. Beyond the quality of vet, I really do not see any downside of it.
If your definition of exotic means endangered species, it can be a whole different meaning. I believe endangered species should be left alone, except for the ones who had been domesticated for generations. Trade newly caught wild endangered species for pets is not right.