Like your dad said, the puppy just hasn't been around people that much but since he's used to his breeder, of course it will tolerate him. What your dog needs is time and patience to get used to his new environment. If you've already had taken it to the vet and there's nothing physically and internally wrong with it, chances are he's just scared and anxious. Separation anxiety is also not uncommon to them.
The first thing you should do is to be calm and patient yourself because dogs can easily sense their owners' emotions. You must not be too worried because it will give them the idea that's something's going on that would make them even more scared. Don't rush it into doing something you'd want it to do. You don't have to lose sleep over it as well because all you need to do is to show it your understanding and to make it used to a routine like his feeding schedules, play time, etc.
