Do most dogs treat people based on their physical apparence?

You are right in that they know peoples smell. However if you remind him of someone that he had a bad experience with, that would be enough. In his doggy mind if you are anything like the offender you probably are not to be trusted. People are not much different. Ever meet someone you took an immediate dislike to? Even though they did nothing to cause you to dislike them? Subconsciously you picked up on something that caused you to dislike them. Sometimes it could be they remind you, in some way, of someone you dislike.