Do most dogs treat people based on their physical apparence?

There is this little dog always standing in front of someone's house, and he would only look at me from a distance when I pass by ever since the first time "we" met lol. I don't know the owner, so the dog never met me before either. Once the dog was closer to the road when I passed by, he turned and ran back home when I got close. I was absolutely not in a threatening manner.

The other day, I was walking near that place again, and saw a little lady passed by that house like I used to, and the dog immediately barked at her.

What do you think? Why did the same dog treat us differently while we were both strangers to him?
Because I am taller and bigger than the little lady?