My Cats Keep Following Me Into The Bathroom

I'll be honest, as much as I love my cats, it creeps me out when they do this. The last thing I want to be doing is petting my cats while I am trying to go to the bathroom - it just seems really unsanitary and gross to me.

The funny thing is, they actually get annoyed if one of us happen to see them using the litter box. They act like we caught them red handed doing something bad. They'll stop and give us this neurotic stare until we walk away and leave them alone.