Most pet birds can easily become scared. Loud noises of any kind is the most common source of fear for pet birds at home.
I do not believe your birds freaked out by a hat the way you think.
Many years ago I have watched something on Animal Planet about a scientist got close to a wild eagle. The eagle got used to her presence and would see her as friendly. As soon as she started to wear a hat, the eagle didn't recognize her any more and tried to attack her when she got close to the nest. I believe your birds weren't scared of the hat itself, but rather they see you as a stranger when you wear a hat. Animals without good sense of smell might use the appearance to define individual human beings. What we wear might become a part of our appearance to some of them.
