Some cats are stressed by being petted.

I think you put your finger on it when you said "when they want to be petted". Cats are really obvious when they want petting - they kind of make it a challenge not to pet. The tail in the air, the on your lap whether you want them or not, the head buts, the miaows - difficult to ignore and totally different from food demands.

If a cat obviously wants petting then pet (or you may get bitten - I did know of a cat who would bite if ignored) if it is hiding or reluctant (again obvious) then leave it alone.