It really does sound like you have one of those cats that just want to be left alone ! At least it is good to know that some cats are just this way, and that he is happier not being petted; and you don't have to keep worrying that you are leaving him out.
I have read that some times animals can "see" our visual thoughts, and understand what we are thinking that way. If you want to try an experiment, you could visualize this cat really enjoying it when you pet him. As you are able to stroke him, think about him happily purring, tipping his head up to have his chin scratched, and maybe even rolling over for a tummy rub; all the time, with a look of pure contentment on his little kitty-cat face.
It might not do anything, but then again, it might change his attitude about being petted when he realizes it should be a joyful experience for him.
In any case, it will be fun for you to try, and let us know how well it works. A week or so of doing this should make a difference if it is going to make one.
