I can relate! I haven't been on a vacation since I adopted my bird over two years ago. It's partially because I simply can't afford to take any trip at the moment, but the thought of leaving my bird's care to somebody else scares me. My parents are fine with bird-sitting and I know they'd do a fine job taking care of him, but I worry about him thinking I abandoned him or something. I so wish there was a way we could communicate "Don't worry, I'll be back in a few days" to them. I also worry about rushing home after work to check on him.
The only recommendation I can think of (and one I'll probably do myself once I have the funds to do it) would be to go away for just one night. Before you go, have your pet sitter spend some time with the cat while you watch, so they get comfortable with each other. When everything turns out okay for you and the cat, try going away for a weekend next. That way, you'll be a little less nervous if you have to go away for longer, and your cat will learn that going away for a little while doesn't mean "never coming back."
