Not sure if the chameleon was backing off because of the toy or because of your hand lol. 
You can try to put the chameleon toy down, and observe the behavior of how the real chameleon interact with the toy.
By the way I had no idea how small they are. You've shown photos of your chameleon before. It looked like a dinosaur. With your hand in the picture, I just realized they are tiny.
| 
