Why is my horse so grumpy?

My horse - also tb gelding off the track, pastured for 2 years and back to work last year - does EXACTLY the same! He does it when I groom his belly, put on his saddle, tighten his girth or do the straps up on his blanket. I think it is because he hated racing and his saddle probably didn't fit properly then. We've had it checked (make sure you do the same) and it fits perfectly. I think he now just ociates me doing his girth up (and the rest) with racing and pain. (I have nothing against racing but Hicks probably wasn't very well cared for being a Sh*te racehorse!) I am just living with it at the moment but it's getting better after a year. Make sure you have his back and saddle checked. If it really bothers you, go to a specialist about it and see what they say. Hope this helps!