Horse Swishing his tail?

When I'm grooming a certain horse at my stable, he will constantly be swishing his tail, despite the lack of flies. Although I do know that swishing his tail often means he's frustrated/angry, its so consistent that that doesn't seem to be the case. He also does this even if I just touch him or pet him. What's going on?