A blonde

I’m a manager in a male dominated work environment. One of my colleagues sometimes refers to having a “blonde moment” when he forgets things or similar. I find this offensive because it is never used to mean blond men and so it belittles me, especially in large meetings where I am the only woman.