Verbally defenseless Black Belt

I had taken karate for 9 years and I am a well respected black belt at the karate school I got to. 14 at the time, I was working with a boy whom I always worked with because we were the same size and rank. We were doing a standard drill that involved hitting your partner without full force in the stomach. Then, a man in his twenties said to by partner, “No fair, you get to work with a girl.” He was implying that either I do not hit as hard because I am a girl, (which was not true), or that my partner was getting some sexual advantage by working with me. Although he was older than me, he was a lower rank. I could have yelled at him, or made him do pushups, or anything that deemed an acceptable punishment since I was a higher rank, but i just froze. I did not know what to say and felt defenseless.