I actually take a different approach and tell the individual that I'll be their point of contact even if they ask for a man. If they flat out refuse to work with me; I then ask my manager to back me up.
A good leader will support you and tell the individual that this is how the company does business with you as the point of contact for this stage of the conversation.
By letting them off the hook, I find that reinforces their behavior.
And typically when I don't let the comment phase me and hold my position, the individual then chooses to do business with another company or chooses to fall in line.
Regardless - annoying to deal with!