Gregg Wallace has issued a defiant response to mounting allegations against him, dismissing his accusers as "middle-class women of a certain age".