JOIN USBritain's Press Association news agency said Cameron told journalists at a press lunch about the time Health Minister Simon Burns backed into Speaker John Bercow's car. When Bercow said he was not happy, Cameron said Burns retorted: "So which one are you?"
Bercow stands about 5 foot 6 (1.68 metres).
The Walking with Giants Foundation charity, which helps people with restricted growth, said Cameron's gibe was "downright appalling." Its co-founder, John Connerty, said it was "totally unacceptable" to have "the most influential person in the country making distasteful references about a person's lack of height."
Cameron's office said his "lighthearted comments" were not meant to offend.
The speaker's wife, Sally Bercow, hit back on Twitter, writing: "If Mr B's a dwarf then I'm Snow White. And David Cameron is definitely Dopey."