Raising anti-Britain slogans, the furious protesters, most of them students from Tehran universities, urged the government to expel the British ambassador from Iran, Xinhua reported.
Some of the protesters clashed with the embassy guards and climbed over the gate of the compound.
The rampage occurred a day after a vote by Iran's highest legislative body, Guardian Council of the Constitution, unanimously approved reducing diplomatic ties with Britain in response to its "hostile" policy against Iran.
Britain severed all links with the Iranian banks last week as part of its new sanctions imposed in the wake of a report by the International Atomic Energy Agency (IAEA) that expressed concern over Tehran's controversial nuclear programme.
A witness told Xinhua the protesters set fire to the British flag after pulling it down and replacing it with Iran's national flag.
Local TV showed some protesters smashing windows of the embassy building.
Riot police failed to prevent the students from entering the embassy, the semi-official Fars news agency reported.
