The UK High Court had previously backed government claims that an inquiry into whether abuse was systematic was not needed as a team had already been set up to look into the allegations.
But three judges of the UK Court of Appeal ordered the Ministry of Defence (MoD) to reconsider its decision.
The MoD said it would examine the judgment "very carefully", the BBC reported.
Some 128 Iraqis have complained of torture and inhuman and degrading treatment by British soldiers and interrogators in Iraq between March 2003 and December 2008.
Lead claimant Ali Zaki Mousa, from Basra, alleges he endured months of beatings and other abuse in the custody of British soldiers between 2006-07.
Last December the High Court ruled a fresh inquiry was unnecessary as the government had set up the Iraq Historic Allegations Team (IHAT) to investigate.
But appeal judges ruled that the IHAT lacked "practical independence" because it included members of the military police, who might themselves be accused of wrongdoing.
They also found that other inquiries had failed to fully meet Article 3 of the European Convention on Human Rights, which protects against inhuman and degrading treatment.
A Ministry of Defence spokesman said: "We will examine the judgment very carefully and consider next steps."
Defence Secretary Philip Hammond has until 30 November to decide whether to challenge the ruling in the Supreme Court.
Phil Shiner, lawyer for Iraqi claimants, said the MoD was "deploying every dirty trick in the book" to avoid accountability.
"There must now be a judicial inquiry into the UK's detention policy in south-east Iraq. It is something we have been calling for since 2004.
Two public inquiries have already been launched into similar claims.
The first inquiry, into the death of 26-year-old hotel worker Baha Mousa in UK military custody in 2003, reported in September and blamed "corporate failure" at the Ministry of Defence for the use of banned interrogation methods in Iraq.
In November, 2010, the MoD announced details of a second public hearing into allegations that 19-year-old Hamid Al- Sweady and up to 19 other Iraqis were unlawfully killed and others ill-treated at a British base in May 2004.
