The Anti-Defamation League joined the debate against the building of a mosque and Islamic Centre and said this is was not a question of rights of Muslims but a question of what is right.
The proponents of the Islamic Centre have said it would be a centre to propagate a message of peace and inclusiveness and promote the moderate faith.
But the opponents say the establishment of an Islamic Centre would hurt the sentiments of the victims and relatives of those who were killed in the September 11 attack.
Jewish organisation, the Anti-Defamation League, said in a statement that the bigotry expressed by some people against the idea is unfair, and wrong.
"The proponents of the Islamic Centre may have every right to build at this site, and may even have chosen the site to send a positive message about Islam... But ultimately this is not a question of rights, but a question of what is right," it said.
"In our judgement, building an Islamic Centre in the shadow of the World Trade Centre will cause some victims more pain – unnecessarily – and that is not right," it added.
The debate about a building a mosque on the Ground Zero site has been raging in the US for several months.
In May, a New York community board approved the building of a 13-story mosque and Islamic cultural centre close to the site but the project still faces resistance from some groups.
The plan is being pushed by a Kuwaiti-born imam, Feisal Abdul Rauf, and will cost a USD 100 million.
"My colleagues and I are the anti-terrorists. We are the people who want to embolden the vast majority of Muslims who hate terrorism to stand up to the radical rhetoric.
"Our purpose is to interweave America's Muslim population into the mainstream society," Imam Rauf wrote in the New York Daily News in May.
He said the centre will be open to all regardless of religion and will be a centre for all New Yorkers.
"What grieves me most is the false reporting that leads some families of 9/11 victims to think this project somehow is designed by Muslims to gloat over the attack".
Parties that oppose the building a mosque, which will be called Cordoba House, insist this project is inappropriate since the terrorist attacks were carried out by extremist Muslims.
The groups that support the plan assert that building a mosque will be a symbol of tolerance in New York City, which is home to people from all around the world practicing different religions.
They also say that the Islamic centre will be a venue to promote moderate Islam and practice interfaith dialogue.
New York City mayor Michael Bloomberg, for instance, came out in a powerful defense for the Cordoba House to be built near Ground Zero.
"I think it's fair to say if somebody was going to try, on that piece of property, to build a church or a synagogue, nobody would be yelling and screaming," Bloomberg said in May.
"And the fact of the matter is that Muslims have a right to do it, too".
The Islamic center will have a swimming pool, basketball court, meeting rooms, a 500-seat auditorium, banquet facilities, theatrical programming, art exhibitions and cooking classes, according to Rauf.
But the Jewish group noted that "the controversy which has emerged regarding the building of an Islamic Center at this location is counterproductive to the healing process".
"Under these unique circumstances, we believe the City of New York would be better served if an alternative location could be found," it added.