The WHO said the Games would "not significantly alter" the spread of the virus that has been known to cause serious birth defects.
To press on with the Games in Rio, the second most affected city in Brazil by the Zika crisis, would be "irresponsible" and "unethical," the letter argued yesterday.
"Our greater concern is for global health. The Brazilian strain of Zika virus harms health in ways that science has not observed before," said the letter, signed by experts in the US, Britain, Canada, Norway, the Philippines, Japan, Brazil, South Africa, Turkey, and Lebanon, among others.
"An unnecessary risk is posed when 500,000 foreign tourists from all countries attend the Games, potentially acquire that strain, and return home to places where it can become endemic," it said.
"Should that happen to poor, as-yet unaffected places (eg, most of South Asia and Africa) the suffering can be great," the letter added.
Zika can cause birth defects, including a devastating syndrome known as microcephaly in which babies are born with unusually small heads and brains.
Nearly 1,300 babies have been born in Brazil with the irreversible defect since the mosquito-borne Zika began circulating there last year.
The World Health Organization and top US public health officials have called on those travelling to Brazil to take precautions against mosquito bites, and have said pregnant women should avoid areas where Zika is circulating, including Rio de Janeiro.
The Olympics and Paralympics, set for August 5 through September 18, "will take place during Brazil's wintertime when there are fewer active mosquitoes and the risk of being bitten is lower," WHO said this month.
And on Thursday, the top US public health official, Tom Frieden, said "there is no public health reason to cancel or delay the Olympic Games."
But the open letter, signed by doctors and researchers at leading universities around the world, expressed worry that officials are not adequately protecting the public against the danger posed by Zika.
"It is unethical to run the risk, just for Games that could proceed anyway, if postponed and/or moved."
The letter urged the WHO to "conduct a fresh, evidence-based assessment" of the situation in Brazil and its recommendations for travellers.
Given the big financial investments at stake, the letter questioned whether the UN health agency was able to give a non-biased view of the situation.