As the soft glow of the sun began to rise over the piazza at the heart of this city’s Olympic celebrations, the statue of the Viscount of Mauá, from whom this square takes its name, silhouettes against the amber sky. The Olympic flame will soon be burning nearby. An extravagant Museum of Tomorrow, designed by the Spanish architect Santiago Calatrava, juts out into Guanabara Bay, while a gleaming Olympic Boulevard stands as the centerpiece of Rio’s redeveloped port.

The Viscount, a 19th century industrialist, became one of the richest men in the world at the height of slavery in Brazil, though the history books record that he was an opponent of the practice. His presence is a reminder that Rio de Janeiro cannot so easily escape its past. Slavery is woven into the fabric of the old city. Rio hosted one of the biggest slave ports in the world, and its legacy echoes down the ages, from the foundation of the first favela shantytown nearby to the growth of samba music, created by former slaves. Though the slave wharf and many public records about the extent of slavery have been destroyed, the past—with which Brazil has never truly reckoned – cannot easily be forgotten.

More than 5.5 million slaves set sail from Africa bound for Brazil between the 16th and 19th centuries. 4.9 million arrived alive, and of those, more than two million docked in Rio de Janeiro, with half of those coming in the first half of the 19th century alone. In contrast, less than 400,000 slaves disembarked in mainland North America in total. “The consensus for a long time was that the British were the main slave traders and the Caribbean was the main destination,” says Daniel Domingues da Silva, a historian at the University of Missouri, Columbia, who works on the Trans-Atlantic Slave Trade Database. “Now it’s becoming clearer that the Portuguese were the main traders and Brazil was the principle destination for slaves on the Atlantic.”

From when Valongo Wharf, one of the biggest slave ports in the world and a short walk from Praça Mauá, was built in 1811, hundreds of thousands of Africans disembarked there. Historical accounts of the market tell of slaves constrained by shackles, chains and whips sitting next to the bodies of those who died in the Transatlantic voyage. Up to 40,000 were buried in the nearby Cemetery of the New Blacks, where the bones of the dead were often crushed to allow more burials. Slaves were sold all over Brazil—at the time still a Portuguese colony—to work on sugar and coffee plantations. Two years earlier, in 1809, the Military Division of the Royal Police Guard had been formed; a precursor to the city’s current Military Police. Then as now, the division’s main purpose was to maintain public order. The public beatings they meted out to slaves, who made up half of the city’s population, became infamous.

By the time the slave trade was finally ended in 1850—though slavery as a practice would continue for decades more—Valongo had already been buried to allow for the construction of a grander wharf. It was then largely forgotten for more than a century and a half. “I believe they buried it to eliminate the evidence of slavery,” says Washington Fajardo, president of the city’s World Heritage Institute. Brazil was the last Western country to outlaw slavery in 1888, a year before it declared itself a republic. In 1890, the government authorized the destruction of most records about the trade, with the aim of erasing the “stain” on the country’s honor.

Over the years the port remained strongly Afro-Brazilian, with descendants of freed slaves joined by their brethren from the country’s northeast, and the neighborhood became known as Little Africa. In 1897, the city’s first hillside favela shantytown, built over the port and called Morro da Providência, was formed by poor locals and veterans of a recent civil war. At a nearby square called Pedra do Sal, at the foot of a rock where slaves unloaded salt, the Rio variant of samba was born. (The word may derive from the Angolan Kimbundu word semba, which means an invitation to dance. Native African languages were spoken here well into the 20th century.)

Even after slavery, a physical divide endured in the city, with blacks living on the port side of a main thoroughfare in the city center, an area blighted by poverty and homelessness. “The fact that this side of Avenida Presidente Vargas was forgotten by the government undoubtedly had a racial aspect,” says Giovanni Harvey, a port resident and a recent advisor to President Dilma Rousseff on racial equality. In the 20th century its colonial buildings and narrow, paved streets fell into disrepair.

After the Olympics were awarded to Rio in 2009, a $2 billion Marvelous Port project was unveiled to regenerate the area. The plan was to update transport links and turn the area into a new business district, including Trump-branded skyscrapers. In 2010, Morro da Providência became the latest favela to get a police pacification unit (UPP), a community policing initiative to take control back from drug gangs. As part of the renovations of the port, the remains of the Valongo Wharf were discovered. Artifacts recovered included amulets believed to have been worn by slaves on the voyage to guard them from harm. “This is one of the rarest historical sites in the world to tell the story of slavery,” says Fajardo.

The developments led to fears that the area’s Afro-Brazilian character was being forgotten. Morro da Providência, set far above and a world away from the skyscrapers under construction in the Marvelous Port, got a $23 million cable car as part of the project, but many basic services there are still lacking. “The story of this favela is principally one of racism,” says lifelong resident Eron Cesar dos Santos, 49. “It is important that the people here have a part in this story.”

The favela was “pacified” by a police unit in 2010, but now – despite sitting just above the state security headquarters – the gangs are back in control, openly brandishing guns on the street. Last year five Military Police were arrested after a cell phone video showed them executing 17-year-old Eduardo Santos and planting a gun in his hand to make it look like self-defense. He is one of 2,600 people killed by Rio de Janeiro police since 2009, mainly by the Military Police. “What happened always happens in Rio. It’s just that there was a camera,” Santos says. “Its an undeclared war.”

The hillside neighborhood of Santa Teresa, where colonial homes for the rich sat overlooking the port, also became neglected in modern times but has also seen a revival in recent years. One such mansion is now a five-star boutique hotel, patronized by Amy Winehouse, Alanis Morissette and Robert Plant. But even here, there is an echo of the past. In 2014, federal prosecutors accused its then-owner François Delort of subjecting six workers to “conditions analogous to slavery.”

The workmen had been recruited from the interior of the North-East state of Bahia and had gone without food for the 18 hour journey, prosecutor Fábio de Lucca Seghese said in a denouncement, before being put up in a stinking rat-and-insect-infested house while working on a renovation of the $350-a-night hotel’s ballroom. At the time, the Hotel Santa Teresa, which is now under new ownership, denied any wrongdoing. The case is pending at Brazil’s Superior Court of Justice.

The case is typical of Brazil’s fight against modern slavery, which it defines as forced labor, debt bondage, degrading conditions that violate human rights or overwork that threatens life or health. Since 2003, raids by government inspectors have rescued more than 45,000 workers from such conditions. “Brazil abolished slavery more than a century ago but did not change its worker exploitation model,” Luiz Camargo, the country’s chief labor prosecutor, said last year. “We only changed the people who are exploited. Back in the 18th or 19th century, you had only black slaves, and today you have slaves of all colors.”

The Valongo Wharf is just a few steps from the Olympic Boulevard, a pristine pedestrianized highway alongside red-brick warehouses with Coca-Cola branding. Last week, a huge mural called Ethnicities by artist Eduardo Kobra was unveiled there for the Olympics. It depicts the faces of members of indigenous tribes from five continents. A plan exists to memorialize Valongo, too, by creating a landscape invoking the trees and red clay soil that exist in both Africa and Brazil, along with benches in the shape of white ribbons, an African tradition used to mark where ancestors lie. “It’s an allegory for Afro-Brazilians, that they could flourish in both environments,” says Sara Zewde, its designer. But that plan has not gotten official approval. “The legacy of slavery and racism is still here,” says Harvey. “We have an artificial paradise in Rio de Janeiro.”