On the internet these days, Wall Street is often blamed for killing the American Dream of homeownership.

From TikTok videos claiming that institutional investors are bulk-buying homes at inflated prices and forcing people to rent until they die, to a Medium article claiming 44% of all single-family home purchases last year were by private equity firms, Wall Street has emerged as a villain in public discourse. The narrative has persisted, despite economists and analysts pointing out the widespread claims are wildly incorrect.

Who is right? It’s difficult to blame institutional investors like Blackstone, BlackRock, and Invitation homes for the nationwide housing crisis; most estimates put their ownership at less than 5% of single-family rentals and less than 1% of all single-family homes. But desirable markets like Atlanta, Dallas, and Charlotte tell a different story—Wall Street owns more than 4% of single-family homes in Atlanta, for one, and could be a factor in rising housing costs.

Still, hedge funds, corporations, asset management firms are often blamed for unaffordable housing by everyday Americans; home prices and rents rose substantially during the pandemic-fueled housing boom, mortgage rates more than doubled shortly after, and so many people are house poor. Politicians jumped on the bandwagon blaming Wall Street; Democrats in Congress introduced a bill toward the end of last year, banning hedge funds from owning single-family homes in the country.

“You have created a situation where ordinary Americans aren’t bidding against other families, they’re bidding against the billionaires of America for these houses,” Senator Jeff Merkley, who introduced the bill alongside Representative Adam Smith, said, according to the New York Times. “And it’s driving up rents and it’s driving up the home prices.”

Wall Street’s share of the multifamily rental market is much higher and traces back much further. But in the aftermath of the Great Financial Crisis, institutional investors entered the single-family rental space, and because so many homes were foreclosed on, they scaled quickly, buying houses in bulk for cheap. While they benefited immensely, institutional operators also kept the housing market from hitting rock bottom. Interestingly enough, their existing home portfolios still reflect that entrance into the market more than a decade ago.

“They map almost one to one to metros that suffered the highest rates of foreclosures and deepest housing price cuts,” Moody’s Analytics Senior Economist Ermengarde Jabir told Fortune, citing Phoenix as an example.

The truth about why housing has gotten so unaffordable is much more nuanced and goes back more than a century.

Wall Street doubles down, location matters

In January, Blackstone announced the $3.8 billion acquisition of Tricon Residential, a deal that will give it the third-largest single-family portfolio across the country, following Progress Residential and Invitation Homes, according to Parcl Labs, a real estate analytics company. Collectively, these three institutional buyers would hold more than 200,000 single-family homes across the country. But institutional investment isn’t distributed evenly, Parcl Labs’ cofounder, Jason Lewris, explained to Fortune.

“They went into the exact same markets and almost half of that national portfolio is in six U.S. housing markets,” he said: Tampa, Charlotte, Atlanta, Houston, Dallas, and Phoenix. “When you look at the share of ownership within those markets, it’s 20x or above,” with respect to their national holdings.

To make matters worse, they’re not evenly distributed within those markets. “In almost all cases in those top six markets, they’re concentrated in just a handful of zip codes … now you’re looking at a situation where they own more than one in 10 homes,” Lewris said.

Think about Atlanta, the housing market with the largest institutional presence, where firms own 4.4% of single-family homes. Investors have a substantial impact within the market, and their presence in Atlanta and other heavily concentrated metropolitans can be a burden to the average buyer. “They have special financing vehicles, and when they buy in a market, they really buy in a market, so they account for a big share of all trades,” Lewris said.

The average home value in Atlanta is $387,216, nearly 13% more than the average home value nationally. Meanwhile, the median rent for all bedrooms and all property types in the city is 5% higher than the national median. From an analysis on institutional investors, Jabir and her colleagues found rents for single-family rental properties are higher in metropolitan areas where institutional owners operate.

Institutional investors are also buying homes where people want to live, Taylor Shelton, a geographer and assistant professor at Georgia State University, said. Buying a home in rural Mississippi is a different ballgame than the metropolitan Atlanta area, where as previously mentioned, institutional buyers largely are. Shelton and his colleagues’ research also suggests there are a few places where institutional investment is concentrated, with Atlanta being the most prominent. Institutional operators look to invest in places with some distinguishing characteristics, such as a lack of strong tenant protections and a growing demand for housing.

Metropolitan Atlanta’s population increased by nearly 67,000 people between April 2022 and the same month last year to more than five million people; the city of Atlanta’s population grew by more than 14,000 people, which is nearly three times as much the prior year, to over 500,000 people. With a growing population comes an increased demand for homes.

“These firms are all trying to capture some corner of the market and really exercise significant market power,” Shelton said. “It’s rarely in a pure monopolistic sense, but in a kind of oligopolistic sense.”

Institutional firms have algorithmic buying strategies; they can buy homes as soon as they go on the market and place an all-cash offer over asking, Shelton explained. “Institutional investors are not to blame for all aspects of the housing crisis everywhere equally,” but they play a role in Atlanta, and in the Sunbelt, for starters, he said.

How did housing get so expensive?

To start, the country just doesn’t have enough homes to house its growing population. There are several varying estimates on the housing shortage, although it’s generally understood as a deficit between one million to more than six million homes. As Tobias Peter, a senior fellow and co-director of the American Enterprise Institute’s housing center, sees it, the government promoted demand for housing via lax underwriting and down payment assistance programs against a limited supply.

All the while, building homes is harder. In the 1920s, Herbert Hoover, as the secretary of commerce before his presidency, created a model law for zoning that was implemented across individual states and localities. Our housing shortage developed from there, and worsened in the years following, Peter explained, because they’ve “hindered the market from building more housing.”

Zoning laws, coupled with a shift in the American mindset and the environmental movement of the 1970s, complicated development to the extent that what little housing gets built is mostly expensive housing that can offset steep costs and regulations. There’s a missing middle, and it’s a consequence of “a self-inflicted wound,” Peter said. In his view, for politicians in particular, “Wall Street is always a convenient scapegoat.” The real issue, he said, is government regulatory failure that has resulted in a market that’s not building enough housing. For Wall Street firms, housing is a good investment, particularly when there’s a widespread shortage of it, Peter said.

“Housing has been unaffordable long before these Wall Street firms came into the marketplace,” he said. So if we were to build more housing, homes would be more affordable and institutional operators wouldn’t have the market power they currently have.

Institutional investors are not the only problem, and we can’t simply say institutional investors are the reason why Americans can’t afford single-family homes, Moody’s Jabir said; she pointed to the disparity between home price appreciation and wage growth over the last 30 years. “The ratio of a median house price to median household income in the 80s was half of what it is now,” Jabir said, and “since the Great Financial Crisis, building has plummeted.”

“It’s a confluence of factors where the institutional ownership component is a part of the problem, but it’s certainly not the only problem,” Jabir continued. “It goes back to, where have we gone wrong on the path to the American dream?”