▲ 213 points
back
143 comments
While there are some decent points in this post, it ignores another possibility for why Zillow Offers got shut down.
The first explanation, when it was just being paused, was that there was a shortage of labor and materials to do the renovating. Probably true, but not the real reason. It is entirely possible that the second explanation, that it was too hard to model accurately the price 6 months out, is also true but not the real reason.
The real reason might be something the Zillow CEO doesn't want to say out loud. Like, we are near a top in house prices, and it looks like it might fall a lot, or for a long time, or both, before we get back to reasonable house prices that have a chance of going up again. Zillow does not want to say "real estate is about to bust, as an investment strategy", because they are still tied to the real estate market. But, flawed as their analysis might have been, they might have been able to see enough to know that no matter how good their algorithm could be made, they're not going to be able to buy low and sell high if the prices keep falling.
Now, maybe Zillow is wrong again, and housing is not in for a rough ride in the very near future. But maybe they're not wrong, and shutting down Zillow Offers (rather than fixing it) will look like a quite prudent move in a year's time.
Median housing prices have been consistently rising [0] since Zillow started their property buying program in 2018. In their most recent earnings report, Zillow announced $422M in Q3 losses from their home buying program and expects $240M to $265M in losses for homes they plan to buy in Q4 [1]. Rising sale prices coupled with big losses implies Zillow has been systematically paying above-market rates for homes, and there's a lot of evidence corroborating that hypothesis [2]. Maybe we're near a peak in home prices, but prices haven't started falling yet, and if Zillow was paying market rates, they wouldn't be losing money yet.
[0] https://fred.stlouisfed.org/series/MSPUS
[1] https://investors.zillowgroup.com/investors/news-and-events/...
[2] https://fortune.com/2021/11/03/zillow-house-flipping-overpai...
But if we are "near the top", that doesn't really explain why Zillow Offers lost so much money. If we're still on the upslope, and the problem is just that Zillow expects a bust soon, wouldn't they have just been able to sell their houses for a profit? Like I don't think Zillow's model was to buy houses and sit on them for five years with the expectation that they continue to rise in value, so should an approaching downturn really cause them to lose hundreds of millions?
I don't think they lost $420 million on it because they knew what they were doing and were good at forecasting home prices and know things others don't about them. I mean, it's an interesting story you tell, but doesn't seem like the most obvious explanation to fit the data -- which would be, actually, they were bad at forecasting home prices, is why they lost $420 million trying to predict home prices.
The only time I've seen house flipping work at scale is 2010-2012 when those with huge piles of cash could buy foreclosures in a few heavily affected metros, do nothing for 3-6 months then put them back on the market for a tidy profit.
After that brief window closed, you either had to become a landlord for a couple years or employ a crew of handymen on staff to fix all the surprises that come with being a homeowner. You're either a landlord or a builder if you want to actually make money outside of a recession.
Rising interest rates will be a drag on the market. It’s clear those will eventually come, but maybe not for five years.
Pundits have called 23 (or more) of the last 2 housing market drops.
I'm looking at the current RedFin value of my house, the statement that "A lot of these properties are purchased by companies" and "Zillow is getting out of the flipping business" and thinking it was a problem of their own making. They've overheated the market to the point that they can't make any money, and a lot of people were having to spend more cash than necessary on a house that wasn't worth what they paid for it.
I don't have to move, I expect a 30% drop in value...it probably won't be as bad as 2008.
The post seems more like an explanation for how Zillow they lost a bunch of money going long housing in a real estate boom than why Zillow is shutting down Zillow Offers.
I think the issue is around labor.
When a contractor g team has the skills necessary to do a house renovation sufficient to flip, then Zillow is really just financing it. Financing is not really an issue for house renovations because homeowners provide the financing, or small-scale flippers self-finance. In any case a loan is really all that’s needed. I can’t imagine a contractor ever doing contracting-for-hire by a mega corp when there’s so much demand from homeowners flush with cash as-is, to the point where skilled contractors can basically name their price.
If their Zestimate is any indication of their algorithm's proficiency, then that might point to the problem. The range of values they give for a home are frequently ~30% wide. Then they just pick a midpoint. So, it's "we think it's worth $700K, give or take $100K. Not exactly rocket data science.
I previously just chalked that up to the whole estimate being a marketing thing anyway. And figured that if they did have some brilliant data science accurately predicting home prices, why would they give it away?
Now that I see they were consistently overpaying for houses in a rising market--even with a presumed premium baked in for the convenience of an easy sale process--it's looking like their algorithms are indeed suspect.
I doubt it. It would mean Zillow had some type of insider knowledge regarding interest rates or the macro economy. We aren’t in a 2008 scenario either regarding subprime. Perhaps they have an insight that no one else sees?
I think you should also consider first and foremost the CEO's primary stated reason, which is introducing too much volatility for the company. He's said that Zillow aimed to be a market maker with +/- 200 basis points, but that Zillow Offers ended up swinging +/- 700 basis points. This volatility makes Zillow less like a reliable publicly traded company with regular growth and earnings, and more like a housing hedge fund where they need to hold onto a huge pool of capital to cover liquidity swings.
> We have been unable to accurately forecast future home prices at different times in both directions by much more than we modeled as possible, with Zillow Offers unit economics on a quarterly basis swinging from plus 576 basis points in Q2 to an expected minus 500 to minus 700 basis points in Q4.
> Put simply, our observed error rate has been far more volatile than we ever expected possible and makes us look far more like a leveraged housing trader than the market maker we set out to be.
> We’ve got these new assumptions that we’d be naïve not to assume will happen again in the future, we pump them into the model and the model cranks out a business that has a high likelihood, at some point, of putting the whole company at risk, not just the business, but in the more normal case, just causes a ton of volatility in earnings, which is not a great look for a public company. That’s basically what it boils down to.
He also acknowledged the operational issues, but in reality it seems like the volatility + scale of the capital needed is too much for Zillow to risk on top of their existing business and status as a public company.
Stratechery had a good article on this which is where I pulled those quotes from, but it is behind a paywall.
I keep hearing this line that Zillow must have Cassandra-like visions of a housing collapse and that's why they're bailing. The obvious question is, if they have such good foresight, what possessed them to overpay for houses so much in the first place?
I believe we are approaching a top. A lot of otherwise-undesirable places to live have doubled and tripled in price recently (while homes in desirable cities haven't moved much at all), fueled by COVID-19 and remote work spiking demand temporarily. Basically, city-dwellers paying city prices for rural real estate. I don't think this is a sustainable state of affairs and I believe this has created a giant bubble in home prices outside of cities. Something has to give. Either the price of homes in cities doubles, or the price of rural/suburban homes falls back down to earth.
Eh, I think we'll see the nationwide housing market basically do what the Bay Area market did in 2018. Buyers dried up because prices outran incomes, the overbid vanished, and houses basically sold at list, leading to a ~10% decline from what they were selling for at the beginning of the year. Then it stagnated for a couple years until the COVID reopening, and has since shot up ~30-40%. The increase appears to be durable; unlike in many other regions, Bay Area techworker total comp has gone up like 50% since COVID, so houses are actually more affordable.
I feel that there are lots of contractors that can flip houses, with a better profit margin than Zillow. Contractors have the capital to outright purchase a home, no middle person as they do the upgrades, they also know the local market.
Perhaps with large enough capital, the money is in building smaller condos, commercial units. As the initial capital layout is greater than what a small contractor can afford to do.
What markets was zillow operating in? If it was in a particularly constrained market, there is no top over say a decade of ownership. Zillow could hold and rent properties and in 10 years, even if we had 2008.2 tomorrow, they'd be worth more than they are today. Then again that sort of longterm investment thinking doesn't click with modern shareholder controlled companies.
I feel like labor and materials would be included somewhere in the forecasting model for pricing. Fairly hard to ignore either in, well, any business.
Housing is a local game, even at the peak of the housing crisis some places did not fall by more than 10%.
Hi I'm the author of the Prophet article https://www.microprediction.com/blog/prophet which for better or worse is why many people have recently questioned Prophet as a time-series method. (There are better, and certainly more formal critiques before and after - I referenced several).
I just hope those who push back on your article read all the way to the last paragraph. I'll repeat it below. Very well put.
I'm happy to answer technical questions about Prophet from anyone here but again, this is somewhat beside the point, which is...
The requirement that people come to your company knowing how to use piss easy baby tools is an extremely dumb and lazy hiring practice. It is also, unfortunately, a common practice in data science job postings. The aggregate effect of this practice being widespread is that talented people with unusual backgrounds get gatekept out of good paying jobs that they’d be exceptional at. Making fun of the job posting and using Prophet has been compared to gatekeeping. To be clear, the Prophet prerequisite is an actual form of gatekeeping being undertaken by a major company that has actual material impacts on people’s careers. The job post excludes people not based on aptitude, but based on whether they have previous experience and familiarity with a tool they could be introduced to and then master in under 15 minutes. A tweet making fun of the job posting is not gatekeeping. Get over it, LinkedIn clout chasers
I would add that since posting my own less-well worded version of this astonishment I have received numerous DM's from people at large companies who are aghast at the way Prophet is a favorite of management. So whether or not this was a problem at Zillow beyond, say, 2015, it might well be the case elsewhere.
If claims to have good knowledge in TS, but had not played around with Prophet (which is dead simple, takes a few hours - or 15 minutes for you - to see some problems), how interested/knowledgeable are they in TS? (I assume that they look for candidates knowledgeable in that field, and just playing around with prophet dosnt make you knowledgeable)
Especially in TS is saw many PhDs, really focussed on one specific method, not looking left and right and neither are pragmatic about their choices. Which is essential to solve anything in real life. Filtering candidates out, based on criteria like "played around with different stuff" is a good indicator for open-minded people IMHO.
Your blogs are awesome !
I check on them every few weeks to see, What you wrote next :D
This was pretty stupid straw-man. I doubt anyone who has used Prophet is under any illusions as to what it is doing under the hood (after all, the only input is a single time series dataframe...). Companies use it to do quarterly/yearly goal setting and anomaly detection, not deploying it to production to power product features. This whole thing has the tone and substance of a giant "i am very smart" faff.
The author also seems annoyed that people get paid 200k for such little "technical" skill, to which I would ask why he cares?
>to which I would ask why he cares?
Do you think he didn't think enough about the reasons he stated in TFA for why he cares? Or they're not good reasons?
He says, from the employer's perspective,
"$200k a year can attract people who actually know what they’re doing. Maybe a math or econ PhD. Maybe a Microsoft Excel pros with 10 years of finance sector experience";
from the prospective employee's,
"The aggregate effect of this practice being widespread is that talented people with unusual backgrounds get gatekept out of good paying jobs that they’d be exceptional at. ... The job post excludes people not based on aptitude, but based on whether they have previous experience and familiarity with a tool they could be introduced to and then master in under 15 minutes."
If you read the article further it morphs from "you gotta have a better time series tool bro" to an article about "if you don't know what adverse selection is, no amount of highly-paid machine learning engineers will help you".
This, I think, it a more valid point (and it was not only made by the author).
Or, let's say, it is at least a fun implication: When companies like Amazon and Google hire hundreds of Econ PhDs to design platforms, market ops and auctions, it IS amusing to imagine that Zillow thought they could just "Machine Learn" their way through it and they fall flat on their noses.
Very likely this was not a real job post, but rather the statutory post required for a visa application (H1-B, etc). Zillow HR looked at the current employee's background for a set of skills that is rarely found in one person, to keep the number of qualified outside applicants at zero. "Must be a a senior person with recent hands on experience in this novice tool". What they didn't say is, their exist employee used Prophet in a class unrelated to work.
Similarly in Singapore - a lot of jobs require that, even if a direct candidate is ready to be hired, the job is advertised for a month (government mandate). As a result job ads are poor quality and extremely specific to get around visa constraints.
This is an excellent point, and whether or not it happened here it is often what happens, not only in developer jobs.
BTFD.
Zillow issued $4bn corporate bonds that they only have to pay 2% on annually while they pay it back over 10 years. This is way below their annual revenue which was already $2.7bn annually. The bonds were for the expansion of Zillow Offers, but based on their prior revenue. And now, they are trying to sell a bundle of homes for $2.8bn all at once “at a loss”, with no information about the percentage of loss. -5%? -25%? Does anyone else see that it doesn't matter even if it was 100% loss which it clearly isnt?
People out here acting like this is the great financial crisis they’ve missed 12 years of alpha just waiting for.
This is the least leveraged market participant in real estate lol. They flipped a few (thousand) houses maybe at an overall profit, made some of their developer and overleveraged home owner friends happy, and found an excuse to fire a bunch of their workforce!
This game is obvious
> The main reason why competitive market price movements tend to be stochastic and I(1), and tend to exhibit little highly profitable predictable seasonality, is because if that wasn’t true, then people would be able to make free money and markets would no longer be weak-form efficient.
Oh no! Someone invoked weak-form efficiency! I guess every market maker should shut down now?
OP is trying to criticize the hypothetical Prophet-trader because Prophet seems to rely on trivial seasonalities for forecasting. But he is ignoring that the bare minimum configuration for Prophet requires several injections of domain expertise (or at least bias):
- Prophet predicts event frequencies, not prices/events. So presumably it must be used in conjunction with another price model.
- The Prophet user must acquire and group data for their forecast -- which is itself a form of locale-sensitive regularization, and is often >50% of the challenge of calibrating a predictive model.
- Prophet users configure market sizes, change points and event streams ("holidays" are a special case, not the only example) based on external data.
Obviously none of these things saved Zillow, but they are all outside the conditions supported by weak-form efficiency, which gives you a good idea of how useless that concept is in practical trading.
I have zero investment in Prophet and wouldn't use it for a trading model, but it's annoying to see Data Twitter's culture of assuming others are dumb in order to quote grad school stats classes getting rebroadcast elsewhere.
I see some decent points in the post, but they are mostly about modelling.
What I'd like to say is even though you have a good model, that doesn't tell you what to do.
If you know it's going to rain next week, should you go corner the umbrella market?
If you are going down that route, there are a bunch of non-weather things that you'll need to know. Perhaps where to find a cheap source of umbrellas, likely places where people will be the most susceptible to buying, and so on. Basically domain expertise.
The major issue you will have is whether your own presence causes your model to be inapplicable. If a guy sees you at the umbrella factory, maybe he will decide that's a signal that the weather will rain next week? Maybe people will just stay home in that case?
The main thing I see with Zillow is a lack of due diligence. There seems to have been an idea that you can just have some data people, pour in data, and out comes money. In reality there's a lot of legwork in any business, and you pay for experience with money. In this case also your job.
Another ignored issue I see is that people choose the modeling methodology they use, the scoring methodology, and the data that goes into the model.
The reactions act as though the model has a life of its own and that some unlikely error led to these results.
Purchased data/models can frequently be wrong or unhelpful to one’s modeling objective.
Finance also has a number of risks vs rewards with every decision.
Is it possible that Zillow artificially inflated the housing market?
They bought up bunch of houses while having the most popular pricing tool? That's a strong incentive to predict ever increasing prices. Plus it whipped up a frenzy and site traffic. Dangerous incentives all around. You can see how this could be self reinforcing with systems that all want the numbers to go up.
Per this US Census and HUD report [0], there have between 600k (2018) to 1M (2020) single family home sales per year over the years since Zillow started Offers. Per this Barron's article [1], Zillow bought 9680 homes in Q3 2021 and less than half of that in Q2 2021.
So, yeah, they've probably driven the price up a little bit, but their purchases represent maybe 2% of all sales, so I assume the impact that has on the market in general would be pretty small.
[0] https://www.census.gov/construction/nrs/pdf/newressales.pdf
[1] https://www.barrons.com/articles/zillow-opendoor-stock-price...
The question should be, 'To what degree did Zillow influence the housing market?'
Interest rates are low so people have cash. New housing development was lacking even before Covid disrupted supply chains. Now we are facing a labor shortage that could play out in a number of ways, not limited to proletariat uprising or in contrast, a new age of entrepreneurialism.
Like with most market making operations, your main risk is adverse selection. I strongly suspect that this was what killed the zillow operation. Suppose zillow is on average right with their house price prediction, but in 50% of cases they offer 10% too much and in 50% of cases they offer 10% too little. In their backtest this will show as a fair price prediction. However in reality, they will mostly end up buying the houses where they overpay 10%. Only their bad quotes are being hit so they keep consistently overpaying for properties.
Imagine being this angry about a job listing mentioning a Python library...
I don't know, maybe, or only I liked it better when there was so much new houses being built that this wouldn't of happened. Some of us People of Age remember growing up in houses built in post-war baby boom buildings, so there was enough of a housing increase that people weren't paying so inflated prices for houses. Maybe this is what we get when interest rates are so low and so little money is needed for down payments, I know a bunch of people who are started getting into real estate in the last year because even though it's lots of money banks make it easy if you have credit.
It's normal to assert that open source generates millions or even billions of dollars that isn't captured.
In this case, it seems like an open source project destroyed billions of dollars.
That's a new idea for me.
A good prediction is one which is made BEFORE the outcome. All of these armchair data scientists had the opportunity to short Zillow BEFORE it was obvious that zoffer was losing money. No one did? The best outcome which could flow from this is a sort of adverse peer review in data science based on asset trades. To be honest, the zillow data team should have sought this themselves, the community could have debugged them.
Extremely well written article for someone who would like to dig deeper into forecasting, actually giving someone ne valuable links about the prophet tool that i used and recommended. I agree with the author that it is strange that the job posting only mentions the prophet tool, which is rather like a very basic and one particular library that i usually recommend to undergrads for their senior project.
I'm surprised that nothing here has been written about human in the loop. All time series models are flawed in some capacity, and are going to have difficulties in the current market given how it won't necessarily match historical seasonalities. Just having more human safeguards to flag wacky outputs seems to be a more reasonable explanation than not handling Prophet's shortcomings properly. Zillow would have likely faced similar issues with other time series packages as well
Blind is suggesting another reason this went wrong - misaligned incentives and overriding the predicted prices
I use Prophet quite a bit because it allows me to add exogenous variables (temperature, wind, solar etc.) easily and assess their impact with reasonable outcome for electricity load analysis.
One thing that baffles me is when people try to use it to predict weather itself.
It's instructive to recall that the original use case for Prophet at FB was 'good enough' forecasts for a large number of time series. Both the modeling choices and interface are optimized for that point in design space.
I’m surprised I haven’t seen this rumor yet
There isn't a company or individual who has models that can accurately predict the future of housing prices. There are only charlatans who claim they can achieve the impossible. This guy can do no better, but he'll do his best to convince capitalists that he's got the right stuff.
My opinion is that we are not at the top of the housing market. People are realizing that it is not fun to live in dense areas.
Local markets may ebb and flow, but desirable houses will rise.
Phoenix is particularly dense and the house prices are cheap, which is why I believe Zillow sold them all-- they realized their actual profits weren't as good as what they projected.