back
155 comments
I always feel that a good deal of data-driven anything is basically modern divination. The median data-driven decision might as well have been based on haruspicy.

You can construct experiments to demonstrate damn near anything is true, and without (or even with) a rigorous background in statistics and scientific methodology, it's very hard to examine the supposed evidence.

I know I say this a lot, but constructing solid experiments is time consuming and difficult. It's hard even if you you're trying to be fair. Professors routinely get this wrong despite a literal decade of formal training in the scientific method and half a lifetime of practical experience with this exact craft; despite how fucking up puts their career, reputation and livelihood in jeopardy. Not only does this happen a lot, bad science routinely slips through peer review as well.

That's the state of affairs in the spaces that in general aren't extremely adversarial, where the people involved in constructing the experiments in general don't have a a vested interest in trying to mislead you. (Scientific fraud still happens of course, but Jan Hendrik Schön is the exception and not the rule)

There's always the famous case of Uber made 120M cuts[2] in their 150M adspend, nothing absolutely changed in their app installs. [1]

[1] https://thehustle.co/01072021-uber-ad-spend/

[2] https://www.forbes.com/sites/augustinefou/2021/01/02/when-bi...

Meanwhile, people with no scientific training are entrusted to run A/B tests and other web experiments.

Practically all web marketing experiments and analysis is invalid, and not replicable. But it pays the bills for everyone to pretend its legitimate.

> ...don't have a a vested interest in trying to mislead you.

I witnessed this.

I briefly worked on the recommendations team for a fashion retailer, doing yeoman plumbing and maintenance. Basically an in-house equiv of RichRelevance.com.

So, so much data and logging.

Alas, after cutting out all the obfuscation layers and steps, the "lift" attributed to the team's exquisite algos proved illusory. Some >80% of the conversions came from the "recently viewed" rule. Whereas the conversation attributed to the algos consuming most of our dev and compute resources was basically noise.

I wouldn't say our (very smart) data scientists were fooling themselves, necessarily. More like the feedback loops were never fully validated. (Familiar story of non-programmer domain experts hoisting a POC, which then was promoted to production. Maintenance hell.)

Coincidentally, FWIW, my bro has worked in ad tech (Avenue A/Microsoft, RichRelevance, others?). He's got a math skillset. Was a true believer. It took a long time, but he eventually lost his "faith" that digital advertising (as they were doing it) was effective, on the balance.

Also FWIW, during my brief stint, I advocated "personalization". What I imagine StitchFix is doing. I'd love to know, for real, how they're doing. Like the split between algos and human "style consultants".

> You can construct experiments to demonstrate damn near anything is true, and without (or even with) a rigorous background in statistics and scientific methodology, it's very hard to examine the supposed evidence.

One time I was at this silly daylong corporate team building event where they assigned us to groups and had us do "experiments." As it happened one of my group members has a PhD in Particle Physics and the look on his face was something else. Needless to say he didn't bother telling the take charge type that her experiment design was entirely nonsensical, because why bother, but I was amused.

One doesn't even need to observe all this to recognize the simple fact that craigslist is much nicer to use than twitter, reddit, tiktok and craigslist hasn't changed in decades I dont think.
Which explains why I've been getting ads for brides' maid dresses.

However, it also seems as if a tuned LLM could look at the corpus of my on-line profile (mail, FB, HN) and produce a model of my attention and commentary (positive/negative). Then that model (or several models) could be tested against various ads to determine which are more likely to produce positive responses. Right now I think that would be too expensive except for very high value ads/targets, but perhaps pricing will come down and non-keyword based advertising might appear.

Hmm, I feel like this analysis comes from trying to rationalise what’s happening. From my experience the truth is simpler, and much more disappointing: Advertising is about statistics, and if some wonky new data model creates better conversions the advertisers don’t care how it’s underlying rule sets work.

The cost of pushing ads to people who already have mattresses is insignificant to capturing those that are about to buy them, statistically speaking.

The disappointing thing about this trend is that even if the rule sets are obviously wrong, yet still provide positive results means that their operators have little reason to change them. Statistics don’t work for individuals.

Case in point is credit rating and insurance underwriting

> The cost of pushing ads to people who already have mattresses is insignificant to capturing those that are about to buy them, statistically speaking.

This is called conversion rate and for some businesses it can be surprisingly tiny while still making that business completely viable.

I would hazard that rates in the mattress industry are almost preposterously low (<5%) but margins are so high that it doesn't matter in the slightest

Yep, in the past they would advertise in newspapers or magazines or TV or radio where the vast majority of the audience had zero interest in buying a mattress anytime soon. Narrowing that down to "people who have recently searched for a mattress" is a win, even if that includes some people who already bought one. Despite what people think about Google being all-knowing, it's less likely that they know you bought a mattress than they know you searched for one.
There are a whole class of fallacious modes of thinking that revolve around assuming the world is smarter than it actually is and that people are more competent than they actually are.

The most common these days is “conspiracy theory,” which is really an optimistic view that starts with the assumption that someone understands what is happening and can coherently control it. If these conspiracies were true it would mean that all we’d have to do to fix major problems is get “them” out of power or convince “them” to become the good guys. Very, very optimistic.

I think I've outlined this before on HN, but I've worked with a proper sleazy DSP, who I found out were basically targeting vulnerable folks with ads. These guys have zero ethics and less morals and have an attitude of "if it works it works, and to hell with the consequences".
Those advertisers are responding to signals in the junk data the OP is talking about. You’re both right.
This is yet another of the recurring "online advertising doesn't work" articles, which have been published regularly for the past 20 years.
The way to measure whether the data-driven ads are a scam is not by saying "this ad is irrelevant to me". It's by the advertiser looking at their ROAS (return on ad spend). If the algorithm is optimizing for showing you an ad, there's a pretty fair chance that the advertiser has done that math and decided that in a world of imperfect data and algorithms, you approximate enough the type of person who is worth showing an ad to... statistically.

In fact, getting to a world where ads are perfectly targeted would be one hell of a privacy nightmare.

> there's a pretty fair chance that the advertiser has done that math

As an occasional admittedly naive experimenter with ads – it’s far more likely that the platform has done the math and showing you the ad is profitable to them, not the advertiser.

Google et al love nothing more than to burn your entire ad budget on cheap clicks from countries that will never buy anything. But the number of “ad conversions” looks high and graph go up!

Getting these platforms to do what you want is a literal full-time job. Hell, in bigger companies you have entire teams doing nothing more than bending over backwards to get these platforms to do the useful thing instead of setting money on fire. The algorithms love setting money on fire. It’s a lot like giving your 3 wishes to a genie, you have to be very specific.

I don’t think so as only the advertiser has data you need.

Let’s say sales increase but your advertiser just burns 20% of your spend because it can and you won’t know. That’s the scam part.

It’s not that your spend doesn’t increase sales. It’s that part of your spend is wasted and you could increase sales more if it wasn’t scammed away as wastage.

I just bought a dozen bagels for $20. I’m happy, pretty good deal. But if the store was throwing away a dozen bagels for every one I bought, that’s wastage. Just because I’m happy doesn’t mean it’s not a scam.

Before reddit shit the bed, I wanted to advertise there, only to learn that the subreddits I wanted to target were too small to be able to be targeted. So much data wasted.
Meanwhile, in some subreddits, your competitors are doing some combination of:

* participating in the subs in good faith, with full disclosure;

* astroturfing the posts&comments;

* rigging upvotes/downvotes;

* building relationships with (maybe bribing) the mods; and/or

* entirely running the sub already.

Small subreddit targeting is just a matter of shilling. I’ve seen it on certain niche subreddits, brands becoming the subreddit darlings and added to the faq and wiki and sidebars, constantly recommended in the comments under any “what do I buy for x” thread. Even if its a junk product. Sometimes the brand even has an account active on the subreddit giving away coupons or free stickers and such.
There’s one thing that advertisers agree on: ads work in the aggregate.

You can tell by turning them all off. Ask Kraft Heinz.

Beyond that it gets fuzzy. Is it inefficient? Yes. Does anyone agree on the actual mechanics? No. Is the data wildly inaccurate? Sure. Is there grift? Totally.

People also forget how inexpensive and ephemeral individual ads are. A display banner on a quality site costs less than a tenth of a penny.

And it’s like a room on a cruise ship — the boat is gonna sail one way or another. So it has to be sold.

Advertisers might choose to let Facebook arbitrage those impressions into a cost per click model — at pennies per click instead of hundredths per view. Tradeoffs galore in that model.

When you aggregate these numbers into a $600b industry, you start to see how sweating some of the finer details just doesn’t matter.

What happened to Kraft Heinz?
What happened to Kraft Heinz? I assume they are still doing very well.
Many here already pointed at the fact that likely P(buying mattress in the future|googled mattress) is much higher than P(buying mattress in the future) thus for an advertiser with a finite budget it makes perfect sense to target users who googled mattress irrespective of whether they bought it already or not.
The author's point is that, with the data available to the platform owners, they could easily deduce the subset where that P value is nearly zero (having already bought the mattress).

Places that would also make sense to advertise mattresses: around universities and dense rental housing, or new developments of single family houses. I required no spy networks to make this list.

It only makes sense if you consider that the end all be all to target your customer. Maybe late night television ads are even more effective than google searches, because they are cheaper than prime time slots and are perhaps enriched for people with poor sleep. Maybe right now some subreddit is shilling mattress penny stocks and that is masking the true signal from your internet search data. You have no way of knowing that though, because you’ve reduced all of reality down to a single factor, an internet search, without considering how much of the observed variance could even be explained by this factor nor how the results might be confounded with other factors.
Isn't it the case that you can't target specific individuals. Only big group sizes are allowed to be targetted through their portals.
> Marketing theory is never tested rigorously.

Of course its tested "rigorously". Performance marketing is always about testing and optimizing CAC/CPA. It can make or break your company. If you can acquire customers at 10% lower rate(at the same volumes) than you competitors and you have similar COGS, you will dominate them and will ultimately will be the market leader. Monetary incentives are there to ensure efficiency just like there are incentives around COGS for efficiency.

The only evidence the author offers are her personal experiences. While Im no fan of advertising and question its efficacy as well, I don’t really think this is a convincing argument against them.

Statistically, these advertising segments will be very fuzzy and there will be outliers (probably many of them?).

So much of advertising is lies that I don't do any advertising online any more.

I've pulled all my google ads, facebook ads, instagram ads and podcast ads.

After about 10 years of doing ads (for my coffee club) and really attempting to track what was happening (from a technical point of view) I couldn't actually attribute any of it (the holy grail of digital marketing). "Word of mouth" is still our biggest channel. And all the time whenever I speak to "marketing" people it's the same low value analysis with zero mathematical or scientific vigor. They look at what happened (High CTR on this ad) and draw some conclusion from it, and then next week when CTR flatlines on the ad they have no comment and are busy looking at the non-significant noise coming from another ad.

And that's before even talking about the pure lies coming from the podcast people who swear up and down they know exactly how far into a podcast someone plays, when the spec they quote explicitly states it doesn't really work on apple products and 90% of the listernship is Apple. So many lies. It is a grift built on grift. Somebody in the organization knows it's a pile of lies, but the sales people actually believe it I think or don't care.

Remember, the primary job of an advertiser is to sell their advertising services to companies, not to sell a company's products to consumers. That comes second and is only done insofar as it facilitates the first.
I really hope in the near future there is some shift in the advertising business where they realize the ridiculous amounts of money they are lighting on fire and decimates the ad driven internet as we know it.
Their job is to sell advertising to advertisers, not to please their users.

I thought it was common knowledge. The more complicated and graph-filled the dashboard is, the more work that the advertiser can claim to have done to their bosses

> Marketing theory is never tested rigorously. The <del>common sense</del> incredibly sound scientific view based on heaps of scientific evidence view – showing your ads to people more likely to buy your product is more efficient because they’re more likely to buy your product anyway – is entirely untested.

Before this, the classic life style segment in media analysis was magazine audiences aged 15-25. Notably, this was not the part of the population that could afford the goods advertised immediately, or even anywhere in the near future. The idea was that this would be formative, that this would effect in an invaluable baseline of expectations and anticipations that could not be competed with, when it came to real acquisitions. Well, nowadays it's all just-in-time, often even just-after-buy, which seems to be the most efficient.

"Amazon knows exactly when you have an outbreak of aphids because you buy things to kill the nasty little beasties, and it probably also knows when you’ve had a nasty breakup because nobody listens to Fleetwood Mac’s Rumours on repeat at 3am when they’re in a good place."

Just great writing, this.

Ehh I find it hard to believe that a multi-trillion dollar industry is a grift based on this single person's collection of anecdotes. If online ads didn't result in measurable customer conversions, I have a feeling that the millions of business using them would stop.

Disc: Google employee, not ads though

So OP claims that data-driven ads are a scam using the anecdata that she is getting served irrelevant ads.

I'm not going to argue about whether ads are "worth it", but working on the other side (serving and optimizing ads), I can tell that yes, we run the numbers, and the math (mostly) works out.

As per your examples, we have run experiments by applying targeting ads based on narrow segments, and others without. As it turns out the data shows that "casting a wider net" works better, statistically.

Make of it what you will.

It’s possible that an audience that is very likely to buy a mattress are recent mattress buyers.
I do dislike digital ads refining their profile of me, via my involuntary contributions to their models. Definitely feels like I'm being spied upon.
Maybe advertisers don’t actually get signals when people click pay in some third party site? I mean, why would that third party want to expose your purchase to the advertisers?

So, you’re left with this great signal that someone is shopping for a mattress. Seems like a very good reason to recommend other mattresses to them.

Woman who knows nothing about digital marketing claims an entire multiple-billion dollar industry is scam, nice.
We can argue about whether data-driven ads work for the OP or for buying an extra mattress, but in aggregate they absolutely work. Contra OP's anecdote, if FB ad targeting stopped working, then FB would notice the difference in engagement within a day - probably within the hour.
I have a friend with a business in a niche market. His business is driven almost entirely by ads place with google. He’s tried other platforms and had much worse results. Something must be working.
At some point we tried using a major network…. That predicted our target market to be 50% female, and heavily present in middle east. Had a laugh and moved on.
Contrary to the belief that advertising is less data-driven, the complexity and dependency on feature-rich data sets has increased over time. Redundant targeting sometimes happens due to the ad objective of maximizing expected revenue over cost. However, it's more nuanced in practice.

Advertisers aim to optimize expected revenue over cost within a 'payback period', which is typically 1-3 years for large advertisers. This is calculated through customer acquisition cost, retention, and incrementality (the probability of an ad causing conversion).

Only advertisers themselves can effectively calculate incrementality due to access to their specific conversion/retention metrics. This incrementality, along with the optimal mix of channels and ad spend, is the ongoing challenge for sophisticated advertisers. It involves multi-objective optimization across millions of ad assets, campaigns, and targeting criteria.

Privacy regulations since 2015 and subsequent laws like GDPR and CCPA have led to more reliance on probabilistic modeling for targeting. The entire pipeline of targeting, engagement, conversion, and retention forecasts are now based on probabilistic models.

While ad networks offer simplified scaling solutions like 'target roas' and 'campaign budget optimization', they're more useful for average advertisers with limited internal resources - eg, they can't justify hiring ML SWEs, quant traders, technical PMs, etc.

Advertising has become even more data-driven and arbitrage gains for sophistication have increased. Profound gains can be made with investments in marketing and forecasting science, similar to the operations of a quant trading firm.

Source: I've managed $10B+ through automated ad spend systems since 2012.

This entire essay is just full of ideas I disagree with.

First is the old chestnut: Why does company keep advertising product to me after I've already bought it!? This is one of those child-like criticisms that keeps rearing it's ignorant head. Literally, you are ignorant as to why they are doing it. Your ignorance as to why doesn't imply their stupidity. Amazon does it. Google does it. I'm sure Facebook does it. So either three of the biggest companies in the world are full of stupid idiots or perhaps you are just ignorant to the reason they are doing it.

> Their ethos is that if a human can do a task, a machine can do that task better, and not cost them anything such as salary, pensions or or a basic level of respect

No, their ethos is - we can't possibly have any reasonable number of humans do the task. In what universe could we imagine hiring a sufficient number of humans to sift through the data Google has collected? They have immense data on literal billions of humans. How much time would it take any person to understand you as a person sufficiently by painstakingly reviewing your data so that Google could send you perfect ads? The very premise of this is ridiculous beyond reason.

So the conclusion is based on the premise that having humans do the task is impossible. We have no other option than to use machines and algorithms to attempt the task because it is the only viable alternative.

> Advertisers, then, are getting served a steaming turd on a plate rather than the medium-rare filet mignon they were promised.

Advertisers are making a value judgement based on the options they have available to them. They can advertise on TV, radio, or magazines. They can advertise on podcasts, YouTube channels or Instagram feeds by directly contacting the creators and doing sponsored content. They can put up billboards and wrap public buses. They can put people in public squares and hand out flyers, do personal demos and offer prizes. I couldn't even scratch the surface of what is available to modern companies for promotional purposes. The fact that Facebook, Amazon and Google are as large as they are suggest there is some value being derived by companies given that they have plenty of alternate options available to them.

> There isn’t any evidence to suggest that an ad targeted to 35 year old men with children with an interest in football is any more likely to result in sales of Football Dad socks than a poster for Football Dad socks at a bus stop. But an entire industry is based on pretending that this is the case.

I really don't think they are. But let's compare the idea to the other advertising avenues I listed. TV ads are broadly targeted based on the time of day the show runs at as well as the content of the show. Magazine ads are broadly targeted based on the content of the magazine. Junk mail and flyers are broadly targeted to people living in a particular geographic area.

But I guess this MSc in marketing knows more than the entire industry they claim to be from.

I’m very curious what events she is seeing that exclude people over 30.