▲ 213 points
back
76 comments
It's the bitter-lesson to feature-engineering lifecycle.
When a technique or technology is new people are making massive gains by just applying it to some use case, or gathering more data for training, or giving it more resources.
As time goes on those "bitter lesson" gains start to hit the shallow part of the logistic curve and companies have to start investing more and more effort into engineering for each small, incremental gain.
I got a very different message from this, actually much closer to the problem of incumbent advantage.
The known-good thing has been heavily optimized for performance, making it much harder for new technologies to prove that they are better. This is similar to the problem of gas vs electric engines - we had a century of optimization and ecosystem development around gas engines, which creates an uphill battle for electric motors even though they are (eventually) superior on every way /except/ having that massive ecosystem.
The problem isn't as bad here, because software is much more flexible than hardware, and scaling laws give a reasonable way to try things out at smaller scale before going whole hog.
what is the known-good thing? The whole point is that LLMs were not optimized at all, they got better results than older ML algorithms just because they are able to use all of the GPU, where older algorithms are designed for 10yo GPUs and can't make use of modern GPUs. But now you do in fact have to optimize, to the point that transformers look a lot more complicated than "attention is all you need."
the known-good thing is whatever was previously used for the application you're using llms for in any given case. This is often the structured application of human intelligence.
I assume the choice of phrase "bitter lesson" is intentional irony (since the original concept is that you get better results by just scaling up and not trying to be clever with domain-specific knowledge)?
I assume that the bitterness of the bitter lesson is not for engineers but for subject matter experts. I can only imagine how it would feel to discover that your decades of hard-earned expertise don't amount to a whole lot when it comes to domain-specific ML modeling, compared to simply throwing more compute at the problem.
My read of GP is that there is a cycle economy between "bitter lesson" style domain-general scaling and domain-specific adaptation once the scaling plateaus for the latest tech.
Specifically the you get better results with techniques that can scale up to the amount of data you have. People often think of it as a "just brute force it" but for the lesson to apply you do need to come up with how you're gong to get the data and how you're going to use it.
Maybe the ultimate bitter lesson is that entropy always wins in the end.
Well put, thanks.
Why didn't this author compare Llama 3 with GLM 5.2 (released 1 week ago) which is a more standard attention based LLM? To compare 2 separate families of LLMs and then pointing out that they are different is not a surprising result and detracts from the point the author is trying to make.
https://sebastianraschka.com/llm-architecture-gallery/?compa...
If you look at it, the diagrams are very similar, but the main differences are that the feedforward is replaced with a MoE (router to multiple feedforwards) and the model has a different attention implementation.
The author is correct, the model architecture is now much more complicated. You can see this if you use llama.cpp and follow the project. The earlier models were always fully implemented. Yet with more contributors, as of today tons of latest models only have partial implementation. DeepSeekv3.2 isn't fully implemented, same with KimiK2.6, GLM5.2+, DeepSeekv4 has no implementation, MiniMaxM3 not supported yet, Hy3-preview no implementation. The latest models are just bare bones to run with lots of support missing for the advanced features.
The architecture is not much more complicated. And llama.cpp not implementing something is more likely an issue of its business model and financial incentives than it being due to raw complexity.
indeed, there's even a (pretty solid) custom server just for DS4 https://github.com/antirez/ds4
-- works very well on high-RAM Macs
I don’t k ow what I’m doing wrong. Everyone says ds4 is faster than a lot of models around the same size, but I’m getting 2t/s with DSv4 vs 12t/s with Minimax 2.7 (16Gb 5080 + 16gb 5060ti + 128gb ram).
ds4 is optimized for systems with unified memory. it works best on apple silicon with 96GB+ of RAM
> Why didn't this author compare Llama 3 with GLM 5.2 (released 1 week ago) which is a more standard attention based LLM? To compare 2 separate families of LLMs and then pointing out that they are different is not a surprising result and detracts from the point the author is trying to make.
The entire point of the comparison is that LLMs look vastly different today than before. Comparing more similar LLMs would detract from the point I thought the author was trying to make.
It is misleading the reader since most current LLMs look the same as before. It is cherry picking an example to make a point when it's not necessary at all to make the argument he is trying to make.
> It is misleading the reader since most current LLMs look the same as before.
But most of them do not? They do look vastly different from the earlier incarnations of GPT and Llama.
> If you look at it, the diagrams are very similar,
The page links to the same site you do. No wonder it is similar -- the source is the same!
The source is the same in the original article too. He is using a different diagram from the same site on the right to justify his point on how much more complicated things have become.
Yeah, not a great apples-to-apples comparison.
I think the point stands: MoE, a myriad of complex attention approaches, shared layers, you name it. And making it all work together well is a huge trial-and-error pain even for small models, never mind getting to efficient hardware utilization.
It’s written by AI.
I am _very_ familiar with Claudish, and to some extent, the other AIs' writing styles. This article is human-written and features human writing quirks.
The very first sentence
> Back in 2022 and 2023 there were two big branches of machine learning happening at Meta.
is unmistakably human. That's not how a LLM would phrase this sentence, and if it did, it would have put a comma after 2023.
You can just prompt an llm to make causal grammar mistakes
[[citation needed]]
I am a professional writer and have been for over 30 years. (I do not use any form of LLM ever.) This means I read a lot. This also means that I have 30+ years of experience of readers not understanding what I wrote, or not getting further than the title, or not getting the main message, or inverting it in their heads, or inserting their own message and then complaining when I diverge, and an endless list of Ways People Do Not Get It.
I am also a trained TESOL teacher. Ability to capture gist is a skill we test for and measure, and many, maybe the majority, of native speakers don't have it and don't know.
In recent years I constantly see people going "this is written by AI" and I have yet to see a single of of them able to coherently prove their point. It's all just feelings and hunches.
So I am calling you on this:
How do you know? Show your working. Demonstrate your case.
Claude's writing style is at least as distinctive as any human's personal style. It has a long list of favorite words, verbal tics and common structures. On top of that, LLM writing is often bad in a very particular way: it's weak on actual things to say, but with an overheated style.
Some days, I spend over 4 hours a day reading walls of text written by Claude. If I couldn't recognize Claude's default "voice" by now, something would be wrong. It would be like a Hemingway fan not being able to recognize Hemingway. Except more so, because Claude's writing style is getting worse from version to version, descending into self parody.
On the statistical side, Pangram's model identifies AI-authored text with a 1-in-5,000 false positive rate, measured against hold-out texts from before 2022. My "ear" also agrees closely with Pangram. If I think something sounds AI written, Pangram virtually always comes back with "AI, confidence: high."
I don’t think TFA is written by AI.
But AI written pieces do have a certain feeling. A sort of saccatto in the succession of ideas that does not feel natural. They emphasize certain points, and you as a reader, you just wonder why is that. There is the “This thing, not just that thing”. There are also the three successive propositions (mostly in one sentences) to accentuate an idea and “Negation. Strong positive idea in the same direction”.
In general try reading one (vocally) to yourself and it will feel really weird.
I want Scrabble rules for HN AI challenges. If someone finds an AI-generated comment, the commenter has violated HN guidelines and the comment should be deleted. But if the accusation is wrong, there should be a penalty for the often massive disruption the accuser has caused to the discussion.
(As of now, that four-word low-effort comment has generated over a thousand words in response, none of which improve this article's discussion.)
You need to start using LLMs a lot and then you will know how we know.
Edit: You know how you can recognise someone just from their gait while they walk towards you? I would struggle to describe that for an individual person but it doesn't mean I can't identify them from that alone.
I’m not sure if it is written by an LLM, but anything being called “load-bearing” (formatted that way and all) sets off my alarm bells
Highly doubtful
Grammarly and GPTZero say 0% AI.
- with all due respect, from a ux perspective, could you kindly add a page where i can see just the titles of all your blog posts
- https://ianbarber.blog/blogroll
- https://ianbarber.blog/archive
- https://ianbarber.blog/posts
- none of the above links work
- i really dont want to scroll 200 pages just to see what your blog articles are
There is https://ianbarber.blog/feed
firefox directly downloads it, i cant see what it is without downloading a file
lol yeah I guess the best move right now is to fetch their /feed and iterate through <post>s
For someone like me who's never done any hands-on work in ML, the blog is really hard to understand. Whoosh, over my head.
But, I think the underlying problem is that we don't understand how this sh*t works. So, it's just an empirical, iterative mess.
Like physics in the the years shortly before relativity and quantum mechanics.
My read was: "The first 90% of the work takes the first 10% of the time, and the remaining 10% of the work takes the other 90% of the time." And that we are now squarely in the remaining 10% of the work.
> Claude Telenovela
Nice, hadn't seen this one before.