back
160 comments
This is why it’s so critical to have open source models.

In a year or so, the open source models will become good enough (in both quality and speed) to run locally.

Arguably, OpenAI OSS 120B is already good enough, in both quality and speed, to run on Mac Studio.

Then $10k, amortized over 3 years, will be enough to run code LLMs 24/7.

I hope that’s the future.

Open source models could be run by low-cost cloud providers, too. They could offer discounts for a long term contract and run it on dedicated hardware.
Every business building on LLMs should also have a contingency plan for if they needed to go to an all open-weights model strategy. OpenAI / Anthropic / Google have nothing stopping them from 100x-ing the price or limiting access or dropping old models or outright competing with their customers. Building your whole business on top of them will prove to be as foolish as all of the media companies that built on top of Facebook and got crushed later.
Many of the larger enterprises (retail, manufacture, insurance, etc) are consistently becoming cloud-only or have reduced their data center foot print massively over the last 10 years.

Do you think these enterprises will begin hosting their own models? I'm not convinced they'll join the capex race to build AI data centers. It would make more sense they just end up consuming existing services.

Then there are the smaller startups that just never had their own data center. Are those going to start self-hosting AI models? And all of the related requirements to allow say a few hundred employees to access a local service at once? network, HA, upgrades, etc. Say you have multiple offices in different countries also, and so on.

I am looking forward for the AMD 395 max+ PCs to come down in price.

The inference speed locally will be acceptable in 5-10 years thanks to those generation of chips and finally we can have good local AI apps.

> In a year or so, the open source models will become good enough (in both quality and speed) to run locally.

"Good enough" for what is the question. You can already run them locally, the problem is that they aren't really practical for the use-cases we see with SOTA models, which are just now becoming passable as semi-reliable autonomous agents. There is no hope of running anything like today's SOTA models locally in the next decade.

IMO local models is kind of inevitable.

Hardware vendors will create efficient inference pcie chips and innovations in ram architecture will make make even mid-level devices capable of running local 120B parameter models efficiently.

Open source models will get good enough that there isn’t a meaningful difference between them and the closed source offerings.

Hardware is relatively cheap, it’s just that vendors haven’t had enough cycles yet on getting local inference capable devices out to the people.

I give it 5 years or so before this is the standard

What's performance of running OpenAI OSS 120B on a Mac Studio as compared to running a paid subscription frontier LLM?
Even if they do get better. The latest closed-source {gemini|anthropic|openai} model will always be insanely good and it would be dumb to use a local one from 3 years back.

Also tooling, you can use aider which is ok. But claude code and gemini cli will always be superior and will only work correctly with their respective models.

Why bother mentioning this model? From what I've seen, it only excels at benchmarks. Qwen3 is sorta where its at right now; Qwen3-Coder is pretty much at "summer intern" level for coding tasks, and its ahead of the rest.

Shame anyone is actually _paying_ for commercial inference, its worse than whatever you can do locally.

Problem is that it really eats all resources when using a llm locally. I tried it. But the whole system becomes unresponsive and slow. We need minimum of 1tb memory and dedicated processors to offload.
After trying gpt-oss:20b, I'm starting to lose faith in this argument, but I share your hope.

Also, I've never tried really huge local models and especially not RAG with local models.

It's not hard to imagine a future where I license their network for inference on my own machine, and they can focus on training.
>In a year or so, the open source models will become good enough (in both quality and speed) to run locally.

Based on what?

And where? On systems < 48GB?

Its not, capitalism isn't about efficiency; it's about lockin. You can't lockin open source models. If fascism under republicans continue, you can bet they'll be shut down due to child safety or whatever excuse the large corporations need to turn off the free efficiency.
I'm not sure where the author gets the $100k number, but I agree that Cursor and Claude Code have obfuscated the true cost of intelligence. Tools like Cline and its forks (Roo Code, Kilo Code) have shown what unmitigated inference can actually deliver.

The irony is that Kilo itself is playing the same game they're criticizing. They're burning cash on free credits (with expiry dates) and paid marketing to grab market share -- essentially subsidizing inference just like Cursor, just with VC money instead of subscription revenue.

The author is right that the "$20 → $200" subscription model is broken. But Kilo's approach of giving away $100+ in credits isn't sustainable either. Eventually, everyone has to face the same reality: frontier model inference is expensive, and someone has to pay for it.

So convenient a future AI dev will cost as much as a human developer, pure coincidence
> Both effects together will push costs at the top level to $100k a year. Spending that magnitude of money on software is not without precedent, chip design licenses from Cadence or Synopsys are already $250k a year.

For how many developers? Chip design companies aren't paying Synopsys $250k/year per developer. Even when using formal tools which are ludicrously expensive, developers can share licenses.

In any case, the reason chip design companies pay EDA vendors these enormous sums is because there isn't really an alternative. Verilator exists, but ... there's a reason commercial EDA vendors can basically ignore it.

That isn't true for AI. Why on earth would you pay more than a full time developer salary on AI tokens when you could just hire another person instead. I definitely think AI improves productivity but it's like 10-20% maybe, not 100%.

At some point the value of remote inference becomes more expensive than just buying the hardware locally, even for server-grade components. A GB200 is ~$60-70k and will run for multiple years. If inference costs continue to scale, at some point it just makes more sense to run even the largest models locally.

OSS models are only ~1 year behind SOTA proprietary, and we're already approaching a point where models are "good enough" for most usage. Where we're seeing advancements is more in tool calling, agentic frameworks, and thinking loops, all of which are independent of the base model. It's very likely that local, continuous thinking on an OSS model is the future.

There is nothing new here and the math on this is pretty simple. AI greatly increases automation, but its output is not trusted. All research so far shows AI assisted development is a zero sum game regarding time and productivity because time saved by AI is reinvested back into more thorough code reviews than were otherwise required.

Ultimately, this will become a people problem more than a financial problem. People that lack the confidence to code without AI will cost less to hire and dramatically more to employ, no differently than people reliant on large frameworks. All historical data indicates employers will happily eat that extra cost if it means candidates are easier to identify and select because hiring and firing remain among the most serious considerations for technology selection.

Candidates, currently thought of 10x, that are productive without these helpers will continue to remain no more or less elusive than they are now. That means employers must choose between higher risks with higher selection costs for the potentially higher return on investment knowing that ROE is only realized if these high performance candidates are allowed to execute with high productivity. Employers will gladly eat increased expenses if they can qualify lower risks to candidate selection.

This is the goal. Create a reason to shave a bunch off the top of SWE salaries. Pay them less because you "have" to pay for AI tools. All so they don't have to do easy rote work - you still get them to do the high level stuff humans must do.
Author just choose a nice number and give no argument to it
No justification for a $100k number. For $100k a year or about $8k a month, you will end up using 1B tokens a month (that too a generous blended $8 per million input/output tokens including caching while the number is lower than that). Per person.

I think there is a case Claude did not reduce their pricing given that they have the best coding models out there. There recent fundraise had them disclose their Gross margins at 60% (and -30% with usage via bedrock etc). This way they can offer 2.5x more tokens at the same price than the vibe code companies and yet break even. The market movement where the assumption did not work out was about how we still only have claude which made vibe coding work and is the most tasteful when it comes to what users want. There are probably models better at thinking and logic, especially o3, but this signals the staying power of claude - having a lock in, it's popularity, and challenges the more fundamental assumption about language models being commodities.

(Speculating) Many companies woudl want to move away from claude but cant because users love the models.

What is everyone’s favorite parallel agent stack?

I’ve just become comfortable using GH copilot in agent mode, but I haven’t started letting it work in an isolated way in parallel to me. Any advise on getting started?

> charge users $200 while providing at least $400 worth of tokens, essentially operating at -100% gross margin.

Why are we assuming everyone uses the full $400? Margins aren't calculated based on only the heaviest users..

And where are they pulling the 100k number from?

This makes sense as long as people continue to value using the best models (which may or may not continue for lots of reasons).

I’m not entirely sure that AI companies like Cursor necessarily miscalculated though. It’s noted that the actual strategies the blog advertises are things used by tools like Cursor (via auto mode). The important thing for them is that they are able to successfully push users towards their auto mode and use more usage data to improve their routing and frontier models don’t continue to be so much better AND so expensive that users continue to demand them. I wouldn’t hate that bet if I were Cursor personally.

Fyi Kilocode has low credibility. They’ve been blasting AI subreddits with lots of clickbaity ads and posts, sometimes claiming things that are outright false.

As far as spend per dev- I can’t even manage to use up the limits on my $100 Claude plan. It gets everything done and I run out of things to ask it. Considering that the models will get better and cheaper over time, I’m personally not seeing a future where I will need to spend that much more than $100 a month.

Never heard of kilo before, pretty sure this post is just an ad
> The difference in pay between inference and training engineers is because of their relative impact. You train a model with a handful of people while it is used by millions of people.

Okay, but when did that ever create a comparable effect for any other kind of software dev in history?

I started https://www.vantage.sh/ - a cloud cost platform that tracks Infra & AI spend.

The $100k/dev/year figure feels like sticker shock math more than reality. Yes, AI bills are growing fast - but most teams I see are still spending substantially lower annually, and that's before applying even basic optimizations like prompt caching, model routing, or splitting work across models.

The real story is the AWS playbook all over again: vendors keep dropping unit costs, customers keep increasing consumption faster than prices fall, and in the end the bills still grow. If you’re not measuring it daily, the "marginal cost is trending down" narrative is meaningless - you’ll still get blindsided by scale.

I'm biased but the winners will be the ones who treat AI like any other cloud resource: ruthlessly measured, budgeted, and tuned.

Don't know about the numbers but is this not the cloud all over again. Promises about cheap storage and you don't maintain it developed into maintenance hell and storage costs steadily rising instead of dropping.
I think what this model actually showed is a cyclical aspect of tokens as a commodity

It is based on supply and demand of GPUs, the demand currently outstrips supply, while the 'frontier models' are also much more computationally efficient than last year's models in some ways - using far fewer computational resources to do the same thing

so now that everyone wants to use frontier models in "agentic mode" with reasoning eating up a ton more tokens before sticking with a result, the demand is outpacing supply but it is possible it equalizes yet again, before the cycle begins anew

"The bet was that by the following year, the application inference would cost 90% less, creating a $160 gross profit (+80% gross margins). But this didn't happen, instead of declining the application inference costs actually grew!"

This doesn't make any sense to me. Why would Cursor et al expect they could pocket the difference if inference costs went down? There's no stickiness to the product; they would compete down to zero margins regardless. If anything, higher total spend is better for them because it's more to skim off of.

Tools like Cursor rely on the gym model—plenty of people will pay for a tier that they don't fully utilize. The heavy users are subsidized by the majority who may go months without using the tool.
@g42gregory This would mean that for the certain devs, an unfair advantage would be owning a decent on-prem rig running a fine tuned and trained model that has been optimized for specific use case for the user.

A fellow HN user's post I engaged with recently talked about low hanging fruits.

What that means for me and where I'm from is some sort of devloan initiative by NGOs and Government Grants, where devs have access to these models/hardware and repay back with some form of value.

What that is, I haven't thought that far. Thoughts?

└── Dey well

How many parallel agents can one developer actively keep up with? Right now, my number seems to be about 3-5 tasks, if I review the output.

If we assume 5 tasks, each running $400/mo of tokens, we reach an annual bill of $24,000. We would have to see a 4x increase in token cost to reach the $100,000/yr mark. This seems possible with increased context sizes. Additionally, we might see additional context sizes lead to longer running more complicated tasks which would increase my number of parallel tasks.

An interesting metric is when token bills per dev exceed the cost of hiring a new dev. But also, if paying another dev's worth in tokens getting you further than 2 devs without using AI will you still pay it?

I wonder how the economics will play out, especially when you add in all the different geographic locations for remote devs and their cost.

give me $50k raise and I need only $10k/yr.

seriously, I don't see the AI outcome worth that much yet.

On the current level of ai tools, the attention you need to manage 10+ async tasks are over limit for most human.

In 10 years maybe, but $100k probably worths much less by then.

If you are throttled at $200 per month, you should probably just pay another $200 a month for a second subscription, because the value is there. That’s my take from using Claude.
Maybe this is why companies are hyping the "replacing devs" angle, as "wow see we're still cheaper than that engineer!" is going to be only viable pitch.
It's not hard to imagine a future where I license their network for inference on my own machine, and they can focus on training.
Why even stop at 100k/yr? Surely the graph is up-and-to-the-right forever? https://xkcd.com/605/
Honestly we're in a race to the bottom right now with AI.

It's only going to get cheaper to train and run these models as time goes on. Modes running on single consumer grade PCs today were almost unthinkable four years ago.

Title modded without merit.
What is Kilocode?
"Tokenomics."
tl;dr

> This is driven by two developments: more parallel agents and more work done before human feedback is needed.