I am not a proper developer and only use AI for faster research of topics so please forgive my ignorance. Could one not save a lot of money on tokens by using the 80/20 or 90/10 rule in that 90% of AI usage is on local models and save that last 10% or less for the frontier models where the local model did not meet the needs? Did they cover this and I misunderstood?
back
4 comments
Unlees you invest thousands your local ai won't even come close to the cheap cloud llms. Local is only worth it if you care about privacy or have a legitimate usage for the hardware otherwise. Money wise it isn't worth it
I think you bring up a good point about privacy. Companies probably should not be handing over intellectual property to cloud AI third party vendors. Perhaps all of that should stay on-prem and then the cloud vendors could be used to fill in the gaps.
Enterprises running in public clouds have Zero Data Retention policies with the vendor. Routing the ai spend through Azure or AWS usually bring these policies along with it, ie enterprise data and useage is not ingested to retrain the model.
the qwen 27b is quite good for doing a mvp
I mean, if you're using one of the $200/month plans, you'd recoup the cost in a year or two.
You're not running an Opus 4.8/GPT 5.6 tier model on $5k of hardware at a useful Tok/s
This is like the "half of my marketing spend is wasted" quote. The complexity is finding out which half.
I think the idea or methodology is something along the line of one starts with the local models then when they hit a wall then continue with their current results in a frontier model, thus the other half is those last bits one could not compute locally.
If you are only using chatbots for code gen then for sure this is a reasonable approach. If you're trying to learn something then your best bet is to not have the lowest jpg quality version of human knowledge be your teacher.
We could probably get the same results by offloading 20% of obviously smaller / simpler / well defined tasks to smaller models and keep the 80% for ones that benefit from the big iron. Not the 80/20 they were referring to, but still it's something.
I find myself wasting time on smaller models or wasting money on frontier models. I only have so much of either.
To run a local AI that is half decent at research at usable speeds requires hardware that costs thousands of dollars.
Spending thousands of dollars on hardware to save dollars per month on tokens does not make financial sense.
If you run the numbers you'll probably find that using cheaper cloud models makes more financial sense than running those same models locally.
I suppose I was thinking in terms of companies that already had a slew of hardware and are just missing external GPU's, FPGA's or whatever people are using these days. A company might have a $50MM budget for tokens but could save a chunk of that using their existing hardware with some modifications.
Maybe that makes sense if the company already bought the hardware.
But I currently think the fundamental reason the cloud models often make more sense is based in how the models work. My current understanding is that once these models are loaded into GPU memory, the model can respond to multiple prompts in parallel. So a cloud vendor that receives lots of user prompts at the same time can achieve really good hardware utilization. These cloud vendors can then spread the cost of the hardware over many more users than you can with a local model, leading to lower prices per prompt in the cloud.
If you are an individual, almost certainly, unless you are using LLMs very heavily.
If you are a business employing many people all of who use LLMs reasonably heavily? You can still spread the hardware cost over many users and respond to multiple prompts in parallel. At some point buying the hardware will make sense.
I would not get a dedicated server to run a single small VPS. On the other hand if I needed a lot of capacity a dedi is cheaper than a big VPS (or lots of small ones).
There is also a benefit for anyone who has confidential or secret information.
That can work, especially for privacy or repetitive tasks. My experiment focused on subscriptions I already had, but local models are a natural addition to the routing layer.