▲ 395 points
back
342 comments
I had my formative years in programming when memory usage was something you still worried about as a programmer. And then memory expanded so much that all kinds of “optimal” patterns for programming just become nearly irrelevant. Will we start to actually consider this in software solutions again as a result?
You're right in terms of fitting your program to memory, so that it can run in the first place.
But in performance work, the relative speed of RAM relative to computation has dropped such that it's a common wisdom to treat today's cache as RAM of old (and today's RAM as disk of old, etc).
In software performance work it's been all about hitting the cache for a long time. LLMs aren't too amenable to caching though.
I've actively started to use outlook and teams through chrome to free up some of my ram, easily saves 3-4gb. It's gotten ridiculous how much ram basic tools are using, leaving nothing for doing actually real work
I doubt it. I predict in a few years, maybe sooner, one/some of the AI companies buying up the supply will either have achieved their goal or collapsed, and then the market will be flooded with a glut of memory driving prices low again. Or, conversely, the demand stays high for a sustained period of time and the suppliers just increase supply. There's no hard bill of materials/technical reasons for the memory prices to be this high, unlike 20+ years ago.
> And then memory expanded so much that all kinds of “optimal” patterns for programming just become nearly irrelevant.
I don't think that ever happened. Using relatively sparse amount of memory turns into better cache management which in turn usually improves performance drastically.
And in embedded stuff being good with memory management can make the difference between 'works' and 'fail'.
I never really bought in to the anti-Leetcode crowd’s sentiment that it’s irrelevant. It has always mattered as a competitive edge, against other job candidates if you’re an employee or the competition of you’re a company. It only looked irrelevant because opportunities were everywhere during ZIRP, but good times never last.
It's not like most developers are wasting memory for fun by using Electron etc. It's just the simplest way to deploy applications that require frequent multiplatform changes. Until you get Apple to approve native app changes faster and Linux users to agree on framework, app distribution, etc., it's the most optimal way to ship a product and not just a program.
RAM didn't get more expensive to produce. It just got more desirable. The prices will come down again when supply responds. It may take some time, but it will happen eventually.
We would have, if the expensive memory was a long term trend. It is not - eventually the supply will expand to match demand. There is no fundamental lack of raw materials underlying the issues, it is just a demand shock.
I just heard in a podcast, they talked about how powerful our devices are today but do not feel faster than they did 15 years ago and that it's because of what you write here.
When I train some leetcode problems, I remember the best solution was the one that optimised cpu (time) instead of memory. Meaning adding data index in memory instead of iterating on the main data structure. I thought, ok, thats fine, it's normal, you can (could) always buy more RAM, but you can't buy more time.
But well, I think there is no right answer and there always be a trade off case by case depending on the context.
> I had my formative years in programming when memory usage was something you still worried about as a programmer.
As 'just' a user in the 1990s and MS-DOS, fiddling with QEMM was a bit of a craft to get what you wanted to run in the memory you had.
* https://en.wikipedia.org/wiki/QEMM
(Also, DESQview was awesome.)
I've recently started a side project for the N64, and this is very relatable!
Working within such tight constraints is most of the fun.
Depends on the machine you’re targeting.
I do embedded Linux and ram usage is a major concern, same for other embedded applications.
I’m partying like it’s the 90s, on a 32-bit processor and a couple hundred MB of ram.
I think Europe should invest into manufacturing RAM. RAM isn't going anywhere, all of modern compute uses it. This would be an opportunity to create domestic supply of it.
If you are on Linux you can 'download' some RAM. Enable zram, configure sysctl variables to make good use of it.
Note that it won't help you if your workload makes use of all your RAM at once.
If you have a bunch of stuff running in the background it will help a lot.
I get 2 to 3 compression factor at all times with zstd. I calculated the utility to be as if I had 20GB extra RAM for what I do.
i am working on my side-product [1] where i was exploring a Rockchip which required external memory (just 1G) which went from $3 to $32 and completely destroyed economics for me. I settled with one with embedded memory and optimizing my code instead :)
Maybe this RAMmageddon will trigger a wave of optimized softwares that don't need GBs of memory for anything and everything.
Only a matter of time before you hear about missing shipping trucks being stolen. China is opening up more production, but I don’t see any relief coming soon.
Jeez. I'm glad I "splurged" for the 24gb RAM in my macbook air. Should last me a few more years..
There were years in the 1990s and early 2000s when it was easy to get faster runtimes by using more memory, back when even on a multi-user system like Linux or BSD was typically running one main program at a time. We had multiple hardware web servers, multiple hardware email servers, and multiple hardware database servers. Getting the most CPU performance out of the system for your applications was the order of the day.
Now, almost everything on the server side is a VM or a container. We have lots of neighbors who want to share the CPU and the RAM, and the RAM is the bigger constraint because the CPUs have 192 cores and each of those cores does a dozen times as much work as a decade ago. Heck, we used to have the memory controller on the motherboard and the last level of cache was a chip or module of SRAM outside the CPU.
We also have a situation now in which the multiple in speed of the CPU over RAM has skyrocketed, but the caches have gotten far larger and much smarter. Smaller things arranged differently in RAM make things run faster because they make better use of the cache.
Now that RAM is expensive, shared, and program and data size and arrangements are bound to cache behavior, optimization can lean heavily into optimizing for RAM again.
Some of these arguments hold true for desktop systems as well.
I have wondered for years when the time will come that instead of such huge and smart caches, someone will just put basically register-speed RAM on the chip and swap to motherboard RAM the way we swap to disk. HBM is somewhere close, being a substrate stacked in the package but not in the CPU die itself.
for those shopping for new PCs and are open to build one, it seems like the overall BOM has been overall lesser than a few months back. the big caveat being that you need a decent GPU.
having been in the market for one, i did make some compromise for the build (single stick of 16 GB for now; a non "future-proof" GPU within the budget). however for a decent spec (last-gen x3D CPU, mid-range RTX) build, the GPU price reduction made up for the premium on RAM.
the sad reality since the turn of the decade is that the $1000 mark for a well-rounded (gaming) system has now bumped up to $1500-$2000.
crazy time to be alive, where on laptop side, macs are now a "decent" value. especially if you were going to get the higher spec to unlock the specific memory tier. thanks work for setting me up well with one!
It's so bad that at work we're looking into shipping a big batch of prototypes off to someone to desolder and reball DDR4 chips.
We can't get any new chips. At all. We can't launch our new product because nobody could afford the memory even if we could get some.
Incredible.
I might not have bought NVDA or timed BTC correctly, but at least I have 512 GB of DDR5 in my server and 128 GB in my Macbook Pro haha. The reality is that these are insanely huge amounts of RAM. I'm glad to have them because I don't need these tab suspender extensions a bunch of my friends use, but really I'd prefer if GPUs were a bit cheaper, and server hardware was generally easier to get. An SXM5 based motherboard is really hard to get these days despite the fact that you can get super powered Epyc 9755s for comparatively nothing.
It reminds me of the heady days of Thai floods when hard drives were inaccessible.
A conspiracy theory I’m entertaining right now is that hogging RAM manufacturing by AI companies is not so much because they _need_ the RAM, but because they want to cripple existing and potential competitors, and that includes on-device models.
One thing that might support this is the fact AI companies are purchasing uncut wafers of DRAM. One use might be to hoard and stockpile them somewhere in a cave, so that no one else gets to them.
Another thing that might support this is that precisely the same strategy had been in use by software companies during the COVID hiring fever. Companies used to hire people for ridiculous pay with little actual work to perform so that among other things, competitors wouldn’t whisk those people away and be at an advantage.
This, of course, ended with massive layoffs once the reckoning came about, and I’m wondering about what is going to happen when (there’s no “if”) the reckoning comes for big AI, too.
Recently order a number of machines with 32Gb of RAM. Wanted 64, was told prices couldn't be guaranteed nor could delivery dates. Under the pressure of urgency settled for whatever was available that day.
"Bill of materials" (BOM) is not a monetary invoice, only an itemization of constituent parts. :)
This is like in 1993, when I bought a 486-DX2 with mighty(!!! ;-) 4 MB of Ram. MEGAbytes, not GIGAbytes :-D
(Graphiccard memory was back then 256KB or 512KB or 1024KB, amount what we have today as L1 cache on throwaway CPUs)
Raise your hand if you have been there too! :-))
Isn't there a full wafer ai chip mainframe for data centers now that blows anything needing ram out of the water?
I don't understand the ram shortage exists companies have surpassed nvidia.
Expensive PCs/homeservers means more people on mobile crap + someone cloud, means students who do not learn PCs FLOSS when they have time and so on. That's the real point.
I think China is about to step in and take every last bit of non-ai market share, and then when the bubble bursts companies like micron and samsung are going to be begging governments for a bail out.
Is there any hope for RAM to stabilize in prices again?
I think we’re at the peak, or close to it for these memory shenanigans. OpenAI who is largely responsible for the shortage, just doesn’t have the capital to pay for it. It’s only a matter of time before chickens come home to roost and the bill is due. OpenAI is promising hundreds of billions in capex but has no where near that cash on hand, and its cash flow is abysmal considering the spend.
Unless there is a true breakthrough, beyond AGI into super intelligence on existing, or near term, hardware— I just don’t see how “trust me bro,” can keep its spending party going. Competition is incredibly stiff, and it’s pretty likely we’re at the point of diminishing returns without an absolute breakthrough.
The end result is going to be RAM prices tanking in 18-24 months. The only upside will be for consumers who will likely gain the ability to run much larger open source models locally.
Kind of funny, with the help of AI finding some historical price sheets and 'design' a computer. So like 70s to 80s and I was blown away, how large the RAM cost was. A huge part of the BOM. It totally change the way I think about computer design in this area and why some decisions were made.
I guess something needs to be done about the RAM (and to a degree SSD/NAND) production cartel if it can so easily take hostage a major part of the society & starve it of critically needed components that are needed for a functioning modern society.
90s: so little memory, programmers have to optimize their code to run properly.
2010s: so much memory, programmers used electron and chrome wrapping everything in js.
2026: so little memory, programmers have to optimize AI code to run properly.
The last spike in RAM price was after an earthquake in Taiwan in April 2024. Now, the shortage will continue until about 2027 when new factories will start shipping.
Will this RAM shortage also affect the price of mobile phones?
How much are these price hikes affecting the ability for people in poor countries to access compute?
>AI-driven “end-to-end planning processes"
Behold, the RAM cost is being optimized with AI.
I'm willing to bet RAM prices start to fall by June of this year.
What!? I always thought we could just download more RAM!
Then make RAM out of less scarce sustainable inputs.
Big tech wants all the chips and they get them. That is Stalinist level of absurd planning.
People is missing the point. Mega-corporations distort the market. This is not capitalism this is old aristocratic ruling by power. If all these monopolies were divided in smaller chunks and regulated to not allow them to abuse that power we will not be here.
This situation is not normal, big tech is currently above the law and above the market economy and if they fail their plan is to make us pay *AGAIN* for their bad decisions. All businesses and individuals are already paying higher prices for big tech folly, we will be left with the bill when the AI boom fails, too.
This is a fairly odd statement given that BOMs are managed in manufacturing systems and for accounting and engineering purposes in multiple different ways. This can be for anything to do with sales data for a client or for guys on the factory floor or for the accountants. There are sales BOMs, manufacturing BOMs procurement BOMs and nested BOMs etc all for different parts of the business process...you would have BOMs within the organisation that were probably nearly 70% etc or those that were 0%!
I asked ChatGPT directly how it was fair that OpenAI bought 40% of the world’s RAM supply.
It denied this saying that the figures quoted were estimates only, that such massive RAM contracts would be easily obtainable public knowledge and that primarily the recent price increases were mostly cyclical in nature.
Any truth to this?
Edit to add: I am actually curious; I was under the impression that this 40% story going around was true and confirmed, rather than just hyperbole or speculation.