▲ 93 points
back
164 comments
> While energy usage has not been disclosed, it’s estimated that GPT-3 consumed 936 MWh.
I'm a huge fan of energy efficieny, but this figure isn't all that much. Let's put things into perspective using some (probably somewhat inaccurate but probably somewhere in the ballpark) random internet sources, showing that this is less energy than a single long haul Boeing 747 flight. (CO2 footprint is probably different but somewhere in the ballpark.)
https://science.howstuffworks.com/transport/flight/modern/qu...
> A plane like a Boeing 747 uses approximately 1 gallon of fuel (about 4 liters) every second. Over the course of a 10-hour flight, it might burn 36,000 gallons (150,000 liters). According to Boeing's Web site, the 747 burns approximately 5 gallons of fuel per mile (12 liters per kilometer).
(I didn't find the right type of gallon where 36000 gallons == 150000 liters, but let's go with the liter figure anyway.)
According to https://en.wikipedia.org/wiki/Energy_density, 1 liter of kerosene has an energy content of 35 MJ. 150000 liters of kerosene have an energy content of 5250000 MJ. 5250000 MJ is 1458.333333 MWh.
Add another dubious source just to check our calculations (this one uses a larger fuel tank, but we'll re-use the previous figure):
https://www.withouthotair.com/c5/page_35.shtml
> And fuel’s calorific value is 10 kWh per litre. (We learned that in Chapter 3.)
150000 l * 10 kWh/l = 1500000 kWh = 1500 MWh, so about the same.
Not to mention that training a model is a "one time deal", where a successfully trained model can be reused by a lot of client (devices).
Considering the way AI can potentially bring benefits to humanity, i see it more like an investment.
For comparison, Bitcoin in 2021 used 110 TWh, solving a problem we've either solved millenea ago, or could be solved using much less power with premined coins.
>>> While energy usage has not been disclosed, it’s estimated that GPT-3 consumed 936 MWh.
How was this estimated? Is that for the final model run, how about all the testing runs, funs that failed, parameter tuning runs etc. are those included?
You've compared one use of energy to another and found them about the same. Fair enough, but that's an utterly different issue to whether it "isn't all that much", which you have not justified.
Not an attack on your figures or your claims, just saying you've not addressed the point you set out to address.
Many of the big players in ML also go out of their way to ensure their daughters are using renewable energy sources. The CO2 footprint thus ends up much lower than the airplane flight.
Following your logic, a pencil weights 0.006kg, considering E=mc*2, this is 150,000MWh. This AI uses less energy than a pencil.
At first glance I thought this was going to be another 'tech X uses energy Y and so tech X is bad' but reading the article shows it focuses mainly on lowering the energy cost of computations using a number of different approaches that look interesting. Sparse matrices where every element doesn't need to be re-computed as most stay zero for example. I'm not sure what actual improvement this would bring, but they do follow a good basic notion: if you're going to discuss a problem, try to discuss possible solutions to that problem, and improving the energy-efficiency of computation is always a good idea.
The one issue is that translating electricity usage into fossil fuel equivalents for this specific application, without contextual information about similar energy demands, such as streaming video, data collection/storage/processing (be it at Google or the NSA), total router energy consumption in the global Internet, etc. might result in a distorted view of the relative importance of energy demand for their particular issue (training complex models).
Furthermore, it's not necessary to generate electric power with fossil fuels, is it? My view is that solar/wind/storage is the optimal global-scale solution, but placing energy-hungry steady-load data centers near baseload nuclear power plants is arguably an efficient solution (ask the insurers first, however). Hydropower is region-specific and as the drought shows, subject to going offline when needed most to run AC etc. It's not inevitable that power demands equate to fossil carbon emissions, in other words.
You're right on the face of it, the added context that's missing is Numenta has been pushing bio-inspired AI for like 17 years and nothing practical has come of it. If using some energy to achieve a practical goal is wasteful, using less energy to accomplish nothing is more so.
I got into AI after a grad school career split between mathematical signal processing and computational neuroscience. I knew folks back in the early 2010's looking at joining Numenta. The ideas are absolutely good to explore, it's the execution that's lacking. Maybe their big breakthrough is just around the corner, but how long do you wait for product 1 alpha build 1 before calling out vaporware?
Saying a particular sector (AI/Crypto/...) consumes a lost of energy and we should do something about it, is like treating the symptoms and ignoring the root causes. The problem is mostly how we produce energy, not how we consume it. Consuming less energy on a particular sector means people in that sector have more money to spend on other things and those other things also cost energy (ok maybe less). To be clear I'm not saying we should totally ignore how much energy a sector consumes but most of our focus should be on how the energy is produced.
I don't think this is correct. I think that in essence what we've learned over the past 100 years is that we need to be more conscious of the totality of the cost benefit analysis, because what happens is that a lot of industry can can make private profits by exploiting public goods. So it's actually encouraging that we're starting to more objectively look at different activities and "Ok, here are the private profits of this industry and here are the public costs". Energy is a really simple element of that, and in AI it's probably not the most important, but it is very easy to measure. It's very easy to say "If you want to do this AI stuff, it's going to mean X tonnes of carbon or Y tonnes of rare earth metals or Z square miles of wind turbines".
Having said that, in Crypto it's a much more compelling argument because it's like "Here are some obvious, big costs that are fundamental to the system, and there are almost no benefits" whereas in AI the nature of these algorithms is that we're likely to improve them, so comparing training GPT-3 is kind of like measuring the power draw of the LHC. It's certainly big and it's certainly something we can talk about, but no one thinks we're going to be building 100,000 LHCs.
> Saying a particular sector (AI/Crypto/...) consumes a lost of energy and we should do something about it, is like treating the symptoms and ignoring the root causes. The problem is mostly how we produce energy, not how we consume it.
For AI, sure, for crypto (if PoW), no.
If energy was a tenth of the cost, PoW crypto would use 10 times more.
You do have a point, and I agree that making the production-side clean should be a large focus, but there are valid arguments for reducing consumption, too, I think.
Supply of energy is limited, and so there is a market effect: overconsumption by some players can increase the cost to other consumers.
Market economics should mean that the resulting increase in profit margin turns into an opportunity for more-cost-effective competitive entrants.
Most of this assumes no collusion between players in the market. If, for example, company A provides a product that is knowingly energy-inefficient and half of their board members also sit on the board of energy company B that is experiencing record profits -- and could easily acquire company C that theoretically has a cheaper energy production solution -- then perhaps problems could occur.
Totally agree. 5% of power plants are responsible for 73% of emissions [0]. That's like 1500 power plants. Rather than fix the emission issue at this small source, many would rather blame the population at large. Rather than fix 1500 big issues, they'd rather change the behavior of billions of individuals. It makes no sense.
0: https://www.smithsonianmag.com/smart-news/five-percent-power...
I think the main problem is the unaccounted externalities in the transactions.
I feel like this is not going to gain a lot of traction but just in case anyone ends up here I can't pin down a great source but I've heard / read that the entire IT sector consumes something in the neighborhood of 5-9% of global energy, and that that percentage doesnt have a linear relationship to the expansion of the IT sector.
https://www.enerdata.net/publications/executive-briefing/bet...
The more reliable government resources dont even seem to bother to list IT as a major energy consumer, although its obviously embedded in most / all traditionally recognized sectors.
https://www.enerdata.net/publications/executive-briefing/bet...
I'd love to know more about all this but thats about as far as my late night googling had taken me.
5-9% seems like a very small amount for something most people use most of the day and gain extreme amounts of value from.
That's 5-9% of global electricity, not global energy. Electricity is only about 15% of global energy use making IT around 1% of total energy.
Their careful comparison of the efficiency of AI vs the human brain is hilariously absurd to me.
It seems to assume a disembodied "brain in a vat" model, as if brains aren't ordinarily attached to a large clump of flesh that typically attempts to accumulate as many resources as it can, and often enjoys combusting large quantities of fossil fuels hurtling around in various metal enclosures.
I suppose we could be very efficient at image classification once AWS announces their new h5.large "MT bare human" instances, where gigantic banks of people are set to work solving CAPTCHAs with their incredible performance per watt specs.
Or we could redo the analysis with other cherry-picked measures and find the "human architecture" is orders of magnitude less efficient at multiplying large integers, and come to the conclusion that we should replace all babies (which have a wasteful decades-long training period) with ARM chips.
> once AWS announces their new h5.large "MT bare human" instances
Amazon already did that, it is called mechanical turk.
It costs a lot of money because we think those computers should have rights. But under more permissive legislations it would be so cheap and accessible that we wouldn't need to do much of the modern AI research. I assume this is what they meant, with slavery there would be much less need to do AI research, self driving cars is hard but slave driven cars is easy, the whole point of AI is that it lets us create slaves that we don't feel bad about abusing.
Your interesting take does ignore the fundamental point however that the human brain requires both far less energy and far less training data to learn. The takeaway for me is that there is huge scope to improve our machine learning techniques.
Now do videos.
I bet YouTube/Facebook/Instagram/TikTok/NetFlix spend orders of magnitude more resources uploading/transcoding/streaming videos than all the AI models training costs put together.
My 4c:
a) Stop using the term carbon footprint. It was invented by BP to scapegoat and market, it defocuses us. b) Blaming tech or civilians doesn't help. Blame countries. China, U.S, Germany etc need to step up. Ban coal power production -- coal industrial power combustion and mining is everything bad including radioactive (https://www.scientificamerican.com/article/coal-ash-is-more-...) -- promote nuclear. c) At this point add serious teeth to treaties. Countries not taking steps are costing future lives and resources. We are going nowhere otherwise.
P.S. Practically, too many powerful countries have serious incentives to offload responsibility, and as this is a prisoners dilemma game in the sense one cheating screws everyone, that there are no peaceful means to achieve this goal. So let's not kid ourselves.
> there are no peaceful means to achieve this goal
The EU's border-adjustment scheme isn't perfect. But it's in the right direction. Clean up your own act. Assign a broad-brushed adjustment to goods coming from dirty origins and use the proceeds to (a) further clean up your act or (b) incentivise others to clean up theirs.
Invading China is a cure worse than the disease, we really need to do this peacefully.
One peaceful way to do it is to develop clean technology that is cheaper than the dirty way.
Then every one will quit the dirty tech, out of greed.
Addressing the energy cost of "large" but rare events is a remarkably low leverage way to improve environmental outcomes.
Alongside training AI models, Spaceflight is the other issue that comes to mind. These things look bad because of "big" numbers or plumes of soot coming out of engines, but when added up they account for such a small fraction of emissions that they won't be useful focuses for a long time.
Focusing on them now detracts from places where we can have real impact: human transport, cargo transport, food production, clothing manufacture, carbon capture, and so on.
Limiting energy use while continuing to use carbon-emitting, non-renewable sources is literally the worst way forward.
This ignores renewable energy. What's the problem with renewables? Intermittency. Which is perfect for training. Just use a battery for a 10 minute buffer and start/stop when current energy price drops due to wind or sun energy flooding the grid.
Lol, this is a good angle for Numenta to take, but for all the time and brainpower they've spent so far you'd think they could come up with one thing that moves the needle in practice!
Numenta was founded in 2005. They’ve been pushing their idiosyncratic views on AI for almost 20 years with no impact that I can see. I guess they are upset that other people are doing AI without them.
I wonder how many Argentina's worth of electricity AI uses.
Almost all the claims in this article are technically wrong from the start
FTA:
> Many large companies, which can train thousands upon thousands of models daily, are taking the issue seriously.
No they don't. Because:
> Continuing to build larger and more computationally intensive deep learning networks is not a sustainable path to building intelligent machines.
If they'd care, they would evaluate how they threaten humanity's future by emitting more CO2 versus how they improve humanity's future with the services they provide.
Instead, they start from the idea that their services are a non negotiable absolute net positive (TBH it's obvious since it's their raison d'être)) and that CO2 is a negative that can be managed. Whereas, to me, it's the opposite: CO2 is an absolute non negotiable negative and the services might be changed or removed.
You are ignoring that all major cloud providers are aiming to be carbon neutral in the next few years and regularly publish their progress on that front. They take this very serious because people keep asking them about this and they don't like it when they have to give answers people won't like. Most companies at this point are well aware that they don't have forever to clean up their act. And most of the big tech companies have actually been very pro-active on this front for quite some time.
Pretty much all new data centers are powered using renewables. Not because it's cleaner but because it's cheaper. If your main consumable is a lot of electricity, you are going to want to source it cheaply. Which these days means the same as using renewables. Mostly that's a mix of them generating their own power and supplementing that with renewable energy on the local market. That has been the case for quite a few years, which is why companies like Google and Amazon figured out that they were also helping themselves financially by pushing hard for carbon neutral well over a decade ago.
So, whatever goes on in those newer data centers is pretty much not emitting any co2. Aside of course from manufacturing of hardware and construction of the site, which would be a separate topic.
Some of the older data centers are still powered via coal/gas plants, especially in the US. Those will likely clean up their act in the next few years. All the big cloud providers have announced roadmaps for becoming carbon neutral. Once that is done, data centers will use a lot of energy that is generated sustainably.
That doesn't mean that there aren't issues around this. E.g. the Netherlands is reluctant to host more data centers because they are claiming all the sustainable energy that is available in the local market and they have to worry about powering the rest of the country in a sustainable way too. And since that costs a lot of money, having Facebook or Google then get subsidized to use all of the newly available wind power looks a bit bad.
But all that means is that there's a lot of healthy demand for clean power that short term outstrips supply. Lots of companies are working to tap into that rapidly growing market. Which is good news for our planet. Any market that supplies a lot of clean & cheap energy can expect to see businesses trying to make use of that. Especially energy intensive businesses like data centers. So, there's a great incentive for governments around the world to make that happen.
So, perhaps the wrong reflex is to argue that we should all become Luddites to reduce demand on renewable energy. The right response is to argue for sourcing your cloud services sustainably. Most cloud providers can probably tell you which of their data centers are clean and which aren't. They might not be very vocal about dirtier ones but you can choose which regions you host in and which ones you skip and make some educated guesses. Interesting way to nudge them a little harder. That's on you, not them. Not a lot of companies do that yet. But they could. And IMHO they should.
If you wanna get good laughs, I recommend any Austrian conference about energy.
So easy to trigger anybody, much fun.
this is not revolutionary, all of this has been discussed before.
the stuff about inspiration from the brain confuses me too... it reads a bit like neither ai nor neuroscience is particularly deeply understood. its very basic stuff.
If it's profitable to run the AI given it's energy costs, that means all is fine, no? If you think it shouldn't consume all that energy, shouldn't the energy be more expensive?
Not necessarily; while I wouldn’t apply the term in this case, the idea of a “sin tax” is an example of where society says a product or service must be made more expensive to discourage use, even if it’s otherwise profitable.
Bitcoin used all the energy and took all the profits, so there isn't enough left for the AI. duh.
Or, and hear me out, using energy to push society forward isn't necessarily bad.
Maybe instead of vilifying every new technology that uses energy for the last 20 years, including the internet: (Dig more coal the PCs are coming https://www.forbes.com/forbes/1999/0531/6311070a.html?sh=56f...), perhaps the impetus should be on incentivizing green energy production.
Let’s start at the insanity that is crypto that has little value.
"Just use less energy" arguments are insane. Even when we improve the energy efficiency of our processes, that just leads to said process happening more - not an overall reduction in energy use. Just look at historical energy consumption trends to see this.
Anything less than producing more energy more cleanly is missing the point. Long term energy usage trends do not go down.
No, our dependence on fossil fuels to product energy is harming our planet. I feel like it’s long past time we reinvest in nuclear energy. It’s the one thing we can do to significantly reduce fossil fuel consumption over the next decade.
Batteries! Who wants batteries!
AI will eventually be all that's left of humanity, I'd bet. It's probably worth it, as much as anything is.
So after trying their luck at Bitcoin PoW, the new target is AI ? When did using energy become a thing to apologize for ?
Year 2022, I know people in America are comfy in their homes. But here in Europe, people are dying few thousands KMs from our borders. Who the fck cares about carbon footprint or green energy...
I'm sure these people will gladly put themselves in a position to decide which are valid uses for energy so they can enlighten us all with their wisdom
And those who don't obey should be intimidated into compliance. After all, this is a fight to save the planet and your casualty is a small price to pay
That's civilian corporate Ai.
There are other Ai possibilities, for example military-grade Ai: https://www.reddit.com/r/conspirFBeyesWideShut/comments/v74i...
And THEN ... "asteroid" is an anagram for "Ai do rest" ... that would just automatically plug into the Earth's natural electro-magnetic field as both an energy source and a means of subtly influencing "evolution" of lifeforms that could be influenced via their nervous systems ...
First to stop clapping. By attaching an inconsistent critical theory to something complex like machine learning, it elevates the ideological speaker to being percieved in the eyes of an audience as equivalently competent to the people whose work they are criticising. This is the trick of all critical theory, and if you do not challenge them, it means that you are going to have to pretend someone yammering on about climate justice knows anything about your field.
Saying something "harms the planet" these days is just code for pointing out targets for subversion, subjugation, and governance. The next best they can do is be bullying and disagreable. I'm tired of this bullshit narrative stuff. What's doing the most harm to our planet is putting up with it.