back
8 comments
Sounds like Microsoft is pricing AI services too low (possibly as a monopolization strategy) given capital costs and constraints of scaling up capacity.
This reads to me like, "$2.4 trillion dollar software company throws a public tantrum because they didn't order the parts they needed in time"
During every GPT-4 request, eight of these[0] cards are utilized to 100%. EUV wafers are borderline black magic, only so many can be produced in a day.

[0] https://www.newegg.com/p/1FT-0004-00814

I've heard this line before, but I have a hard time understanding it.

Does that mean that a single user has an exclusive lock on 8 of these things for the duration of the request? I'm surely missing something.

> Does that mean that a single user has an exclusive lock on 8 of these things for the duration of the request? I'm surely missing something.

You're not missing anything, that's exactly what it means. There is a reason OpenAI severely ratelimits the number of GPT-4 prompts users can make. Every token (approximately 4 letters) requires many trillions of floating point operations.

Large language models are only possible because of decades of exponential transistor density growth.

Yes, these large language models are extremely constrained by RAM. Traditional PC architecture separate RAM and VRAM, so GPUs are required to have the appropriate amount of memory onboard, in this case achieved via 8 A100s synchronized to act as one compute module.

This is why developers such as myself have opted for Apple's new unified architecture to run smaller models locally, as the GPU can access all 64GB. But these small models are still a fraction of the size of GPT-3.5-Turbo and GPT-4.

I was very sad to discover that my 10GB 3080 can't run any compelling LLMs on initial overview, and that the A100s are well above 5grand in price. I wish something like mythic ai's analog chips would come out in the consumer arena.
That is entirely a problem that can be solved by throwing money at it. Why are they complaining to the public about their own organizational torpor?

Microsoft should have the maturity, money, and any other resources needed to solve their own problems without making a public spectacle of it. It's embarrassing and childish.

I am being critical of them, but their net worth is nearly 1/10th of the debt of the entire United States of America. If they can't get it together with that level of resources, then they should be ashamed of themselves.