▲ 435 points
back
141 comments
4bit Quantized versions that run on an A100-40G or 2x3090/4090 24GB: https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ
Inference is very slow right now but it works!
How is it possible to run this model on 2x 4090s ?
I thought that 4090s were "nerfed" and nvlink support removed - https://www.windowscentral.com/hardware/computers-desktops/n...
The hardware requirements on these models is basically at a fixed floor, and the democratisation will come from cheaper, possibly specialised, hardware, not reduced requirements, right?
This will require a cpp port first to run on e.g. apple silicon?
How can it be deployed on a Huggingface Space or Colab notebook?
is there a model version where it can be deployed using huggingface's tool?
For people who directly want to check the benchmark - https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
Very cool but how does this compare to GPT-4 (before it was nerfed)?
I feel like the best benchmark atm is the orig gpt-4 version.
How does it compare to OpenAI's GPT4? It's not listed in the table.
Am I the only one who finds this very sketchy? They had the whole license things, there's been some loud complaining by the HF CTO on social media that this model is not getting enough attention, and there are also press releases about how Falcon tops the "leaderboard":
https://www.morningstar.com/news/business-wire/2023052900504...
I've never seen this kind of "strategy" with an ML model before. Maybe I'm seeing something that isn't there...
It could be a question of not being used to see blatantly commercial advertising in places we're used to being about software. Feels like we're moving more towards the bro-ification of AI.
Not sure I care to be honest. With an open license the community can now take this and roll with it.
What PR noise they make is secondary to that in my mind.
maybe attention IS all you need
Probably PR to attract subsidies/investment. UAE might be throwing more money on tech/innovation branded projects than China.
I saw this, I still am highly suspicious about the future of these models and later attempts at monetization. Did they really just drop their 10% royalty thing and decide they'll just open source all their models now?
Another comment mentions llama may get an open license, and there are other emerging alternatives. In six months there will be lots of options. I would not spend my time building anything around a model that started in such a sketchy way.
It would be interesting to hear about why they decided to change their license and what their plans are for the future.
I admit Falcon has really pissed my off because they pretended they had an open source model when they really released a sleazy freemium thing.
Do you consider the Unity model "sleazy"? But I agree that it was poor form to call it open-source at the time. Sadly, it seems to be standard practice in the LLM space to release models with all kinds of restrictions on use and call them "open source".
Well now Falcon is open source. Given they are giving away something that was very expensive to train, I am grateful.
Well if its Apache 2.0 now, can't you clone it as-is and have it be open "forever?"
This is such a bizarre comment to make about something that is Apache licensed. Who cares what happened before now that it’s truly open?
This Falcon-40B royalty free license may force Meta ... that LLama-7B/13B may soon be fully open sourced as Meta wants open source LLM advancements and contributions on its own LLM architecture.
Yann LeCun: "No. But it's not because we don't want to. It's because of complicated legal issues."
Why do people think that Meta released their model in order to get open source coders to improve their models? They will get absolutely no competitive advantage from this. Every other team developing a closed source LLM can easily copy the innovations that open source coders have applied to Llama on their own, closed source models.
There's no advantage here. Meta just spent $10 million on releasing fun chaos into the world and increasing their recruiting power.
Research in the LLM space is moving so fast, I doubt any architecture will "stick".
Meta's eventual, internal LLM architecture will be totally different than the open source LLMs, right? They don’t need to run on commodity hardware.
Maybe I am cynical, but I dont see the incentive for Meta to contribute an open model.
Serious question - why doesn’t someone crowd source the funds to train a GPT scale model for open source? I assume it’s not just a matter of a ton of GPU instances?
Doesn't one need to have a bunch of "very good" data to train on? I'm under the impression that sourcing costs are large.
For fast inference, the HuggingFace cofounder, Thom Wolf recommends their text-generation-inference library https://github.com/huggingface/text-generation-inference
In terms of building something that's usable (considering cost, speed, scale, etc) if comparing an OpenAI API call to these, it's difficult for me to see a current path where these have any viable application outside some niche scenario.
From what I understand, even to run locally you/your team needs to be able to afford a machine with a 4090. These are super expensive in some countries.
I played around with the smaller Llama/Alpaca models and it wasn't really viable to build anything with.
Not really seeing a use-case for fine-tuning either compared to just few-shot prompting.
Can someone fill me in on what I'm missing? It feels like I'm out of the loop
I'm running Vicuna on a free 4core Oracle VPS, and it's perfectly usable for a Discord bot. Responses rarely take more than 15 seconds with <256 max token limit, and the responses are much more entertaining than GPT 3.5. I'm not using the streaming API my server software[0] offers, but if I did it would probably load somewhere between the speeds of GPT-3.5 and GPT-4. It's more or less the same time a human would take to compose the same message.
So... not exactly a serious use-case. But it's what I'm using, and now I'm saving 10s of dollars on inferencing costs per month!
[0] https://github.com/go-skynet/LocalAI
I'm also using this to improve acceleration - https://cloudmarketplace.oracle.com/marketplace/en_US/adf.ta...
Fine-tuning is a much better proposition than you’re giving it credit for. Papers are coming out demonstrating that 7B parameter models can outperform GPT-4’s quality when trained on a limited set of tasks. Yet, a 7B model offers comparatively cheap and fast inference. Furthermore, for a lot of use cases, few-shot prompting is infeasible because you need to supply 2-3k tokens worth of few-shot examples with every prompt in order to fully specify the behavior you want. (As an example, think of long-form summarization where you want the summary to adhere to certain rules.)
>From what I understand, even to run locally you/your team needs to be able to afford a machine with a 4090. These are super expensive in some countries.
That's because we're only half a year into LLMs becoming mainstream. Give it 3-4 years. The advancements in bringing down model size, optimizations, and newer GPUs, SoCs from Nvidia, AMD, Apple, Intel, Qualcomm, etc will make it so that top LLMs will run on a highend laptop/desktop.
This is bleeding edge stuff.
All advances in this direction do indicate that it will be easier and easier for more people to do things with it.
This doesn't need to work for everyone.
A 4090 costs today 2k, the 3090 with also 24gb costs today 1k and costed 2k.
Not specific to this model, but beyond the large players (OpenAI, Cohere, etc) are there any free hosted versions of the open(ish) LLMs? Even the smaller 7B parameter ones? I'm prototyping out a project and using OpenAI for now, but it feels like there has to be a hosted alternative somewhere.
I spent some time today exploring HuggingFace's Inference API but if the model is sufficiently large (> 10gb), HF requires you to use their commercial offerings.
> HF requires you to use their commercial offerings
Some of which are quite affordable ($80 per month). Larger ones can be like 2000 a month which is still ok to prototyping phase. You're basically paying for aws/gcp infrastructure.
I quite liked the UX of it, very intuitive. My trouble was finding a model that executes out-of-the-box tho. All of the GPT ones crash on startup.
40B is pretty large, right? I expect it would take 70GB or so of RAM to run it. That's some expensive hardware ($10,000 or more).
Some people use second hand P40 GPUs, which go for around 200-300$. Combine 3 of them with SLI and you've got 72GB of VRAM for less then $1000
> https://github.com/ggerganov/llama.cpp/issues/1602#issuecomm...
Here somebody quantized it down to 29929.56MB .
Amy implementation for this akin to llama.cpp?
Hypothetically speaking, if Falcon 40B could out-perform GPT-3.5, would that force OpenAI to open-source GPT-3.5?
I wish these releases had short but meaningful descriptions for their models.
E.g. 10 layers x 2048 tokens x 1024 embedding model using full attention.
I am an amateur when it comes to these models. What can I do with this model and how?
It's interesting, that intruction-tuned model has lower performance on ARC and HellaSwag.
Oh wow. That’s great news.
Falcon seemed good till I read the license fine print about pre approvals and what not. This seems to fix that
I’m curious to know the real reason for the change. Is it related to the data they used for training?
If it turns out that there is a consistent architecture which works really well, how long before we see an ASIC?
Is this multilingual?
Chatting with Falcon 40B feels like chatting with GPT4, very capable model.