▲ 1129 points
back
509 comments
Benchmarks for Gemma 7B seem to be in the ballpark of Mistral 7B
+-------------+----------+-------------+-------------+
| Benchmark | Gemma 7B | Mistral 7B | Llama-2 7B |
+-------------+----------+-------------+-------------+
| MMLU | 64.3 | 60.1 | 45.3 |
| HellaSwag | 81.2 | 81.3 | 77.2 |
| HumanEval | 32.3 | 30.5 | 12.8 |
+-------------+----------+-------------+-------------+
Thank you. I thought it was weird for them to release a 7B model and not mention Mistral in their release.
Only 8K context as well, like Mistral.
Also, as always, take these benchmarks with a huge grain of salt. Even base model releases are frequently (seemingly) contaminated these days.
According to their paper, average of standard task of Mistral is 54.0 and for Gemma it's 56.4, so 4.4% relative better. Not as big as you would expect for the company which invented transformers and probably has 2-3 order more compute for training it vs few month old French startup.
Also for note on their human evaluations, Gemma 7B IT has a 51.7% win rate against Mistral v0.2 7B Instruct.
Came here to post the same thing for Phi-2:
+-------------+----------+-------------+
| Benchmark | Gemma 2B | Phi-2 2.7B |
+-------------+----------+-------------+
| MMLU | 42.3 | 56.7 |
| MBPP | 29.2 | 59.1 |
| BoolQ | 69.4 | 83.3 |
+-------------+----------+-------------+
[1] https://www.microsoft.com/en-us/research/blog/phi-2-the-surp...
the real gold will be when this gets finetuned. (maybe by mistral...)
https://www.youtube.com/watch?v=1Mn0U6HGLeg some test vids came out on the 7b model. Shock it doesn't perform well at all.
In my subjective tests it's not even close to Mistral.
While my local gemma is quantized, so is mistral.
But I also tried gemma on huggingface.co/chat which I assume isn't quantized.
Honestly, this is more of a PR stunt to advertise the Google Dev ecosystem than a contribution to open-source. I'm not complaining, just calling it what it is.
Barely an improvement over the 5-month-old Mistral model, with the same context length of 8k. And this is a release after their announcement of Gemini Pro 1.5, which had an exponential increase in context length.
The terms of use: https://ai.google.dev/gemma/terms and https://ai.google.dev/gemma/prohibited_use_policy
Something that caught my eye in the terms:
> Google may update Gemma from time to time, and you must make reasonable efforts to use the latest version of Gemma.
One of the biggest benefits of running your own model is that it can protect you from model updates that break your carefully tested prompts, so I’m not thrilled by that particular clause.
Hello on behalf of the Gemma team! We are really excited to answer any questions you may have about our models.
Opinions are our own and not of Google DeepMind.
I notice a few divergences to common models:
- The feedforward hidden size is 16x the d_model, unlike most models which are typically 4x;
- The vocabulary size is 10x (256K vs. Mistral’s 32K);
- The training token count is tripled (6T vs. Llama2's 2T)
Apart from that, it uses the classic transformer variations: MQA, RoPE, RMSNorm.
How big was the batch size that it could be trained so fast?
https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2/bl...
Is there a chance we'll get a model without the "aligment" (lobotomization)? There are many examples where answers from Gemini are garbage because of the ideological fine tuning.
The fact Gemma team is in the comments section answering questions is praiseworthy to me :)
I personally can't take any models from google seriously.
I was asking it about the Japanese Heian period and it told me such nonsensical information you would have thought it was a joke or parody.
Some highlights were "Native American women warriors rode across the grassy plains of Japan, carrying Yumi" and "A diverse group of warriors, including a woman of European descent wielding a katana, stand together in camaraderie, showcasing the early integration of various ethnicities in Japanese society"
Stuff like that is so obviously incorrect. How am I supposed to trust it on topics where such ridiculous inaccuracies aren't so obvious to me?
I understand there will always be an amount of incorrect information... but I've never seen something this bad. Llama performed so much better.
There are some pretty impressive benchmarks on https://ai.google.dev/gemma. Even the 2b model looks fairly not awful?
I guess my weekend is going to be spent exploring this.
Go back 5 years and ask anyone on this site what companies do you think will be the most open about AI in the future OpenAI, Meta, or Google. I bet 10/10 people would pick OpenAI. Now today Meta and Google, both trillion dollars companies, are releasing very powerful open models with the ability to be used commercially.
Ironic.
mmm..
> what are the top 5 tallest man-made structures in the world
Statue of Unity (Tallinn, Estonia) - 368 m
Leaning Tower of Pisa (Pisa, Italy) - 330 m
Supertree Grove (Orlando, Florida) - 275 m
Skylon Tower (Kuala Lumpur, Malaysia) - 271 m
Statue of Liberty (New York City, USA) - 186 m
The richest country in the world is Saudi Arabia with a gross domestic product of US$2.3 trillion.
A foot is equal to 0.0254 metres (ft).Parameter counts notwithstanding, it’s an objectively funny outcome that Meta, Microsoft, and Google are all releasing cutting edge open models, while OpenAI keeps theirs closed source.
If you are looking for a nice chat UI to try out Gemma (and other offline + online models) locally, I'm working on an app [1] that is offline and privacy focused.
I've just added support for Gemma 7B.
[1]: https://msty.app
Already available in Ollama v0.1.26 preview release, if you'd like to start playing with it locally:
This is commendable, but there's room for improvement. Up until now, SOTA-level "open-source" LLM models (LLaMA, Mistral, etc.) have usually only made their inference code and model architecture public. While these elements are not insignificant, they are somewhat trivial when compared to the training code and training datasets, as these two factors largely determine the performance of the model. This is not open at all. It goes without saying that sharing the training datasets and process with other AI researchers is crucial. This transparency would not only help to improve the model(for others could contribute to it) but also contribute to the whole community, as they usually advertised. Otherwise, it will be difficult for these efforts to truly promote the development of LLM.
They have implemented the model also on their own C++ inference engine: https://github.com/google/gemma.cpp
Taking a page out of metas book with open models. I wonder what the game plan here is.
Nice that it allows commercial use!
Tried inference with the 7B model and without flash attention this is soooooo slow. With flash attention the fine-tunning requires A100 or H100.
Also the inference doesn't always stop generating resulting in garbage being added to the response.
It is surprising how willing Google is to stretch the truth in the marketing for its AI initiatives. Although it's being compared against 7B models, Gemma "7B" is actually much more than 8B parameters in total.
Congratulations on the release! How can we download the model and run inference locally?
The 2B model seems underwhelming. For instance, compared to the recent StableLM2 1.6B model that is slightly smaller and probably wastes some "English metric points" by being multilingual.
The latter (and other similar open models) seem to do similarly well in benchmarks (much better in Math?) with way less fancy stuff. For instance, public data and no secretive filtering with pre trained models or synthetic data.
My take is that using the vanilla approaches take you really far. And many of the latest tricks and hours-of-work buy you little... Will be interesting to see how this plays out, especially for the open source community.
Can this run on my AMD Vega VII on Windows 11? As always, AMD is missing:
> Optimization across multiple AI hardware platforms ensures industry-leading performance, including NVIDIA GPUs and Google Cloud TPUs.
Gemma-7B (instruction tuned version) is now on the Vectara HHEM leaderboard, with 100% answer rate and 7.5% hallucination rate. Pretty good for a model with 7B params.
I applaud the Google team openly engaging on HN here.
Q: how sure are you that the newer models trained from trillions of tokens - a huge chunk of open web, hasn't been accidentally polluted by slurping test data?
I really don't get why there is this obsession with safe "Responsible Generative AI".
I mean it writes some bad words, or bad pics, a human can do that without help as well.
The good thing about dangerous knowledge and generative AI is that you're never sure haha, you'd be a fool to ask GPT to make a bomb. I mean it would probably be safe, since it will make up half of the steps.
Available on Ollama?
Thr landing page on ai.google.com seems to be machine translated, for Huggingface it uses the literal German translation (Umarmungen Gesicht)
I wonder if people will get confused with the naming
Gemma, Gemini pro, Gemini advanced, Gemini ultra
To a layperson it is not obvious which one is better than the other
Has anyone found the context length for these models yet? So far I haven't seen it mentioned in their write-up or the model card
It looks like it's pretty resistant to quantization. ollama 4bit 7B doesn't work very well, but the 16bit 2B does
Is there any research on using smaller, lower capability models to act comparable to high quality models? Even if it's just prompt engineering or doing lots of attempts to accomplish the task?
If somehow that is possible it means we only need a capable enough model and can use it reliably for lots of practical things.
Nice to see more open models. Props to the team for coming to the HN comment section to answer questions
> Open models feature free access to the model weights, but terms of use, redistribution, and variant ownership vary according to a model’s specific terms of use, which may not be based on an open-source license.
does a model being "open" say anything about how it was trained?
The scariest difference between OpenAI and Google right now is: Ask Gemini who owns the code it writes, and it'll confidently say that Google does. Ask OpenAI, and it'll say that you do. It's that easy to choose which one is the better decision.
Hopefully, they re-release this under an open license. Making everyone go through the exercise to authenticate and agree to terms hasn't worked for any model to-date. It just limits it's reach. We saw the same thing with Phi-2.
Looking forward to Gemma 7bx8 moe
Are these any good? I have been trying the non pro version of Gemini, and that seems awful at code generation. I am more keen on getting access to the best model and I would pay for it if I wasn't already paying for ChatGPT 4.
Nice, more choices are good. I just saw that the Ollama project already has these models available (date stamp is 58 minutes ago), so I will use that rather than Colab (I live Colab, but I like to run stuff locally).