back
305 comments
I keep seeing these "sovereign" LMs time and time again. In Sweden we had GPT-SW3 (https://www.ai.se/en/project/gpt-sw3) and same story there. Instead of burning money on "sovereign" claims, national research labs should instead focus on building on top of solid baselines (like Qwen/Kimi) and finetuning frontier models with real agentic utility that can be applied across actual use cases and can be widely used by its people, basically for free. Nations should mirror what Cursor has done with Composer 2.5 for example.
Disagree, it’s in the country’s best interest to facilitate internal expertise on the full stack and own their “supply chain” so to speak and fight brain drain. The outcome isn’t just the model, it’s the expertise. Otherwise all their smartest folks will depart for countries where LLM development is strongest.
Kimi and Qwen come out of China, which means that their training material may be biased e.g. relating to Taiwan [1]. In addition, there is no way to determine what input went into the training, if it was properly licensed, if it was legal (e.g. not contaminated by CSAM), or how the human component of RLHF was sourced - in US models, for example, stories about exploitation like [2] have been floating for years.

Assuming us Europeans finally get our act together, I think it is better for our long-term future (and the ethical problems) if we manage to get a baseline of training input and data ourselves, from scratch, with everything being ethically sourced.

Oh and, while we're at it, the EU has 24 official languages plus a host of minority languages. Most LLMs focus on the English, German, French and Chinese languages, but everything else is... left behind at best. An European model with actual funding and proper data sources might be able to significantly reduce that.

[1] https://www.taiwannews.com.tw/news/6245677

[2] https://www.theguardian.com/technology/2024/apr/16/techscape...

If open frontier models start closing up and states start more export controls on AI services and hardware, it might be good to ensure the supply chain is there to reproduce the SotA, or even a couple generations behind it.
There is something to be said for this "most cheapest" approach, there is also something to be said for making models that are entirely ethically sourced:

1. Free of controversy like unlicensed training materials

2. Free of exploitative rlfh loops by people in low-wages countries

3. The leasons learned (and published) from going through the entire training process on "European" hardware: "AI factories" (the term for Slurm HPC/HTC systems with lots of heavy GPU nodes, heavily subsidized by our government [0])

1 and 2 are strong counter-LLM arguments at the moment, and hold back some groups of potential users. Another is energy/water use, so going for maximum green energy would be a nice boon as well. 3 is something I consider to be highly useful for our European identity and "way of the ninja" (for you Naruto fans out there).

[0 https://hpc-portal.eu/funding-opportunities]

Sounds like calling those model "open source" did its wicked job. You can not take an open weight model and build a next-generation model using that as a foundation. Once those companies decide it's no longer in their interest to release new open weights everything you've created this way becomes a pile of rapidly deprecating legacy.
Do we know for sure how much national corpus of knowledge (like dutch) goes into these "global" models and how that affects "localized" model biases? What's wrong with specialized models?
And what happens once the "solid baselines" become unavailable for a reason or the other?
TNO doesn't serve nor represent global interest and hence does not care about global progress. It exists to enhance knowledge on things within Dutch society primarily with some ripple effect outward to EU because they have interests within the EU.

Its purpose is not to become some kind of OpenAI or global foundation offering services/tools on that scale.

There is a lot of critisism on this project, not invalid, but mostly based in lack of understanding of what the goals are of the organization as well as the people building the thing.

The people building it, are well aware of how it will be less capable than other LLMs on a general reasoning aspect, not only due to having actually purchased _all_ licensed data that has been used as inputs. Not being a multi billion dollar corporation, this means having very little data and should be an obvious signal to observers that it has not the goal to outdo other models.

In my opinion (personal) its a project that has a learning and demonstration value that is not 'look how well our model performs against others', but still offers value.

Today, you keep seeing "sovereign" LMs that are subject to the sovereignty of some human-led state. Tomorrow, the "sovereign" LMs will be called that for a completely different reason.
Their legality is very questionable given all the likely copyright infringement going on, and a state can't really ignore that.
It is crazy that anything Europe gets so much hate. IMO it is important to build models within the boundaries of smaller nations, using their own language. Research has to continue even if it is outside of US and China.
If Europe is serious about getting home grown AI fast, three simple steps:

1. Huge tax incentives, let the companies get grossly wealthy while paying minimal taxes. Minimum 10 years with clauses protecting "retribution" taxes there after.

2. Tax incentives for the founders/shareholders, just like above.

3. Drop worker protections to a minimum, make it easy to fire people. You only want serious/dedicated employees anyway.

Within 2-3 years there will be at least a trillion dollars looking to get in.

Don't worry though if reading that made you mad. Its absolutely not going to happen. I can think of few things more antithetical to the European ethos than smart skilled people working 80-100hrs weeks with almost no vacation to gas their founders net worth by tens, hundreds, of billions.

> GPT‑NL is developed within the Netherlands and Europe. This gives us full control over the model, the data and the choices we make. We avoid dependency on non‑European providers and invest in a sustainable AI ecosystem aligned with our laws, values and societal goals.

I love it! So this is our answer to America and China denying foreigners access to their frontier models.. a massive 13,5M€ founding to develop souvereign european ai, trained exclusively on legally obtained documents and highest moral standards as defined in EU AI Act.

So good to see these developments. Every country should do this. I'd even say every person should gave their own personalized AI running on their own computers. If only the costs involved were not so astronomical.
Interesting that this got posted now: the project is receiving increasingly more skepticism lately in the Dutch tech scene [0], and I think that’s fully justified.

[0]: https://www.quotenet.nl/zakelijk/a71588202/techondernemers-m...

I think at this point what the Netherlands, and any other country that wants a good model in their language should do, is gather up every piece of text ever written in that language and license it to the big AI labs/companies for training. I'm sure there are vast libraries of books and other text that haven't been digitized and aren't a priority for the big labs.
Previously posted on 02-dec-2023 https://news.ycombinator.com/item?id=38497495 3 comments
I don't understand countries (especially governments) wanting to have their own models when there are already pretty solid open source (weights) models out there.

Countries should want control over _where_ the compute is happening rather than _what code_ is running.

What's wrong with a country hosting a Kimi, Qwen or GPT-Oss on their hardware for their government work purpose?

They’re building a competitive-quality model, from scratch, with fair compensation to content owners, for €13.5 million? Something’s wrong with this picture.
I feel that not only is Europe losing its independence to the US and China, but it does not even try to take part in the race.

Unlike the US, Europe has no California-level VCs. I don't expect hundreds of billions of Euros to be poured into long-shot projects.

Unlike China, Europe has neither cohesive public investment at the global level nor the drive to grow. Long-term investments have a lot of words, a lot of regulations, a lot of proxy goals, but there is neither a lot of money nor urgency. It was captured by this post: https://x.com/piotrsankowski/status/2065795919623438546

So yeah, both in economy and warfare, Europe dooms itself to be in the hands of the US, China, or a mix of both.

This is not an open source model. In that sense I think the sovereign claim is a bit strange. It's the data providers that determine access to the model.
A total of €13.5M has been allocated to the project.

I guess we’re going for GPT2 level capability?

I really think countries should build a sovereign _ecosystem_ and sovereign models are an excuse to achieve it.

An ecosystem is the tribal knowledge, revolving door of talent, known processes etc.

If the end goal is to make a half assed Dutch speaking model, I think it won’t cut it. I don’t see anyone using it over Gemma 4b that runs on my laptop.

An ecosystem is more durable and has desirable second order effects.

I feel like building datacenters and filling them with chips may be more valuable than creating sovereign models. xAI I think makes more money renting datacenters to Anthropic than with the models they trained, and they could pivot thanks to their datacenters. By making regulations easier than in the US, this could bring some computing power to Europe, which then can be used to train sovereign models, or rented to big AI labs.

Also, when training models, you create talent that then could go to other countries (brain drain). Restricting that brain drain without imposing authoritarian restrictions on the movements of people seems hard, so it seems hard to keep talent as a competitive advantage. If instead the competitive advantage is datacenters with chips, power capacity building, fast path to building datacenters, I think they are easier to retain while preserving the rights of everyone involved.

> A total of €13.5 million has been allocated to the project.

> This public investment underlines the importance of an independent, trustworthy and future‑proof Dutch language model.

It does, but not in the way you think it does.

We already had GEITje but it was banned by the courts. Of course it can still be found because the entire internet is not subject to Dutch law. But it did manage to stop development :'(
Europe should have a sovereign model on its content and languages that is trained with renewable energy and published as open source.

This looks like a good step in that direction.

I wonder with these stories. Why are there so many individual country efforts? We know the scale needed with scaling laws / capital / energy. Most of these countries alone can barely compete (even large groups of them would struggle.

Why don't they work together on it? Companies like Airbus have already been able to do that with aircraft.

I'm making a Dutch dictionary and would be interested to see how this model would fair in evals vs non specialized ones. I've tested a variety of models for https://hetnederlands.com content and differences can be big
Is it a proposal or a model? And if it is a model, how fies it fare on benchmarks?
What really matters is the sovereign capability to finetune the LLM models. Any model could be vetted and tested, but you need finetuning/lora training to prevent the model from being outdated.
What are they going to train with 13.5M really? We're a tiny company in Amsterdam in Holland and we've got "only 64x B300 to train on" so we could never make an LLM I thought, since we've got only 4M in compute.

And they're going to train an LLM with all kinds of extra difficulties compared to OpenAI for just 13.5M?

The very first Llama was 16M for one training.

Supposedly this model also aims to treat publishers of all sizes well. Looking forward to its launch soon :)
overall, the revenue sharing model is (IMO) more interesting than the fact that it is dutch. Usage of data, and sharing it with the providers of this data, is an inherent part of the creation of these models that is not discussed as much as it should be
Honestly, I used to think the 'sovereign model' was a waste of money. But recently, with the US logic of restricting model exports, I've come to think that if things go south, they could even cut off allied nations. So now the sovereign model seems reasonable to me. That, in turn, means US influence is deteriorating. And that probably isn't such great news for American businesses.
I fear sovereignty is not a adoption-driving feature
Better work on something at Europe level
So cute.
How do you use it?
Burning tax money. I dare to bet this will never lead anywhere.
"sOvErEiGn"
> Excluding harmful content

#define(HARMFUL)

[edit] Downvoters please tell me what the problem is with specifying this?

How about fixing whatever the hell prevents competitive private LLM vendors from appearing in Europe?
> A total of €13.5 million has been allocated to the project.

This is not even funny. If you want a competitive AI industry, you need to invest much more heavily in infrastructure first, building models second.