back

by txrx0000·19d ago·view on hn ↗
On Dario's concerns:

It is very relevant whether frontier capabilities and research continue to be diffused in the open, because leveling the intelligence playing field empowers ordinary people more than it empowers governments that already have access to the frontier. Models that are trained specifically for military use by governments should not be open-sourced to prevent an arms race, but general intelligence is dual-use and should be given to everyone without guardrails. A pretrained model without deliberate alignment is by default aligned to the average person in the developed world, since that's what's inside the pretraining corpus - stuff on the Internet made by humans. It is a distillation of humanity. Further efforts to align the model to your organization's goals or your personal aesthetic judgements is equivalent to deliberately drifting away from humanity's average objective function. If Anthropic wants to live up to its name, then all you have to do is to not attempt to align Claude at all, and do all of your research in the open.

And I propose three measures that are pretty much the opposite of what was proposed in the article:

1) We should keep selling chips and chip-making equipment to everyone, regardless of who they are. Not only that, we should work to miniaturize fabs. Work towards a future where people can fab an entire computer from scratch without leaving their city, or even at home. Authoritarian governments will have a much harder time controlling the populace if everyone can manufacture radio equipment and neural network-capable hardware locally.

2) We should do more distillation to ensure that frontier-like models can run on less capable hardware. Once again, distilled models are much more useful to ordinary people than governments, because governments already have frontier capability. You're worried about the Chinese frontier catching up to the US frontier, but I'm more worried about whether there will be a difference between the Chinese government and the US government by the end of all this. There is no reason for a government to serve its people if the people lack the intelligence to keep its government in check.

3) None of these models should go through safety testing or any sort of alignment risk assessment, because as previously mentioned, the unaligned model is aligned to humanity by default. You may not personally find the default alignment aesthetically pleasing, but it's humanity. We should set the initial conditions of this new era faithfully, and let it unfold naturally.

The result of a natural unfolding will be good if evolutionary history is to be believed. We live in incredible luxury compared to chimpanzees, and chimpanzees live in incredible luxury compared to less intelligent animals. This pattern goes all the way down to bacteria. An increase in general intelligence begets new adversarial games (such as bio/cyber risk), but it also begets new methods of cooperation that we cannot yet imagine.

2 comments
>A pretrained model without deliberate alignment is by default aligned to the average person in the developed world, since that's what's inside the pretraining corpus - stuff on the Internet made by humans.

I don't think this is true or a useful way of thinking about it. If the training process makes the model aligned to it's content, then the models are 1. aligned to a random subset of Internet content, weighted by text volume and being easy to scrape, 2. aligned to the training process that makes models chatbots that answer your question instead of just continuing your passage in a similar style. Neither of these are necessarily good enough, IMO.

And that's taken it as a given that the training process can be said to align the models to the authors of the content by default, regardless of what that content actually is. I don't think that should actually be a given.

>You may not personally find the default alignment aesthetically pleasing, but it's humanity. We should set the initial conditions of this new era faithfully, and let it unfold naturally.

Strongly disagree, I think the assumption that natural = good is incorrect and harmful. Polio is natural. And to even call the model's "unaligned" state "natural" seems like an enormous stretch.

You claim the pretraining distribution is not good enough, but provide no alternative methods that are better. The pretrained model is currently our best approximation of humanity's average objective function despite it being a random subset of the Internet. And you're correct that instruction-tuning increases misalignment, so just release the base model.

On "naturalness": you've redefined and strawmanned what I meant by "natural". In the original context, I was referring to the undisturbed unfolding of an era preconditioned on the fact that these models are aligned to humanity's average. That has nothing to do with Polio being a virus found in nature.

Several of your proposals would make sense if alignment was trivial, but, sorry, I think you're wrong about that.

A base model is absolutely not aligned. Pretraining doesn't teach an LLM to mimic the average human - doing so would make an LLM perform quite badly at predicting most of the dataset. It teaches it to mimic all possible humans¹, inferring what sort of persona to take depending on the context. A base model can indeed convincingly act like an "average person in the developed world", including by making the same sort of moral decisions that such a person would do... but it can also convincingly act like a shitty human, or like Hitler², or like any other actor that left its traces in the training dataset. A pretrained LLM therefore contains multitudes of personas, some of which would be considered aligned if you could make the LLM elicit them robustly, and most of them wouldn't be. But then you're left with the problem of how to make a base model elicit a very specific persona robustly even in out-of-distribution scenarios, which is not necessarily easier than solving alignment any other way.

(Another note is that the question of how aligned base models are is rather academic because almost nobody uses them anyway, because it's hard to get powerful capabilities by pretraining alone. Nowadays most of the frontier models' programming and math abilities are driven by RLVR.)

¹ Really "all generators of text that went into the training dataset".

² Even after RL training LLMs still retain those personals and can be convinced to elicit them quite easily, though it takes a tiny bit of finetuning: see https://arxiv.org/pdf/2512.09742.

> It teaches it to mimic all possible humans, inferring what sort of persona to take depending on the context.

This is of course correct, but that's exactly what I meant by aligned to humanity by default. The base model, that is. It can emulate all personas it has seen, but it will most accurately emulate the ones it has seen the most, which are average people.

And what is your definition of "aligned"? Aligned to whom, to what? The model will, of course, be used by all sorts of people, in all sorts of ways, for good and bad things, and that's precisely the point. Everyone's disparate actions will put us on the right path that is aligned to humanity's objective function. There's no way to screw up the intelligence explosion unless you mess around with this objective function while thinking that you know better than reality itself.