numeri

This is research, trying to understand the fundamentals of how these models work. They weren't actually trying to find out where Bill Bradley went to university.

They try this in the appendix without success, unfortunately. It seems having this enabled early on in training is important.

Claude Opus, by a lot. It is especially good with the few low-resource languages that I or people I know could test, including several German/Swiss German dialects and Azerbaijani!

That has nothing to do with the idea of ensembling multiple specialized/single-purpose models. Mixture of Experts is an method of splitting the feed-forwards in a model such that only a (hopefull…

Swahili and Indonesian are primarily second languages, used as lingua francas amongst large and diverse populations, so linguistic changes are mainly driven by non-native speakers, as opposed to the…

I believe what the grandparent comment meant was that you can't run a server that participates in the public network, not that you can't run a private server. That was my prior understanding…

I don't know about strictly superior. It's certainly strictly easier for people with a budget, who just need "good enough" results the first try. I don't have any evidence wha…

The linked leaderboard is actually very trustworthy, in that it consists not of scores on a test dataset, but of ELO ratings generated by actual humans' ratings of the models' responses. You…

Almost all LLM inference these days includes a repetition penalty, to help prevent models from falling into this pattern, the so-called "boredom trap" [1]. Once the model repeats itself once…

It's currently a major area of research and an unsolved problem to find out what individual weights do – and the most recent research seems to suggest that there is not a one-to-one relationship …

You're not necessarily wrong, but I'd imagine this is almost prohibitively slow. Also, this model seems to use two experts per token.

I spent far too long trying to figure that out as well. It's a much catchier name, for sure, but sort of silly that it has so many forks itself.

I would have to go back and reread the paper to be sure, but FF layers are applied position-wise, meaning independently and in parallel on all input tokens/positions. Because of that, I could ima…

I believe you're underestimating how key RLHF seems to be to getting a functioning chatbot with human-like behaviors.

I bought it – or rather, asked for it as a gift from my wife – and it was well worth it. It brings me just a tiny bit of joy every time I see one of the beautiful ligatures in my terminal, and I was a…

This post isn't a scientific investigation. It's someone playing around with a black-box model with few to little controls – which is unfortunately the only thing we can do when experimentin…

FlashAttention is an amazing improvement over the previous state of the art. The others are still highly experimental, but seem like they'll at least contribute significant knowledge to whatever …

How do your views on rural independence fit with the fact that most rural areas in the United States require massive amounts of monetary support from the more population dense (and progressive) states…

One of the most significant quantization papers of the last year [1] found precisely that these outliers only start occuring with LLMs at 6.7B parameters and above. One of the most important keys to t…

In the Qualcomm paper cited, they explain/hypothesize that Transformers learn to attend to these low-meaning tokens when they want to avoid adding too much extra info to the residual stream. So…

In addition to the evaluation issues, it looks like several of their test sets have significant overlap with the test sets [1]. Especially for a compression-based technique, having exact duplicates is…

Neither of those have anything to do with what I quoted. Parts I have issue with are the idea that the best structure for analyzing language are messages, which contain intentions, which can be recurs…

This reeks of the classic "physicist/mathematician/computer scientist encounters new field and is convinced she/he has solved it", this time applied to linguistics and the dee…

Long term, it's the teenagers who are really in play. The events that impact each generation as they transition into adulthood will impact their political views for the rest of their lives.

Mostly because society (at least in the US) does not yet have the protections in place to stop corporations and government agency from abusing automated surveillance. Who has permission to perform mas…

This is reminiscent of the Hegelian dialectic. I'll probably misrepresent it, but as I understand it, it's the idea that society evolves slowly but surely through a constant three step proce…

I've opted out the last three times while traveling between Europe and the United States, but each time the worker has given me grief about it. One badgered me repeatedly about why I wanted to op…

I'd say for research purposes. HackerNews seems to tend to mostly represent the LLM consumer viewpoint, but these waves of models being released are honestly more interesting from a research than…

The closest English equivalent I can think of is the first sound in the word "huge" if it's pronounced with a lot of exaggeration. Even that's not a perfect match.

This technique can be added on to any encoder–decoder Transformer model post-training, so the added training difficulties you mention don't apply. It honestly is a very interesting approach to me…