Item — HN Reader

This reeks of the classic "physicist/mathematician/computer scientist encounters new field and is convinced she/he has solved it", this time applied to linguistics and the deeper meaning and structure of language:

> Languages are not created randomly, but with a specific purpose in mind, which is to convey information. Languages are composed of distinct, relatively independent units, such as sentences in natural languages or statements in programming languages. These separate pieces of language are referred to as "messages" in this paper. Each message, represented as x, is in turn composed of a sequence of symbols from an alphabet with varying lengths. A message is created with the aim of expressing a single and definite intention, denoted as θ. The set of all possible intentions constitutes another space, denoted as Θ. We assume that the intention space Θ is a countable set of many distinct intentions. Each θ may represent a simple intention, which is an element from a finite set, or a composite intention that is made up of several simpler concepts or components through concatenation or recursion. Here we only require that the intention space Θ is discrete and complete, and each element in Θ is unique.

1 comments

There's a decent amount of research that supports this view [1][2]. How else do you explain the unreasonable effectiveness of transformer-based language models?

1. https://journals.sagepub.com/doi/pdf/10.1177/014272371986973...

2. https://arxiv.org/pdf/2301.06627.pdf

Neither of those have anything to do with what I quoted. Parts I have issue with are the idea that the best structure for analyzing language are messages, which contain intentions, which can be recursive, or that the fundamental space of meaning is discrete and countable. These are all just assumptions made without justifications by someone clearly approaching the field of linguistics with the attitude that it's an easier field than their coming from, and could probably all be easily solved if someone would finally use some proper math/physics/computer science.

I'm not saying those assumptions are necessarily wrong, either, just that this is a slightly arrogant simplification of a field that has already hashed these kinds of questions through and through, and decided that such simplistic views don't yet have enough experimental evidence to be cut and dry truths.

Aside from that – the paper's main contribution is essentially "I have a pet theory that lets me predict everything we've already seen LLMs do, but nothing more." This is accompanied with a simulation, which shows nothing more than that Transformers can learn an unambiguous, 18-letter/6 sentence toy language generated with a Markov chain better than an ambiguous one. This simulation does not even come close to supporting the claims and assumptions in the rest of the paper.

I see what you're saying, but I don't think this kind of interdisciplinary approach is without value. Siloing can prevent new perspectives.