Item — HN Reader

> the premise is that because of these few existing dsls (like PlantUML mentioned) my "new dsl" will be equally effective. PlantUML has millions of examples in the training data, my new dsls are not (*specially if its not json/yaml* or just function chain based)

I can confirm that having a DSL that is json/yaml helps a ton. Kind of like static type checking, it eliminates entire swaths of syntactical errors, allowing the LLM to focus on the semantics.

> because if then you are going to ask the llm to "compile the dsl to implementation" we are back to square 1.

I think this is an edge case; 99% of the time you (and/or the LLM) would have access to the implementor so it wouldn't need to do this.

4 comments

> it eliminates entire swaths of syntactical errors, allowing the LLM to focus on the semantics

As I understood it, the idea the article presents is basically the opposite: The LLM is completely clueless about the semantics, but by constraining the language so that semantic errors are syntax errors, we can catch the LLM's bullshit with a simple validation step and make it try again until it produces valid syntax, which limits how wrong it can get the semantics.

It does apply to JSON in a way: An LLM doesn't always produce valid JSON, but it's easy to do a syntax check with a JSON parser and make the LLM try again until the JSON is valid. The JSON might still be semantically wrong on the level of the application domain (have the wrong values), but at least you can be sure that the generated data consists of lists, dictionaries, strings and numbers.

I thought they got like a logit mask that removes illegal next tokens, at least in llama.cpp and GBNF.

You're probably right about that. It looks like llama.cpp lets you specify a grammar file or a JSON schema - is that what you mean? If the syntax of your DSL can be expressed that way, I suppose you wouldn't have to validate it externally and retry.

the grammar support/json-schema applies to a whole response, but I was under the impression a similar logit guide rail can be applied when a tool call is opened.

> DSL that is json/yaml helps a ton

it definitely does, and i would say json/yaml is not a dsl. this example of json/yaml keeps coming in the form of "DSL". i would say your configuration is not a dsl, it a declaration. llms are better at declarative stuff ? maybe but there are hardly that many of complex declarative frameworks.

PlantUML is a real dsl. not just declarative yaml.

I would say any JSON or YAML that carries semantics with the syntax is a language. It may be a configuration language rather than a programming language, but it’s a language rather than just generic data for some program to slurp.

> any JSON or YAML that carries semantics with the syntax is a language

semantics are defined by the converter/compiler/interpreter, and that is the process which is going to consume the said json/yaml. if the json/yaml is going to be consumed by any process then the semantics are inherently defined, so by your definition all jsons/yamls are in their own a "language" (or they are not being consumed at all), which just defeats the purpose of calling it a language at that point.

At that point you can say C has no semantics, because the compiler defines them.

yeah and so llm cannot write C reliably

> and i would say json/yaml is not a dsl

But you can have DSLs that are json/yaml, is my point.

> PlantUML is a real dsl.

PlantUML is a DSL that isn't json/yaml. That doesn't make it better, and you can make the argument that it is worse because the tooling around it won't be as good.

> you can have DSLs that are json/yaml, is my point

i disagree on the semantics of "DSL" vs "config in json". these are different.

and anyways both of them just pretend to be "reliable" by throwing the responsibility to an upper layer of validator/compiler/interpreter

> it is worse because the tooling around it won't be as good

plantuml is good because of the the tooling around it. not sure if we are agreeing/disagreeing there, confused by the wording

That's not a domain-specific language. You can use JSON and YAML in other domains. A DSL is defined by its syntax.

the best language by a mile for dsl-as-data-structure is kdl2

it reads like code and you can easily define a schema too

well that's what i am using

https://kdl.dev/

and https://crates.io/crates/knus/ to define the schema

actually more than that. it would be awesome to have a programming language whose code is also valid as kdl 2. like lisp code being composed of s-expressions

Have you looked into YAMLScript?