It’s hard to communicate about this stuff. I think people hear ‘tools’ and ‘function calling’ and assume it provides an actual suite of tools or pre-made routines that it calls upon on the Anthropic backend. But nope. It’s just a way of generating a structured schema from a prompt. It’s really crucial work, but just funny how obscured the boring truth is. Also FWIW I experience a tonne more schema adherence if I use XML-like semantic tags rather than JSON. XML is so much more forgiving a format too.
back
7 comments
I find this far more useful than a suite of tools or "AI agents" which always work well in a controlled development environment but not so much further than that.
Function calling is a great step towards actually production-izing LLMs and making them extremely robust - I remember when GPT-3 API first came out and I was furiously making sequential calls with complex if/else and try/catch statements and using a couple of Python libraries for the simple reason...I need the output to be a valid JSON. It was surprisingly hard until function calling solved this.
Agree. Can really build a strong chain of functionality with this function calling. I have a harder time seeing the use of something like Langchain - seems unnecessary to learn a new bloated API when I can use the powerful tools from the models themselves, and then chain things together myself.
Yeh agreed. Function calling FTW— just need a bit more reliability/(semi-?)-idempotence.
It’s much more than just generating structured schema. It also understands user intent and assigns the correct functions to solve a query. So for example if we give it two functions getWeather(city) and getTime(city) and ask “what’s the weather in New York?” It will decide on the correct function to use. It will also know to use both functions if we ask it “what’s the time and weather in New York?”.
Open LLMs can use grammar-based sampling to guarantee syntactically correct JSON is produced, surprised OpenAI never incorporated anything like that.
my concern with grammar based sampling is that it makes the model dumber: after all, you are forcing it to say something else than what it thought would be best.
Looks like it’s quite the opposite: http://blog.dottxt.co/performance-gsm8k.html
Yes, the 'function calling' naming is unfortunate. It's really structured output that can be fed as input into any functionality elsewhere in your code.
The difference between the structured output of json mode is that the model can choose which set of structured output (matched to various function definitions). Subtle, but pretty cool and powerful.
I do wonder if a stack-based format would be easier for an LLM. Seems like a better fit for the attention mechanism. My suspicion (without having lifted a finger to check) is that it's the closing tags that make the difference for XML. Go stack-based and you can drop the opening tags, and save the tokens.
XML and other document markup languages are objectively horrible data storage formats. Why is "forgiving" a desired quality in this case?
While some of the downvotes are justified because you're selling this short, I want to point out that your comment about XML is actually valid to a degree. I've found that using XML for prompts lets you annotate specific keywords/phrases and import structure on the prompt which can produce better results.
Getting results back in XML though? That's a terrible idea, you're asking for parsing errors. YML is the best format for getting structured data from LLMs because if there's a parse error you typically only lose the malformed bits.