Well, you still need to decide if structured output is the right choice.
As they point out - this might impact results where deep reasoning is required.
So you might be better off taking the unconstrained approach with feedback.
back
1 comments
The only "solution" with the unconstrained approach is to ask the LLM to regenerate the JSON. This is definitely more expensive than whatever downside from requesting structured outputs from the API.
ESPECIALLY with situations where deep reasoning is required, since those are likely to correlate with longer JSON outputs and therefore more failure points.
Definitely. A lot of what is missing in many discussions is the absolutely essential need to have evals.
The only way to “know” what is the best (or better) approach is to have a significant number of test cases that you can measure performance against.
At the moment, for a lot of people, state of the art is “let’s try a different prompt and see if the answer on my one example is better”