back
2 comments
but have you compared whether or not this strategy is net beneficial? for instance, similar reliable output first time?
It is has been shown many times you can use a large model for tip level orchestration or planning, and then have smaller models do exploration and file editing (e.g. in coding). This keeps a bunch of (wasted) tokens out of the big vllm context (e.g. reading a file only to find out it has nothing relevant for the task)