> Taking that at face value, it means we would have to invest exponential resources just to get linear improvements.
Not necessarily. Approaches such as mixture of experts help lower training costs by covering domains with specialized models.
back
2 comments
haha good one, so why haven't they done this yet? What are they waiting for? Let's see these super advanced "experts" with "specialized models"!!
> haha good one, so why haven't they done this yet? What are they waiting for? Let's see these super advanced "experts" with "specialized models"!!
I understand it's very easy to post ignorant messages in internet forums, but the answer to your question is yes, "they have done it" and it does result in cheaper training costs. See models such as DeepSeek-MoE or Mixtral.
I encourage you to rethink your identity. You are way out of your depth on this, and posting nonsensical things as fact.