I wish Google had spent their efforts building stuff like this into Google Sheets instead of launching a Gemini tab that follows you everywhere and says "sorry I can't help with that" if you try to use it
back
2 comments
When you look into the data structure and organization that is underneath Google Sheets, and if you have any experience with LLMs and structured outputs, Google shit in their sheets as far as having an overly complex internal representation for a spreadsheet. Their data representation needs to be translated into something far easier to work with for machine learning models to work with that data, without a massive amount of filtering just to reach the spreadsheet data itself.
> When you look into the data structure and organization that is underneath Google Sheets
I don't use Google Sheets very often, so maybe this is obvious to someone that does. But, how do you look at the data structure and organization underneath Google Sheets? Are you an employee or is this available to the general public? Through an api?
Look at their API and the data structures it uses, the relationships between them. It's more complex than necessary, ridiculously so, it feels like a student's learning project they abandoned out of boredom. API consistency, what's that?
Or we can make models that are used to accepting spreadsheet data as input. It’s probably just a different kinds of positional embedding even.
Believe it or not, we already do, it is just not obvious and so far, I've only exchanged info with very few that realize this, and how to access that capability.
I think if you trained a MLM to replace certain tokens in a CSV with formulas, you can accomplish that without worrying about the internal representation of a spreadsheet.
My point is that no training is necessary, none at all. The major LLMs already know the internal data structures and APIs of open source spreadsheets, because that information is in their training data, all open source software has their source code in major LLM training data. The trick is accessing that knowledge.
Is it a prompt trick?
Not so much of a "trick" as it is understanding how to create a context the retrieves/generates from the more/most accurate training data.
In the case of open source software, a specific open source app or library or API, often all it takes is to reference a library or API by name, use a few terms specific to that library or service, and you'll get an LLM context that retrieves/generates from that area of knowledge. That context knows information about the library/API that is useful for integration and will tell you all about them just by explaining what you're trying to do. Note this is essentially conversational R&D, which is then backed up by verifying the data structures, library calls and/or APIs are indeed as they are being discussed. I include verifying checks during the conversation, rather than build a strategy and then check if what was discussed is even possible.
If an integration is possible, if the above conversation yields useful data and function/API calls (which they tend to do), then it's just a matter of a well structured prompt that's probably generating structured output. The end result is a process where a useful library/API call retrieves essential data of some sort that the user can ask questions about, modify, and if modified the prompt generates the data that is then fed back into the library/API.
That would have been the future had OpenAI and the other company not ventured to offer AI search power in a chat window. small incremental AI implementations would have been the norm. Like auto reply in email. Or email summary at end of day.