Item — HN Reader

I wonder if telling (or somehow architecturally coaxing) the LLM it has 'skin in the game' will make it more risk-averse? I imagine it does.

This makes me wonder too about the entire premise and worthiness of these evals. They orient themselves around normal one-shot interactions with a likely non-sys-prompted model with no built up context or memory of the person. I doubt the mentioned 'job loss' scenario is even contextually seen as a 'loss'; it is only a circumstance descriptor, a single snapshot without a history. Maybe to get the best advice we actually need to tell the LLM our entire story, not just a narrow request for a question; a question that - itself - is biased to our own imaginings of what problem we perceive ourselves as having, which humans are often bad at.

3 comments

The problem I see with this approach is threefold.

First, from a technical standpoint the required context window would be massive if you're looking at a person's career/life holistically. Probably solvable, but definitely something to be aware of.

Second, privacy goes completely out the window since you're sharing everything. You don't know what's relevant and what's not up front so you need to provide everything.

Third, you would need a training dataset of all those input variables and their outcomes to be able to provide any sort of useful output. The first set of people to share everything wouldn't be able to derive any value from the tool, and I think you'd be hard pressed to convince enough people to do it to get a useful dataset.

This is correct for out of the box AIs. This is why I built Roundtable, our domain aware persistence layer for AI context management. It's what powers Pendragon. Our protections aren't just "trust us", we show you how your data is controlled architecturally and secure.

You don't necessarily need to provide everything. Arthur (our AI) is smart enough to see exactly which information it needs to answer a given question. but, yes, the more information you provide the easier of a time the AI will have in answering your question. Arthur doesn't guess. if there is crucial information it needs he will ask for it. it doesn't have to be a Plaid hook up, a csv or even a simple user response is a start.

On your third point — you'd need an outcomes dataset — that's true for traditional ML, but it's not how this works. The normative layer is finance itself (life-cycle theory, tax rules, amortization) implemented as deterministic calculators, with the LLM doing explanation and elicitation. The paper under discussion is sort of the proof: the models already give theory-aligned advice with zero outcome training. The gap it found is input quality and statelessness, not a missing training set.

> First, from a technical standpoint the required context window would be massive if you're looking at a person's career/life holistically. Probably solvable, but definitely something to be aware of.

Why would it be massive? The application layer typically compacts a profile of information about the users financial situation when offered. I doubt many of us have financial situations that would exceed the context window.

> Third, you would need a training dataset of all those input variables and their outcomes to be able to provide any sort of useful output. The first set of people to share everything wouldn't be able to derive any value from the tool, and I think you'd be hard pressed to convince enough people to do it to get a useful dataset.

Would you 'need' a training dataset of input variables and their outcomes for an LLM? Certainly for traditional ML, but the LLM toolcalling can simulate what an astute user should statistically do in their situation based on information on the internet and reason about the different constraints.

You wouldn't need history. You can probably build a flowchart choose-your-own-adventure that gives good financial advice, and then have an LLM hold the user's hand through that flowchart.

Like, do you have $1,000 in an emergency fund? No? Start there.

Our AI, Pendragon (https://pendragon.foxtrotcommunications.net/) doesn't stop at one and done interactions. It actively works to get to know you and your household's finances so that it can build an effective model. It also produces confidence scores based upon what it knows (and what it doesn't) to always give you a complete picture.

The more information you give pendragon the better your answers will be. Pendragon also produces a history of decisions, memories, and plans so it can keep you on track and have a better understanding of your overall financial health. Pendragon helps you achieve your goals by providing detailed financial advise and specific actions you can take to achieve whichever goal you have.

I look forward to more models like this but don't take this the wrong way, I'm not touching this with a 10 foot pole unless it's running locally on my private hardware.

I see. Do you not trust the cloud, the data storage layer, or is it the AI model itself?

Specifically, yes to all of the above but only with things such as this (financial data) since it's basically a commodity. Entire markets exist around personal data like this for credit and advertising.

Who are you selling the conversation logs and customer info to? If you’re not selling customer financial data (yet), how much are companies offering you for it?

I’d rather get no response than be patronized, so no linking to corporate policy documents on your website please.

we don't sell customer information on an individual level, and I can't disclose the details of any potential deals. The most we would consider would be aggregated and anonymized decision data, such as what families value the most, but we would always treat correct PII handling as an absolute requirement. We aren't selling any information right now.

I know you said that you don't want a TOS link, here's something better. This is our constitution. https://pendragon.foxtrotcommunications.net/constitution

Thanks for the honest response, I appreciate it! It’s a great idea for an AI tool and as long as you aren’t selling directly identifiable data, I don’t see a problem with using it.

I worry about my data being sold if I used a tool like this, but I should probably be more worried about my actual credit card purchase data being sold (because it is).