Launch HN: Human Layer (YC F24) – Human-in-the-Loop API for AI Systems

Startup owner using AI with this need - needless to say, a real problem. I've considered DIYing an internal service for this - even if we went with you we'd probably have an intern do a quick and dirty copy, which I rarely advocate for if I can offload to SAAS. I'm sure you've put a fair bit of work into this that goes well beyond the human interaction loop, but that's really all we need. Your entry price is steep (I'm afraid to ask what an enterprise use-case looks like) and this isn't complicated to make. We don't need to productize or have all the bells and whistles - just human interaction occasionally. Any amount of competition would wipe out your pricing, so no I would not want to pay for this.

I assume your reasoning is something like: if people are already paying out the nose for open AI calls, an extra ten cents to make a human in the loop check probably isn't bad, and realistically speaking ten cents isn't much when compared to a valuable person's time, and I guess the number of calls to your service is likely expected to be fairly low (since they by definition require human intervention) so you need a high per operation cost to make anything.

Even understanding that, the per operation cost seems astronomical and I imagine you'll have a hard time getting people past that knee jerk reaction. Maybe you could do something like offer a large initial number of credits (like a couple hundred), offer some small numbers of free credits per month (like.... ten?) and then have some tier in between free and premium with lower per operation pricing?

It also seems painful that the per operation average of the premium plan is greater than the free offering (when using 2000 ops). Imo you'd probably be better off making it lower than the free offering from 200 ops and up, to give people an incentive to switch. I imagine people on your premium plan using premium features would be more likely to continue to do so, for one. The simplest way to do this would be to bump up the included ops up to 5k I guess. Someone using less than 5k would still have a higher average price, but it seems like it would come off better.

Interesting tool, congrats on the launch!

I was wondering: have you thought about automation bias or automation complacency [0]? Sticking with the drop-tables example: if you have an agent that works quite well, the human in the loop will nearly always approve the task. The human will then learn over time that the agent "can be trusted", and will stop reviewing the pings carefully. Hitting the "approve" button will become somewhat automated by the human, and the risky tasks won't be caught by the human anymore.

[0]: https://en.wikipedia.org/wiki/Automation_bias

P.S. nobody asked but since you made it this far - the next big problem in this space is fast becoming, what else do we need to be able to build these "headless" or "outer loop" AI agents? Most frameworks do a bad job of handling any tool call that would be asynchronous or long running (imagine an agent calling a tool and having to hang for hours or days while waiting for a response from a human). Rewiring existing frameworks to support this is either hard or impossible, because you have to

1. fire the async request, 2. store the current context window somewhere, 3. catch a webhook, 4. map it back to the original agent/context, 5. append the webhook response to the context window, 6. resume execution with the updated context window.

I have some ideas but I'll save that one for another post :) Thanks again for reading!

We must do whatever we can to stay above the API:

https://www.johnmacgaffey.com/blog/below-the-api/

10c per slack API call. I could make a mobile phone call for less than that in 1995. It is expensive...

IFTTT, Zapier, NodeRed, etc. are your competitors.

E.g.

https://ifttt.com/applets/J75VtBA9-get-an-email-when-a-webho... -> https://ifttt.com/applets/KWqQedih-make-a-web-request-when-i.... They have lots of AI things too.

The problem is you are saying "API" call so you are already dealing with devs. They can save $10k by writing their own Slack integration (even easier if they pay IFTTT $150/y), and the enterprises will want you to be all FedRAMP, ISO, SOC, Data Resident etc.

I'm considering this for a workflow agent and would be keen to hear thoughts on this process.

We're a medical device company, so we need to do ISO13485 quality assurance processes on changes to software and hardware.

I had already been thinking of using an LLM to help ensure we are surfacing all potential concerns and ensure they are addressed. Partly relying on the LLM, but really as a method to manage the workflow and confirm that our processes are being followed.

Any thoughts on if this might be a good solution? Or other suggestions by other HN users.

Isn't this precisely how AI started? It was a bunch of humans under the hood doing the logic when the companies said it was AI. Then we removed the humans and the quality took a hit. To fix that hit, 3rd party companies are putting humans back in the loop? Isn't that kind of like putting a band-aid on the spot where your arm was just blown off?

My favorite part of all this is that it’s inevitable. Someone has to solve agent adoption in whatever-the-environment-already-is. And nobody is doing this well at scale. Europe is mandating this. And even though Article 14 of the AI Act won’t be enforced until 2026, I’m glad projects like this are working ahead. Get after it, Dex!

Required reading for everyone considering a human-in-the-loop system:

https://pluralistic.net/2024/10/30/a-neck-in-a-noose/#is-als...

I wish OP the best of luck with their product. I still think that the points Doctorow has made are important to know and consider beforehand.

This is exciting. I am an architect in a startup that has long valued bringing humans in the loop for the moments when only humans can do the work. The key thing missing between the potential seen in the last couple years of LLM-based fervor and realizing actual value for us has been the notion of control and oversight. So instead, we have built workflows and manual processes in a custom way throughout the business. Happy to discuss privately sometime! (email in profile)

Congrats on the launch! I'll be thinking about this for a while to be sure.

P.S., there is a minor typo on the URL in your BIO.

Definitely a problem that everyone needs to solve.

I wonder if you can achieve this workflow by just using prompt and the new Model Context Protocol connected to email / slack.

https://www.anthropic.com/news/model-context-protocol

You’re close. It’s not the humans in the loop in standard tasks you need though, it’s human surrogates for AI agents to do jobs they can’t for a variety of reasons (like missing a body or requiring an internet connection).

I have a request for startups for this: “GraggNet: Task Rabbit for AIs

Surrogate humans for AIs to use before robotics are human level”

https://ageof.diamonds/rfs

This is the first new YC launch I've seen involving AI that I am extremely positive about. I have worked with systems implementing similar functionality ad-hoc already, but seeing it as a buy-in service - and one so easy to integrate - is really cool.

From what I've seen, this will bring the implementation needs for this kind of functionality down from "engineering team" to a single programmer.

Congratulations on launch! We’ve faced this problem with our autonomous web browsing agent https://www.donobu.com and ended up implementing a css overlay to wait for user input in certain cases. Slack would be so much better. Excited to try humanlayer out.

Nice. I guess the issue is that this is such a basic i/o feature that any system with some modicum of customization can already do it.

It's like offering a service that provides storage by api for agents. Yeah, you can call the api, or call the s3 api directly or store to disk.

That said, I would try it before rolling my own.

Congrats on the launch, this is an interesting concept. It's somewhat akin to developers approving LLM generated code changes and pull requests. I feel much more comfortable with senior developers approving AI changes to our codebase, then letting loose an autonomous agent with no human oversight.

Congrats on the launch! Human in the loop is an underserved market for AI toolchains. I've usually had to build custom tools for this which is a PITA.

Make.com has a human in the loop feature in closed beta. https://www.make.com/en/help/app/human-in-the-loop

There's also https://www.gotohuman.com/ that uses review forms.

Looking forward to playing with HumanLayer. The slack integration looks a lot more useful for my workflows than other tools I've tried.

In the demo video and example, you show faked LinkIn messages integration. Do you have any recommendations for tools that can actually integrate with live LinkedIn messages?

I think Human Layer is a great idea. Recently, my baby turned one year old, which made me reflect on many issues. We train AI with a lot of data but overlook the impact that decades of seemingly useless data from human growth experiences have on our brain development. As a result, humans still have an incomparable advantage over LLMs in terms of the so-called "big picture." For example, a recent experience I had was when I asked Claude 3.5-sonnet to write a bash script; it inadvertently modified the PATH variable, costing me a lot of time to fix it. Such attention to detail in work is difficult to avoid through vector db recall or manual context completion. But I believe that a true bash expert would not make such mistakes.

This is a great idea- I hope that you are wildly successful.

I’m an AI skeptic mostly because I see people rushing to connect unreasoning LLMs to real things and as a result cause lots of problems for humans.

I love the idea of human-in-the-loop-as-a-service because at least it provides some sort of safety net for many cases.

Good luck!

There is definitely a need for this.

What I don't understand from quickly skimming your description and homepage: Do you source/provide the humans in the loop? That's a good value add, but how do I automatically / manually vet how you do the routing?

Congrats Dex! Excited to see what people build with this + tools like Stripe's new agent payments SDK (issuing a payment seems like a great place to ask permission).

congrats on the launch dex! this is a problem that i've already seen come up a dozen times and many companies are building it internally in a variety of different ways. easier to buy vs. build for something like this imo, glad its being built!

I knew this was coming, so kudos to you all for getting out of the gate!

I've implemented this in our workflows, albeit a bit more naive: when we kick off new processes the user is given the option to "put a human in the loop" -- at which point processing halts and a user/group is paged to review the content in flight, along with all the chains/calls.

The human can tweak the text if needed and the process continues.

Sounds great. Perhaps an interesting aspect: I haven't discovered the right words for it but: If it is your job, answer the fucking question. This layered approach might prove more of a gain than imagined. They might not show it but some people are terrified to ask even the first question. Others think it perfectly fine to ask 100 questions they already know the answer to.

So this is an automated foreman for the customer's own employees, like a call center controller? Or does HumanLayer provide the human labor, like Mechanical Turk?

The API contains a "human_as_tool" function. That's so like Marshall Brain's "Manna".

"Machines should think. People should work." Less of a joke every day.

It's generally recommended to add a meat-gap interface between AI systems to reduce unexpected results.

Meat-gap. We have your back.

How does it compare with the built-in human-in-the-loop feature from langgraph? Or CrewAI allows humaninput as well right?

The idea is great and necessary. It doesn't seem super hard to replicate but why would anyone build their own solution if something already exists and works fine.

The thing that got me thinking... how do you make sure an LLM won't eventually hallucinate approval -- or outright lie about it, to get going?

Anyway, congrats, this sounds really cool.

The biggest problem is that a lot of times "approval" requires domain knowledge / specific training so handing it off to some random dude will result in lots of errors, no better than just having some AI model decide.

Not to mention anything that requires a quicker response rate.

Is that possible to connect it to an existing website chat widget apps like tawk?

Also, caught a few typos on the website: https://triplechecker.com/s/992809/humanlayer.dev

Congrats! Looking forward to getting HumanLayer integrated into our stuff

Congrats on the launch. Seems quite interesting!

My only queation/feedback is: What policies do you have in place to prevent bad outsourcing or exploitative behavior like what has been done by big-tech companies that turn a blind eye to what happens in, for example, Kenya [0] for the training of AI models?

[0]: https://youtu.be/qZS50KXjAX0

Just an idea: having a little widget in the MacOS menu bar that pops up or sends you a notification to solve a human task wouldn't be so terrible either.

So many uses for this. Excited to see how it develops.

Loving you guys have Typescript support from day one!

I feel like HumanLayer is a great idea, but decision fatigue and bystander effects could pose challenges. If people are overloaded with approvals or don't feel ownership over what they're verifying, the quality of oversight might drop. + also even if approved, you still have to make sure the agents doesn't hallucinate at the execution phase.

This seems generic enough that it could almost be applied to any use case. Have you considered catpcha as a use case?

I think at some point, the term API should be replaced with another acronym to emphasize humans as the focal point.

Neat, this could be a step forward from using something like n8n to manage processes, input and reviews.