Was curious so we checked and yep, Antithesis finds this bug in about 15 minutes. Will post a repro/writeup here soon.
back
4 comments
Oh wow. I checked your site, but I (still) don't see the ability for average joes (and their pet AI agents) to signup. Could be huge, even if just offered in 5 minute slots. Is your email contact still valid in your profile? Want to reach out (Moose is the name). FYI, got a job at Antithesis a while ago after some rigorous interviewing.
We are working on getting self-serve signups to you ASAP. For now, just email us. We don’t bite.
Discussion here: https://news.ycombinator.com/item?id=49277799
Wait, it's AI? AI probably has information about this bug by now, since it was discovered and fixed in March.
Nope, it's a generic PBT harness. The human who built it knew about the bug, but the workload was specifically designed not to specifically target this particular bug, and may have found several others as well (we are in the process of reporting these upstream).
I'm equal parts intrigued and skeptical- I guess if the prompt doesn't lead on there is a bug there then I'm impressed.
Stay tuned!