Great writeup. Writing your own search index and crawler from scratch is a big undertaking, but sounds like the sort of thing you might have to do due to the constraints of Lambda. For the search index, the blog you link to (https://www.morling.dev/blog/how-i-built-a-serverless-search...) does use Apache Lucene within Lambda (compiled into a native binary via Quarkus and GraalVM to make startup time viable, although not a distributed index and doesn't need to be because of its relatively small size), and for the crawler, it sounds like you were nearly there with Colly (except for memory issues).

On the non-tech side:

>"In February 2021 the Australian Greens Party called for a publicly owned search engine to be created and be independent and accountable like the ABC."

Which is an interesting idea, given search has become something of a utility.