▲ 281 points
back
78 comments
Ohi, I'm the original creator of Searx, but due to the limitations of the metasearch concept I'm not involved in the development anymore. My new search project is https://github.com/asciimoo/hister (https://hister.org/).
Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. Storing full page content allows serving offline result previews and the full page content via MCP.
Take a look at how the MCP can be utilized: https://hister.org/posts/give-your-ai-assistant-a-private-me...
Beautiful! Thank you for making this.
I've been trying to find something to use for enriching my own self-hosted LLMs and agentic tools with information I find useful. Metasearch tools like SearXNG make it less likely you'll get blocked by bot detection tools when finding information, but usually it's something I've already found, read, or seen that I want to incorporate into my tooling.
I came to the conclusion that a self-hosted content storage system with a search engine and a browser extension that can extract and save web page content and metadata was the ideal setup for me. Preferably with some sort of federated content sharing ability and the ability to import creative commons content like Wikipedia and Gutenberg.
This looks almost exactly like what I wanted.
It'll be a few weeks before I have time to audit the code and deploy it, but I'm really looking forward to trying it out.
I'm sorry for not taking the time to read the docs, but I have a question.
Some 20 years ago a friend of mine has set up a local proxy (python if I'm not mistaken) that was gathering all his web traffic and served him as a long term memory. The proxy had a web interface and allowed him to quickly find something he saw ca. 10 days ago, or that specific algorithm he recalls but can't remember it's name.
For years I've been collecting links to different work related trivia which I use on a daily basis as a rabbit-from-a-hat solution to answer random question from friends and coworkers. For example someone randomly asked me for an idea for color palette for data charts and I can immediately give them a scientific research into the color palette. Or an obscure algorithm.
But with time the collection has grown substantially and it's really cumbersome to find the proper things.
Would your project be a good fit for my problem?
Hister sounds like something I wanted for a while, but never got around to building. Searching stuff I’ve seen before is most of what I do with a search engine, so having it local and fast would be amazing. Eager to give it a try.
Looks good, but was also curious about the "limitations of the metasearch concept," could you explain this more?
I was considering paying someone to build something like this at some point. With two jobs, I eventually had no time to even organize what I find. It's just piles of links in text files.
Can I give your software a huge list of URL's to index? Or do I need to use browser automation to open them a few at a time with it caching and indexing them?
I also use my own, lets say search engine.
I download only title, description, thumbnail, common og fields.
My index is very lean, I think. I have 2m of pages crawled.
https://github.com/rumca-js/Internet-Places-Database
It has tags, and votes support.
Recently I also launched my first fdroid app
https://github.com/rumca-js/OfflineWebSearch
https://f-droid.org/en/packages/io.github.rumcajs.offlineweb...
YaCY has a proxy mode that automatically index your web-serving. In my experience, the index grow in size very fast and reaches ~100GB or more. How does the index size of Hister compare to that?
Interesting, a local search option. I made the recoll engine for SearX and now SearXNG and still use this daily over a rather large archive of journal articles and other non-fiction texts. Recoll's indexer can extract text from just about anything I throw at it, it also extracts and indexes metadata. Would Hister serve the same purpose and if so is there a SearXNG engine to integrate it into the result stream?
Always excited to see new things like Hister in the search space. What are the scaling limits, as far as you can tell in terms of how much can it hold before queries start breaking down or become too slow to be useful? Could it evolve into a general internet search engine if, say, enough trusted members of a geo-distributed YugabyteDB cluster and an army of crawlers built a sufficient index?
Wow! that looks like a bit of software I have been dreaming about for awhile - will definately check out! You're at least doing something right in communicating the reasons why and appeal for starters! All the best!
I love your idea and wondered why saving and indexing browser visited pages was not being done. Does this handle large amounts of local files, for example 10-20TB across file types like Powerpoint, Excel, Word, and PDF?
Hister is a great idea and the creator is a really nice person, please give it an honest look and consider supporting them (I'm Uruky's co-founder and we sponsored them)!
This is great, like many others I've been thinking of something like hister but only for bookmarked web pages. I presume it should be straightforward with hister to do that?
All the best!
Oh thank god there used to be several tools like this and they slowly went away. I’ve been wanting this to return.
this is really cool, first time hearing about this, is there any org level model for this so you can promote individual's indexed websites into an organization/team owned model?
Is this similar to fastcrw ?
I installed this a while back and honestly I almost never touch it. It turns out that for me searching my history doesn't really replace a search engine at all. The built in extractor list is pretty limited and adding them seems like too much of an ordeal for me to bother.
This appears to be a key tool for providing search to local models.
I'm curious what setups folks use to provide this functionality.
Since the quantized 24B parameter Gemma model came out, I've had good luck with tool calling on a 4070 Ti Super.
Successful tool calling is what finally made the local experience useful.
I should note this is for the general and not coding specific context.
It has a JSON mode that you need to enable in settings and then you can create a simple python script to interact with it or have the agent use `curl` and `jq` to interact with it.
It's at the bottom of this page: https://docs.searxng.org/admin/settings/settings_search.html
SearXNG is my daily internet search now +5 years; with YaCY Backends and else as fallback. I also build internal document search or RAG applications with this setup (SearXNG also support json results).
However, there are some downer I accept because of privacy:
1. Its slower and the results are not that good then with others. But fast and good enough for most of my queries.
2. From time to time you get blocked on the duckduckgo, brave or whatever search and you must solve some captures. You can prevent this by getting and using API-Keys from them.
The nice thing about using your own backend is, that you can prio it in the results and for example, if I crawl the smallweb and other site important for myself, this sites come up first in the results.
I've been self hosting this as my default engine across all of my searches for a few years now. I can't recommend it more highly.
I've been using searxng for several years now. I don't run my own instances because the inhumane network censorship imposed by GFW, and proxy detection enforced by search engines. Instead, I rely on public instances on the list [1] and libredirect [2]. Note that service from a single instance is not guaranteed, but you can always switch to other available instances with little cost within a minute.
I won't say searxng can help you degoogle because metasearch engine calls other search engines, e.g., google, to collect results. However, if you try searxng, you can at least get rid of things like ai reviews in no time.
In the end, thank you people after searxng project and public instances.
TinySearch wraps this and works well for agents. It's better than the native SearXNG MCP because it optimizes the context before it even gets to the agent so as to not waste tokens.
It works well if you connect it the Brave Search API, but using it a scraper is fairly unreliable. Google stopped working a few days ago.
I’ve been a big fan of SearXNG for a while now. My distain for google has only grown, so having the ability to search and avoid things like yk, small AI models being installed on my PC without my consent, is awesome.
I've built https://github.com/denysvitali/searxng-mcp to use this as an MCP for coding agents. Works very well, until you get rate limited by the providers (e.g: DDG).
It also needs a SearXNG server to run, so I recently pivoted towards a self-contained solution: https://github.com/denysvitali/search-mcp
I've been using SearXNG for a few years now, however I've been trying out Degoog as a SearXNG alternative since I've had issues with engines constantly failing or being slow since day 1 of using SearXNG, but Degoog has worse results with the same engines. It's a shame since I'm having to pick between slower but better results, or very fast but worse results.
I’ve always liked this tool, but I’m of two minds regarding the privacy gained by sending my searches to 280 companies instead of just one.
I've been using this for some projects. It's exceptional and I recommend it highly.
I actually included a recipe to deploy it to kubernetes in typekro, my TypeScript infrastructure-as-code project for kubernetes: https://typekro.run/api/searxng/
Years of regular use here, has been great even before I started using it as an agent tool.
Highly recommend this. I set up a self-hosted instance and been using it exclusively for months. It's better than DDG and i don't miss Google whatsoever.
Image search is worse i guess mostly for lack of CDN so it's slow but whatever.
Yeah, I find that searx results are way more relevant to what I’m actually looking for than a single engine. There’s so much manipulation going on that if you don’t aggregate multiple engines, it’s near impossible to get what you want.
I have used SearXNG hosts like https://searx.be/ but stick with Brave search for the most part. Are there other good hosts people tend to use?
Have used it in my homelab for the past 2 years, can recommend it. Easy to run in docker and helps both get better search results and keeps your search history local and in your control.
I've got SearXNG set up but lately I've been using uruky instead.
For those not familiar think EU-ish Kagi...but crucially their plans comes with very legit API limits too. Their recent show HN: https://news.ycombinator.com/item?id=48396004
Not on the same wavelength as searxng stallman wise but I prefer it for AI use. Faster, better quality (subjective) and much less tokens per search - 3.5k vs 23k. Searxng returns a bunch of extra crap that just pollutes AI context. e.g. Link to the thumbnail of the page. I guess you can build a proxy to clean it up but meh.
[not affiliated]
Been a fan of searX for a while. Not sure if this is the same thing but there were plenty of hosted versions too.
Friendly reminder that if your user and traffic count is low, your traffic is still unique and able to be profiled. Love and use this project, though.
This is really great!
I prefer 4get.
searxng is fantastic for AI agents
how do i configure which specific search engines SearXNG pulls its results from?
Can we extend it to onyl search Stack Overflow and GitHub