Item — HN Reader

Very cool! - Most search problems are solved this way - interesting index data modeling to the needed use cases, not magic machine learning neural network unicorns. Many want to sell you the latter. The thing that gets the job done is usually the former.

One big reasons for this is the need for really good training data to do supervised learning on search problems. If you’re Google or Wikipedia, you can get this data. But many struggle to understand what a good result was for a query based on clicks and other user data due to low volumes of traffic.

4 comments

As an aside: most of the Wikimedia Foundation's analytics data is available for anyone to download (https://wikitech.wikimedia.org/wiki/Analytics#Datasets). There is some link-rot to wade through, but try asking on #wikimedia-analytics on freenode if you can't find something.

Then you're using magic machine learning wrong. I'm part of a small team who uses it exactly for the purpose of giving intelligent search to non-Googles and non-Wikipedias, and we have a very good track record.

The key is to use machine learning to generalize user behavior across multiple items in roughly the same category, rather than individual items. In fact, with tiny amounts of data you should pretty much never deal in individual items -- always in groups of related items.

Undoubtedly I’m doing a lot wrong :)

Yup that’s one method I teach people in Learning to Rank training. In addition to reducing dimensionality on queries, you can also do so by geo and user depending on the domain. But do you want to do this much munging of training data?

Even still the basis for how you group similar information needs isn’t always obvious, and most teams prefer a simpler solution they can understand and manage that’s not ML based. A lot depends on the team that’s ultimately going to maintain the solution.

When you said that you work in groups, the groups are discovered with a clustering technique and then you apply the learning to rank algos ?

In most learning to rank data sets, the only notion of a "query" is a numerical identifier that groups a set of features and a grade. Your goal is to optimize the ordering in that group. That group can really be a user, a query pattern, a location, or a bazillion other groupings likely to share similar rankings.

For example, a common training data storage format looks like

grade(0-4) queryId list of feature values...

Such as:

4 qid:1 1:0.5 2:0.24 # a doc with these features is 'good' for this query

0 qid:1 1:-0.5 2:0.24 # a bad doc for query id 1 is described by these features

Features in an LTR context are some relationship between the query and doc (such as scores computed from Solr/Elasticsearch queries)

Of course turning clicks/session info into grades isn't easy. One way to go about it is with click models (https://pdfs.semanticscholar.org/0b19/b37da5e438e6355418c726...). Another way is to equate conversions in a session with one of the grades (a purchase is a 4).

Needless to say, such usage data is sparse for long tail queries (queries that rarely happen - 'fuscha shoes'). Luckily it's not sparse for head queries (common queries - 'shoes'). So those head queries can provide some benefit, helping to generalize to the rare long tail queries.

You'll see a lot of patterns that are modifiers on head queries. Like 'shoes' is a head query. But 'blue shoes' and 'purple shoes' and other 'color shoes' are tail queries.

You can do a lot with that insight

- Supervised grouping of queries into one 'query id' by watching how users refine queries. For example users type "shoes" then in the same session add one color adjective "red shoes". Using that to group '<color> shoes' queries

- Unsupervised clustering of similar sessions or searches where users seem to expect ranking to perform similarly based on user behavior

The downside/tradeoff is you lose a lot of nuance that you're getting with per-query training data. But it's a technique people use.

It also substitutes one ML problem for another, which may or may not be harder than the original one

It's more of a hierarchy than groups, actually. If a user indicates that a pair of maroon walking shoes were relevant to their query for "red sneakers", then we have learned that "red sneakers" is associated, in order of decreasing strength, with e.g.

- maroon walking shoes - walking shoes - casual shoes - footwear - clothes

And obviously the same thing can be applied to generalize over the query. These hierarchies are constructed statically and dynamically with unsupervised learning, and then associations from query to groups happens dynamically.

Very cool, how exactly do you generate the hierarchies? Is there an existing site taxonomy or categorization you’re using? Or associating query strings with the docs clicked and using refinements to see the hierarchy? Or maybe LtR training data per site category?

They are constructed from a probabilistic similarity measure defined in terms of the metadata available for items, where their path tends to be weighed fairly heavily. Does that answer make sense?

Cool stuff, appreciate the answer. Makes more sense. BTW feel free to join us on relevance slack community, a lot of us we’re curious what you were doing

http://o19s.com/slack

I’d also be curious who your team is exactly and what are their results? Be very interested in learning more

I'm gonna be that guy and ask you not to refer to OCD in a flippant/casual sense please

Ah thanks for catching that. Changed to “obsessed”

>Most search problems are solved this way - interesting index data modeling to the needed use cases, not magic machin learning neural network unicorns. Many want to sell you the latter. The thing that gets the job done is usually the former.

occam's razor