Item — HN Reader

HN should detect duplicates on submit.

1 comments

They do when they're identical. Unfortunately, all these links are subtly different.

    When someone says "I want a programming language
    in which I need only say what I wish done," give
    him a lollipop.

        -- Alan Perlis ACM-SIGPLAN '82,
           Epigrams in Programming

I invite you to suggest a solution to the problem of duplicated content.

No something changed, I have seen several links I and others have submitted from tech sites duplicated, they are 100% identical.

Example:

http://news.ycombinator.com/item?id=4749228

http://news.ycombinator.com/item?id=4749070

Same exact title, same exact link.

I can't check, because one of them has been deleted, but I know that there are rare occasions when this happens. I also know that most of the time when a duplicate link is submitted it acts as an upvote on the existing item.

I also know that this doesn't work unless the earlier item is in the cache.

I also know that the duplicate detection mechanism is currently working because I just tripped it, so it's either more complicated than you think, or I haven't understood you fully.

HN does (usually) detect exact duplicates, and your comment seems a bit simplistic. The duplications of these videos about the voting machines are not identical URLs, and I haven't recently seen an example where the URL was identical, and yet was not detected. As I say, your references can't be checked because one has been deleted, but I have several times seen people claim a duplicate was not detected, only the find that the URLs were not, in fact, completely identical.

I am unconvinced that anything has changed.

Here is another duplicate of the same story:

http://news.ycombinator.com/item?id=4749252

Simplistic, but exact matches none-the-less.

Nope, not exact duplicates, because one has a trailing backslash, the other doesn't. Sometimes that matters.

Here are the resulting links to each of the three examples I showed:

http://arstechnica.com/gadgets/2012/11/surface-disk-space-a-...

They are 100% identical, no?

They are as you quote them, and they are what they resolve to eventually, but one of them at least is not the URL actually in the item.

This item: http://news.ycombinator.com/item?id=4749070

Look closely at the URL as it's actually in the submission, not in the URL bar after you click it. There is no trailing slash on the URL, so you are not quoting the URL as it was submitting. That matters.

And when I say that it matters, it's possible that two URLs that differ only in a trailing slash actually deliver different pages to you. For that reason, URLs that differ in a trailing slash have to be considered different. Indeed, URLs that are completely different sometimes end up delivering identical content.

The DupDetector I wrote actually downloaded headers and titles, resolved URLs, followed links, and generally tried to work out what would actually been delivered. It was never production standard, and it's hard.