You might want to check out "Nutch"/"Wikia Search" ( http://re.search.wikia.com/about/get_involved.html )... They let you download their copy of the web ( http://search.isc.org/download/ ) Not exactly Yahoo quality, but it might be enough to get started on some smaller scale projects.