main.htm: How to search the web, by fravia+ main search engines

~ Main search engines ~

				Main search engines

Version June 2002

LATEST SEARCH ESSAYS
Google's wild side
Using Fuzzy Logic
A Re-ranking trilogy
Steganography and just-in-time info on the web
Some thoughts on CSE results
Hitting The BullsEye

Go at once to the Instructions and skip MAPA ~ f l a n g e o f m y t h

Fravia's searching MAPA (masks and pages) 1-10: width (1-10): freshness
Best s.e.	1 (2) Google (US/Wasp)		2 (1) Fast (Europe)		1 (3) Wisenut (Far east)
Main s.e.	3 (6) Yahoo!	4 (3) Hotbot	5 (3) Northernlight (dead...? :-)	(cf fast) Lycos	4 (4) Inktomi	6 (5) Alta! simple
Auxil s.e.	Go.com	MSNsearch	Ouverture	Kart00	Webcrawler	10 Excite (ill)
Other	(n/a) Wayback		8 (n/a) Teoma		n/a FTPsearch

@ PHP	[Our searching scrolls!]			[600 engines for next to nothing]
@ fravia's	Pointers	Local	Regional	Compound	Usenet	Accmail
@ fravia's	Live	PageProvid	Combing	Details	Databases	Allinones

Instructions & Caveats

Just copy this page onto your harddisk as c:\main.htm (or whatever), and then use it (after having edited anything you fancy) in order to perform EFFECTIVE searches on the web (and elsewhere) using the main search engines.

Note that just because one, hundred, or thousand pages from a site are crawled and made searchable trough one of the main search engines, this does not guarantee that every page from an indexed site has really been crawled and indexed. This shortcoming hits not only 'new' pages, that can take MONTHS to be indexed: beehives of spiders harvesting a site often MISS whole subdirectories, old and new. Useful material may be all but invisible to those that only use 'main' search tools to seek. You would be well advised to use regional engines, specialized or targeted search tools, combing techniques and your own bots as well, when searching.

Note that you can also easily search and find targets that do not exist any more :-)

Google or Fast? That is the question nowadays. The only correct answer is: "both, and many more".

SEARCH ENGINES FORMS
(Use the MAPA to navigate)

ALTAVISTA ADVANCED SEARCH [Only 400 results viewable]
AND,OR,(),NOT,NEAR,",*
link:text (search for links to 'text') anchor:text (search for links with the description 'text') url:text (search for given text in the url) domain:targetdomain (search files within 'targetdomain') host:hostname (search files on 'hostname') title:text (search 'text' inside the title tags) applet:text (search Java applets named 'text') image:filename (search images with such 'filename')

Read the [Altavista in depth] page! Altavista's algos main drawback is that they are very easy to spam, so you'll get most useless results in the first 20-30 positions: "hic alta, hic salta"... experienced searchers mostly jump directly in the middle of altavista's results lists. How many results? Seems to depend on the hour of your query / servers' workload and if you ask the first (less 'results' reported) or the last of the 100*10 results pages (more 'results' reported).
AltaVista includes paid results from the Overture (GoTo) search engine
Altavista is the 'dead links' champion among the 'main' search engines. Use the Simple search (which defaults to OR) ONLY if you really know what you are doing :-)

Simple search - Graphic Version

ALTAVISTA SIMPLE SEARCH [Only 400 results viewable]
Read the [Altavista in depth] page! and read more global info about alta here
No boolean! It defaults to OR... hence very useful for quick searchprobes! For boolean operators use Advanced Altavista instead!

Kart00
A "Graphical" search engine, rather interesting result clusters.
Here follows the text search form, but by all means try the cartographic interface

The Wayback machine
This is not only a -powerful- search engine, but also an incredible stalking tool! Explore the Net as it was!

YAHOO [Only 677 results viewable]
",*

EXCITE [Only 1000 results viewable]
AND,OR,(),NOT,,",
Spider: Architext
2001/2002: A very ill and now de facto dead searchengine. Its spider seems moribond as well, hav'nt seen it for ages. Now Architext has gone bersek: it scans for non-existing pages. Excite@home is a classical example of just another 'ignoble corporate merge'. This applies to all merges btw: attempts to escape the fate of all pyramide schemes that always forebode catastrophes

Special syntax that worked upon a time: site:, url:,

Excite tried to 'guess and figure out' what you really want, a precursor of Google, Teoma and Wisenut-similar non boolean engines: a mixed blessing to say the least.
Excite had a "precision" search feature. In Excite non-English pages could only be searched by specifically selecting them in the Advanced Search. An interesting Excite-specific algo is that shorter urls rank much higher, which is not so stupid after all. Unfortunately, all these features have disappeared: since excite now evidently rank for money (a consequence of their idiotical merges), the commercial bastards in charge did manage to ruin this once interesting search engine :-(

Describe what you want to find using Excite...
I want to search		Do not use query syntax in the search forms below
My search results MUST contain
My search results MUST NOT contain
My search results CAN contain

Display my results by document with	and	results per page.
Display the top 40 results grouped by web site.

Google [Only 999 results viewable]
+,-,",OR,(),  AND is the default boolean operator
not case sensitive  no stemming, seems to do 'something' with NEAR
Special syntax: site:, link:, inurl:, intitle:, filetype:
Use your keyboard to navigate trough google's results... unfortunately not with Opera :-(

Read the [Google in depth] & the [Google moves to Linux] pages
Google searches inside PDF files! Moreover, it locates the text most relevant to your specific query and highlights your keywords and its context! Very quick and very accurate (until sommer 2001) because of its algos, it is very useful for all kind of stalking purposes because of its CACHED pages! Use + to force stopwords for instance: +"index +of/mp3" +dylan :-)   "Well, yes, Google seems to do 'something' with some boolean operators... we just don't know exactly what..."   USE the special syntax! For instance the extremely useful filetype:... "+fravia +filetype:pdf" will give you 'high level' docs, whereas "+fravia -filetype:htm -filetype:html" will help to avoid 'redundant' pages when seeking (thank Nemo!)

Simple Google

Advanced GOOGLE

G. Univ search ~ G. 'news'
G. Classical :-)

Since most search engines are just keen on making money no matter how, Google represented a breath of fresh air, and (mostly) held the promise of delivering high relevancy results without all the extraneous and often ridicolous and annoying 'services' of the larger portals. Google is expanding quickly and had now swallowed Deja's huge usenet database as well:

Recently Google introduced its own Images search service!

	Advanced Search Preferences Search Tips
	Search images Search the Web Mature content filter is Off

Google's Zeitgeist (Search patterns, trends, and surprises according to Google)

!!: Since google doesn't support stopwords anymore, the old trick of using something like "advanced of searching" in order to fetch both "advanced internet searching" AND "advanced web searching" won't work anymore... but they 'forgot' to take off 'the', so you can use this article 'the' as a jolly * between words: "advanced the searching" will still give you both "advanced internet searching" AND "advanced web searching" results... (and "advanced searching" tout court as well, btw).

!!: The well-known 'provisory indexes' of google, at www2.google.com and www3.google.com seem to use -at times- completely different algos! (Not just a pool of rotating algos, algos that are 'weird' if compared to the usual google's ones). We do not know if this is just a temporary phenomenon or if it means something else.
Just try www2 by yourself...

Early spring 2002 (Shally Steckler): The * happens to be a wildcard that replaces an entire word. This is not a documented Google command but if you use the * connected by any of the characters Google ignores like - = , ; \ / < and > then it acts as a place setting for "any word" like this: three-*-cats or nice=*=spring or fravia<*>site and so on. An interesting thing is that if you use another connector and another asterisk then it returns results with two words between the first and the last term like this: fravia-*-*-site.

Precious
Item

LYCOS [As many results viewable as you get!]
AND,OR,(),NOT,NEAR,",

"Part Man, Part Machine" ~ Open Directory & DMOZ used. Uses especially Fast's index, with updates at greater intervals than FAST. Major sin: Has closed the VERY useful Trondheim ftp-search facility.

Lycos advanced: fields Lycos advanced: language Lycos advanced: link referrals
Lycos help page

FTP search [the famous "Trondheim" server]
Now unfortunatley DISCONTINUED!
See either Other 'files search' facilities or, better, the ad hoc section [FTP searches]
Over 200 million files have been catalogued by Lycos, now managing the famous Trondheim engine, and can be searched using the lycos_ftp advanced form (the one below). Do not underestimate the amazing power of this tool for searching purposes! A true treasures searchmachine!

Lycos introduced an "akamai-heavy" form in autumn 2001. In the form below, each small red dot represents one of the 16 (sic!) linked calls to akamai. Thus you will avoid all useless (and snooping) connection overloads you would have made using the "real" Lycos form.

	"Lycos downloads" Search: computers downloads tech help the web

Advanced FTP Search!!

Search Parameters
Search for
Search type: Try exact hits first
Max hits: Max matches: Max hits/match:
Limit to domain: Limit to path:
Minimum size: Maximum size:
From date: To date:
Hide: Packages Distfiles FreeBSD OpenBSD NetBSD Linux

Formatting Parameters
Output fields:
Per-host header: Nothing Just Host Host and Country
Sort by: Nothing (unsorted) Host (path) Size Date
Truncate hostnames (longer than characters.)

Other 'files search' facilities

Filesearching
http://www.filesearching.com/
http://www.filesearching.com/advanced/
Use Author as query string for quick and dirty ebooks searches
Other, regional FTP services
For instance http://www.filesearch.ru/ Russki
http://www.filesearch.ru/cgi-bin/s?q=Sice2&t=f&w=a
Slow but... hey, it works!
Archie gateway
http://www.uni-jena.de/net/archie-gate.html/
The web of old...
Internet Cache Archive Service (ICAS)
http://archie.edu.tw/
Woah!

WEBCRAWLER (powered by the now quite "ill" Excite")
AND,OR,(),NOT,,",*
WebCrawler has been bought by Excite. Before it was a neat minor engine that used its own index of web pages to provide fairly accurate search results and had a real niche of aficionados, hence real long term value. Now search results are identical for both engines. Since Excite itself is going down the drain, thank to the 'short-fuse' brains of the actors of the 'new merging economy', Webcrawler chances of survival are next to zilch
TSS "searching+tips":
http://search.excite.com/search.gw?s=%22searching+tips%22&c=web&showSummary=true&lang=en&start=0&perPage=25&lk=webcrawler
TSSR "searching+tips":
http://search.excite.com/search.gw?s=%22searching+tips%22&lk=webcrawler
TSSI YAK3:
http://search.excite.com/search.gw?c=spider.mial&lk=webcrawler&s=yak3

GO.COM (powered by Infoseek)
Infoseek was the "Proximity champion", Expert ~S~eekers always used Infoseek for proximity queries. Note the "Search within results" option. Unfortunately Infoseek has been transformed into GO.COM and the proximity commands do not seem to work anymore. Also: GO.COM offers an automated translation service à la Altavista. GO.COM servers are often overloaded :-(

A list of Infoseek's old beautiful proximity operators :-(
ADJ, ADJ/#, OADJ, OADJ/#, NEAR
NEAR/#, ONEAR, ONEAR/#
FAR , FAR/#, OFAR, OFAR/#,

[gocom "powersearch"]

Spider: Sidewinder. Does not go trolling for unsubmitted pages, doesn't crawl inside sites, just indexes (very slowly) individual submissions. Answers display either metadescription or first 200 chars. Spider IPs are around 204.162.96.xx

INKTOMY RAW

Incredible 'raw' access to Inktomi's data (pointed out by Sally)...

HOTBOT [Only 1397 results viewable]
AND,OR,(),NOT,,",*
Special syntax: domain:, linkdomain:, originurl:, title:
Examples: linkdomain:searchlores.org title:searching

Offers FULL boolean searching. Owned by Lycos. Very large! Uses the Open Directory, Inktomi indexing services, but ranks results using "Direct Hit" algos data and it's own internal data. Its 'popularity' result engine is a mixed bless ("clicking" algo: the more people click on your site the more it weights). Moreover it seems to give a lot of weight to older pages. Note the "Search within these results" option!
Try the advanced options: [Hotbot BETA supersearch form]

Indexing service: Inktomi Spider: Slurp. Apart from Hotbot, Aol, Snap and MSN, Inktomi serves also private databases with its spider.

NORTHERN LIGHT [You can (awkwardly) view its deep results]
AND,OR,(),NOT,",

Now defunct for the mass-public, alas, because the search service has been taken offline :-(
Yet we can still use it, see below.
Had a unique folders feature (dynamically generated by the search results!) to refine your query (very useful & powerful). Note that this engine automatically recognized and searched variants and plurals of your query.

Select:

All Sources Search the World Wide Web & Special Collection

World Wide Web Search the entire World Wide Web

Special Collection 1 million articles not on other search engines

northern light power search ...and other commercial problems...
Northernlight's powersearch has been recently castrated (LIMITED to the 'premium collection' crap)
Should Northernlight not work, (or rather not seem to work), try using the following query, substituting 'searchlore' with whatever you are searching...
http://www.northernlight.com/nlquery.fcg?qr=searchlore
But the substitution MUST BE MADE 'per hand' on the string you use, not on the resulting mask 'offered' by northernlight (and you also better use a cookies killer à la proxomitron

FAST ("Alltheweb")
European search technology... I like fast, because I list before google :-)
[Only 4010 results viewable]
+,(),-,,",
Special syntax: normal.title:, url.host:, url.all:, link.all:.
Domain exclusion (VERY useful): -url.tld:com (no com crap), +url.tld:no (only norway).
Also: link.extension:jpeg
spiders ending with sac2.fastsearch.net

FAST is indeed pretty fast :-) But the real value of this engine resides in its linguistic capabilities (very useful for "non-wasp" searches) and in being a good old BOOLEAN one (no stopwords... try FAST's to be or not to be and compare with google's same search :-)
FAST delivers moreover a huge number of 'unique hits' (it does not overlap too much with the other engines).

Note that you can restrict the advanced search to a single domainname (leaving every other parameter unchecked) and thus get a 'sitemap' of a given site. Example: FAST's Altavista ad hoc ['sitemap']
Note also that you can find your favourite mp3 songs using the name of the author (for instance dylan) and stating that the document must include "index of/mp3" in the title: here is FAST's Bob Dylan's ad hoc [mp3 search] (thank SaF)
Use the url.host: syntax to find out how many pages of a given site are indexed, e.g: q=url.host%3Awww.fravia.com.
For the links, use the link.all: syntax, e.g.: link.all:www.searchlores.org

Fast index: Do use the very useful 'Result Restrictions' options to find most recent links to matters you are interested into... for instance: &q=searchlores &age=3m &hits=100 &nooc=off (see source for complete string)

Fast FTP-search: http://alltheweb.com/?cat=ftp&lang=any&query=, is one of the most effective ftp-search possibilities around. (See the others here)

Fast indexing timespan: a large portion of fast's index, the so called "popular content", is re-spidered every seven to 11 days, the remainder is being spidered every month or so. It's spiders did not always use to go deep inside, but they are now catching up: see if the depths of these three mirrors of mine are still far apart: fravia; searchlores; searchlore.

OUVERTURE

This used to be "Go To", the commercial clowns changed the name because this "reinforces our leadership in performance-based search", haha :-) Uses Inktomi, like Hotbot. Ranks results by how much a company is willing to pay for listings and is heavily akamai infested :-(
The mask below is relatively "clean", you should NEVER use ouverture's site own mask to perform your searches (has visitor tracking sniffing annoying logging options aplenty).

WEBTOP
Winter 2001: For some nutty reason webtop, a 'sleeping giant' that had very good search algos and a huge database, has been discontinued (or, more probably, privatized). Click this link and ask them why...
http://www.smartlogik.com/commercial_bastards_why_did_you_discontinue_webtop?.htm

Example string:
http://www.webtop.com/search/vanilla/results.htm?WEBSITE_SEARCH=1&QUERY=fravia&EXPANDED=web&Search.x=40&Search.y=10
help powersearch
European search engine developed at Cambridge uni. Runs on Linux (of course :-). Probability and Baysian inference applied to the search process. Hence no booleans: beware! Its results may be utterly weird because instead of the traditional method of searching for a matched keyword in a document, the 'probabilistic techniques' focus on the relative value of a word - either in the search expression, or in the document being indexed.

WISENUT [Only 300 results viewable]
default to AND phrase searching: use "" use - for NOT
no truncation use + to force stopwords

Example string:
http://www.wisenut.com/search/query.dll?q=%22advanced+searching+techniques%22

WiseNut is a "Korean/Japanese" new 'main' search engine. has good customization feature and one single huge database of indexed Web pages. It lacks almost all advanced search capabilities, yet it seems useful because it gives results that you will not find elsewhere.

Search for Web pages...

... WITH ALL of these words
... WITHOUT ANY of these words
... WITH this EXACT PHRASE

TEOMA [Only 194 results viewable]
default to AND phrase searching: use "" use - for NOT

Example string:
http://s.teoma.com/q?t=%22advanced+searching%22&submit=Search
Askjeeves foray into real searching. Has an useful 'folder' collation à la Northernlight


Find this Phrase

MSNsearch

Example string:
http://search.msn.com/results.asp?q=%22advanced+searching%22&FORM=SMCA&cfg=SMCINK&v=1&ba=0&f=any&sort=&rgn=&lng=&dom=&depth=&d0=&d1=&cf=
Note however that -as usual with Microsoft's malbehaviour, the PREVIOUS QUERY you have made is indicated inside the new querystring...
http://search.msn.com/results.asp?q=%22Microsoft+sniffing+practices%22&origq=%22advanced+searching%22&RS=CHECKED&FORM=SMCRT&v=1&nosp=0&cfg=SMCINITIAL

Actually this search engine is not that bad, not as crappy as you would expect from Microsoft programmers... but it is indeed quite commercial infested. It is target - basically - for "AOL type" lusers and commercial zombies, and uses therefore the Inktomi indexing services (infamous for its PPC - Pay per click - schemes).
Note moreover that this is a 'puritan' s.e. and will not retrieve adult content everytime someone uses 'banal' adult search words. But it nevertheless will fetch any sort of filth if it has an 'unusual' searchquery input. A doomed attempt of course, as all censorship attempts are :-)
Quote: "For research, they are useless, but honestly, how many people that need to do research on the net would really use MSN? AOL? IWON?"

f l a n g e o f m y t h

	Worldwide web English web
more options To use the best of KartOO, try the cartographic interface.

Search for
FAST's [Advanced Search] FAST's [help] FAST's [news] FAST's [pictures]