The Search Guy: Interesting search topics and random musings

All of a sudden everyone is putting out a desktop search tool. Note that these are all things that you can actually download and run on your machine. I suspect it's not a good idea to run them all at the same time, but hey, it's your machine.

Of course, the big two that I didn't mention are the search facility that's supposed to be in WinFS and Apple's Spotlight, which is coming in OSX 10.4.

Slashdot has had it's discussion, based on a Slate article that compared a few of the current challengers. It's indicative of how fast things are changing that they missed out on Yahoo!'s new entry. So why is this such a big deal so suddenly as Joel Spolsky points out, a lot of the technology that these engines are based on has been around since the 1970s?

I really think it's a convergence of a couple of things. The first is the cost per megabyte (I guess that should be gigabyte these days) of hard drives. My first hard drive was 40 megabytes. At the start of grad school, I had to save up to buy a 300MB drive for my PC and a few years later, I had to beg my supervisor to buy me a 2GB drive (for more than $2K!).

At these prices and capacities, you had to do actual work to keep your drive from filling up: compressing things, deleting things, moving things to floppies. In the act of doing this, I think you got to have a pretty good idea of what stuff was on your drive(s) and where they were. This mental model of your drive meant that you didn't need search.

Last Black Friday I went to Best Buy and spent $30 on a 160GB drive. I put it into my home machine that already had 200GB of disk. At work I have a 530GB filesystem to hold text corpora along with a couple of 120GB drives for other interesting stuff.

While the prices on hard drives were falling and the capacities rising, the amount of stuff that was available to put on the drives was increasing fast. All of that email that everyone started getting. All of those Web pages people were pulling down.

At this scale, you can't keep it all in your head. At these prices, you don't need to delete anything: you just need to go buy a new drive. So, when you want to find something your only recourse is search.

Unfortunately, I don't think that John Udell is right about desktop search being too late. I think people are going to continue to keep more and more stuff on their local storage. It's just a lot easier.