Quiet ramblings with purpose. 404 File Not Found

I was doing some work and thought, "Wouldn't it be nice to have my own cluster?" I'm guessing not many people have those types of revelations, and probably fewer that decide they should go ahead and solve the problem. I wanted a cheap, small, easy to pack, light, quiet, low-power cluster that I could sit on my desk, and not even think about it. I did a little bit of research of what equipment to use and decided on the following:

(1) EEE PC-901 -- A good option for a management node due to it not only being small and portable, but also includes a screen to allow as a front-end to the cluster.
(3) fit-pc2 -- A tiny PC in a box. Although not considered a workhorse, the Atom-based system can get the job done with its two 32-bit cores.

The network for the cluster is just using any 10/100/1000 switch laying around (in my case a spare unused wireless router).

The OS for the system will be CentOS 5.3 (i386) with all of the cluster tools being supplied by the Sun HPC Software, Linux Edition (sorry, consider this part an advertisement for the project I work on). OK, with all that selected, let's build us a cluster.

Installing the EEE PC 901

BOOTING

Because the EEE PC 901 doesn't have a CD-ROM drive, and the ethernet card in the machine isn't recognized by the CentOS installer (therefore no network install), I found a USB stick that was big enough to hold the CentOS DVD installer.

To create this, I did the following:

• mount the CentOS iso (/mnt)
• fdisk /dev/sdb (set to type b)
• mkfs.vfat /dev/sdb1
• mount the USB drive (/media/disk)
• rsync -rv /mnt/isolinux /mnt/images /media/disk/
• mkdir /media/disk/syslinux
• syslinux -d syslinux /dev/sdb1
• mv /media/disk/isolinux/* /media/disk/syslinux/
• rmdir /media/disk/isolinux
• mv /media/disk/syslinux/isolinux.bin /media/disk/syslinux/syslinux.bin
• mv /media/disk/syslinux/isolinux.cfg /media/disk/syslinux/syslinux.cfg
• Copy the CentOS iso image to /media/disk/linux.iso

Now, boot onto the USB drive. I've noticed that the EEE seems to have random issues of detecting flash drives as bootable. If that's the case, you might need to cycle a few times and check your boot device settings in the BIOS options. It'll work, it just might take a little magic.

Once the machine boots, you should be at a very unpretty CentOS installer screen. Click through the defaults until you get to the "Installation Method" screen. At this point, you want to choose "Hard Drive" and then "/dev/sda1". If all goes well, the installer will continue on look a lot more "normal".

PARTITIONING

There's always a question of how best to partition the hard drive of a machine. At one time I was pretty hard core about what partitions were used, how big they are, extended or primary, etc. Now, I've realize that really just using the default partitioning scheme works best for most people. One thing to keep in mind is that for clustering, your management node could have a need for more disk space in certain places; this is namely to store the images you'll be using on the rest of the cluster. Depending on your diskless of diskfull provisioner, the location of these images could change, so be aware of those decisions. In this setup, the images will mainly lie in /var/lib/oneSIS; so as long as /var has a fair amount of disk space we should be good.

And now a quick note about the EEE PC 901. Mine is the "linux" version, which includes a 20GB Solid State Device. This is physically represented as two separate drives; a faster 4GB drive and a slightly slower 16GB drive. Normally, installing linux on an EEE you would want to put as much of the OS onto the 4GB drive and save the rest for /home; for this layout I'm going to devote the 4GB disk to the initial CentOS install, and then over the course of this document, carve up the 16GB as needed.

** NOTE **
The CentOS installer is going to want to include any USB drives it finds in the partitioning scheme. This includes the one you booted off of, so make sure that it's only paying attention to the drives you want it to.

I noticed that the installer attempts to install grub onto /dev/sda (which is thoroughly unuseful) by default. So, make sure that you remove the USB flash drive from the grub options (not required) and click the "Configure advanced boot loader options". Then click next, and on the next screen click the "Change Drive Order" pushing /dev/hdc (or your actual hard drive) up to the top position. Clicking OK, you should now see your actual hard drive set for the MBR selection to load grub into (phew).

PACKAGE SELECTION

At this point, we'll go ahead and just leave the default package selection (which is just "Desktop - Gnome"). While you can install anything you want from here (it's your box, do what you want), I like to keep it minimal and do things later. One thing to note on the EEE PC 901 is that the ethernet and wireless networking aren't supported by default. So, perhaps also installing the "Development Tools" would be helpful here.

Well, if all went well, the system should be installing; this can take a little bit of time ... so, entertain yourself in some way.

** NOTE **
I noticed that on my system, the CentOS installer saw the hard drives as /dev/hdc and /dev/hdd I didn't really do anything about it because I didn't know what the reprocutions might be (from what I can tell, there aren't any, that just seems to be the placement Asus put the SSD on the bus). After the installation and reboot, I also noticed that my system won't boot without the USB stick available (seems that grub installed onto there instead of the hard drive). So, if this happens to you, here's what I did (though you might need to run through the next section of this guide to get to a login prompt).

1. Boot the system with the USB stick, but make sure it actually boots into Linux and not the installer.
2. Login to the system as root (you can either login as root, or su to root later; sure wish CentOS setup sudo by default).
3. Pull up a terminal and "cd /boot/grub"
4. cp grub.conf /tmp
5. rm -rf /boot/grub/*
6. grub-install /dev/hdc (make sure this matches whatever your boot device should be)
7. mv /tmp/grub.conf /boot/grub && ln -s grub.conf /boot/grub/menu.lst
8. Modify the /boot/grub/grub.conf so that the CentOS entry looks like this (specifically the root part):

   title CentOS (2.6.18-128.el5)
   	root (hd0,0)
   	kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/VolGroup00/LogVol00 rhgb quiet
   	initrd /initrd-2.6.18-128.el5.img
   

After a reboot, system came right back without worrying about the USB drive.

SYSTEM CONFIGURATION

After the initial reboot, there are some quick configurations that CentOS want you to do. Included are Firewall and SELinux options. For both of these, I usually just disable them since in a cluster they tend to be a hindrance. If you do need added security, the firewall is your best bet, but I'd recommend leaving SELinux disabled.

At this point, you should be at a login screen and ready to start really configuring the management node. As a long time user of sudo, this is a good moment to go ahead and get it running. In a terminal, the first thing we need to do is enable the "wheel" group in the sudoers file:

1. As root run visudo
2. Look for the line that looks like:

   # %wheel	ALL=(ALL)	ALL
   

   and remove the leading "#".
3. Save the file and quit.

So, at this point anyone in the "wheel" group can use sudo (with a password) for all commands on the system. So, now we need to add ourselves to the wheel group:

usermod -G wheel <your username>

For the change to take affect, you'll need to log out and back in; then after that sudo will be available.

If you're using an EEE PC 901 like I am you'll notice that you don't have wireless or ethernet. CentOS doesn't include the needed modules for the ethernet card that comes with the 901 (Attansic L1 Gigabit Ethernet Adapter). Looking online, it seems that the module that's needed is the "atl1e", so we'll install it:

1. On a working system: wget http://elrepo.org/linux/elrepo/el5/i386/RPMS/kmod-atl1e-1.2.40.3-1.el5.elrepo.i686.rpm
2. Copy the rpms to your flash disk and move it over to the EEE.
3. On the eee: rpm -Uvh <path to>/kmod-atl1e-1.2.40.3-1.el5.elrepo.i686.rpm
4. Reboot

A good idea at this point is to take a break and let the computer churn on updates. You generally want to make sure your system is up-to-date, so we'll do that now before we start really configuring the cluster software. So, run "sudo yum update" accept the updates it wants to do, and then go and do something else for a bit.

<-- Time Passes -->

So, now we have what amounts to a laptop running CentOS. Not overly exciting, nor the goal of this little ditty, so let's keep going. First thing I want to do is give my system (and cluster for that matter) a name. I'm not going to judge or help offering up names, instead I'll just use the name "frosted" (thanks for Michael MacDonald for the name Frosted Mini-Cluster). With that, I have a naming scheme for my cluster:

Node Type	Name	IP-Address
Management/Login	frosted	10.0.10.1
Compute	frosted0	10.0.10.2
Compute	frosted1	10.0.10.3
Compute	frosted2	10.0.10.4

For a cluster like this, we'll just go ahead and double up the responsiblities of the management node by also making it the login node. All the rest of the nodes in the cluster can all be used for computing (or whatever). At this point, we're really building a cluster, so I'll just point to the Sun HPC Software, Linux Edition documentation. One thing to keep in mind is that this cluster is 32-bit, that SHSLE is a 64-bit OS. Most of the tools that are a part of SHSLE can easily be rebuilt for 32-bit, it just takes time. The rest of this write-up is the result of my rebuilding the stack and making my own ISO. I'm not quite sure if we (the Giraffe team) can do anything to help others with a 32-bit version; but I might be able to at least make my ISO available in some way.

When going through the OneSIS configuration, a very important step is making sure you correctly build the initramfs so that the fit-pc2 has a working network device. Here's the command that I used:

mk-initramfs-oneSIS -w r8169 -f -s 8192 /tftpboot/images/centos5.3-onesis/initrd-2.6.18-128.1.10.el5.img 2.6.18-128.1.10.el5

With that, you should be good to go with booting the nodes off of this image.

CONFIGURING THE FIT-PC2's

All the way up till now, the shiny new fit-pc2's have been sitting there unused. Now's their moment. The first thing we need to do is get the ready for network booting. The fit-pc2 has built-in support to boot over the ethernet, but by default it's disabled (at least on my fit-pc2s). So, we need to turn them on, go into the CMOS settings (hitting F2 frantically), and setting the following options:

Main -> Large Disk Access Mode: -> Other
Advanced -> Installed O/S: -> Other
Advanced -> Boot-time Diagnostics Screen: -> Enabled
Advanced -> QuickBoot Mode: -> Enabled
Intel -> Chipset Control Sub-Menu -> LAN PXE BOOT: -> Enabled
Exit -> Exit Saving Changes

So, with that, we should be able to turn them on and bask in the glow of our shiny new cluster.

USING THE CLUSTER

There are a number of tools that come included with the Sun HPC Software Stack that help us utilize the cluster. One such tool is pdsh (parallel shell), which allows us to run commands on all of the nodes in parallel. So, for example, let's say we want to check the time on all of the nodes, we can do the following:

[makia@frosted ~]$ pdsh -a date
frosted0: Sat Jun 27 14:41:51 MDT 2009
frosted1: Sat Jun 27 14:42:04 MDT 2009
frosted2: Sat Jun 27 14:42:03 MDT 2009
[makia@frosted ~]$

Or, if we want to see combined output, we can also include dshbak:

[makia@frosted ~]$ pdsh -a "df" | grep /home | dshbak -c
----------------
frosted[0-2]
----------------
10.0.10.1:/home        3850336   2525376   1129376  70% /home
[makia@frosted ~]$ 

As part of the SunHPC Software Stack, we also include slurm (and also a way to configure it). Once it's configured, it can help us run jobs across the system (including MPI jobs, using OpenMPI which is also included):

[makia@frosted mpi]$ mpicc -o hello hello.c
[makia@frosted mpi]$ sinfo
PARTITION AVAIL  TIMELIMIT NODES  STATE NODELIST
compute*     up   infinite     3   idle frosted[0-2]
[makia@frosted mpi]$ salloc -N3
salloc: Granted job allocation 3798
[makia@frosted mpi]$ squeue
  JOBID PARTITION     NAME     USER  ST       TIME  NODES NODELIST(REASON)
   3798   compute     bash    makia   R       0:07      3 frosted[0-2]
[makia@frosted mpi]$ mpirun -mca btl self,tcp hellofrosted0: hello world from process 0 of 6
frosted1: hello world from process 3 of 6
frosted2: hello world from process 5 of 6
frosted0: hello world from process 1 of 6
frosted1: hello world from process 2 of 6
frosted2: hello world from process 4 of 6
[makia@frosted mpi]$ exit
exit
salloc: Relinquishing job allocation 3798
[makia@frosted mpi]$ 

Another nice tool that comes along with the stack is monitoring tools. Here's ganglia output for the cluster (it even had a slight load on it, just for added drama).

CONCLUSION

That's it. A minicluster. Not to difficult to build, takes up little-to-no desk space, extremely quiet, and kinda cool to have sitting around. Usefullness varies with whatever you see the need for a cluster. I decided I wanted to see how the Selfish Detour benchmarks would hold up. Here's the graph (it's pretty, but also shows the nodes are pretty noisy):

Feel free to leave comments or questions.

Before I begin, this isn't going to be one of my normal posts (which is somewhat noticeable from the title). I had a few things to say, and figured I might as well post them out to the rest of the world in the hopes of getting them off my chest.

First, I should explain a little bit of a background. This morning, I received an email from my mother that my Grandmother, who was 93 years old, passed away in her sleep last night. This got me to thinking about things; namely the fact that I wasn't really sad in any way and didn't know if that was right or wrong. Not that there's anything particularly wrong with my Grandmother, we were just never really all that close; and no one's really to blame on that. For most of my life she's lived on the other side of the country, periodically visiting but mostly existing as a random voice on the other end of a phone during various holidays. I never really knew what to talk to her about, cause I didn't really know her and she didn't really know me. Our lives were always seemed very different, and so there was physical and emotional distance. I liked her as much as any person, but how close can you really feel to someone when you barely even know or interact with them.

So, on the news of her death, I tried to feel something. I tried to use memories, or stories, or anything I could think of to actually find a feeling. But, nothing came. And then I started feeling bad about it; she is related to me. If she didn't exist, I probably wouldn't either, and I kind of like existing. But in all honesty, in my day-to-day life she rarely, if ever, popped into my thoughts. So, how do you really mourn for someone that exists so little in your life; are you required to just because you're related? As much as I hate to admit it, I'd feel more sad if the local sushi chef died, simply because I happen to see him far too many times in a week.

But, then I realized a few things. First, I don't have to feel bad about not feeling bad; you really can't push something that you don't feel. I loved her, she was my Grandmother (weird usage of past tense), and I'm sad we weren't closer, but there's not much we can do about it now. In fact, I know I shouldn't feel bad, because even if someone said, "Just kidding she's not really dead" it's not like I'd rush to her side and start working on getting to know her; that sounds cruel, but it's the truth. Second, I do feel sad, but not because of how her passing affects me, but how it affects those I'm close to. My father lost his mother--that has to be hard--and my mother lost a friend. I realize that I feel sad for this fact and for them. So, I guess in a way I am mourning; but more for their loss.

I know that my Grandmother died peacefully and lead a long life (not always happy, not always sad; so a good life indeed). It might be wrong of me to think this; but I love the fact that she passed while playing bingo (she got tired, went to rest and never came back). I hope in someway she was winning, because how cool of an exit is that, "Yeah, I could beat you; but I've got better things to do."

OK, so this entry will probably hurt in the future (especially if AT&T actually reads my blog), but I'm going to put it out here anyway, because this is the one thing that makes my buying an iPhone 3G (there Apple, I even capitalized it right ... that counts for something, right?) an actual justification.

So, with other phones, I was able to tether fairly easily (see the WM6 write-up I did earlier). For people like me that don't want to carry around a 3G card for their laptop (or have a tmobile, AT&T, and who knows what other wifi hotspot account), tethering is about the only way to make sure and be able to jump online and get some work done while in between destinations. The 3G network, while not the zippiest thing out there, at least provides a respectable connection for email, some chatting (work based, of course), and some light web browsing. Heck, if YouTube can work on the phone in 3G mode, I should be able to use it on my laptop as well.

Earlier incarnations of tethering on the iPhone involved specifically using 3proxy (available from the Cydia installer). And, believe me, this worked quite nicely. Sure, there were some manual steps involved (e.g. setting up proxy settings for firefox and thunderbird), but in general it worked in a pinch. But, then along came PDANet to make things even better.

A port of the old PDANet application, this basically turns your iPhone into a WIFI router. Create an Ad-Hoc network, join your phone to it, and off you go. No longer do you need to change your proxy settings, just make sure that your default routes point to the IP Address of your phone. All network apps work without an issue (unless you count slowness an issue ... ok, so maybe one issue).

Now, I don't condone this approach. My feeling is that Apple completely missed the mark by creating an "open" development environment (App Store, how many flashlight programs do you really really need?) instead of an open one (see, no quotes). And by disabling some core features (e.g. tethering, bluetooth syncing, etc.) they've opened the door to people killing their market share (if only these "people" would get busy designing) as new phones come to market. And with each passing firmware update of nothing more than flash instead of substance (cut/copy/paste? really? still not there?) the lack of attention becomes more and more noticeable.

So, yes, I own an iPhone (and even spelled it with the stupid correct capitalization). Even worse, I own the first and second generation iPhone. Do I have a good reason for this? No, no I don't (but I could make one up just to justify it, if I really need to). So, why am I admitting to this? Because if I didn't, then I wouldn't be able to talk about how the best part about the iPhone is the community that is behind it jailbreaking it and actually turning something that's just "cool"ish into something that actually is quite useful. So, another admission ... yes, I voided my warranty; I jailbroke my iPhone and am looking at the tools out there. I will not hide in shame, instead I'll just hope that I don't need my warranty anytime soon.

Now... why, some might ask, would you want to jailbreak the phone? With the latest OS, you now have access to the App Store (which is a needed change from the first iPhone release, since relying on Web 2.0 for all of your answers is a huge mistake in my opinion). While there is a lot to be seen in the App Store, there is still more that the phone can do, if only we had the right people working on it. So, here are a few of my favorite and most used features (all of which can only be used after a jailbreak).

Themes (Winterboard)

OK, this is all just nothing more than frills; but I hate the standard iPhone look. It's nice (everyonce in a while) to just have something look dumb/cool. So, here's a look at my iPhone's home screen ... doesn't that just say "hey, that's my iPhone!"

Shell Applications

This is really where the power is. After installing "MobileTerminal", you can now pull up a console on the phone and run console based apps from the phone. Add in the networking features of your phone, and now you have a very powerful diagnostic tool. Have a server on your network that you want to know what services are running, just run an nmap against it. Have some weird network traffic, tcpdump is right there. The list goes on, especially as more standard tools are ported.

Tethering

OK, well this one is not only warranty voiding, but perhaps also bill increasing, but I'm going to mention this anyway. Tethering is very important to anyone that finds themselves out on the road and in desperate need of getting online (desperate might be overblown in this sentence, but let's just pretend). There is an App being offered through the App Store, but from what I can tell it keeps appearing and disappearing, making the conspiracy theorist in me wonder. On the other hand, a Socks proxy is freely available via the jailbreak apps. I won't get into the details (as they're publicly available and easy to find via our friends and Google), but having access to the 3G network to write a few quick emails (using a real keyboard, and cut/paste ability) is extremely handy.

Well, that's really all I wanted to say. While I don't advocate breaking laws, or ignoring rules, sometimes it's good to look at a tool and see what it really can do. If it were up to me, Apple would just offer these options without having to get around things (cause isn't it bad enough that we can't even get MMS messages? I mean ... a RAZR can do that!).

I figured I'd throw a plug for myself on a talk I did at LUG that is now online. Go ahead and visit the Giraffe blog for the links (this way I can also tempt people into going to the Giraffe blog.

As this is my personal blog, I can mention that the talk at least shows pretty well how awkward I am around people, but hopefully still able to get my point across.

So, enjoy. As usual, any criticism is always welcome.

<sorry>I'll apologize early for what will basically turn into a ranting session on my hatred of graphical user interfaces (GUI's) when people attempt to apply them in the high performance computing (HPC) space.</sorry>

I was doing some investigation into some of the advanced options on the service processors we are using (especially virtual cd-rom mode), and came across something very disturbing. Apart from the basic node control (e.g. power on/off/cycle) which can be accessed on the command line (either through IPMI or the ssh-command line), these other options can only be accessed via an embedded web server on the service processor. Even worse, there's a java web start requirement to even properly see these pages (thus making expect or perl scripts to access it somewhat difficult).

So, the question is, how can we fix this? In the HPC space (think hundreds to thousands of nodes) how can we think that this is acceptable to manage these systems? Moreso, in a lights out facility with limited access, we're requiring an administrator to X11-forward a web browser from the internal network to check on these settings when there could just as easily be a command line equivalent.

There is this GUI frame of mind that everyone seems to be in. GUI's are nice to look at and give everyone a warm and fuzzy feeling because there's a button to click, but in the end GUI's are not scalable and slow down process. I've seen it range from cluster installation tools, to node management tools, to job launchers, to even some companies only providing power features through the embedded web services (at least this one has mostly gone away). How is this acceptable? Windows clusters with their fancy buttons and <insert whatever feature you actually get from it> are nice and all, but there's a reason that most of the HPC community is using Linux and AIX.

If I have one request, it's that we finally realize that GUI's have a purpose but are not THE solution to everything. Also, HTTP (and whatever else goes with it) is great for online banking and reading the occasional educational material, shouldn't be the sole source of the management of a large-scale HPC system. The command line interface isn't as scary as it's made out to be.

As an aside, this isn't just tools and applications. This includes all those special interfaces to switches and everything else that requires configuration through a web application. We all have to start thinking large scale (managing hundreds rather than tens).

Tags: HPC, gripe, GUI, CLI

OK, let's say you need to print something, but you don't have a printer.  For some reason you have a fax machine, but no printer.  Seems strange, but I *might* know someone like that.  Well, what do you do?

A long time ago, I always thought that in that situation, I'd go ahead and find some way to e-mail fax the document to myself and turn it to a hard copy.  Seemed easy enough, but why not get a printer instead?  Well, as it turns out, I needed to print something quickly, didn't have time to buy a printer (have I mentioned that there are far too many printer options out there), and was staring at my fax machine cursing it's lack of inputs.  But, that's when I decided to put my plan into motion.

So I scoured the internet for something that would do my dirty work.  That's when I stumbled on Fax Zero.  It's an ad-driven service that gives you three (3) free faxes a day, and all you have to put up with is an Ad on your coverpage.  "PERFECT," I exclaimed in my head.  It seems that they are able to take PDF as input (as well as a few others document types) and then you tell them where to send it to (which, of course, is myself) and off it goes.  In less than five minutes, I suddenly get a phone call, and soon after a hard copy of my PDF document.

I still don't have a printer.  I should remedy that.

Tags: faxprinting

I saw this spot on TV about Hulu last night, and got quite interested.  It's an interesting idea: at the cost of watching a few ads here and there, you get to watch TV shows and movies for free (or at least it seems to be).

It's a pretty big tax on your network connection (as the movie is being streamed to you), but it seems to work well.  The video quality is passable (all of you HD people out there might be a bit disappointed, but I've noticed you are all disappointed easily for some reason), and the sound is also passable (you audiophiles can take a look at the HD note and apply to yourselves).  The selection of videos is quite broad (a big woohoo for "The Big Lebowski").

Perhaps I'll update sometime on my trials and travails of Hulu, but I'll probably just forget about it in not too long.  I put future in the title of this entry ... I should probably take that out cause I don't really know the implications of Hulu on the future (as there are probably none).

Recently for work, a few of us were trying to come up with an acceptable time for everyone to get together. With the busy schedules of everyone these days, this seems to get harder and harder, whether it be for work of just getting a bunch of friends together for dinner. Well, during this latest conundrum a co-worker pointed out a cool online tool: Doodle.

This tool allows you to create an easy survey that helps select the best intersection between the options and peoples choices. All you need to do is go to the site, create a poll, and then invite people to the poll and see what happens. Better yet, this doesn't just have to be scheduling, this can be any kind of choice; want to know where to go for dinner, let people decide on their own choices.

So, for a little while now I've been looking for ways to tether my HTC Mogul (Windows Mobile 6 based PocketPC phone) to my laptop for those moments when wireless connections are sparse. With a Windows based laptop, this is trivial, since both ends are Windows. But, what if we are Windows-less (I'll refrain from reasons why I think you should be), or more pertinently, OS X based? Well, this post is my first from my new tethering between my phone and my MacBook Pro, so there does seem to be an answer. I should start talking about syncing, but there's quite a few resources out there to help with this for OS X and WM6. The best/easiest resource I've found is Missing Sync from Mark/Space. Perhaps some other day I'll go ahead and elaborate on my setup (which includes being to sync via bluetooth or USB, even while using bluetooth for tethering), but for now I just want to tether (hey, remember tether-ball ... weird game, but kind of fun). Now, back to tethering. I found a few resources out there claiming how to get this to happen, but most were pretty confusing or didn't quite work. This isn't to say they're wrong, I'm just hoping to be a little more concise here on how to get it to work (just in case I get hit in the head, forget, and need to do it again later). There are a few key ingredients needed for this to work:

• Internet Sharing on the phone (apparently some firmware versions include this, some don't, so mileage my vary). My phone has this listed in the Programs menu from the Start menu.
• Bluetooth, as this really makes everything a whole lot more simple.
• A good data plan on your cell phone.

So, now for the steps (which you've probably scrolled well past all of that text to get here ... I know I would have).

1. Pair your devices -- Now this is where most of the solution is. A lot of people say to start with your phone, and pair to the laptop. This works in general, but doesn't get you to where you really need to be. Instead, start with your laptop, and pair it with your phone. There really isn't much to this, except that when setting up the device, make sure that "Use device as a network port" is clicked. With that, you should be all paired and happy.
2. Set up Internet Sharing -- Open the Internet Sharing application. Under "PC Connection" select "Bluetooth PAN", and set "Network Connection" to whatever your phone uses for it's data service.
3. Get your phone online -- Windows Mobile 6 is a bit annoying in the ways it detects that it's time to get online. The best method is to open Internet Explorer and let that take care of setting up the connection (once it's up, it stays up). When you have that going, head on back to the Internet Sharing application.
4. Start Internet Sharing -- Hit the connect button on the mobile phone. You should see "Device setup finished. On the PC, connect Bluetooth PAN."
5. Connect the laptop and phone -- Under the bluetooth icon, find your devices name, and select "Connect to Network" under the submenu. If this goes well, your phone will say "Connected", and you're now connected.

At this point, if you look at your network connections, you'll see that bluetooth PAN is all nice and connected (with a private network given to your laptop by your phone) and you're off and surfing. From here on out, if you want to tether, use step 5, and to disconnect use step 5 to disconnect. Not the speediest connection, but still nice to have.