Friday March 06, 2009

Good Bye, Sun. Thanks for 4 great years. Today was my last day at Sun. I'm leaving and heading to Carpathia Hosting, Inc. Carpathia is a small hosting company that's growing fast, and the lure of getting in early at what is essentially still a startup environment despite having been in business for a number of years. My job will be pretty much the same, as I'll be a Senior System Administrator. I'll be working with several former coworkers from 3 of my past jobs. I'm excited about the opportunity, even though I'm sad to leave the truly extraordinary team that I'm leaving behind. Not much else to say. Have a good weekend, I guess. Posted by tkblog ( Mar 06 2009, 09:48:14 PM EST ) Permalink Comments [1]

Friday November 07, 2008

wordle analyzes my drivel Interesting. A friend of mine tipped me off to the Wordle analysis he did of Jonathan's Blog. I thought it looked neat, so I ran it on my own blog. Here's what it found:

Friday May 16, 2008

diskread: reading beyond end of ramdisk (& how I recovered)

We had to do a maintenance to replace a NEM module in a Sun Blade 8000 Modular System.

Two of my team mates went on down to the datacenter on other business and graciously offered to SWAP the NEM for me. The pulled the old one out, stuck the new one in.

That's as simple as it should have been.

Should have been. I wish. Instead, the chassis started to freak out, cycling it's power over and over, and somehow was taking the CMM with it. In between one set of cycles, I was able to connect to the CMM via console and paste in a bunch of commands to shut down chassis power. I let it sit for a moment, then began to power up the system. First the chassis, then the individual blades. One blade came up, no problem. The next two, though, were very much less than happy, spitting out errors like:

diskread: reading beyond end of ramdisk
	start = 0x2000, size = 0x2000
failed to read superblock
diskread: reading beyond end of ramdisk
	start = 0x2000, size = 0x2000
failed to read superblock
panic: cannot mount boot archive
Press any key to reboot

The GRUB menu was coming up OK, though, so I pressed the trusty any key, booted into Solaris 10 Failsafe mode. This was no picnic either.

SunOS Release 5.10 Version Generic_120012-14 32-bit
Copyright 1983-2007 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Booting to milestone "milestone/single-user:default".
Configuring devices.
Searching for installed OS instances...
NOTICE: /a: unexpected free inode 5825, run fsck(1M)
/dev/dsk/c2t0d0s0 is under md control, skipping.
To manually recover the boot archive on a root mirror,mount the first
side (the one that the system boots from) and run:

        bootadm update-archive -R 

umount: /a busy

No installed OS instance found.

Starting shell.
#

My immediate thought was "WTF? No installed OS instance found?" Closer inspection revealed that it had in fact found two possibilities, but one c2t1d0s0 was inconsistent and needed a fsck, and the second c2t1d0s0 was under md control, and so being skipped.

An fsck of /dev/dsk/c2t0d0s0 revealed a few inconsistencies. Here's an example. I think this was actually the 3rd of 4 fscks I ran on this dev:

bash-3.00# fsck /dev/dsk/c2t0d0s0
** /dev/rdsk/c2t0d0s0
** Last Mounted on /
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3a - Check Connectivity
** Phase 3b - Verify Shadows/ACLs
** Phase 4 - Check Reference Counts
UNREF FILE  I=1457  OWNER=root MODE=100644
SIZE=657 MTIME=May 15 18:01 2008
RECONNECT? y

UNREF FILE  I=1458  OWNER=root MODE=100644
SIZE=675 MTIME=May 15 18:06 2008
RECONNECT? y

** Phase 5 - Check Cylinder Groups

CORRECT BAD CG SUMMARIES? y

CORRECTED SUMMARY FOR CG 0
FRAG BITMAP WRONG
FIX? y

FRAG BITMAP WRONG (CORRECTED)
CORRECTED SUMMARY FOR CG 4
CORRECTED SUMMARY FOR CG 12
CORRECTED SUMMARY FOR CG 30
CORRECTED SUMMARY FOR CG 70
CORRECT GLOBAL SUMMARY
SALVAGE? y 

Log was discarded, updating cyl groups
46737 files, 1720899 used, 24099860 free (21460 frags, 3009800 blocks, 0.1% fragmentation) 

***** FILE SYSTEM WAS MODIFIED *****

So far, so good. Let's reboot, and see if we an come up in a multi-user state. So ... reboot ... wait ... wait ...

CRAP! Same panic as our previous boot. We're missing something. A further delve into google reveals that I need to recreate the ramdisks for boot. A boot into failsafe mode again, allows me to fsck c2t0d0s0, which is mounted on /a, and remount it -o rw. bootadm update-archive fails, due to fs inconsistency. Another fcsk, we're in single user, nothing is using that disk, so I just ran the fsck without remounting -o ro. Now, let's skip bootadm and just move straight along to /boot/solaris/bin/create_ramdisk.

bash-3.00# /boot/solaris/bin/create_ramdisk -R /a
Creating ram disk for /a
updating /a/platform/i86pc/boot_archive...this may take a minute

That's it! That's the little piece of magic that fixed it. After that, I was able to reboot, and the server came right up into runlevel 3. Not without a few minor errors, but at least it was up.

SunOS Release 5.10 Version Generic_127112-11 64-bit
Copyright 1983-2008 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Hostname: generic
NOTICE: /: unexpected free inode 9193, run fsck(1M) -o f
NOTICE: /: unexpected free inode 5961, run fsck(1M) -o f
WARNING: /: unexpected allocated inode 9637, run fsck(1M) -o f
Loading smf(5) service descriptions: 1/1
/dev/md/rdsk/d60 is clean
/dev/md/rdsk/d30 is clean
/dev/md/rdsk/d20 is clean

generic console login:

At this point it was pretty simple to complete the fix, which, not wanting to reboot into failsafe mode and fsck a bunch more to recover from the unexpected free and allocated inodes, I wrote a script to: by turns, detach each have of the root mirror, clear the detached metadevice, newfs the raw device, re-create the metadevice, and attach it once again to the mirror. Let it sit long enough to complete the resync, and repeat the same steps on the other half of the mirror.

#!/bin/sh
#
# fix-mirror.sh
#
# 05-16-2008 Tim Kennedy 
#
# This script will take one argument, which should be the 
# metadevice of the mirror you want to rebuild.  This script
# will determine the Submirrors, and one at a time, detach,
# clear, newfs, re-init, and reattach them.
# For me this has solved problems with ailing filesystems,
# while replacement storage is procured.
#
# YMMV.  Use at your own risk.  This is not in any way to
# be considered a Sun Microsystems product, and is not in
# any way supported by Sun Microsystems.
#
 
PATH=/usr/bin:/usr/sbin
export PATH
 
MIRROR=$1
 
check_return () {
        RETURN=$1
        if [ $RETURN = 0 ]; then
                printf "%-6s\n" "[ok]"
        else
                printf "%-6s\n" "[err]"
                echo 
                echo "please check the last step manually to see why it failed."
                echo
                exit 1
        fi
}
 
for m in `metastat $MIRROR | grep "Submirror of $MIRROR" | cut -d: -f1`; do
        echo "Found Submirror $m"
        DEVICE=`metastat -p $m | awk '{print $NF}'`
        printf "%-72s" "    -- metadetach $MIRROR $m"
        metadetach $MIRROR $m >/dev/null 2>&1
        check_return $?
        printf "%-72s" "    -- metaclear $m"
        metaclear $m >/dev/null 2>&1
        check_return $?
        printf "%-72s" "    -- newfs /dev/rdsk/$DEVICE"
        echo y | newfs /dev/rdsk/$DEVICE >/dev/null 2>&1
        check_return $?
        printf "%-72s" "    -- metainit $m 1 1 /dev/dsk/$DEVICE"
        metainit $m 1 1 /dev/dsk/$DEVICE >/dev/null 2>&1
        check_return $?
        printf "%-72s" "    -- metattach $MIRROR $m"
        metattach $MIRROR $m >/dev/null 2>&1
        check_return $?
        printf "%-72s" "    -- checking resync status before continuing "
        while [ 1 ]; do
                STATE=`metastat -c $MIRROR | head -1 | grep resync`
                if [ "x${STATE}" = "x" ]; then
                        printf "%-6s\n" "[ok]"
                        break;
                else    
                        sleep 60
                fi
        done
done

Now these blades are happy once again. We'll see how long that lasts or if they continue to have problems of any sort. My hope is for the former.

Have a good weekend.

Posted by tkblog ( May 16 2008, 10:46:57 PM EDT ) Permalink

Saturday March 22, 2008

solaris 10, sun cluster, zfs, and zones not creating devices

I had this strange problem where zones weren't properly creating devices. This is on a newly Jumpstarted Solaris 10, Sun Cluster 3.2, cluster. The strange part is that we have 8 other clusters that all run zones, and none of them have the problem. I thought it might have been something wierd with ZFS, but other zones on ZFS roots can create devices.

As it turns out, the Jumpstart server that we built this cluster with is not as 'up to date' as our 'main' Jumpstart server, and was not patching new builds up to the latest kernel patch.

Updating the system with kernel patch 127111-11 resolves this issue. This patch patches a ton of stuff.

The specific bug that hit us was bugID 6608977 can't add device to non-global zone in S10 update 4.

That bug was actually fixed in 127111-03 but 127111-11 fixes a ton of stuff.

Monday October 08, 2007

Parallel trends in hardware, software, and services

There is *A LOT* of content here at CEC 2007. There are hardware groups presenting on trends in processor technology like CMT (Chip Multi-Threading) like the Niagara Processors. In some ways Sun's Niagara processors consolidate entire systems onto a chip. They certainly consolidate multiple processors into a single chip. There are Software groups presenting on a variety of consolidation and virtualization techniques like Logical Domains, Containers (Zones), and Virtual Machines (VMware and Xen xVM). This would allow you, for instance, to consolidate 10 single purpose servers that spend most of their time less than 10% utilized into a single multi-processor (CMT) system, with ten zones (or more), that would allow for one server that's closer to 100% utilized, to replace those other ten older systems. Then we have Services groups presenting on new services like Data Transport Services that will allow us to modernize old services and develop new services, on a common framework that would provide value-add services like consolidated registration services, consolidated authentication and authorizations services, and the previously mentioned data transport services, out of the box.

It seems that the whole premise of these 3 parallel trends in hardware, software, and services is all to abstract each layer from the others. It's all part of Redshifting and Blueshifting. Abstracting the hardware layer from the application layer (through the OS layer and virtualization) we can expand our support for Redshift applications. Those applications that are growing faster than single purpose servers can continue to support them. We can make many systems appear as a single consolidated system to the OS and application, allowing for horizontal scalability for redshift applications. Those same developments in hardware and software (particularly the OS) allow us to support Blueshift applications just as well, by leveraging a different perspective of the same stack. Instead of making multiple systems appear as one, we can make one system appear as many, though the use of virtual machines, logical domains, and containers. Then we can take applications and systems that typically underuse systems, and combine them, onto a common platform, without them even knowing it.

It's an exciting time to be in IT, if you're the sort that enjoys always being challenged, and always feeling like you can never keep up with enough of the developments out there.

Tuesday July 31, 2007

OUCHIES! I broke my big toe this morning!

I can't beleive I broke my big toe. Not, I think I broke it. I know I broke it. I went to the hospital and had them X-RAY it. It's broke!

I was carrying my son (15 months old) down the steps, and I slipped. My only though was "Don't let Jason get hurt."

So I grabbed him and wrapped my arms around him, as my left foot missed a step, and my right foot slipped off the step it was on, hitting the step below that toe first, which toes consequently folded underneath that foot at the same time as they became the primary weight bearers for all 250 lbs of me.

Jason was not hurt. I think he was scared that daddy was screaching like his 19-month old cousin Jade when they're fighting over a toy (actually he's the screacher, not her), but he was fine.

Here's a pic of the X-RAY:

After I hurt myself, I took about 5 minutes to gather my wits, then I took Jason to daycare, and drove
myself to the hospital, which is quite pleasant at 8:45 AM.

A few X-RAYS, and a silly post-op shoe later, and here I am on a diet of Advil and ice-packs. Hopefully the bones won't need to be pinned in place, but I won't find out till the end of the week, when I have a follow-up with the orthopedic specialist.

Friday July 27, 2007

IRS Loses In Court On Income Tax Legality!

The Internal Revenue Service has lost a lawyer's challenge in front of a jury to prove a constitutional foundation for the nation's income tax, and the victorious attorney now is setting his sights higher.

The jury in U.S. District Court in Louisiana voted 12-0 to find Cryer, of Shreveport, not guilty of failure to file income taxes for two years. He had been indicted in 2006 on charges of failing to pay $73,000 to the IRS in 2000 and 2001. The next step in his personal case will be up to the IRS and prosecutors, if they choose to continue the issue, he said.

"There are three points that are important," he told WND. "There's no law making the average working man liable [for income taxes], there's no law or regulation that allows the IRS to contend that earnings are 100 percent profit received in exchange for nothing, and the right to earn a living through any lawful occupation is a constitutionally protected fundamental right, and it is exempt from taxation."

Here's the article at Digg.

Maybe this will promote the idea of a flat tax!

Thursday July 05, 2007

T-Mobile announces Hotspot @ Home service

I was just reading this story about a T-Mobile announcement that got lost amid the iPhone hub-bub.

It seems that T-Mobile may have found a way to compete on some levels with the likes of Skype. For the grand old introductory price of $10/month, or $20/month for family plans, you can add Hotspot@Home to your service. When you do this, your cell phone calls will hand-off to wifi hot-spots when you're in range of a wifi network, seamlessly no less. And vice versa when you move from a wifi coverage area to a cellular coverage area.

Pros: Wifi calls are FREE. Who ever heard of a phone company doing anything for free?
Calls initiated on wifi are free, even if you later hand off to the cell network!

Cons: Lack of selection of phones. That's a big con for me. If it worked with my T-Mobile DASH,
I would sign up today, and never look back.

It's an interesting start though.

Wednesday June 20, 2007

Using Solaris 10 Update 3, Sun Cluster 3.2, Zones, & ZFS in a Multi-Node Cluster of Sun Fire T-2000s

It all started with a conference call with one of our customers. We wanted a way to set up some highly available systems that could be used for various beta or QA purposes, or production services, or anywhere in between as needed. We also wanted a way to maximize the resources. We had 4 servers available to us, all Sun Fire T-2000s. If we used them as straight servers, they'd be great at anything they do, right? 8 cores, 4 threads per core, 32GB of RAM. Nice. Capable of running dozens of zones without skipping a beat. Perhaps even hundreds of zones.

Zones make perfect development boxes, right? You can blow them away and re-install them in a matter of minutes, or even seconds on ZFS. Zones also do pretty good as production environments as well. We're currently using a large number of zones in production, to supply a variety of services.

Zones on ZFS make particularly good dev boxes because you can take frequent snapshots and roll back as desired.

** Zones with their zoneroot on ZFS do encounter bug #6356600, which relates to how the live upgrade scratch zone used for installing packages into local zones can't access ZFS filesystems to upgrade zones with a ZFS zoneroot.

Sun Cluster 3.2 introduced support for ZFS as a failover filesystem, and for failover zones as well. We decided to make use of both of these features.

We built a 4-node cluster out of the 4 T-2000s, and began exporting individual disks from our SAN. Put 3 disks in a ZFS pool as a raidz filesystem, and installed the zone root at a ratio of 1 zone per zpool. (We're still doing some testing with our SAN and comparing performance of ZFS on individual disks, or ZFS on a RAID5 LUN exported by the SAN, but so far the way we're doing it is working nicely.)

So we built the cluster, and got it all configured and running. We then installed the first zone onto the ZFS pool. Then I copied the relevant portions of the zones configuration (in /etc/zones/*) to the other nodes in the cluster.

We then created a resource group in Sun Cluster 3.2, and added the zone into the resource group. We also added the ZFS pool into the resource group as an HA-Storage resource, and created a quick set of control scripts to start and stop the zone. The zone itself takes care of bringing up it's ip addresses, and starting the various applications installed within.

End result: Highly available servers, on a failover basis, that take less than 30 seconds to fail over from one host to another.

So far it's working really well. We're already getting more requests to build more of these multi-noded clusters, with zone/zpool combo's as the resource group. It's been a great solution for us.

Thursday June 14, 2007

2 birds with 1 stone. The IDstride pedometer badge holder. What a great thing.

http://www.idstride.com.

Just what I need. A badge clip with a built in pedometer.

I'm big on consolidation. Pretty much any time I can consolidate two or more of the things I find myself carrying around on a regular basis, I take the opportunity. Last year I consolidated my PDA and my Cellphone into a T-Mobile Dash, which has all the functions of both items that I use. It also has really good reception, so that's a plus.

I've already got a badge holder. Doesn't everyone?

I've already got a pedometer, but it's just one more think to connect to myself. I already feel like I need a Batman Utility belt. With a phone, a bluetooth headset (when I'm not wearing it, and I only wear it in the car because I think people look stupid wearing their bluetooth headsets in public places), a swiss army knife (the Cybertool 34), a badge, two secure access tokens, wallet, two sets of keys, pedometer, and assorted pocket sundries (rubber band, paperclip, lint, etc), I'm full up.

I haven't accepted a Murse as an option yet. My brother in law carries a man bag. It's like the one Indiana Jones carried. On him it's a manly masculine accessory, so I may have to start thinking about it.

After watching "You on a Diet" with Dr's Oz and Roizen (from Reader's Digest, and Oprah), I've been trying to keep track of how far I walk each day. They recommend at *least* 10,000 steps per day. I've had my pedometer for about 2 weeks now, and wanted to spend a month seeing what I do now, without augmenting my routine with any additional exercise.

So far I seem to be averaging only about 4500 steps per day. Pretty sad.

I think I'll pick up an IDstride, though. Maybe if I consolidate 2 items into 1 item that weighs less than the original 2 items, I'll feel more like walking. :)

Wednesday December 13, 2006

How to buy your wife or girlfriend a present they'll like

Yesterday I was watching the Today show on NBC, and Jean Chatzky showed how little she knows about the inner workings of men, and simultaneously provided great insight into women, and a problem that most of us men have experienced: What do you buy your wife or girlfriend for Christmas?

First off, she said that women like to surprise men. They like the feeling of suspense when they buy us something they know we'll like. She seemed surprised to hear that men would like some big bold neon sign -like hints as to what a women wants for Christmas. She thinks it should be obvious to men what women like, like it's obvious to women what men like. Cute, eh? Unfortunately for her, men's brains are not wired that way. We're wired to look for meat.... hunting, fishing, computing. We're wired for taking things apart, rarely for putting them back together.

She then went on to ask, like it's obvious, "Why don't you just go to your wife's closet, look through her clothes, and look at the tags. You'll get the brands we like, you'll get our size, and you'll get an idea of the styles we like. Or notice what perfume we wear, and buy us the whole line. We'll be happy."

I sometimes wonder at how the male brain is wired. None the less at why I never thought of that. Why in all the years that I've been married, or previously, had a girlfriend at Christmas, have I never thought to look at the brands and size of her clothes, or at her perfumes, to buy her a gift? Is it because men are terrified at the thought of buying clothes for a woman? Maybe, but with the right brand, style, and size at hand, it's much less terrifying a prospect.

Thank you Jean Chatzky for getting on national television and giving the men of America (and beyond) a big, bold, neon sign -like hint. :) (yes, we really do need it!)

Here is her full article on the subject. It has a more big bold hints, like "Talk to her friends", and "Don't over look electonics", and "Kitchen appliances are not sexy gifts".

Now I feel much more confident about the Christmas present I'm getting for my wife.

Monday November 06, 2006

In 2004, 53% of Eligible Americans Voted

Please Vote tomorrow.

Vote for change, or vote to keep things the same. Just get out and vote. Take a sick day or vacation if you have to, or vote on your lunch break. It's worth it, really.

Think of the last Presidential Election. Pretty contested, right? Not how you think.

Only 53% of the eligible voting population actually got off their duffs and voted. That means that 47% of the population either beleives that voting makes no difference, or that there's not a significant difference between Bush and Kerry. No matter what... that's the real poll that we should be worrying about. That almost half of the population

Dubya was elected by a majority of 50.7% of the vote in 2004.
Out of 55.3% of the eligible population that turned out to vote.

In 2000 he was elected by the Supreme Court a 47.9% majority.
Out of 51.3% of the eligible population that turned out to vote.

Bubba was elected by a majority of 49.2% of the vote in 1996.
Out of 49.1% of the eligible population that turned out to vote.

He was elected by a 43% majority vote in 1992.
Out of 55.1% of the eligible population that turned out to vote.

Even less people vote on the off years.

Like in the 30-40% range.

Please vote. And continue to vote. Tell your friends to vote.
Vote Green, Vote the Donkey, Vote GOP, Vote Libertarian.

Excercise your Freedom and Vote.

Don't overlook the simple answers!

Today, I spent a good part of the day troubleshooting an Oracle 10g database who's db_recovery_file_dest kept filling up. Now, I'm not a DBA, by trade, just a technical generalist with a penchant for Googling. I increased the size of the db_recovery_file_dest, and 4 hours later, it was full again. I could not for the life of me figure out why the archiving and log rotation RMAN scripts weren't working. I ran them manually, and voila! problem fixed again, for a limited time.

That's when it occured to me to look in /var/cron/log. Sure enough, I found the answer to all my problems. Well, not ALL of my problems, but enough of the ones I was dealing with today that I rated today a success.

The oracle user's password had expired.

That was it. The root cause of two database outages due to the recovery log destination filling up, and the database refusing connections, and hours of troubleshooting.

An expired password.

This brings me to a lesson I know well, but often forget.

Never overlook the simple answers!

Sadly, I often forget that, and try to solve a complex problem that's not complex.

Thursday September 14, 2006

SCARY: Diebold AccuVote-TS Voting Machine Security Analysis

Security Analysis of the Diebold AccuVote-TS Voting Machine

From the site: "This paper presents a fully independent security study of a Diebold AccuVote-TS voting machine, including its hardware and software. We obtained the machine from a private party. Analysis of the machine, in light of real election procedures, shows that it is vulnerable to extremely serious attacks. For example, an attacker who gets physical access to a machine or its removable memory card for as little as one minute could install malicious code; malicious code on a machine could steal votes undetectably, modifying all records, logs, and counters to be consistent with the fraudulent vote count it creates."

See the whole thing here.

Then, once you've watched it, write your congressman or senator or something.

Wednesday September 13, 2006

$800 for a Tivo Series 3? Without Service? Really?

We have an HDTV at home. More precisely we have an HD-Ready TV. A nice Sony 57" rear-projection monstrosity that really makes itself apparent in the living room of our tiny condo. But my gadget-hound wife got a good deal on it, and actually bought it before I met her, which may have contributed to the marriage itself. :D

We have not, however, had an HD tuner for the TV. Last year we decided to get the HD package from COX Communications here in Fairfax VA, and voila! we were finally able to watch HDTV on our HD-Ready TV. What a beautiful picture. With the Olympics, it felt like we were there, in some super-VIP box, where we could reach out and give the athletes high-fives when they won, the picture was that good.

Earlier this year, after COX had announced they had an HD-DVR available. The HD-DVR is the Scientific Atlanta HD DVR (HD8300). We got it home, plugged it in, and WOW! We thought it was great....

...for about 30 minutes. Then we started getting into the nitty-gritty finer details of setting up recordings, and setting up our preferences on the box itself....

... and we realized that although it had a couple of good features, the cons greatly outweighed the pros. And it's not like the cons were minor... at least for us. Most of the cons are deal-killers all by themselves. None of the pro's are deal-makers. Not even all together.

So we dumped it back in the hands of COX, and went back to a regular cable-box. Not even the HD-box. We'll go back to HD when there are some better options for HD-DVRs.

And for me the Tivo Series 3 doesn't cut the mustard. $800.00? Really? The best deal available is the $800 Tivo Series 3, with 3 years pre-paid service for $300.

And that's just for the Tivo. Each Tivo Series 3 is capable of hosting 2 cable cards. Cable cards from COX Communications in Fairfax are $1.99/month according to their website. What their website doesn't mention is that, according to the COX rip-off artist representative I spoke with, each cable card also requires a device called a "Gateway". Each "Gateway" costs $5.99/month. And there is NO mention of that anywhere on the COX - Fairfax website that I could find.

That's $8.00/month! PER Cable card.

For less hardware than a cable box, you pay 2x-3x as much. It doesn't make much sense to me, but I suspect it has to do with the lost advertising revenue due to people who don't see the Interactive Channel Guide, and lost pay-per-view revenue.

So, to recap, the average monthly cost for me to replace 2 cable boxes and 2 ReplayTV units would be:

Or, I can stick with my current costs, averaged over the next 3 years as is:

Even $75/month for is crazy, but that's the rule of supply and demand. And they have all the supply here in Fairfax. Leaving me with only demand. That's not very good leverage.

It's better than doubling my costs for the next 3 years, just to have a great picture and really cool gadgets though.