Dave's Bit Bucket

"Veni, vidi, Vader"

As you'll already have seen on Alec's blog, Sun UK had "a bit of a fancy-dress party" last Thursday. I own precisely one fancy dress costume - the idea being that, even though my face isn't visible with the helm on, folk I know will very quickly realise it's me - and having suitably kitted-up for the event I managed to pick up a bottle of reasonable bubbly for my trouble, being awarded "best costumed male". Woohoo!

The outfit is seriously hot to wear - the suit isn't breathable, so I left the zip up the back mostly undone so I had a reasonable area through which to try to circulate air. The lenses in the helm both darken and distort the (limited) view out, so it takes a lot of practice to walk around a crowded room without thumping into people. Fortunately, the inner robe and outer cloak do an excellent job of hiding the mess of webbing and straps used to hold all the bits of costume in place. After standing up and walking around in full kit (occasionally minus helm - one of the rooms the party moved between had the lights down, making it impossible to see otherwise), I've probably lost about 3 pounds in weight, but it was huge fun - I've never had so many folk taking pictures of me!

Speaking of pictures, probably the best one I've found of me from last Thursday so far is on Simon's Flickr page - specifically, here...

"Also available for weddings, parties, barmitzvahs, etc..." ;-)

(2007-01-29 01:02:56.0) Permalink Comments [0]

"People, policy, procedure..."

It's often said that "security is 70% people, policy and procedure, and 30% what you do to the computers".

This was brought home to me on the train into London this morning, when I sat next to a young lady who was reading a bunch of documents contained in a bulging lever-arch file. The documents had Crown Prosecution Service header pages on them, and appeared to be all the trial documents associated with a rape case - witness statements, forensic reports, the lot.

Some of them were even protectively-marked as RESTRICTED in the header and footer on each page.

I was able to see all the details - names of plaintiff, accused, witnesses etc - as were some folk who got on at the next station and stood next to us.

Sometimes I wonder why we bother with computer security - on the other hand, I wish I had been able to figure out how to get her details, to pass to someone who could get her clearance revoked...

(2007-01-23 07:52:20.0) Permalink Comments [0]

Thoughts on Disk Scrubbing

The range of views on disk scrubbing - "scrubbing" being the writing of various patterns of data to disks in an attempt to remove all traces of existing data from them before disposal or re-purposing - is as wide as the range of our customers. Some folk are happy with the "purge" mode of the format(1M) command (where available), others write sets of 1s followed by sets of 0s (or vice-versa), still others write repeated fixed bit strings, and others write data from a random number generator. Of course, there are others who do varying combinations of all of the above - 7-pass scrub cycles are not unheard of.

The significant point to remember is that any software-based scrubbing technique is always "an attempt" to remove all traces of pre-existing data. Remanence issues mean that there will almost always be some old data left at track edges, where the head happens not to reach during the scrubbing cycle, so if a Bad Guy was to get physical access to the drive (eg after disposal) he would be able to get something back via judicious use of electron microscopy. If the disks are merely being re-purposed, and their new custodians aren't expected to have physical access to them (such as if the disks are in the same managed datacentre before and after), then scrubbing is of very little benefit.

It's surprising how intensive a process scrubbing actually is. I gather from the grapevine that if you take a fully-stocked 3510 array, it takes over 100 hours to write contiguous "1"s to it from a server running moderately-recent 2GBit Fibre Channel, and several times that to write data sourced from /dev/random. In fact, the real-world customer seeing this is even considering putting an SCA6000 in the server to speed up random number generation!

The thing is, there's potentially another way of looking at this problem. We have a 3510 array to deal with here, which has a decent RAID controller in it. For this - and for our bigger arrays too - why not put it to work? In fact, in extremis:

• blow away the current configuration in the array, putting all the disks to be scrubbed in one LUN
• pick a disk (any disk) within that LUN
• create a volume which has that disk as the principal target, and has all other to-be-scrubbed disks mirrored to it
• using your server, scrub this size-of-one-disk volume and let the RAID controller do the rest :-)

While Your Mileage May Vary depending on the capabilities of your RAID controller (in terms of how many mirrors you can have in a volume - 3500-series kit being happiest doing things pairwise - and the overall performance of the mirroring function within the controller, etc), it seems pretty obvious to me that the time required for a scrubbing process could be cut at least in half by adopting this idea; at worst, you can create a volume of two aggregations (either stripes or concats) mirrored to eachother and scrub that. A hardware RAID controller should be significantly better at doing mirroring than a server, at any rate, leaving the server to get on with the crunching involved with random number generation and shoving the data to the volume.

If our current RAID controllers aren't happy doing massive multi-mirroring, this might be a useful RFE :-).

It's also worth noting that, for RAID controllers which have a "cloning" capability, cloning a scrubbed disk after the fact isn't The Right Thing to do; it would be rather like getting the result of a SPECint calculation versus calculating SPECint 50,000 times and timing it. With scrubbing, the emphasis is on the process, not the result.

(2007-01-22 04:09:43.0) Permalink Comments [0]

It's that "Awards" time of year again...

Following on from Simon's idea of examining the Infoworld winners in his particular area of expertise, I decided to have a look at the winners in the Security section.

The bulleted text below is verbatim from the site (with links added by me):

• Vontu 5.0: Repeat champion Vontu still can't be beat for finding confidential information on file servers and PCs, monitoring network traffic, stopping transmission of restricted data, and enforcing security and encryption policies. Vontu 5.0 sets itself apart with blocking, scalability, finding data at rest, and adhering to privacy safeguards.
• Elemental Security Platform 2.0: ESP 2.0 does an excellent job of identifying and grouping devices attached to your network. Policy deployment is easier than in the previous release, and the new regulatory polices are well-rounded. Reporting gets a massive overhaul with nearly four times as many reports as before, and the ESP server can now scale to 10,000 managed agents.
• Blue Lane PatchPoint G/450: Blue Lane PatchPoint patch proxy appliance is a sorely welcome addition to the IPS field. PatchPoint excels at protecting unpatched Windows, Linux, and Unix servers against many publicly known exploits for popular applications by analyzing network packets and removing harmful bits. It delivers the gains of patching, without the pain.
• SonicWall Pro 2040: The SonicWall Pro provides the best all-around mix of features and functionality among the UTM appliances we've tested. Setup and policy creation are straightforward, and UTM services work across all traffic types in all situations.
• Symantec Mail Security 4.1: Symantec Mail Security offers best-in-class performance with a spam catch rate of more than 97 percent. It is easy to use and has a relatively low cost per user, with compliance and digital asset security features standard. Great policy-driven filtering and e-mail firewalls add value to the anti-spam and anti-virus functionality.
• IronPort C-Series Version 4.0: The IronPort C-Series offers easy setup, strong monitoring and troubleshooting tools, and excellent spam filtering with no tuning necessary. Registering the fewest false positives of any solution we tested, IronPort also provides a wide array of security features, including content filtering for objectionable language, information security, and compliance with legal requirements.

I find it an interesting mix of technologies; Vontu appears to amount to "significant elements of Enterprise DRM without the need for pervasive PKI" and IMHO merits further investigation (particularly with regard to how it goes about distinguishing between sensitive data and other stuff), and Symantec and IronPort are both praised for their anti-spam capabilities (and interestingly, it's IronPort who are particularly praised for their low false-positive rate, which is IMHO the most important element in any traffic filtering solution at the application level).

PatchPoint is a very interesting idea, and while its likely purchasers will be organisations who have not succeeded in getting to grips with system management, there may also be purchasers who have sufficiently mission-critical vertically-scaled systems that they feel they can't afford to fail a clustered service over in order to go patching the nodes (although they would be advised to talk to us if this is so), or have traditional fault-tolerant hardware...

(2007-01-17 07:46:23.0) Permalink Comments [0]

From the Trenches; More Fun with Trusted Solaris

As mentioned a couple of posts ago, I had a good, productive time last Friday; although it wasn't without its sessions of head-scratching. In short, the customer is looking at a traditional SNAP system (this being Sun Ray thin client-serving software and a few other things on top of Trusted Solaris 8), and wanted to arrange things such that, when a user logged in, a couple of copies of rdesktop would automatically start, with each copy being at a different label to facilitate strongly-segregated connectivity to servers on different networks.

Starting multiple copies of a service at different labels is a capability which glories in the name of polyinstantiation - however, Trusted Solaris only makes it nice and easy (using sysh, the System Shell) to do this for server applications which provide a constant service, rather than client applications which start and stop in response to user activity.

So, what's the recipe for such client-side polyinstantiation? Following much-appreciated assistance from my pal Bart (who doesn't blog, hence the absence of a link), it goes like this:

• consider the command you want to issue; in my case, it's "/usr/local/bin/rdesktop", and it's installed at ADMIN_LOW so all users can see it.
• create an executable wrapper script for each label at which you wish to issue the command; in my case, for a two-label system which had a very simple schema of LEVEL_1 and LEVEL_2 as sensitivities, and no compartments, I created /opt/rdp-l1 containing "/usr/bin/pfsh /usr/local/bin/rdesktop app-level1" and /opt/rdp-l2 containing "/usr/bin/pfsh /usr/local/bin/rdesktop app-level2" (app-level1 and app-level2 being servers to which hardwired connections needed to be made, on each segregated network).
• create a rights profile "RDP" in which /opt/rdp-l1 is given a label of LEVEL_1 and /opt/rdp-l2 is given a label of LEVEL_2, and assign the profile to all users who need to use rdesktop
• in order to have rdesktop start when any user logs in (this being what the customer wanted), edit /etc/.login to include "/opt/rdp-l1 &" and "/opt/rdp-l2 &"
• trust to user clearances such that, if a user doesn't have LEVEL_2 clearance, they can't run /opt/rdp-l2 (really, this works just fine)

Granted, editing a .login file isn't elegant. However, I failed to be able to replicate CDE desktop actions across users (and am still not sure why, to be honest), and as rdesktop needs to be started from a shell rather than by an action, any windows it has open at user logout time are not caused to persist for next login. Also, putting commands in $HOME/.dt/sessions/sessionsetc or the global equivalent in /usr/dt/$LANG/ doesn't work on Trusted Solaris, for good reasons I won't go into.

So, what happens is that .login runs at LEVEL_1 when the user logs in, and the pfsh profile shell picks up the label property of rdp-l2 to run its contents at LEVEL_2 correctly. Job done :-).

(2007-01-15 10:20:00.0) Permalink Comments [1]

Thoughts on "Web 2.0 Security"

I saw an internal presentation on Web 2.0 last week - it was mostly a very well-expressed introduction and review of basic concepts, rather than something Earth-shatteringly innovative and radical - nonetheless, it set me thinking.

Principles and Priorities

I think it's fair to say that many Web 2.0 companies have a very different approach to security, and often work to something other than the "conventional set of Enterprise rules".

The primary security-related property that Web 2.0 companies really appear to care about is availability. Service-wrecking attacks, resulting in outages and effort to rebuild, are the scenario everyone dreads most - outages or performance issues are the things most likely to cause a user to leave for good.

Blogs, Wikis and Photo / Video sites

Defacements of wikis and posting of inappropriate material to photo and video sites, or inappropriate blog comments, tend to be dealt with in "Red Army mode" by throwing people at the problem; while this has issues of scalability for Wikis and photo / video sites to a degree (the master editors at Wikipedia seem to keep remarkably well on top of the process of monitoring edits, but something incorrect or offensive can still get up to an hour of "live" time even if it's written in English), the principle with blogs is that the comments are policed by the blogger.

Ultimately, there's a human discipline involved; received wisdom is that "security is 70 percent people policy and procedure, and 30 percent what you do to the computers" and has been for as long as I've been working in the industry, but the Web 2.0 approach to defacement throws the numbers even further in favour of the human element, automatic filtering or not.

Manual blog comment filtering does scale in that "one blog = at least one blogger = at least one human filtering engine", but this only holds true when the blogger operates a discipline of regular comment review. Folk I know who have experimented significantly with automatic comment filtering have anecdotally reported varying degrees of effectiveness of such systems; Bayesian filters work to some degree, but the most effective anti-comment-spam measure appears to be blocking of the string "http://".

Also, while a blogger is required to authenticate to the blog server with at least a username and password in order to post (or delete - unauthorised deletion being a special case of availabiility issues) an entry, the only thing a commenter is (usually) required to do is pass a little test to show that they are a human rather than a machine. This mitigates the risk of "comment spambots", but does nothing to prevent someone damaging someone else's reputation by posting some offensive or daft comment in their name. While Liberty would appear to be an ideal solution to this problem, the matter remains of commenters needing to sign up with Identity Authorities to gain the ability to comment on a blog, and of blogging infrastructure owners forming reciprocal agreements with Identity Authorities to allow Liberty users to make authenticated comments, and somehow distinguish an authenticated comment from an unauthenticated comment when presented on the blog. In the fast-moving, somewhat laissez-faire philosophy that much of Web 2.0 appears to cleave to, this appears "a bit heavyweight".

Mashups

In the world of the mashup, it becomes impossible to verify that the second-hand data presented to the mashup user (eg a map from Mapquest or GetMapping via Google via the mashup to the user) has not been modified or transformed in a way which causes its accuracy to be lost (at least, without attempting the practically infeasible in terms of pursuing globally-pervasive PKI).

This may be part of the reason why so much Web 2.0 software carries a "Beta" designation, no matter how mature the mashup's code is - it hides the fact that people won't pay for something they can't treat as authoritative. Also, where a service relies on another service over which it has no control, a change in the service "further up the chain" could cause things to break unexpectedly at the point of the mashup. To put this pithily, "Beta status is inherited by dependency" :-).

Regulation

Finally, the business model of many Web 2.0 companies doesn't involve making - or indeed, dealing with, to any great degree - money. YouTube's business model was to be bought by a web giant (they made continuous losses every quarter from founding), and Google duly obliged. Any business which doesn't handle financial transactions with its users will have less onerous auditing infrastructure requirements placed upon it by regulatory bodies, and a business which does not need to interact with banks in order to transfer subscription funds, etc will need less back-end interfacing infrastructure.

I suspect this article will be getting several updates as more thoughts turn up on this :-).

Update:

I was reminded over lunch that there's one area I've failed to address; that of activities which seek to promote a website's page rank on Google or an article's ranking on Digg, etc (aka "click fraud"). I need to think about this some more and blog a suggestion or two.

Further update:

There's more. A recent lawsuit (link to be inserted when I find it) involved "misrepresentation as a consumer", where an author masqueraded as a book purchaser on amazon.com to give his own book a glowing review. Unless identities get federated from the outset, this problem will persist.

Also, my pal Efi (who doesn't blog, hence no link) gave a presentation the other day which made me realise something interesting - "Web 2.0 mashups" are almost "cross-site scripting attacks gone legitimate".

(2007-01-15 07:56:40.0) Permalink Comments [0]

Spyware for Symbian

Following a rather triumphant day's work last Friday at A Customer Who Cannot Be Named, I'm catching up on a surprisingly large email backlog.

One of the more interesting items I've been sent involves an apparent full demonstration of spyware for Symbian-based mobile 'phones; the code is apparently able to not only copy the contents of SMS messages and the numbers to which they are sent, but also track which hex the 'phone is interfacing via and - in extremis - open the microphone as an old-fashioned audio bug.

More details can be found here; at least for the time being, as the site doesn't appear to offer a permalink feature. When the site gets updated and this link becomes obsolete, search for an article dated January 12th 2007 by Dani Sinha.

I've heard a lot about the increasing maturity of technologies to track mobile 'phone locations by signal strength correlation between adjacent hexes, but this is the first I've seen about genuine spyware for handsets. Regrettably, the article doesn't discuss the vector by which the software was installed on the handset in the first place; this is something I need to look into further.

Update:

I've heard, by word of mouth from a colleague, that the spyware was manually uploaded to the handset via Bluetooth. It seems that it hasn't been incorporated into a virus at this time (but await some musings I'm writing-up on the history of, and trends in, viruses for more thoughts...)

(2007-01-15 06:23:26.0) Permalink Comments [0]

Happy Anniversary

It's 8 years ago today, that I joined Sun as a Project Engineer, having managed to sneak in under the purview of the graduate recruitment programme (despite being 28 years old and having been working for 5 years).

Much has happened, since. Here's hoping for many more years of mind-bending, wage-paying entertainment.

(2007-01-11 09:04:35.0) Permalink Comments [0]

From the Trenches: a Niagara Jumpstart "gotcha"

Occasionally, things happen which remind me how subtle networking can be, even at the lower levels of the OSI stack.

I was with a customer yesterday, and they were upgrading their nice shiny new T2000 to Solaris 10 Update 3 using JET.

They were having an odd problem with this. The box would do the usual RARP / ARP and tftp its kernel and miniroot over from the server, but would hang just after whoami returned "no domain". Basically, it stalled just before the point where it would report that it was configuring its devices.

After a few 'phone calls to their Jumpstart guru, the problem was found to be that the T2000's network interfaces were trying to autonegotiate speed and duplex at this point; as the switches it was connected to had these properties set statically, the Jumpstart stalled.

The fix is to set these properties statically when initiating the Jumpstart; instead of the familiar "boot net - install", we used "boot net:speed=100,duplex=full, - install".

NB. Note the second, oddly-placed comma after "full" in the line above - it is necessary!

(2007-01-04 03:01:16.0) Permalink Comments [0]

Thoughts on Global Server Load Balancing

As well as doing all manner of stuff relating to Security, I occasionally get to do a bunch of Networking work; as with security, I like to do the whole piece, from requirements capture through design to implementation. I first encountered Layer 4-7 load balancing on Alteon ACEDirector 3s back in the days of WebOS 5.2 - which I figure must have been about 2001 - and found it really rather cool, especially when it came to deciding which load balancing algorithm to use based on the connection and state model of the protocol being balanced.

Later on, when Global Server Load Balancing (GSLB) was introduced, I initially wondered what the reasoning was behind it, then figured that there were all sorts of shortcomings with it which added up to a verdict of "avoid at all costs", then eventually realised that there's one or two cases where it's actually The Right Thing to do. I occasionally hear from various customers who are part-way down the path I've taken, so I figured I'd get some thoughts down on what GSLB is, when you should use it, and when there are better ways to solve the problem.

Why Customers Want "GSLB"

Back in the dotcom days, we'd get folk coming to us with words to the effect of "here's my $10 million in venture capital, build me a resilient service infrastructure to present my creation to the world at large".

We delivered on their requests, many times, including disaster recovery (DR) environments for those whose industry requirements stipulated them, or those who simply wanted them.

Then, when folks' money started getting tight, their requirements shifted a bit. They came back to us, saying "I've got this DR site, sitting there soaking up power and rack rental space, and while my main site is running normally, this other site is not actually doing me any good. How can I put the kit there to work, delivering my creation to the world such that it can still pick up as the DR site if the main one goes down? Oh, and if this site is in another country from the main one, how do I do this without having to put a huge expensive link between the two?"

Thus was GSLB born; the first time I came across it was in Alteon Web OS 8, probably around 2002-3. The following year, pretty much all the load balancer manufacturers had an implementation.

Taxonomy of GSLB

Most GSLB implementations work by exploiting the fact that queries tend to arrive at infrastructures based upon client resolutions of service addresses by DNS, rather than explicit references to IP addresses. Switches performing GSLB across the various datacentres are set up as the domain's primary external DNS servers, where DNS services themselves are typically backed-off to a pair of load-balanced DNS "slaves" hidden from external view. The switches handle DNS requests from clients, and use round-trip time (RTT) timing to determine whether a given client is closer to one GSLB-equipped datacentre than another.

In the event of a load-balanced service failing at a datacentre, where the virtual IP address (vip) associated with the service is presented via the external DNS, the GSLB system in the switches resolves the DNS-advertised address to the other site's vip.

Also - and this is the main reason why folk wanted GSLB - if multiple sites are up, the GSLB switches can compare their RTTs to the client's DNS server, such that the site with the lowest RTT is the one to which the service request is pointed. Thus, we get GSLB as "active-active bandwidth-weighted load balancing by DNS", which fulfils the customer requirements of "making the DR site do useful work".

Implicit Assumptions of GSLB

Perhaps the most significant implicit assumption in GSLB - which Alteon later claimed to have figured out a workaround for, although I never quite sorted out the nature of this workaround in my head - is that the client's DNS server (which contacts the GSLB-presented DNS address) is located logically close to the actual client trying to access the service being presented. By "logically close to", I mean "sufficiently far away from the GSLB infrastructure itself, in terms of hop count and link bandwidth bottlenecks, that the speed of the effective aggregate link between client and infrastructure is the same as that of the effective link between the DNS server chain the client is using and the infrastructure".

This assumption isn't always correct, especially when you consider that DNS is hierarchical and thus the server which makes the request of the GSLB-presented DNS service may be some levels removed from the client. Also, the "logical closeness" assumption doesn't wash too well when you consider entities such as the AOL mega-proxy.

When, and When Not, to Use GSLB

GSLB works best when you have multiple independent datacentres which you want to have doing useful work, such that the bandwidth of any existing link between them is small (assuming that GSLB negotiation traffic is light when compared to live service transaction traffic), and the datacentres do not need to perform back-end synchronisation or multi-phase commits between them.

This qualifier is what has relegated GSLB to the small niche it now occupies. While http and many other protocols are stateless, the transaction data they carry very often isn't. In the event that a non-read-only transaction was performed with one GSLB site and that GSLB site subsequently went down, you'd want the other site(s) to have a record of the data written in the transaction. This usually results in back-end databases at each site needing to either do regular synshronisations or multi-phase write commits, at which point it's frequently the case that the bandwidth of the links between the sites needs to be raised to the point where, rather than use GSLB, you might as well do regular active-active load balancing. While active-active is rather trickier to set up and maintain than active-standby, it's still simpler than GSLB and has the advantage that it's easier to weight the distribution of load across your sites, to cater for any differences in hardware performance between them.

Conclusion

So, in the (relatively unlikely) event that your services are read-only and the bandwidth between your datacentres is small, go right ahead and use GSLB. For all other circumstances, there's usually a better way to approach the problem, based on the fact that there's more inter-site bandwidth available.

Update:

There's actually a couple more points worth considering. First, if you're going to avoid GSLB, it's often useful to source the uppstream links from your datacentres from the same supplier, in a manner such that any vips you need to fail over between datacentres are in the same upstream subnet. Second, if you are going to go down the route of GSLB, beware the latency involved with DNS map updates; while a GSLB device will readily do a map "push" to its upstream server as part of a failover event, you are at the mercy of the "ripple carry" latency involved in reaching the client-end DNS server, before a client will be redirected to the live site.

(2007-01-02 07:50:07.0) Permalink Comments [2]