Random footnotes of Engineering

Map reduce is getting a lot of publicity these days. I came across an article in Harvard Business Publishing that specifically described this as Google's secret weapon. This problem set seems to fall squarely in the realm of what people want to solve with cloud computing.

In the same vein there is a different project, Hadoop that is attacking this problem as an Open Source project. I came across another article that talks about Hadoop and how it works. We even have an OpenSolaris Project that provides a live Hadoop CD.

So fundamentally these technologies are interesting when we have problems that are inherently separable and parallelisable. An common example is indexing text files. Hadoop grew out of the Lucene project which does that sort of thing.

Anyway, I'm going to take some time and look into this because it all seems very cool.

I'd been working on this blog entry for a while, but it seems that Jonathan beat me to the publish button. He is covering something more XvM specific, I've included what I was working on below just for some comparison...

We hear a lot about "Cloud computing" all over the place. Often it's used to refer to to web 2.0 type resources that are out there. Things like Amazon S3 or Google's multitude of applications. What I'd like to discuss today is some thoughts I've got around how a storage "cloud" would look like inside a data center. What are the interesting pieces to the problem?

• Horizontal Scalability
  
• Managability
  
• Reliability
  

This is likely to be a gross oversimplification, but heh, that's never stopped me before! A big requirement I've heard from a number of different places is that people want to store their data, they don't particularly care where, and just know when they come back to ask for it that it will be there. A key consideration for these folks is that the storage system is easy to manage, the fact that they don't care about anything other than if their bits are there implies that they would like as close to no required management as possible. The fact that they want their bits to be there when they come back to ask for them implies data reliability to me. It also seems implicit that as the storage requirements grow the horizontal scalability needs become drastically apparent.

So why do I raise any of this since it seems incredibly obvious? I think that we need to think about the way we approach building a 'Storage Cloud' within our customers data centre. There will be a lot of folks who aren't prepared to trust a cloud out there on the internet somewhere. If we can come up with solutions that create these clouds in the data centre we'll be onto a winner.