Chalfant's Blog

Follow the Money

Most people remember the manta of Jerry McGuire – Show me the Money! When it comes to storage, customers would be wise to 'follow the money'. Everyone in the vendor community understands the insider quip regarding sales people – they are said to be ‘coin operated’. While this is a friendly jab at the sales people, it is also fairly true. Sales managers know that when you build a compensation plan for the sales force, you have to be really careful. Sales people will figure out how the plan works and do what it takes to best leverage that plan.

Business is very much the same. I was recently out on a sales call where I was once again reminded that businesses will also do what they are compensated for. Here is the situation.

About a year ago, I was a speaker at a large golf tournament. Everyone that has seen me present knows to expect brutal honesty and some sound advice on how to bring their operating infrastructure into alignment with the needs of the business. And so it was at the event a year ago. One long term customer heard some new things we are doing which involves an assessment that peals back the layers of complexity in the data centers, and exposes how inefficient storage and servers can be. They decided to give it a try, and so we contracted to perform a business value assessment.

The interesting thing about this engagement is that this customer has multiple data centers. One of their data centers is managed by a service provider, which has lots of disk from one vendor in particular. Our team showed up at that data center with tool kits in hand to begin the collection process which leads to an irrefutable exposure of how much space is commonly wasted. The cool news is with this in hand, the customer can cut their costs in a big way, because they can re-allocate and use space that is otherwise hidden, undiscovered and unused.

Only one problem, the service provider and the disk vendor get compensated based on how much capacity they use. If we go in there and uncover how much space is wasted, and can reclaim it, the service provider and the disk company have a major revenue hit. They don’t like that. So instead of inviting the team in for the greater good of the customer, they blocked the doors and wouldn’t allow any work of any significance.

Now interestingly enough, I have met with this service provider, over and over and over. They are afraid the approach we tout will make them loose business. I have told them this is an ‘eat your own children’ problem, and that if you don’t – someone will do it for you. But – in a time when corporate pressures push revenue and operational efficiency at the same time, it seems impossible to change the business model enough to not only make a reasonable profit, but also do what is right for the customer. So, at the end of the day they blocked the effort.

Well the day of reckoning may be on the horizon for them. I had a meeting with that same companies CIO today, and exposed the game that is played by vendors pushing capacity (as a value for their own top line growth, from the expense line of our customer). I showed him and his technical team how to expose it, and what to do about it. Right in the middle of my presentation he stopped me and asked if we knew the guy that is in charge of this service provider locally. He said he wanted to get up ‘right now’ and bring him into the meeting to find out what they were doing.

Follow the money. If your service company and your vendors are comp’ed to sell capacity, they will sell capacity. What you need to figure out is how to comp them for selling the minimally correct amount of infrastructure that allows the business to operated at peak performance. Period, full stop.

I love the awakening I see going on. Business leadership knows fair well that the IT infrastructure is critical to business. They have suspected for years that there is a better way to align the needs of the business with a balanced implementation of the infrastructure. Now that the tools, technologies, and consulting practices are in place to deliver that, the game is up.

Fair warning to all vendors that are over selling for personal gain, the game is drawing to an end.

Posted at 08:27AM Jun 27, 2007 by chalfant in Personal  |  Comments[2]

Disk only Solutions are Suffering from Atrophy

It seem only fair that if the disk only companies insist on propagating the ridiculous assertion that tape is dead, that I can launch my own ridiculous campaign, also with a medical metaphor.

Here it is – Disk only solutions have Atrophy.

Like human Atrophy, Disk only solutions are the partial or complete wasting away of a part of the disk. Causes of atrophy arise from disuse, mismanagement, or lack of exercise.

This results in a visible bloating of a data center, and has long been feared and shunned by senior managers as a dangerous to the overall health of their organization. Even though this condition can be effectively treated today, the disease still causes widespread anxiety. As a result, organizations suffering from Disk Atrophy often suffer psychological and social problems. Vendors will offer free tickets to major sporting events to help minimize the impact, but when IT professional return, the Atrophy has grown. Sad really.

Yesterday I mentioned that 70% of disk space is wasted on average. Let’s take a closer look at that.

Typically there is far more storage available to users than they appreciate or understand. Sun completed a study performed at 200 user sites globally. In the study, tools were installed to uncover the allocation and utilization efficiency. The results were surprising.
• 10% of all the capacity found had once been allocated to a server, but the server had gone away for various reasons, without the space been returned free. We define that as allocated but orphaned.
• 15% of all the capacity was allocated but unused. Typically someone had requested space from a storage administrator, but no file system or file was stored. It can also be represented as unused database table space.
• 5% of the space is inappropriate use. This is typically unstructured data commonly JPEG, MPEGS, WAV, AAC, etc., that should not be on the system.
• 40% of the space was inert. This was data that had not been referenced for six months or more.
• That left only 30% of the data been well used, along with some capacity for growth.
There are some interesting lessons here. The first one is that the combination of orphaned, unused, and inappropriate space represents some 30% of the typical capacity that is wasted. That being said, when we look at the role of the typical storage administrator, much of the focus of what they do is yielding fairly significant amounts of wasted capacity, which has a large financial impact on their company. Commonly, this issue is never addressed.
The opportunity is to identify the parts that are being wasted though allocation inefficiency and then reclaim it. While that sounds simply, and actually is, remember, literally every site we examine is struggling with this issue (Atrophy is everywhere!). There is a cure fear not.
The second interesting lesson is in the next category of ‘inert space’, which represents another 40% of the wasted space. The inert data space is where effort should be applied to move unused data to more appropriate tiers of storage to reduce capital costs. The basic premise is to implement a policy manager that can migrate active data inside of a storage hierarchy based on specific usage behavior while improving costs.
A Hierarchal Storage Manager can move infrequently referenced, unimportant or old files to less expensive storage without changing the way users or applications access the files. There are a variety of policy based migration controls, among them are Age, Last Touched, Size, and Extension. You can even use HSMs to Speed up Database Performance by moving the index from high performance disk to capacity disk. Again, this isn’t secret sauce; various companies can do the same thing. Sun and Veritas are examples. What’s important is that you use them.
Besides health, what is there to gain for this effort? Plenty!
Customer Example
Here is an example. There was a large international bank that had a growing Enterprise Resource Planning (ERP) business application. Their growth rate was at about the industry expected standard for commercial business which is 30%. They initially went to their primary storage provider to ask for recommendations for how much more storage they would need to sustain the growth rate of their business. The disk vendor came back with their proposal which indicated that the expected need was 100 Terabytes. We suggested to the CIO that they give us the opportunity to assess the real requirements when efficient storage management practices are employed. The customer agreed to the assessment. The results were dramatic. After discovering all of the allocation and utilization inefficiencies, we mapped their growth into the misused space. We then did a gap analysis to determine how much more they would need, and discovered that the real requirement was for 2 Terabytes. This resulted in a savings of $3.8M over three years versus the first vendor’s recommendation. The key to this value was understanding the true nature of data usage behaviour, and mapping that over an appropriate storage infrastructure.

The Paradox
The truth often sounds paradoxical.
- Lao Tsu (c.604–531 B.C.) Great Chinese philosopher, founder of Taoism

While there is no doubt that managing the infrastructure will make the operation staff more productive, not everybody will happily follow this idea. The concern most operations staff will have is based on the fact that they are stretched too thin now to manage what they currently have, not to mention the growth. They often feel that any change is tantamount to adding additional complexity to the environment which will have serious operational impact, and will therefore result in them pushing back on any changes. This is exacerbated by the fear that the introduction of any new technology will only add to the problem. If things are done with a parochial mentality, often they are right. But not always.

The solution to this dilemma is found in doing the right things, not the popular things. Doing the right things is dependent not only on good technology, but also a deep understanding of the operational characteristics, the environment, business processes and application deployments across business applications. With these things in mind and some practical experience, one can architect a solution that will realistically reduce operational complexity, ultimately reducing operating expenses and capital expenses.
It is a sort of forest and trees challenge. Compounded by the lack of resources in a fast paced and growing environment, once the general architecture of an IT infrastructure is implemented, seldom do people go back and question the architecture implemented. The net impact is that often the benefits of the architecture and infrastructure are no longer efficiently delivering peak value to the business. It is no surprise considering that many businesses are growing, and that of course has a direct impact on the load and growth placed on the IT infrastructure. There are always a shortage of qualified people, which is what leads this paradox, people begin to simply buy and use more capacity as a way of dealing with the overload of efficiently managing what they have.

Slowly all people realize the infrastructure is strained, as the business has grown beyond the abilities to efficiently deliver IT services from what was originally conceived and implemented. That’s where things get exciting. With the lack of improved insights on how to manage the infrastructure, and with the lack of enough resources to manage the complete environment, the strained technology staff is forced to adopt a plug the dike strategy for the growth of the infrastructure, in an attempt to keep the business applications running at expected service levels.

This is a wonderful opportunity for self serving vendors to over sell capacity. Even that is understandable. If servers and storage were free of cost, one way to approach the issue is to hide the complexity of managing resources behind vast amounts of capacity. However, not only is this expensive from a capex point of view, from an opex point of view, eventually this will fail also.

Instead of a methodical analysis of how the infrastructure is tied to the needs of the business, and mostly because of industry hype, the effort is always focused on data backup, migration, archive, and tiers of storage for gaining value in the inert data category. All of which are good and interesting, however in the implementation phase mostly inefficient and costly.

Here is a basic outline for what needs to happen to gain efficiency.
1. Identify and manage the wasted space to achieve allocation efficiency
2. Identify and manage inert data to achieve utilization efficiency
3. Build a protection architecture that is justified based on the contribution the application makes to the business ( I will cover this in another BLOG)
4. Consolidate, which will refine the value achieved above
5. Virtualize to further refine

The road to health requires an examination, diagnosis, and the proper treatment. Disk Atrophy is ugly, but it is also curable, and is a big win for you when it is.

Posted at 08:22AM Jun 01, 2007 by chalfant in Sun  |