Useful stuff for your blog-reading pleasure.

All | General

Wednesday March 25, 2009

Think Twice Before Deleting Stuff (Or Better Not at All!)

No, this is not going to be another "Remember to do snapshots" post. I'm also not going to talk about backups. Instead, let's look at some very practical aspects of deleting files.

So, why delete a file? "Trivial", you think, "so I can save space!". Sure, dear reader, but at the expense of what?

Let's stop and think for a minute. Our lives try to center around doing cool, worthwhile, meaningful, useful stuff. Deleting files isn't really cool, nor fun, it is a necessity we're forced to do. Don't you hate it when that dreaded "Your startup disk is almost full" message appears while you're in the middle of downloading new photos from your latest exciting vacation trip?

Actually, the seemingly simple act of deleting is really a challenge: "Will I need this again?", "Wouldn't it be better to archive this instead?", "Last time I was really glad I kept that email from 2 years ago, so why delete this one?". Sometimes I surprise myself thinking a long time before I really press that "ok" button or hit "Enter" after the "rm".

The reality is: Storage is cheap, so why delete stuff in the first place?

To put things in perspective, let's try an ROI analysis of deleting files. Let's say we need about 6 seconds of thinking time before we can decide whether a particular file can really be deleted without regret. Let's also assign some value to our time, say $12 per hour (I hope you're getting paid much more than that, but this is just to keep the numbers simple).

Storage is cheap, and last time I checked, a 1 TB USB hard drive cost about $100 at a major electronics retailer, with prices falling by the hour.

Now, how much space does the act of deleting a file need to free up so it justifies the effort of deciding whether to delete or keep it?

Well, our $12 per hour conveniently breaks down to $0.20 per minute, which allows us to perform 10 delete-it-or-not decisions per minute at $0.02 each. Fine. Deleting seems to be cheap, doesn't it?

Now, for that $0.02 you can buy a 1/5000th of a 1 TB hard drive. Wait a minute, 1TB/5000 still amounts to 200 MB of data per $0.02! That's more than you need to store a 10 minute video, or a full CD of music, compressed at high quality! Or 20 presentations at 10MB each! Not to mention countless emails, source code and other files!

So, unless the file you're pondering is bigger than 200MB, it's not really worth even considering to delete it. I'll call this 200MB boundary the "Destructive Utility Heuristic (DUH)".

The result is therefore: Save your time, buy more harddisk space (or upgrade your old hard drive to a bigger one before it dies) and move on. Life's too precious to waste it on deleting stuff. Create good stuff instead! Only think about deleting stuff if the file in question is bigger than 200MB.

I can hear some "Wait, but!"'s in the audience, ok, one at a time:

"But I can delete much faster than 6 seconds!"
No big deal. So you can delete 1 file per second, that's still a threshold of 33MB, more than 5 songs worth or even the biggest practical business presentation or the source code to a major open source project. And harddisks are getting cheaper every day, while your time will become more and more precious as you age. Yes, if you're dead sure that file is useless junk and don't need to think about it, go ahead and delete it, but why did you save it in the first place?
"But I like my directories to be clean and tidy!"
Congratulations, that's a good habit! Keeping files organized doesn't mean you need to delete stuff, though. Set up an "Archive" folder somewhere and dump everything you think you may or may not use again there. Use one archive folder for each year if you want. File search technology is pretty advanced these days so you should be able to find your archived files quicker than the time you'd take to decide which ones you'll never want to find again. Then, you can still decide to delete your whole archive from 3 years ago because you never used it, and it will likely make some sense, because its size may be above the destructive utility heuristic, but chances are you won't really care because storage will have become even cheaper after those 3 years so you won't save a big deal, relatively speaking.
"That still doesn't help me when that damn 'Your startup disk is almost full' message comes!"
You're right. The point is: It's often hard to sift through data and decide what to keep and what not. That's why we dread deleting stuff and instead wait until that message comes. I'm only offering relief to those that felt that the act of having to delete stuff isn't really rewarding, and it isn't (at least while you're below the DUH). Go buy a bigger harddrive for your laptop, it's really the cost effective option. Use the numbers above to help you justify that towards your finance department.
"I'm still not convinced. I actually kinda like going through my files and delete them once in a while..."
Sure, go ahead. Just know that you could use that time to do more productive stuff, such as checking out the Sun cloud, installing OpenSolaris or testing our new Sun OpenStorage products.
"Wait, aren't you supposed to write about OpenSolaris, ZFS and this stuff anyway?"
I'm glad you mentioned that :). Actually, OpenSolaris and ZFS make it even easier for you to both not care about deleting stuff while keeping your files organized at the same time. The amazing ZFS auto snapshot SMF service will create snapshots of your data automagically every 15 minutes, so it won't matter whether you delete files or not. You can then choose to either not delete them at all and just move them to some archive, or you can delete whatever you want, without the 6 seconds of thinking (just to keep stuff tidy), knowing that you'll always be able to recover those files with Time Slider later. You could then use zfs send/receive to dump your data incrementally to a file server as a backup mechanism and the hooks are already there to automate this.

See, once you think of it, there's not really a need to delete files at all any more. At least not for mere mortals like us with file sizes that are typically below the destructive utility heuristic of currently 200MB (and rising...) most of the time. Music has already reached the point where a song can be stored at studio quality with lossless compression at manageable file sizes so that kind of data won't see significant growth any more. And photos and videos will soon follow. This means we'll need to care less and less about restricting personal data storage. Instead, we now need to focus more on managing personal storage.

Now there's a completely different problem that'll keep us entertained for some time...

"Think Twice Before Deleting Stuff (Or Better Not at All!)" has been brought to you by Constantin's Blooog.
This entry was created on 2009-03-25 07:07:19.0 PST and is associated with the following tags:
efficiency file management opensolaris productivity roi tip useful zfs
You're welcome to use this Permalink , add a comment below or send your feedback to constantin at sun dot com.
Comments [3]

Monday April 21, 2008

On Knowledge Management, Community Equity and Ontologies

Last week, I attended a meeting of the BITKOM Working Group for Knowledge Engineering & Management at the Sun Frankfurt office. The meeting was very nicely organized by Mr. Weber, Mr. Neuwirth and some colleagues from Sun in Germany (Hi Hansjörg, you should really blog!) and Peter Reiser from Sun in Switzerland. Therefore, I got to play host of the meeting without having to do too much work :).

Peter asked me to present his work on Community Equity (see also this interview with Shel Israel and this other one with Robert Scoble) and the CE 2.0 project to the group. The working group was very interested in how to encourage communities to participate and how Community Equity mechanisms can be used towards this goal. We had quite a few positive discussions during the breaks.

Image illustrating Community Equity

But, some people seem to be concerned with tracking community contribution and participation on an automatic basis, for example, see Mike's post on the subject and Alec's reaction to Peter's interview. These are all very valid thoughts, and indeed nobody wants to see their work or life be reduced into a couple of numbers.

As always, the threat is not in the technology, but in the way we use it:

Measuring stuff is a good thing, if you know what you measure and how accurate that measurement is.
Telling people how their work is being received is also a good thing. I always get a kick out of the HELDENFunk download statistics (We should probably start publishing them), or my own blog's metrics. This is a huge motivator.
Telling people about how other people's work has been received is also a good thing. Nobody would put the kind of trust into eBay if it weren't for their rating system. How many books have you bought on Amazon based on other people's recommendations, stars, etc. on their site?
Web 2.0 style commenting, crosslinking, social networking, tagging and rating is also a good thing. Much of the web 2.0 world today would be untrusted, unnavigationable and unuseful if it weren't for those mechanisms.
The next step is to take these concepts, and apply them to an enterprise context. This is what Peter's Community Equity work is all about. The goal I see here is: If you do a good job, others should be able to notice (including, but not limited to, your manager). If you're looking for an expert on topic X, you should be able to find people that may be able to help you. If you are talking to person Y or if you run into that person as part of a team, you should be able to see what kind of work that person has contributed to the enterprise before and what others are saying about them. Think Amazon and eBay and LinkedIn ratings, recommendations, tags etc. as a tool to better navigate the social network and knowledge base of your enterprise.

Notice that the part where discussions become heated is not the technology one, it's the "what do we do with the numbers" part. That, of course, is where we need to be careful. We need to understand how the data is generated, how it has been processed (i.e. the exact rules and formular that is used to generate the Community Equity score) and what it does not tell us. You may trust your latest auction winner to transact with you on that particular sale, but you still don't know if she is actually a nice person or not :).

As long as the process is open, well-understood and transparent, using Web 2.0 mechanisms and Community Equity style metrics can be a very useful thing. You can generate a lot of useful information based on that kind of data: What are hot topics? Which documents are the most used, best rated, most re-used ones? Who are the company internal creators, connectors and consumers of knowledge? What topics have trouble to be picked up by the community? Sounds like fascinating stuff, if you're responsible for your company's knowledge...

Of course, this was only a small part of the BITKOM meeting. We heard presentations by other companies on different applications of knowledge management technologies in a customer service context. Interestingly, all of them (including CE 2.0) mentioned the term Ontology in one way or other. In a knowledge management context, an Ontology is the part of the system that relates "words" or other abstract data to real-world concepts and objects, resolving ambiguities, consolidating synonyms and clarifying user-errors. It's the part of the system that tries to bring in semantic knowledge as opposed to mere processing words.

Ontologies are very hard to do. That's why most of the times they are generated "by hand" which is very time and resource consuming. The holy grail of ontologies is when the system can automatically generate semantic meaning out of naked data by itself, without any help. Some of this systems are seeded with hand-made ontologies that can then expand somewhat automatically.

An interesting approach to generating ontologies might be to analyze web 2.0 style tagging data that has been created by users. An ontology system could then try to identify clusters of tags and assign them to a real world concept, then try to identify relationships between those concepts. As an example, the tags "LDAP", "Directory Server", "DS" all belong to the same concept and they are related to (but not the same as) "Identity Management", "IdM", and "Databases". A search engine then can use this data to find better matches for a user that is looking for "Identity Management and LDAP interoperability".

As you can see, even a seemingly dry and academic workshop on "Knowledge Engineering and Management", organized by an industry association can be an exciting topic, sometimes transcending the boundaries between technology, philosophy and anybody's daily web 2.0 style work.

"On Knowledge Management, Community Equity and Ontologies" has been brought to you by Constantin's Blooog.
This entry was created on 2008-04-21 04:10:00.0 PST and is associated with the following tags:
ce2.0 collaboration community equity knowledge management ontology web2.0
You're welcome to use this Permalink , add a comment below or send your feedback to constantin at sun dot com.
Comments [2]

Thursday November 01, 2007

7 Tips for Enhancing Your Email Efficiency

I think I sent my first email in 1987. We lived in Rome, Italy and my brother and I shared a modem with which we collected our very first online experiences on a Commodore Amiga 500.

Today I receive about 500-700 emails a day on my Sun account. Not counting Spam (most of which is filtered by our mail system anyway). That's a lot, but over time I grew accustomed to dealing with more and more email as efficiently as possible.

Here's what helps me use email as a productivity tool rather than a burden, while still having fun. This is going to be a long post, but if your Inbox currently has more than 100 emails, possibly sitting there for more than a week or two, then I promise you an easy to use way of getting your Inbox to 0.

Zero mails in your Inbox. Once and for all. Still, you will be informed about what's going on and it'll be earlier, with less effort, and more reliably.

The Email Client

In my email carreer, I've use a lot of mail clients. During university days I started with the classic mail(1) on SunOS 4 and it's counterparts on VMS and on an IBM 3090 mainframe. Then I've used Elm for a long time, then Pine. When I joined Sun in 1998, one of the first things I did was to compile myself Pine so I could keep my habit of reading email on a terminal.

Why on a terminal? It's always quicker and more efficient than a GUI (Yes, I'm one of those old-schoolers that still prefer vi as their favorite text editor). It really is. So much that I'd like to make it...

Email-Efficiency Rule #1: Make sure you can use your email client with keystroke commands only.

When dealing with hundreds of emails, the extra time to move the mouse cursor and to click on some buttons etc. really adds up. Learning keystrokes might seem tedious at first, but it will quickly become second nature and you'll be amazed at how quickly you can scan through emails with just one hand sitting on your keyboard, while having your other hand free to drink coffee while reading email.

After a while, I migrated to another email client called Mutt. This introduced two major new features that made my email-life much, much easier: Threads and Filters.

A threading email client automatically groups emails that have the same subject (or that are related to each other based on the header information) into threads. Threads are more efficient to read because they contain all emails related to a certain subject or conversation in one go. And more importantly: You can delete dozens of "Please take me off this list" or "me too" emails and other uninteresting discussions with a single keystroke!

Mozilla Thunderbird supports threads very nicely, so does Apple's Mail and of course GMail, only they call it "conversations" (and they dig up all related mail from the past too, which is very nice).

Message filtering is another powerful feature of modern email clients. It lets you pre-sort email into folders or assign different colors/priorities/etc., based on simple rules. I don't feel comfortable with automatically filing away emails without at least looking at their subject. So I use rules exclusively to assign priorities to emails: Emails that are addressed directly to me or come from my management chain automatically get prioritized highest. Emails where my email address shows up on the CC line or that is addressed to working groups that are dear to my heart get the second highest priority. All other email gets normal priority. Emails from Sun get a different color than external email. Other similar rules are of course possible and can be very useful.

Using filters makes it easy to get a picture of what's going on when you only have a few minutes to check email in between meetings or when on the go, without risking to overlook any important email. Therefore, let's postulate...

Email Efficiency Rule #2: Let your email client do the reading before you do.

I now use Mozilla Thunderbird to read my Sun email. At some point, I just felt that there has to be a way to efficiently read email and still use a GUI, and Thunderbird is quite good at it: It supports keystrokes, threads nicely, you can program complex rules to pre-digest email easily and it is multi-platform, open source and contributed to by Sun.

With threading and rule-based priority sorting enabled, my 500-700 emails a day split into about 10-20 "Highest" and another 30-40 or so "High" priority emails. This is much more manageable as I can work through the higher prioritized emails with a more concentrated mind before quickly scanning through the rest just in case there's something interesting there.

For my personal email, I use Google's GMail, because it completely outsources my need to archive emails, has a great browser-based user interface that can be accessed from anywhere (even a mobile phone) while still feeling like a real application and of course it suppors keystrokes, has very nice threading support and it supports filters too.

After my company gave me a Nokia E61i so I can read email on the go, I had a new problem: Nokia's email client doesn't support threading nor message filters (please tell me if you know a better email client for SymbianOS), and hundreds of truncated sender/subject lines on a mobile phone aren't really useful. So let's have a look at the server side of the picture:

The Email Server

Today, the two main mail server protocols are POP3 and IMAP4. POP3 basically dumps all your email onto your client, then (optionally) forgets about it as soon as you connect to your mail server. Not good if you're on the go. And then you need to take care of all archiving yourself. And what if you access the same mail box from different clients?

IMAP4 on the other hand lets your client choose whether to only pull headers or the whole message, it supports server-side folders to sort your mail into and you can keep your mail on the server while accessing it from multiple clients out of multiple devices and still everything stays perfectly synchronized.

So, whenever possible, choose IMAP4. If you can't choose IMAP4, change your email service. Fortunately, Google just introduced IMAP4 support, in case you want to read your Mail with something else than their web interface.

Thanks to IMAP4, we don't have to organize our mails on our clients, instead we should go by...

Email Efficiency Rule #3: Keep your emails on the server, always.

Really. There's no point in downloading all your email to some client that can suffer a hard-drive crash or a virus infection or whatever. Chances are that your email server is a much more reliable machine than your email client and it minimizes the bandwidth needed to read and manage emails to what your brain can handle without downloading hundreds of emails that you'll never read past the subject line. You can still dump your favorite folders to disk or to a CD for archiving purposes, once in a while, if you want to.

Back to my mobile-phone-can't-thread-nor-prioritize problem. One feature of the Sun Java System Communications Server that we use is server-side filtering. It lets you forward, file or delete mails based on simple rules. Again, I like to be conservative here, so I never want to automatically delete anything, just file away what I know for sure is not important enough to waste my precious mobile phone's bandwidth with.

The utter majority of the emails I get are from internal and external mailing lists that I subscribed to or not or that otherwise find my email Inbox. These are natural candidates for "If the mail was addressed to <insert mailinglist alias here>, then file it to <some folder>" type of rules. Keeping it simple, I only use one folder for this purpose, called "ToBeRead". You could also name it "Inbox2" or "Later" but the important thing here is to actually treat this folder as a real Inbox folder the next time you have some time and a more comfortable client. Don't create a growing monster pile of unread mail because you started playing with rules, it won't really help you.

Email Efficiency Rule #4: Let your email server do some reading, too.

Sorting email on your server is different from sorting email on your client: The former gives you a bandwidth choice that enables the use of mobile devices or helps you quickly check email through a web interface (by pre-sorting email into folders), while the latter helps you look at your email in the right sequence (by threading and prioritizing it).

I just checked my Sun mail through the Nokia E61i after not having checked mail for a day (today is a bank holiday in Bavaria) and I have 47 new mails. I didn't check my ToBeRead folder, but I'm sure it has more mails than I can handle on a mobile device comfortably. Seems to work for me (and I've seen a couple of mails that will make nice new rules to my server-side filter).

I usually don't check emails after work hours, in the weekend or during bank holidays. I seem to be immune to the Crackberry disease, which I guess is a good thing. This brings us to the most important Email efficiency part of all:

Email Workflow

One of the first trainings that Sun sent me to after I was hired was about time-management. This is a fascinating subject by itself but it turns out that a lot of the principles taught under the umbrella of time-management can be applied beautifully to organizing your email.

If you're looking for a great blog on the subject of life hacks (a term for "when geeks start digging into time and self management") then check out Merlin Mann's "43Folders". If you prefer to read a book, then I can highly recommend David Allen's "Getting Things Done" (GTD).

Here's an easy but very efficient email workflow that is very similar to the GTD workflow:

Go to the next email in your Inbox and ask yourself:
"Do I need to do something because of this email?" (or: "Is it actionable?")
- If the answer is "yes", then you either have to reply to the email or do some action that is associated with it. Now ask yourself:
  "Can I do it in less than 2 minutes?"
- File away or delete the email. There's no more reason for it to sit in your Inbox.
Go to 1.

After a couple of iterations, you should have an empty Inbox. Really. 0 emails. Take a deep breath, celebrate and get used to it.

"But now I have this big and long to do list!" I hear you say. Well, that might be true, but a to-do list and an email Inbox are really two different things. Email is for communication, your to-do list is a way for you to organize your tasks. Never mix them up.

The important thing here is to get rid of all those emails in your Inbox. Feel the joy of hitting the delete key or filing away that email with the knowledge that it has been dealt with, once and for all!

Email Efficiency Rule #5: Develop an email workflow that helps you clean your Inbox.

You're invited to try the above workflow or you can develop your own. The point is to have a system that helps you get your Inbox to zero and free your mind for what's really important (Hint: It isn't email). Your workflow should be easy to implement, no matter how, where and when you read email. There should be no excuse left that prevents you from cleaning up your Inbox.

Having an empty Inbox has a great motivational power. You'll feel as if a big weight has been taken off your shoulder. You'll feel free to actually get some work done, instead of looking at all those emails. Try it out just once, but beware: Having an empty Inbox can be highly addictive...

Two things are left now: Dealing with that long to-do list and an easy and efficient way of filing those emails that you've dealt with already. As said, dealing with to-do lists is the subject of a whole science and I can only encourage you to check out one of the many sources on time and self management. This introduction might be a good start.

So what to do about filing emails? I know quite a lot of colleagues with elaborate folder systems that they use to file their emails and stuff in. One can base a filing structure on project names, client names, products, events, themes, priorities, whatnot. My easy answer to this problem is: File everything into one single folder, then let the computer find it when you need it.

Really, it works. Modern email clients are very good at searching through vast amounts of email. In fact, thanks to IMAP, it's actually the server that does it for you. I have just one single folder on my mail server that I use for filing mail away, it has thousands of emails and it is called "file". That's it.

You still think it can't be that simple? Well the ultimate test is: Will you be able to find any particular email quickly and easily? With an elaborate filing system, based on many different folders, this may or may not be the case. I've seen many colleagues try different folders while desperately looking for that one important email. Did I sort it into the client's folder? Wait, it was related to that project so it's probably in that folder. Or was it in "Pending"?

If you only have one folder to file stuff in, you rely on using your email client's search mechanism. This gives you at least four different ways to search for an email:

By person: If you're looking for a particular email, you probably remember its sender or recipient. Search for that person, then find the email in the results. Done.
By keywords in the subject: Think of one or two words that are guaranteed to show up in the subject of the mail you're searching for.
By time: Some study has found that the most brain-friendly way to organize stuff is by date/time. Try to remember the point in time you got or sent that email, then scroll back in time in your filing folder. This works best for emails associated with projects, stuff that is quite recent, etc.
Full text search: If all else fails, do a full text search. This shouldn't happen often, but works as a last resort. And it's reasonable quick on modern computers. Quicker than going through all those other folders...

Of course, combinations work well, too. Searching by person, then subject or time usually works for me 99% of the time. I only need to resort to full text search about once every 6 months.

Email Efficiency Rule #6: File away your email and let the computer do the searching.

Filing or deleting? When in doubt, file! Storage space has become cheap and search algorithms have become so powerful that there really is no reason not to file everything. Google has made this a major point when advertising their GMail service, and they're right.

So we now have found a good email client that supports keystrokes. We teached it how to thread and how to prioritize our emails. We like to keep emails on the server because they're really better off there and we let the server do some pre-work so we can deal with low-bandwidth situations. We've developed an email workflow that empties our Inbox in no time and an easy way to file all those emails too, relying on our computer's ever increasing power to always find what we look for.

We're almost in email heaven now, but we want to make sure to stay there and avoid going back to email hell after the next period of hectic activity or after a long vacation that filled up our Inboxes to DOS-inducing levels. We want to attack the problem at the root.

Remember those server-side rules that said "Email addressed to X should be filed into Y for later review"? Well, why did you subscribe to that newsletter/mailing list/discussion group in the first place? Is email really the right way to stay current on a certain subject?

The truth is: No. Email is a communication mechanism between people who know each other and have to say something to each other. It is not a news delivery mechanism (RSS can do that better and more efficiently). It is not a way to gather and harvest information (Google and other search engines on the internet can do it better). And it is not a discussion forum (Use Newsgroups, IM and chat or web based forums).

So let's go through our server-side rules and ask ourselves: Do I really want to keep subscribed to this service? Why don't I switch to a pull model for staying informed where I'm in control vs. being flooded by all those "informational" emails that I don't have the time to read anyway?

There's also email minimization potential with day-to-day emails to and from your co-workers. Do you really need to forward that email to your 30 or 100 other co-workers that may or may not be interested in that particular news item? Is that joke, video, URL really so funny that your entire office has to look at it? Do you really want to be "kept posted" on all minutiae of that process or just receive a short "done" notification at the end?

Email Efficiency Rule #7: Go on an email diet. Limit newsletters/mailing lists/mass emails to a necessary amount and write/forward emails only when necessary. Especially when addressing a large group of people.

I know that this rule is the hardest. But think of it. It makes sense. It may not be easily implemented everywhere (And I'm known for being an occasionally passionate participant in large email discussions myself), but using the right information resource/channel for the task at hand is often a very good idea.

Let me know if the above tips and rules are helpful to you. Share your own secrets of email efficiency. Let me know how large your Inbox is and whether you like it or not. What is your perfect way of dealing with large amounts of email?

"7 Tips for Enhancing Your Email Efficiency" has been brought to you by Constantin's Blooog.
This entry was created on 2007-11-01 15:56:50.0 PST and is associated with the following tags:
effectiveness email gtd hacks life management time tips useful
You're welcome to use this Permalink , add a comment below or send your feedback to constantin at sun dot com.
Comments [2]