Angelo's Soapbox

Mounting Amazon S3 buckets as a file system on OpenSolaris

Happy New Year.

Recently, I have been trying to make Amazon S3 buckets more "accessible" on my OpenSolaris system. My goal was to "mount" an S3 bucket as a filesystem on OpenSolaris.

I came upon s3fs a filesystem in userspace (FUSE) that uses the Amazon S3 as its data store. Last week I spent a few minutes building s3fs on OpenSolaris.Here are the step by step instructions.

Summary Instructions

• Download and install OpenSolaris 2008.11
• Get a AWS account and create a S3 bucket
• Build and install FUSE for OpenSolaris
• Build s3fuse
• Mount a S3 bucket & access away

Details

• Building fusefs & libfuse on OpenSolaris
  Install needed pkgs
  
  • pfexec pkg install sunstudioexpress
  • pfexec pkg install SUNWmercurial
  • Download SUNWonbld from http://dlc.sun.com/osol/on/downloads/current/
    Install it using pkgadd -d SUNWonbld.pkg
  
  Download, build and install Open Solaris version of fusefs and libfuse
  
  • hg clone ssh://anon@hg.dot.opensolaris.org/hg/fuse/fusefs
  • hg clone ssh://anon@hg.dot.opensolaris.org/hg/fuse/libfuse
  • Read the INSTALL doc for instructions on building and installing fusefs. Here are a few Gotcha's.
    • Make sure to use make and not gmake
    • Add /opt/onbld/bin/i386 to the path
    • cd into fusefs/kernel
    • make
    • make pkg
    • pfexec pkgadd -d packages SUNWfusefs
    • pfexec bootadm update-archive
    • reboot
  • Read the INSTALL doc for instructions on building and installing libfuse.
    • cd libfuse
    • make install
    • make pkg
    • pkgadd -d packages SUNWlibfuse
    
    
• Download and build s3fs
  • Read the details about s3fs
  • Download the source from http://s3fs.googlecode.com/files/s3fs-r177-source.tar.gz
  • Install curl
    • pfexec pkg install SUNWcurl
  • Make one small edit. The s3fs.cpp file includes fuse.h. You need to change that to fuse/fuse.h
  • Use gmake to build
    • gmake
    • gmake install


• Using s3fs
  • First make sure you have an AWS account.
  • Create or reuse an S3 bucket you already have. another-s3-bash is an excellent tool for this purpose. It works well on OpenSolaris. The rest of the instructios assume that you are using another-s3-bash
  • Set up env variables for another-s3-bash
    • export S3_ACCESS_KEY_ID="your access key"
    • echo -n "your secret key" > secret-key.txt
    • export S3_SECRET_ACCESS_KEY=secret-key.txt
  • Create a bucket on S3
    • s3 put fusebucket
  • Put something in the fusebucket
    • echo "Hello S3 World" > /tmp/hello
    • s3 put fusebucket:hello /tmp/hello
  • Mount the S3 bucket
    • s3fs fusebucket /mnt
  • Use it
    • cd /mnt
    • You can do normal filesystem operations like, ls, cp, mv chmod etc
  • s3fs logs everything using syslog. So if you need to see messages look at /var/adm/messages

Hope this helped. Let me know if you have issues. More to come on this front.

Posted at 01:30PM Jan 12, 2009 by angelo in Web2.0  |  Comments[2]