Angelo's Soapbox

Monday Jan 12, 2009

Mounting Amazon S3 buckets as a file system on OpenSolaris

Happy New Year.

Recently, I have been trying to make Amazon S3 buckets more "accessible" on my OpenSolaris system. My goal was to "mount" an S3 bucket as a filesystem on OpenSolaris.

I came upon s3fs a filesystem in userspace (FUSE) that uses the Amazon S3 as its data store. Last week I spent a few minutes building s3fs on OpenSolaris.Here are the step by step instructions.

Summary Instructions

Download and install OpenSolaris 2008.11
Get a AWS account and create a S3 bucket
Build and install FUSE for OpenSolaris
Build s3fuse
Mount a S3 bucket & access away

Details

Building fusefs & libfuse on OpenSolaris
Install needed pkgs
- pfexec pkg install sunstudioexpress
- pfexec pkg install SUNWmercurial
- Download SUNWonbld from http://dlc.sun.com/osol/on/downloads/current/
  Install it using pkgadd -d SUNWonbld.pkg
Download, build and install Open Solaris version of fusefs and libfuse
- hg clone ssh://anon@hg.dot.opensolaris.org/hg/fuse/fusefs
- hg clone ssh://anon@hg.dot.opensolaris.org/hg/fuse/libfuse
- Read the INSTALL doc for instructions on building and installing fusefs. Here are a few Gotcha's.
  - Make sure to use make and not gmake
  - Add /opt/onbld/bin/i386 to the path
  - cd into fusefs/kernel
  - make
  - make pkg
  - pfexec pkgadd -d packages SUNWfusefs
  - pfexec bootadm update-archive
  - reboot
- Read the INSTALL doc for instructions on building and installing libfuse.
  - cd libfuse
  - make install
  - make pkg
  - pkgadd -d packages SUNWlibfuse
Download and build s3fs
- Read the details about s3fs
- Download the source from http://s3fs.googlecode.com/files/s3fs-r177-source.tar.gz
- Install curl
  - pfexec pkg install SUNWcurl
- Make one small edit. The s3fs.cpp file includes fuse.h. You need to change that to fuse/fuse.h
- Use gmake to build
  - gmake
  - gmake install

Using s3fs
- First make sure you have an AWS account.
- Create or reuse an S3 bucket you already have. another-s3-bash is an excellent tool for this purpose. It works well on OpenSolaris. The rest of the instructios assume that you are using another-s3-bash
- Set up env variables for another-s3-bash
  - export S3_ACCESS_KEY_ID="your access key"
  - echo -n "your secret key" > secret-key.txt
  - export S3_SECRET_ACCESS_KEY=secret-key.txt
- Create a bucket on S3
  - s3 put fusebucket
- Put something in the fusebucket
  - echo "Hello S3 World" > /tmp/hello
  - s3 put fusebucket:hello /tmp/hello
- Mount the S3 bucket
  - s3fs fusebucket /mnt
- Use it
  - cd /mnt
  - You can do normal filesystem operations like, ls, cp, mv chmod etc
- s3fs logs everything using syslog. So if you need to see messages look at /var/adm/messages

Hope this helped. Let me know if you have issues. More to come on this front.

Posted at 01:30PM Jan 12, 2009 by angelo in Web2.0 | Comments[2]