Prashant's Perceptions

Prashant Srinivasan's Weblog

« Compiling Ruby 1.8.6... | Main | Control-C Control-V?... »

Wednesday Aug 22, 2007

MogileFS, and Solaris 10

MogileFS is a pretty cool non-hierarchical distributed file system for files that don't need to be stored in an RDBMS. I recently played with MogileFS as a part of some performance investigations on Solaris. While it was pretty straight forward, there are some gotchas to avoid.

Here is how I did it on Solaris/SPARC, using the Studio 11 compilers. It should work on Solaris x86, if you use Sun Studio 12.( I ran into some problems with compiling some perl dependencies on Solaris x86, with Studio 11. They went away with Studio 12, but I haven't gotten around to doing a complete build of MogileFS on Solaris x86 yet.)

I was helped quite a bit by this article by Brett G. Durrett. and I've added (modified)content from it into my howto to make it easy to get all the content in one page.

How do you install Mogile FS on Solaris SPARC?

Start with either a Solaris 10 or a Nevada/OpenSolaris machine. I used Nevada build 46. This is an ancient build of OpenSolaris – build 70 being the latest.
Install Sun Studio 11 into /opt/SUNWspro. This is where it installs, by default. If you have it in another place(such as an NFS mount), create a link to /opt/SUNWspro
Cooltools, are really cool. Get the cooltools AMP stack(from coolstack 1.1). Select the link titled “Apache 2.2.3, MySQL 5.0.33, PHP 5.2.0, English” under the like of downloads for Solaris 10 SPARC. Install this stack, and make sure that /opt/coolstack/bin is prepended to your $PATH variable.
Also install coolstack perl. This should give you Perl 5.8.8. This is available from http://cooltools.sunsource.net too.
Prepend /usr/sfw/bin to your path. This is where the gnu packages are found, in Solaris 10 onward. In case you were wondering, this is different from /opt/sfw/bin, where the contents of the Solaris companion CD are installed(optionally).
Install GNU Coreutils 6.4. - here is a direct link - ftp://ftp.sunfreeware.com/pub/freeware/sparc/10/coreutils-6.4-sol10-sparc-local.gz These are easily obtained from either http://www.sunfreeware.com or http://www.blastwave.org or mirrors in case the above link does not work.
Prepend /usr/local/bin to your $PATH variable. This is crucial because mogstored depends on the GNU df. You'll see disk capacity/usage/free space show up as zero(and also other mysterious failures) when looking at device stats using mogadm if GNU df is not in your path.
Configure cpan with /opt/coolstack/bin/perl. Use http mirrors if you're behind a corporate firewall since they work better. CPAN does not handle ftp:// urls properly(at least for me), when I use Sun's proxies. This leads to corrupt packages. So if you're behind a corporate firewall, use only http CPAN mirrors for better results.
Download Perlbal first - http://search.cpan.org/CPAN/authors/id/B/BR/BRADFITZ/Perlbal-1.52.tar.gzY and go through the perl Makefile.pl; make; make test; make install process. You will see a lot of dependencies that are unsatisfied in the second step. The list looks like this(including some packages that it doesn't mention explicitly):
BSD::Resource 0 (you can install the latest, 1.28 as well)
Danga::Socket 1.44
HTTP::Date 0
HTTP::Response 0
prerequisite Sys::Syscall 0
IO::WrapTie
Compress::Zlib
IO::AIO
Net::Netmask
First try to install them through CPAN, since each of these packages has other dependencies. If CPAN doesn't install some of the above mentioned packages(probably because of failures in “make test&rdquo, then install them manually(by going into /.cpan/build/<Pkg_name> and run perl Makefile.PL, make, make install directly). You wont need to install the dependencies for these, since CPAN would have finished this already(hence making your installation experience easier).

Finally run “make” and “make install” on Perlbal. It shouldn't complain about unsatisfied dependencies now.
Obtain dbi and dbd. install these. -- http://cooltools.sunsource.net/coolstack/faq.html (install dbi, use cpan for the dependencies, and then compile dbi directly). use studio 11(make sure you're running this on sparc, not x86)
By now you're past the difficult section, and can afford to breathe a little. Setup the mogile user:
useradd mogile
mkdir -p /export/home/mogile
chown mogile /export/home/mogile
edit /etc/passwd to change home dir to /export/home/mogile
Setup the mysql user. I installed this on the same machine as the MogileFS tracker. But this may not be most performant.
groupadd mysql
useradd mysql
bash-3.00# mkdir -p /export/home/mysql
bash-3.00# chown -R mysql:mysql /export/home/mysql
Below is how you quickly set up MySQL for MogileFS.

cd /opt/coolstack/mysql_32bit/bin
./mysql_install_db
chown -R mysql:mysql /opt/coolstack/mysql_32bit/
cd /opt/coolstack/mysql_32bit ; /opt/coolstack/mysql_32bit/bin/mysqld_safe &

bash-3.00$ /opt/coolstack/mysql_32bit/bin/mysqladmin -u root password 'new-password'
/opt/coolstack/mysql_32bit/bin/mysqladmin -u root -h MyHostname password 'new-password'
bash-3.00# pwd
/opt/coolstack/mysql_32bit/bin
bash-3.00$ ./mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.0.33-standard Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> CREATE DATABASE mogilefs;
Query OK, 1 row affected (0.03 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'mogile'@'localhost' IDENTIFIED BY 'some_pass' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'mogile'@'%' IDENTIFIED BY 'some_pass' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)

mysql> flush privileges
-> ;
Query OK, 0 rows affected (0.00 sec)

mysql> exit
Bye
bash-3.00$
Install curl support for PHP(if you plan to use MogileFS through PHP). Do this by editing php.ini and add
extension=curl.so . php.ini is in /opt/coolstack/php5/lib/php.ini
Now install mogilefs, using the below steps, which is a modified version of http://mogilefs.schtuff.com/howto for linux. The following steps document the solaris specific stuff, as well as presents an updated version of the Linux howto, in one place.
The snapshot of MogileFS that I used is here. You can use this or pick up the latest from svn. MogileFS seems to be evolving at such a fast rate that I decided to put up the snapshot I used, rather leaving it open ended. You can install Mogile with the following commands:

# cd trunk/server/
# perl Makefile.PL
# make
# make test
# make install

If you get any errors during this process it will probably be errors telling you that a dependent module is missing. If during the 'make test' step you get the error, "t/00-startup....DBI connect('mysql','root',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at t/lib/mogtestlib.pl line 16" it can probably be ignored – it should not be necessary for MySQL to be running on any host other than mogiledb.yourdomain.com <http://mogiledb.yourdomain.com/>.

the following "make test" errors are OK

bash-3.00# make test
PERL_DL_NONLAZY=1 /opt/coolstack/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/00-startup.........skipped
all skipped: Can't create temporary test database: DBI connect('mysql','root',...) failed: Access denied for user 'root'@'localhost' (using password: NO) at /export/pkgs/mogile_new/trunk/server/blib/lib/MogileFS/Store/MySQL.pm line 180
t/10-weighting.......skipped
all skipped: Can't create temporary test database: DBI connect('mysql','root',...) failed: Access denied for user 'root'@'localhost' (using password: NO) at /export/pkgs/mogile_new/trunk/server/blib/lib/MogileFS/Store/MySQL.pm line 180
t/domains-classes....skipped
all skipped: Can't create temporary test database: DBI connect('mysql','root',...) failed: Access denied for user 'root'@'localhost' (using password: NO) at /export/pkgs/mogile_new/trunk/server/blib/lib/MogileFS/Store/MySQL.pm line 180
t/hosts-devices......skipped
all skipped: Can't create temporary test database: DBI connect('mysql','root',...) failed: Access denied for user 'root'@'localhost' (using password: NO) at /export/pkgs/mogile_new/trunk/server/blib/lib/MogileFS/Store/MySQL.pm line 180
t/store..............skipped
all skipped: Can't create temporary test database: DBI connect('mysql','root',...) failed: Access denied for user 'root'@'localhost' (using password: NO) at /export/pkgs/mogile_new/trunk/server/blib/lib/MogileFS/Store/MySQL.pm line 180
All tests successful, 5 tests skipped.
Files=5, Tests=0, 3 wallclock secs ( 2.75 cusr + 0.30 csys = 3.05 CPU)
bash-3.00#
You probably want to install some helpful utilities on each tracker or storage server as well (these will be needed for later configuration). These are located in the trunk/utils directory and can be installed with the following commands (starting in the top of the SVN directory you pulled):

# cd trunk/utils/
# perl Makefile.PL
# make
# make test
# make install

You also want the API – the utilities will require this. These are located in the trunk/api/perl directory and can be installed with the following commands (starting in the top of the SVN directory you pulled):

# cd trunk/api/perl
# perl Makefile.PL
# make
# make test
# make install
Database configuration: The database is empty and will need a schema applied. The ' trunk/server' directory has a utility named 'mogdbsetup' to make this process simple. By default it assumes the database is located on localhost so if you are running it from a different host you will need to provide the host name on the command line.

# ./mogdbsetup --dbhost=mogiledb.yourdomain.com <http://mogiledb.yourdomain.com/> --dbname=mogilefs --dbuser=mogile --dbpass=some_pass
Again, make sure you replace the host and password so that they match you database configuration from above.

The mogdbsetup utility does not specify a table type by default so your tables will match the defaults for your database. In many cases this will mean that you end up with MyISAM tables. If you prefer InnoDB tables you will either need to make sure your database defaults to InnoDB or you can manually convert the tables (both of these are outside of the scope of this document but there are plenty of examples out there).
Tracker Configuration: On each tracker server (mogiletracker.yourdomain.com <http://mogiletracker.yourdomain.com/>), create a configuration file at /etc/mogilefs/mogilefsd.conf with the following:

db_dsn DBI:mysql:mogilefs:mogiledb.yourdomain.com <http://mogiledb.yourdomain.com/>
db_user mogile
db_pass some_pass
conf_port 6001
listener_jobs 5

db_dsn points to your database instance. If you are running the database on the same machine as the storage server you can omit ":mogiledb.yourdomain.com <http://mogiledb.yourdomain.com/>: and it will use the local machine. db_user and db_pass should match the user and password you configured when setting up your database.

The program 'mogilefsd' will not run as root so you will need to run this as a non-root user “mogile” that you created earlier.
Starting the trackers: Trackers will not run as root so you will need to run them as another user. If you created the "mogile" user when seetingup the trackers, the following commands will work (assumes you start logged in to mogiletracker.yourdomain.com <http://mogiletracker.yourdomain.com/> as root):

# su mogile
$ mogilefsd -c /etc/mogilefs/mogilefsd.conf --daemon
$ exit

You can confirm that the trackers are running with the following command:

bash-3.00# ps -ef | grep mogilefsd
mogile 11612 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
mogile 11614 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
mogile 11611 11608   0 15:33:11 pts/2       0:01 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
   root 11635 9604   0 15:34:22 pts/1       0:00 grep mogilefsd
mogile 11613 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
mogile 11618 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
mogile 11621 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
mogile 11617 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
mogile 11615 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
mogile 11616 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
mogile 11619 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
mogile 11620 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
bash-3.00#

If you don't get a list of running processes the trackers are not running.
Storage Server Configuration: On each storage server, create the storage directory (make sure it has access permissions for the user you will use to run mogstored):

# mkdir /var/mogdata

Configure it:

On each storage server, create a configuration file at /etc/mogilefs/mogstored.conf with the following:

httplisten=0.0.0.0:7500
mgmtlisten=0.0.0.0:7501
docroot=/var/mogdata
Adding storage server information to the trackers:

bash-3.00#mogadm --trackers=MyTrackerServerHostname:6001 host add MyStorageServerHostname --ip=10.6.141.125 --port=7500 --status=alive

You can confirm that your host(s) were added with the following command;

bash-3.00# mogadm --trackers=MyTrackerServerHostname:6001 host list
Add a device to your storage server:

# mkdir -p /var/mogdata/dev1

and let the tracker know . . .

bash-3.00# mogadm --trackers=dn15:6001 device add dn15 1
List your devices:

bash-3.00# mogadm --trackers=MyTrackerHostname:6001 device list
MyTrackerHostname [1]: alive
used(G) free(G) total(G)
dev1: alive 4.745 58.722 63.467

bash-3.00#
In order to add/remove/read files, you will need to create a domain and a class. Domains are like buckets to store your files in an otherwise non-hierarchical distributed file system. Classes, seem like no more than specifiers for the replication count.

Create a domain: bash-3.00# mogadm --trackers=MyTrackerHostname:6001 domain add mydomain.sun.com

Add a class to the domain: bash-3.00# mogadm --trackers=MyTrackerHostname:6001 class add mydomain.sun.com Addresses
Now you're all set! But you'll probably want to play around with some files. If you want to use PHP, heres a client that I use - http://svn.wikimedia.org/viewvc/mediawiki/trunk/extensions/MogileClient/MogileFS.php?revision=7483&view=markup(It's pretty straight forward to use.)