Atul's Blog about Clustering, high availability and storage

Many times I have been asked about how to build Lustre from sources. Though the Lustre Manual covers it, some details like selecting OFED distribution are missing. Most likely, you will never have to compile Lustre from sources as Sun supplies all the required pre-built packages including, kernel, Infiniband networking and backing filesystem maintenance utilities. But, you may decide to build Lustre for reasons like:

• Run Lustre on recent vanilla kernel from kernel.org
• To support multiple Clusters with different OS/OFED distributions
• Keep a Lustre build environment to apply Lustre source level patches

In this post, I will explain how Lustre can be custom built from sources.

What you need?

Essentially you need Lustre source, supported kernel source and OFED sources. But before you proceed, I will encourage you to have a look at three documents:

1. Lustre Support Matrix at http://wiki.lustre.org/index.php/Lustre_Support_Matrix
2. which_patch file in Lustre sources located at lustre/kernel_patches/which_patc
3. OFED_release_notes.txt in OFED sources located at docs/OFED_release_notes.txt

These three documents will tell you supported versions of Lustre, Linux Kernel and OFED distribution. For example, if you decide to build Lustre 1.8.1, you will need Redhat Enterprise Linux 5 or Novell SuSE Linux 10, 11 or recent vanilla kernel. After referring to all three documents above, I have decided to use Lustre 1.8.1, OFED 1.4.1 and Redhat/CentOS 5.3 release with Kernel 2.6.18-128.el5 as my base platform.

Compiling Lustre involves three tasks:

1. Patching Linux Kernel with Lustre patches
2. Installing OFED distribution on patched Linux Kernel
3. Compiling Lustre sources with OFED distribution on patched Linux Kernel

Patching Linux Kernel

First of all, you will need to download Lustre 1.8.1 sources. You can either get is from Sun Download website or checkout v1_8_1 branch from Lustre CVS repository. Lustre CVS allows anonymous access, so you need not create an account on CVS.

[root@localhost workspace]# export CVSROOT=:pserver:anonymous@cvs.lustre.org:/lustre [root@localhost workspace]# export CVS_RSH=ssh [root@localhost workspace]# cvs co Lustre cvs checkout: Updating Lustre U lustre/.cvsignore U lustre/COPYING U lustre/ChangeLog U lustre/Makefile.in

[root@localhost workspace]# cd lustre [root@localhost lustre]# ./lustrecvs v1_8_1 lustrecvs: updating lustrecvs cvs update: Updating . P .cvsignore P autoMakefile.am P lustre.spec.in P lustrecvs

If you need to apply custom patches to lustre source, now is the time. For example, you can apply patches mentioned in Bug 20560 to Lustre 1.8.1.

[root@localhost lustre]# patch -p0 < /tmp/Bug20560_attachmentID25766.diff [root@localhost lustre]# patch -p0 < /tmp/Bug20560_attachmentID25833.diff [root@localhost lustre]# patch -p0 < /tmp/Bug20560_attachmentID25896.diff

Lets look at which_patch file in the Lustre sources which holds next clues for selecting Lustre kernel.

[root@localhost lustre]# cat lustre/kernel_patches/which_patch SERIES VERSION COMMENT SUPPORTED KERNELS: 2.6-sles10 SLES10: 2.6.16.60-0.39.3 2.6-rhel5 RHEL5: 2.6.18-128.1.14.el5 2.6-sles11 SLES11: 2.6.27.23-0.1 CLIENT SUPPORT FOR UNPATCHED KERNELS: kernel.org 2.6.16-2.6.27

So, this file tells us that we need "2.6.18-128.1.14.el5" version of Kernel on Redhat Enterprise Linux (RHEL) 5. "2.6-rhel5" is the associated patch series file in Lustre sources. Lustre maintains all the patches required for a particular kernel in series files in lustre/kernel_patches/series/ directory. Series files are nothing but text files with names of patches to be applied in given order. Similarly, lustre/kernel_patches/patches/ directory in Lustre source contains the actual patches.

Next step is to get the kernel source, unpack and patch it. You can either get it from the the OS distribution media, Redhat/CentOS website or from Lustre Downloads website. You will also need "quilt" utility used for patching kernel. Download quilt from either Lustre Downloads site or GNU website.

Unpack the kernel and create symbolic links for kernel related patches and series file at the top of kernel source tree.

[root@localhost workspace]# tar xvfj linux-2.6.18-128.1.14.el5.tar.bz2 [root@localhost workspace]# cd linux-2.6.18-128.1.14 [root@localhost linux-2.6.18-128.1.14]# ln -s /workspace/lustre/lustre/kernel_patches/series/2.6-rhel5.series series [root@localhost linux-2.6.18-128.1.14]# ln -s /workspace/lustre/lustre/kernel_patches/patches patches

Apply Lustre patches to kernel source.

[root@localhost linux-2.6.18-128.1.14]# quilt push -av Applying lustre_version patching file include/linux/lustre_version.h Applying vfs_races-2.6-rhel5 patching file fs/dcache.c Now at patch jbd2-commit-timer-no-jiffies-rounding

Now, copy the corresponding Kernel Configuration file from lustre/lustre/kernel_patches/kernel_configs to top of Linux kernel source tree, compile the kernel and install it. Make sure you replace EXTRAVERSION field in Makefile from "-prep" to "-128.1.14.el5_lustre.1.8.1smp" to match standard Lustre 1.8.1 kernel.

[root@localhost linux-2.6.18-128.1.14]# cp /workspace/lustre/lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config .config [root@localhost linux-2.6.18-128.1.14]# make oldconfig [root@localhost linux-2.6.18-128.1.14]# make -j 2 bzImage modules [root@localhost linux-2.6.18-128.1.14]# make modules_install [root@localhost linux-2.6.18-128.1.14]# make install

Make sure that Lustre patched kernel is selected for boot in /etc/grub.conf and reboot.

Instaling OFED

[root@localhost ~]# wget http://www.openfabrics.org/downloads/OFED/ofed-1.4.1/OFED-1.4.1.tgz [root@localhost ~]# tar xvfz OFED-1.4.1.tgz [root@localhost ~]# cd OFED-1.4.1/ [root@localhost OFED-1.4.1]# ./install.pl --all [root@localhost OFED-1.4.1]# ls RPMS/centos-release-5-3.el5.centos.1/x86_64/ compat-dapl-1.2.14-1.x86_64.rpm libibmad-devel-1.2.3_20090314-1.ofed1.4.1.x86_64.rpm mstflint-1.4-1.ofed1.4.1.x86_64.rpm compat-dapl-debuginfo-1.2.14-1.x86_64.rpm libibmad-static-1.2.3_20090314-1.ofed1.4.1.x86_64.rpm mstflint-debuginfo-1.4-1.ofed1.4.1.x86_64.rpm

Compiling Lustre Source

It is time to compile Lustre 1.8.1 source now with OFED 1.4.1 and patched Linux kernel.

[root@localhost ~]# uname -r 2.6.18-128.1.14.el5_lustre.1.8.1smp [root@localhost lustre]# sh autogen.sh Checking for a complete tree... checking for automake-1.7 1.7.8... found 1.7.9 checking for autoconf 2.57... found 2.59 [root@localhost lustre]# ./configure --with-linux=/workspace/linux-2.6.18-128.1.14 --with-o2ib=/usr/src/ofa_kernel [root@localhost lustre]# make rpms [root@localhost lustre]# cd /workspace/linux-2.6.18-128.1.14 [root@localhost linux-2.6.18-128.1.14]# make rpm

DONE!!

You will find necessary Lustre RPMs in /usr/src/redhat/RPMS/x86_64/, source RPMs in /usr/src/redhat/SRPMS/ and OFED RPMs in OFED sources under RPMS/centos-release-5-3.el5.centos.1/x86_64/

Our group has just posted two new white papers to Sun website. Access them at Lustre File System Product Page.

1. Lustre File System Overview:
This white paper covers basic information about Lustre architecture, networking, high availability and comparison with other shared/cluster file systems.

2. Lustre Networking:
This is a detailed technical overview of Lustre Networking. It covers LNET architecture, routing, configurations for WAN and Site-Wide file systems with a preview of upcoming features. It is a must read for HPC Solutions Architects, field technicians and CTOs.

Introduction to Lustre, the ultimate scalable storage platform.[Read More]

Activities in Day 1 and 2 during Mumbai trip for Lustre training.[Read More]

About official Sun Cluster Blog[Read More]

I visited Pune for University Collaboration Program and met with Principal/Professors of various engineering colleges. This blog summarizes activites on day 1 at Pune.[Read More]

My name is Atul Vidwansa and I am a software engineer working on Sun Cluster in Global Data and Devices group. I am planning to blog about Sun Cluster, OpenSolaris and other activities in general.