Open Source Lab

Introduction

Oregon State University is dedicated to supporting open-source software and the solutions it provides for our campus as well as the world. In support of this, we maintain ftp.oregonstate.edu which houses mirrors for several projects including Gentoo, Debian, Netwinder and Ximian (to just name just a few). Click here for more information about this archive.

As you'll recall a year ago I did some filesystem benchmarks in preparation for bringing up new nntp and proxy servers. The results of these tests can be found here. Its been a year and with it has come 3 new stable kernel releases as well as the 2.5 development kernel. I'm about to replace ftp.oregonstate.edu and wanted to run the tests again to get a feeling for any improvements in the journaled filesystems as well as see what's in store with the 2.5 release.

The Hardware

The following tests are on a Dell PowerEdge 2650 with a 2 x 2.8Ghz P4 Xeon's, 2Gb of RAM with a PERC3/Di controller running Debian Linux unstable (aka sid) running both the 2.4.21 and the 2.5.65 kernel.

The boot partition is a RAID0 mirror with 2 - 18Gb 15k RPM SCSI drives attached to a PERC3/Di controller (AACRAID). The partition for doing the actual tests is made up of 14 - 73Gb 10k RPM SCSI disks enclosed in a PowerVault 220S and attached to a PERC3/DC RAID controller which is essentially an AMI Megaraid card.

Setting up the machine

I needed a few things to get my machine ready for these tests:

• My .config for the 2.4.21 kernel
• My .config for the 2.5.65 kernel
• reiserfsprogs, xfsprogs, util-linux & jfsutils Debian packages

The Debian GNU/Linux sid (currently unstable) distro has the necessary libraries and utilities to support both the 2.4 and 2.5 kernel trees. I did a base install of the machine and then added the testing and unstable sources to /etc/apt/sources.list and did an 'apt-get dist-upgrade'.

On Filesystem Options

I have received all kinds of input from people regarding the use of both of these filesystems. A special thanx goes out to Andrew Morton and Ralf Hildebrand for ext3 usage tips and to Daniel Robbins for his articles on advanced filessytems (see below).

• Introducing XFS | Deploying XFS
• JFS Tutorial | JFS Information
• Linux ext3 FAQ | Introducing ext3 | Surprises in ext3
• ReiserFS Basics | Using ReiserFS

First Impressions

I like the simplicity of all of these filesystems. I don't feel like I have to know a lot to get work done with them. That said, there were a few things that I noticed right off the bat.

The mkfs.reiserfs takes a frickin' long time. Of course, I think its proportional to the size of the volume you're putting the filesystem on but then its like I have to do a mini-fsck everytime I do a mkfs.reiserfs. There is also a similar lag when mounting a ReiserFS device.

JFS was lightning fast with mkfs.jfs and I feel like its a more mature filesystem from just breezing over the documentation on IBM's website. This could be because of JFS's roots in AIX or that its been around for a long time. Even if it is really pretty new to Linux it still _feels_ mature.

ext3 is like the poor man's (person's?!) journaling filesystem. Where it doesn't have a lot of options, it just works and best of all, if it gets really hosed, you just mount it as an ext2 partition and run fsck. I will say that for the same partition I get 860Gb in ReiserFS and only 800Gb out of ext3. That kinda stinks.

I cheated with XFS; Daniel Robbins had a great article on XFS so I just used his info. Hey, its the 21st century, sue me. :-)

The Methodology

I'm no expert of filesystems; I've simply downloaded kernels, compiled and installed them as any system administrator would. I've spent some time learning the most efficient means of using each filesystem; but reading a few articles on the web definitely doesn't make me the master. If you know of a better way to use these filesystems I'd love to hear about it at kveton-at-orst-dot-edu.

Since ftp.oregonstate.edu is a software mirror, I'm most concerned with fast "reads" off of the disk array. I'll test each of the filesystems and have them mounted with options such as 'noatime' and 'nodiratime' because I just don't care about those things in my case. This may not mean that the fastest filesystem in this case will be the fastest for what you are trying to do; if you have data or applications that _have_ to have this information (such as for backups) then you'll want other options enabled. Me? I just want throughput off of the disks _and_ the warm fuzzy feeling that having a journaled filesystem gives me.

Any one of these filesystems can do the job you want it to do (and some will do it better/worse than the outcomes of these benchmarks). However, I just want some stinkin' speed ... :-)

I've decided to use IOZone and Bonnie++ for the benchmarking tools. I picked IOZone because of the purdy graphs it creates and Bonnie++ because, well, everybody else uses it, why shouldn't I?

In my last benchmark showdown I used Postmark. In this test scenario I'm not really concerned about creating/deleting a lot of files because I'm willing to take a performance hit on that if it means reading it from the disk faster.

>> Bonnie++ Results