Welcome to the machine ...

Freitag Mai 29, 2009

FileBench ported to Mac OS X

I recently ported the benchmarking utility FileBench to Mac OSX. The chief attraction of FileBench is, that one can define the I/O operations put on the filesystem in something called the "Workload Definition Language". So - it's not a fixed benchmark, burden the system with the very same load pattern over and over again but an amazingly flexible Software which can simulate various workloads like file servers, databases, web servers etc.

FileBench was created by Richard McDougall who is a Solaris and Systems programming guru and one of the authors of the famous Solaris Internals book. More information about FileBench could be found at the official touch base here. FileBench is now included in OpenSolaris. Take a look here for the sources. There is also a SourceForge-Version landing page for sources which build on Linux.

Here's an example, how this definition language looks like:

set $dir=/tmp
set $nfiles=1000
set $meandirwidth=1000000
set $filesize=16k
set $nthreads=16
set $meaniosize=16k
 
define fileset name=bigfileset,path=$dir,size=$filesize,entries=$nfiles,dirwidth=$meandirwidth,prealloc=80
 
define process name=filereader,instances=1
{
  thread name=filereaderthread,memsize=10m,instances=$nthreads
  {
    flowop deletefile name=deletefile1,filesetname=bigfileset
    flowop createfile name=createfile2,filesetname=bigfileset,fd=1
    flowop appendfilerand name=appendfilerand2,iosize=$meaniosize,fd=1
    flowop fsync name=fsyncfile2,fd=1
    flowop closefile name=closefile2,fd=1
    flowop openfile name=openfile3,filesetname=bigfileset,fd=1
    flowop readwholefile name=readfile3,fd=1
    flowop appendfilerand name=appendfilerand3,iosize=$meaniosize,fd=1
    flowop fsync name=fsyncfile3,fd=1
    flowop closefile name=closefile3,fd=1
    flowop openfile name=openfile4,filesetname=bigfileset,fd=1
    flowop readwholefile name=readfile4,fd=1
    flowop closefile name=closefile4,fd=1
  }
}

On the FileBench homepage and in the distributions "workloads" directory one can find many more examples how to write these definitions.

Here is an console-log of loading and running the varmail benchmark:

$ /opt/filebench/bin/go_filebench

FileBench Version 1.4.4
filebench> load varmail
1364: 10.634: Varmail Version 2.1 personality successfully loaded
1364: 10.634: Usage: set $dir=<dir>
1364: 10.634: set $filesize=<size> defaults to 16384
1364: 10.634: set $nfiles=<value> defaults to 1000
1364: 10.634: set $nthreads=<value> defaults to 16
1364: 10.634: set $meaniosize=<value> defaults to 16384
1364: 10.634: set $readiosize=<size> defaults to 1048576
1364: 10.634: set $meandirwidth=<size> defaults to 1000000
1364: 10.634: (sets mean dir width and dir depth is calculated as log (width, nfiles)
1364: 10.634: dirdepth therefore defaults to dir depth of 1 as in postmark
1364: 10.634: set $meandir lower to increase depth beyond 1 if desired)
1364: 10.634:
1364: 10.634: run runtime (e.g. run 60)
filebench> set $dir=/tmp
filebench> run 10
1364: 26.250: Creating/pre-allocating files and filesets
1364: 26.252: Fileset bigfileset: 1000 files, 0 leafdirs avg dir = 0, avg depth = 0.0, mbytes=4602678819172646912
1364: 26.310: Removed any existing fileset bigfileset in 1 seconds
1364: 26.314: making tree for filset /tmp/bigfileset
1364: 26.314: Creating fileset bigfileset...
1364: 27.009: Preallocated 807 of 1000 of fileset bigfileset in 1 seconds
1364: 27.009: waiting for fileset pre-allocation to finish
1364: 27.009: Starting 1 filereader instances
1364: 27.009: Starting 16 filereaderthread threads
1364: 28.010: Running...
1364: 38.011: Run took 10 seconds...
1364: 38.012: Per-Operation Breakdown
closefile4 95ops/s 0.0mb/s 0.0ms/op 0us/op-cpu
readfile4 95ops/s 1.3mb/s 16.8ms/op 0us/op-cpu
openfile4 95ops/s 0.0mb/s 0.0ms/op 0us/op-cpu
closefile3 95ops/s 0.0mb/s 0.1ms/op 0us/op-cpu
fsyncfile3 96ops/s 0.0mb/s 62.1ms/op 0us/op-cpu
appendfilerand3 96ops/s 0.7mb/s 4.1ms/op 0us/op-cpu
readfile3 96ops/s 1.3mb/s 17.1ms/op 0us/op-cpu
openfile3 96ops/s 0.0mb/s 0.0ms/op 0us/op-cpu
closefile2 96ops/s 0.0mb/s 0.1ms/op 0us/op-cpu
fsyncfile2 97ops/s 0.0mb/s 59.7ms/op 0us/op-cpu
appendfilerand2 97ops/s 0.7mb/s 0.1ms/op 0us/op-cpu
createfile2 97ops/s 0.0mb/s 0.8ms/op 0us/op-cpu
deletefile1 97ops/s 0.0mb/s 3.2ms/op 0us/op-cpu

1364: 38.012:
IO Summary: 12515 ops, 1251.4 ops/s, (192/194 r/w) 4.2mb/s, 3587919346114us cpu/op, 40.9ms latency
1364: 38.012: Shutting down processes
filebench> stats dump "stats.varmail"
1364: 69.662: Stats dump to file 'stats.varmail'
1364: 69.662: in statsdump stats.varmail
filebench> quit
$ more stats.varmail
Flowop totals:
closefile4 95ops/s 0.0mb/s 0.0ms/op 0us/op-cpu
readfile4 95ops/s 1.3mb/s 16.8ms/op 0us/op-cpu
openfile4 95ops/s 0.0mb/s 0.0ms/op 0us/op-cpu
closefile3 95ops/s 0.0mb/s 0.1ms/op 0us/op-cpu
fsyncfile3 96ops/s 0.0mb/s 62.1ms/op 0us/op-cpu
appendfilerand3 96ops/s 0.7mb/s 4.1ms/op 0us/op-cpu
readfile3 96ops/s 1.3mb/s 17.1ms/op 0us/op-cpu
openfile3 96ops/s 0.0mb/s 0.0ms/op 0us/op-cpu
closefile2 96ops/s 0.0mb/s 0.1ms/op 0us/op-cpu
fsyncfile2 97ops/s 0.0mb/s 59.7ms/op 0us/op-cpu
appendfilerand2 97ops/s 0.7mb/s 0.1ms/op 0us/op-cpu
createfile2 97ops/s 0.0mb/s 0.8ms/op 0us/op-cpu
deletefile1 97ops/s 0.0mb/s 3.2ms/op 0us/op-cpu

IO Summary: 12515 ops 1251.4 ops/s, 192/194 r/w 4.2mb/s, 3587919346114uscpu/op

I've created a Diskimage including an Apple Installer package which installs the software into /opt/filebench. Just download the image file and you could install the program in an Apple fashion:


As a next step the process of creating the Mac-version from the OpenSolaris sources should be automated like this:


Here's the first version of the script, the skeleton and a set of patches to automate this.

Dienstag Mai 12, 2009

UNIX DaytimeServer - the Scala way

I'm playing around with Scala. No, frankly I'm in love with it. While reading through "Programming in Scala" (which is a perfect introduction into the subject), I converted the Daytime Server into a Scala version. Get the NetBeans Project.

Here's how it looks like:

 

 1 package daytimeserver
 2 
 3 import java.io.IOException
 4 import java.nio.ByteBuffer
 5 import java.util.Date
 6 import java.nio.channels.SelectionKey
 7 
 8 import com.sun.grizzly.{ ProtocolFilter, Context, Controller, ProtocolChain,
 9                          DefaultProtocolChain, ProtocolChainInstanceHandler, TCPSelectorHandler }
10 
11 import com.sun.grizzly.util.OutputWriter
12 
13 class MyTCPSelectorHandler extends TCPSelectorHandler {
14 
15     override def onAcceptInterest(key:SelectionKey, ctx: Context ) : Boolean = {
16         var channel = acceptWithoutRegistration(key)
17 
18         if (channel != null) {
19             configureChannel(channel)
20             var readKey = channel.register(selector, SelectionKey.OP_READ | SelectionKey.OP_WRITE)
21             readKey.attach(System.currentTimeMillis())
22         }
23         
24         false
25     }
26 }
27 
28 class MyProtocolChainInstanceHandler extends ProtocolChainInstanceHandler {
29 
30     private var protocolChain = new DefaultProtocolChain()
31 
32     def poll() : ProtocolChain = protocolChain
33     def offer(instance: ProtocolChain ) : Boolean = true
34 }
35 
36 class DaytimeWriterFilter extends ProtocolFilter {
37 
38     def execute(ctx: Context) : Boolean = {
39         if (ctx.getProtocol() == Controller.Protocol.TCP) {
40             var channel = ctx.getSelectionKey().channel()
41             var now = (new Date()).toString() + "\n"
42             var buffer =  ByteBuffer.wrap(now.getBytes())
43 
44             OutputWriter.flushChannel(channel, buffer)
45 
46             channel.close()
47             buffer.clear()
48         }
49         false
50     }
51 
52     def postExecute(ctx : Context ) : Boolean  = false
53 }
54 
55 object DaytimeServer extends Application {
56 
57     val PORT : Int = 1405
58     var controller = new Controller()
59     var tcpSelectorHandler = new MyTCPSelectorHandler()
60     var pciHandler = new MyProtocolChainInstanceHandler()
61     var protocolChain = pciHandler.poll()
62 
63     tcpSelectorHandler.setPort(PORT)
64     controller.addSelectorHandler(tcpSelectorHandler)
65     controller.setProtocolChainInstanceHandler(pciHandler)
66     protocolChain.addFilter(new DaytimeWriterFilter())
67 
68     try {
69         println("Starting Daytime server running on port " + PORT)
70         controller.start()
71     } catch {
72         case ex: IOException => println("Exception:" + ex)
73     }
74 }
75 

Calendar

Feeds

Search

Links

Navigation

Referrers