Getting sense of ephemeral store behavior under OpenSolaris (UFS,ZFS)
Tuesday May 20, 2008
To get impression of ephemeral store behavior under OpenSolaris (UFS,ZFS) I use bonnie++, compiled from source with debug output. Pre-compiled binary can be get from [www.sunfreeware.com SunFreeware]
This entry is part of 'OpenSolaris on Amazon EC2' workshopI will skip file based tests and use large file pool test only with size configuration more then 2x of instance memory, because main purpose of test is evaluate "Amazon EC2 documented ephemeral storage feature fist access can be significantly slower ".
Disclaimer
This measurements are done for purpose to get impression of possible bottlenecks, not with any intend to perform deep and log durations performance test.
Hints
On OpenSolaris Nevada line data drive devices are c0dXpX, on OpenSolaris Indiana line data drive devices are c4dXpX
Conclusion
Synthetic test prove then ephemeral storage feature fist access can be significantly slower can be real isuse, we need to take it in account for example wen we will try to backup instance data on OS shutdown.
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
1st ZFSonEC2s 5000M 22925 40 46275 27 27848 12 42671 92 74019 5 296.0 0
2nd ZFSonEC2s 5000M 31250 53 47851 26 32608 14 42355 91 73182 5 272.6 0
3rd ZFSonEC2s 5000M 33538 58 47887 27 32703 14 43116 91 72836 5 287.7 0
Can we do more ? YES
Suggestion are welcome, like:
- Interesting will be to get sense if ZFS feature copy-on-write can affect real user in EC2 env ?
- Can we make scrip which will auto import data pool on reboot (ramdisk is in Dom0 and is not updated) ?
- Can we on Indiana line use ZFS compression to minimalize space used for example by tret based logs ?
- How to best tune ZFS for Amazon EC2 usage: >
- Create one smaller pool and pre-write is with zeros, rest of disk spaces use as slower storage ?
- Tune ZFS memory cache size for each EC2 instance size ?
You can ask you questions at: Help us tune ZFS for usage on 'OpenSolaris on Amazon EC2' project
m1.small disk perf on UFS (1 disk)
m1.small provide to 1x 150GB of Moderate ephemeral storage. Often is used in Linux boxes as plain ext3.
fdisk -B /dev/rdsk/c0d1p0
echo y| newfs /dev/rdsk/c0d1s0
mount -F ufs /dev/dsk/c0d1s0 /mnt
df -h | grep mnt
/dev/dsk/c0d1s0 147G 1.7G 144G 2% /mnt
/usr/bin/time bonnie++ -s 5000 -x 3 -u 0 -n0 -d /mnt -m UFSonEC2s
Write speed on first run was significantly slower, its Amazon AWS documented ephemeral storage feature.
Best results are:
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
UFSonEC2s 5000M 31170 98 54442 75 13512 12 34410 54 71023 32 350.2 1
m1.small disk perf on ZFS (1 disk)
m1.small provide to 1x 150GB of Moderate ephemeral storage. Often is used in Linux boxes as plain ext3.
zpool create mypool c0d1p0
zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
mypool 149G 111K 149G 0% ONLINE -
zfs create mypool/zfs
zfs set mountpoint=/zfs mypool/zfs
df -h | grep /zfs
mypool/zfs 146G 18K 146G 1% /zfs
/usr/bin/time bonnie++ -s 5000 -x 3 -u 0 -n0 -d /mnt -m ZFSonEC2s
Write speed on first run was upto significantly slower, its Amazon AWS documented ephemeral storage feature.
Best results are:
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
ZFSonEC2s 5000M 33538 58 47887 27 32703 14 43116 91 72836 5 287.7 0
m1.large disk perf on ZFS mirror (2 disks)
m1.large provide to 2x 420GB of Hi-Performance ephemeral storage. Often is used in Linux boxes as mirror.
zpool create mypool mirror c0d1p0 c0d2p0
zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
mypool 416G 111K 416G 0% ONLINE -
zfs create mypool/zfs
zfs set mountpoint=/zfs mypool/zfs
df -h | grep /zfs
mypool/zfs 410G 18K 410G 1% /zfs
/usr/bin/time bonnie++ -s 40000 -x 3 -u 0 -n0 -d /zfs -m zfsonEC2x
Write speed on first run was significantly slower, its Amazon AWS documented ephemeral storage feature.
Best results are:
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
ZFSonEC2x 40000M 68032 71 68953 23 34876 15 73828 92 80325 14 374 2
m1.xlarge disk perf on ZFS raidz (4 disks)
m1.xlarge provide to 4x 420GB of Hi-Performance ephemeral storage. Often is used in Linux boxes as RAID.
zpool create mypool c0d1p0 c0d2p0 c0d3p0 c0d4p0
list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
mypool 1.62T 114K 1.62T 0% ONLINE -
df -h | grep /zfs
mypool/zfs 1.6T 18K 1.6T 1% /zfs
/usr/bin/time bonnie++ -s 80000 -x 3 -u 0 -n0 -d /zfs -m zfsonEC2xL
Write speed on first run was significantly slower, its Amazon AWS documented ephemeral storage feature.
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
ZFSonEC2xL 80000M 93203 97 182347 67 88746 45 78486 98 254752 54 557.7 3
|
|
|
|
