Saturday November 11, 2006

As the number of snapshots grows I started wondering how much space they are really taking up on the home server. This is pretty much also shows how much data gets modified after being initially created. I would guess not much as the majority of the data on the server would be:

1. Solaris install images. Essentially read only.

2. Photographs.

3. Music mostly in the form of iTunes directories.

Running this command line get the result:

zfs get -rHp used $(zpool list -H -o name ) |\
nawk '/@/ && $2 == "used" { tot++; total_space+=$3 ;\
        if ( $3 == 0 ) { empty++ }} \
END { printf("%d snapshots\n%d empty snapshots\n%2.2f G in %d snapshots\n", tot, \
        empty, total_space/(1024^3), tot - empty ) }'
68239 snapshots
63414 empty snapshots
2.13 G in 4825 snapshots'
: pearson TS 15 $; zfs get used $(zpool list -H -o name )
NAME  PROPERTY  VALUE  SOURCE
tank  used      91.2G  -
: pearson TS 16 $;

So I only have 2.13G of data saved in snapshots out of 91.2 G of data. Not really a surprising result. The biggest user of space for snapshots is one file system. The one that contains planetcycling.org. As the planet gets updated every 30 minutes and the data is only indirectly controlled by me I'm not shocked by this. I would expect the amount to stabilize over time as the system and to that end I will note the current usage:

zfs get -rHp used tank/fs/web|\
nawk '/@/ && $2 == "used" { tot++; total_space+=$3 ;\
        if ( $3 == 0 ) { empty++ }} \
END { printf("%d snapshots\n%d empty snapshots\n%2.2f G in %d snapshots\n", tot,
        empty, total_space/(1024^3), tot - empty ) }'
1436 snapshots
789 empty snapshots
0.98 G in 647 snapshots

All this caused me to look a bit harder at the zfs_snapshot_clean script I have as it appeared to be keeping some really old snapshots from some of the classes that I did not expect. Now while the 68000 snapshots were having no negative impact on the running of the system it it was not right. There were two issues. First it was sorting the list of snapshots using the snapshot creation time, which was correct, but it was sorting in reverse order which was not. Secondly I was keeping a lot more of the hourly snapshots than I intended.

After fixing this and running the script (you can download it from here) there was a bit of a snapshot massacre leading to a lot less snapshots:

zfs get -rHp used $(zpool list -H -o name ) |\
nawk '/@/ && $2 == "used" { tot++; total_space+=$3 ;\
        if ( $3 == 0 ) { empty++ }} \
END { printf("%d snapshots\n%d empty snapshots\n%2.2f G in %d snapshots\n", tot, \
        empty, total_space/(1024^3), tot - empty ) }'
25512 snapshots
23445 empty snapshots
2.20 G in 2067 snapshots

Only 25000 snapshots, much better, most of them remain empty.

Tags: home server ZFS