The dot in ... --- ...
Chris Gerhard's Weblog
Tuesday June 30, 2009
Using dtrace to track down memory leaks I've been working with a customer to try and find a memory “leak” in their application. Many things have been tried, libumem, and the mdb ::findleaks command all with no success.
So I was, as I am sure others before me have, pondering if you could use dtrace to do this. Well I think you can. I have a script that puts probes into malloc et al and counts how often they are called by this thread and when they are freed often free is called.
Then in the entry probe of the target application note away how many calls there have been to the allocators and how many to free and with a bit of care realloc. Then in the return probe compare the number of calls to allocate and free with the saved values and aggregate the results. The principle is that you find the routines that are resulting in allocations that they don't clear up. This should give you a list of functions that are possible leakers which you can then investigate1.
Using the same technique I for getting dtrace to “follow fork” that I described here I ran this up on diskomizer, a program that I understand well and I'm reasonably sure does not have systemic memory leaks. The dtrace script reports three sets of results.
A count of how many times each routine and it's descendents have called a memory allocator.
A count of how many times each routine and it's descendents have called free or realloc with a non NULL pointer as the first argument.
The difference between the two numbers above.
Then with a little bit of nawk to remove all the functions for which the counts are zero gives:
# /usr/sbin/dtrace -Z -wD TARGET_OBJ=diskomizer2 -o /tmp/out-us \ -s /tmp/followfork.d \ -Cs /tmp/allocated.d -c \ "/opt/SUNWstc-diskomizer/bin/sparcv9/diskomizer -f /devs -f background \ -o background=0 -o SECONDS_TO_RUN=1800" dtrace: failed to compile script /tmp/allocated.d: line 20: failed to create entry probe for 'realloc': No such process dtrace: buffer size lowered to 25m dtrace: buffer size lowered to 25m dtrace: buffer size lowered to 25m dtrace: buffer size lowered to 25m # nawk '$1 != 0 { print $0 }' < /tmp/out.3081 allocations 1 diskomizer`do_dev_control 1 diskomizer`set_dev_state 1 diskomizer`set_state 3 diskomizer`report_exit_reason 6 diskomizer`alloc_time_str 6 diskomizer`alloc_time_str_fmt 6 diskomizer`update_aio_read_stats 7 diskomizer`cancel_all_io 9 diskomizer`update_aio_write_stats 13 diskomizer`cleanup 15 diskomizer`update_aio_time_stats 15 diskomizer`update_time_stats 80 diskomizer`my_calloc 240 diskomizer`init_read 318 diskomizer`do_restart_stopped_devices 318 diskomizer`start_io 449 diskomizer`handle_write 606 diskomizer`do_new_write 2125 diskomizer`handle_read_then_write 2561 diskomizer`init_buf 2561 diskomizer`set_io_len 58491 diskomizer`handle_read 66255 diskomizer`handle_write_then_read 124888 diskomizer`init_read_buf 124897 diskomizer`do_new_read 127460 diskomizer`expect_signal freecount 1 diskomizer`expect_signal 3 diskomizer`report_exit_reason 4 diskomizer`close_and_free_paths 6 diskomizer`update_aio_read_stats 9 diskomizer`update_aio_write_stats 11 diskomizer`cancel_all_io 15 diskomizer`update_aio_time_stats 15 diskomizer`update_time_stats 17 diskomizer`cleanup 160 diskomizer`init_read 318 diskomizer`do_restart_stopped_devices 318 diskomizer`start_io 442 diskomizer`handle_write 599 diskomizer`do_new_write 2125 diskomizer`handle_read_then_write 2560 diskomizer`init_buf 2560 diskomizer`set_io_len 58491 diskomizer`handle_read 66246 diskomizer`handle_write_then_read 124888 diskomizer`do_new_read 124888 diskomizer`init_read_buf 127448 diskomizer`cancel_expected_signal mismatch_count -127448 diskomizer`cancel_expected_signal -4 diskomizer`cancel_all_io -4 diskomizer`cleanup -4 diskomizer`close_and_free_paths 1 diskomizer`do_dev_control 1 diskomizer`init_buf 1 diskomizer`set_dev_state 1 diskomizer`set_io_len 1 diskomizer`set_state 6 diskomizer`alloc_time_str 6 diskomizer`alloc_time_str_fmt 7 diskomizer`do_new_write 7 diskomizer`handle_write 9 diskomizer`do_new_read 9 diskomizer`handle_write_then_read 80 diskomizer`init_read 80 diskomizer`my_calloc 127459 diskomizer`expect_signal #
From the above you can see that there are two functions that create and free the majority of the allocations and the allocations almost match each other, which is expected as they are effectively constructor and destructor for each other. The small mismatch is not unexpected in this context.
However it is the vast number of functions that are not listed at all as they and their children make no calls to the memory allocator or have exactly matching allocation and free that are important here. Those are the functions that we have just ruled out.
From here it is easy now to drill down on the functions that are interesting you, ie the ones where there are unbalanced allocations.
I've uploaded the files allocated.d and followfork.d so you can see the details. If you find it useful then let me know.
1Unfortunately the list is longer than you want as on SPARC it includes any functions that don't have their own stack frame due to the way dtrace calculates ustackdepth, which the script makes use of.
2The script only probes particular objects, in this case the main diskomizer binary, but you can limit it to a particular library or even a particular set of entry points based on name if you edit the script.
( Jun 30 2009, 12:14:33 PM BST ) Permalink Trackback
Saturday June 27, 2009
Follow fork for dtrace pid provider? There is a ongoing request to have follow fork functionality for the dtrace pid provider but so far no one has stood upto the plate for that RFE. In the mean time my best workaround for this is this:
cjg@brompton:~/lang/d$ cat followfork.d
proc:::start
/ppid == $target/
{
stop();
printf("fork %d\n", pid);
system("dtrace -qs child.d -p %d", pid);
}
cjg@brompton:~/lang/d$ cat child.d
pid$target::malloc:entry
{
printf("%d %s:%s %d\n", pid, probefunc, probename, ustackdepth)
}
cjg@brompton:~/lang/d$ pfexec /usr/sbin/dtrace -qws followfork.d -s child.d -p 26758
26758 malloc:entry 22
26758 malloc:entry 15
26758 malloc:entry 18
26758 malloc:entry 18
26758 malloc:entry 18
fork 27548
27548 malloc:entry 7
27548 malloc:entry 7
27548 malloc:entry 18
27548 malloc:entry 16
27548 malloc:entry 18
Clearly you can have the child script do what ever you wish.
Better solutions are welcome!
( Jun 27 2009, 01:25:52 PM BST ) Permalink Trackback
Thursday June 18, 2009
Diskomizer Open Sourced I'm pleased to announce the Diskomizer test suite has been open sourced. Diskomizer started life in the dark days before ZFS when we lived in a world full1 of bit flips, phantom writes, phantom reads, misplaced writes and misplaced reads.
With a storage architecture that does not use end to end data verification the best that you can hope for was that your application will spot errors quickly and allow you to diagnose the broken part or bug quickly. Diskomizer was written to be a “simple” application that could verify all the data paths worked correctly and worked correctly under extreme load. It has been and is used by support, development and test groups for system verification.
For more details of what Diskomizer is and how to build and install read these pages:
http://www.opensolaris.org/os/community/storage/tests/Diskomizer/
You can download the source and precompiled binaries from:
http://dlc.sun.com/osol/test/downloads/current/
and can browse the source here:
http://src.opensolaris.org/source/xref/test/stcnv/usr/src/tools/diskomizer
Using Diskomizer
First remember in most cases Diskomizer will destroy all the data on any target you point it at. So extreme care is advised.
I will say that again.
Diskomizer will destroy all the data on any target that you point it at.
For the purposes of this explanation I am going to use ZFS volumes so that I can create and destroy them with confidence that I will not be destroying someone's data.
First lets create some volumes.
# i=0 # while (( i < 10 )) do zfs create -V 10G storage/chris/testvol$i let i=i+1 done #
Now write the names of the devices you wish to test into a file after the key “DEVICE=”:
# echo DEVICE= /dev/zvol/rdsk/storage/chris/testvol* > test_opts
Now start the test. When you installed diskomizer it put the standard option files on the system and has a search path so that it can find them. I'm using the options file “background” which will make the test go into the back ground redirecting the output into a file called “stdout” and any errors into a file called “stderr”:
# /opt/SUNWstc-diskomizer/bin/diskomizer -f test_opts -f background #
If Diskomizer has any problems with the configuration it will report them and exit. This is to minimize the risk to your data from a typo. Also the default is to open devices and files exclusively to again reduce the danger to your data (and to reduce false positives where it detects data corruption).
Once up and running it will report it's progress for each process in the output file:
# tail -5 stdout PID 1152: INFO /dev/zvol/rdsk/storage/chris/testvol7 (zvol0:a)2 write times (0.000,0.049,6.068) 100% PID 1152: INFO /dev/zvol/rdsk/storage/chris/testvol1 (zvol0:a) write times (0.000,0.027,6.240) 100% PID 1152: INFO /dev/zvol/rdsk/storage/chris/testvol7 (zvol0:a) read times (0.000,1.593,6.918) 100% PID 1154: INFO /dev/zvol/rdsk/storage/chris/testvol9 (zvol0:a) write times (0.000,0.070,6.158) 79% PID 1151: INFO /dev/zvol/rdsk/storage/chris/testvol0 (zvol0:a) read times (0.000,0.976,7.523) 100% #
meanwhile all the usual tools can be used to view the IO:
# zpool iostat 5 5
capacity operations bandwidth
pool used avail read write read write
---------- ----- ----- ----- ----- ----- -----
storage 460G 15.9T 832 4.28K 6.49M 31.2M
storage 460G 15.9T 3.22K 9.86K 25.8M 77.2M
storage 460G 15.9T 3.77K 6.04K 30.1M 46.8M
storage 460G 15.9T 2.90K 11.7K 23.2M 91.4M
storage 460G 15.9T 3.63K 5.86K 29.1M 45.7M
#
1Full may be an exaggeration but we will never know thanks to the fact that the data loss was silent. There were enough cases reported where there was reason to doubt whether the data was good to keep me busy.
( Jun 18 2009, 10:42:10 AM BST ) Permalink Trackback
Sunday June 07, 2009
OpenSolaris 2009.06 After a week of running 2009.06 on my Toshiba Tecra M9 having upgraded from 2008.11 I'm in a position to comment on it. I've been able to remove all the workarounds I had on the system. Nwam appears to work even in the face a suspend and resume. Removalbe media also appears to be robust without the occasional panics that would happen when I removed the SD card with 2008.11.
Feature wise the things I have noticed are the new tracker system for searching files, but it seems to be completely non functional. The big improvements are in the support for the special keys on the Toshiba and the volume control, which unlike the volume on the M2 is a logical control so requires software support. 2009.06 has this support along with support fo the number pad, brightness and mute buttons.
The downside was hitting this bug. This pretty much renders resume useless and I was about to go back to 2008.06 when the bug was updated to say it will be fixed in the first update release and in the mean time there are binary packages. So after creating a new boot enviroment so that I have an unpatched one to switch to when the fix gets into the support repository I have applied the patch. Seems to work which is very pleasing as it has not taken me long to get used to the brightness buttons working.
( Jun 07 2009, 05:45:49 PM BST ) Permalink Trackback
Tuesday May 26, 2009
Why everyone should be using ZFS It is at times like these that I'm glad I use ZFS at home.
pool: tank
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: http://www.sun.com/msg/ZFS-8000-9P
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror ONLINE 0 0 0
c20t0d0s7 ONLINE 6 0 4
c21t0d0s7 ONLINE 0 0 0
mirror ONLINE 0 0 0
c21t1d0 ONLINE 0 0 0
c20t1d0 ONLINE 0 0 0
errors: No known data errors
: pearson FSS 14 $;
The drive with the errors was also throwing up errors that iostat could report and from it's performance was trying heroicially to give me back data. However it had failed. It's performance was terrible and then it failed to give the right data on 4 occasions. Anyother file system would, if that was user data, just had deliviered it to the user without warning. That bad data could then propergate from there on, probably into my backups. There is certainly no good that could come from that. However ZFS detected and corrected the errors.
Now I have offlined the disk the performance of the system is better but I have no redundancy until the new disk I have just ordered arriaves. Now time to check out Seagate's warranty return system.
( May 26 2009, 10:46:56 AM BST ) Permalink Trackback
Sunday May 10, 2009
Another update to Sun Ray access hours script I have made a change to up access hours script for my Sun Rays. Now the access file can also contain a comma separated list of Sun Ray DTUs so that the control is only applied to those DTUs:
: pearson FSS 3 $; cat /etc/opt/local/access_hours user1:2000:2300:P8.00144f7dc383 user2:2000:2300:P8.00144f57a46f user3:0630:2300 user4:0630:2300 : pearson FSS 4 $;
The practical reason for this is that it allows control of DTUs that are in bedrooms but if the computer is really needed another DTU can be used for homework.
Now that bug 6791062 is fixed the script is safe to use in nevada.
The script is where it always was.
( May 10 2009, 10:54:24 PM BST ) Permalink Trackback
Friday May 08, 2009
Stopping find searching remote directories. Grizzled UNIX users look away now.
The find command is a wonderful thing but there are some uses of it that seem to cause confusion enough that it seems worth documenting them for google. Today's is:
How can I stop find(1) searching remote file systems?
On reading the documentation the “-local” option should be just what you want and it is, but not on it's own. If you just do:
$ find . -local -print
It will indeed only report on files that are on local file systems below the current directory. However it will search the entire directory tree for those local files even if the directory tree is on NFS
To get find to stop searching when it finds a remote file system you need:
$ find . \( ! -local -prune \) -o -print
simple.
( May 08 2009, 03:57:16 PM BST ) Permalink Trackback
Friday May 01, 2009
Installing support certificates in OpenSolaris For some reason you only get the instructions on how to install a certificate to get access to supported or extras updates on your OpenSolaris system after you have downloaded the certificate. Not a big issue as that is generally when you want the instructions. However if you already have your certificates and now want to install them on another system (that you have support for) you can't get the instructions without getting another certificate.
So here are the instructions cut'n'pasted from the support page, as much for me as for you:
How to Install this OpenSolaris 2008.11 standard support Certificate
Download the provided key and certificate files, called OpenSolaris_2008.11_standard_support.key.pem andOpenSolaris_2008.11_standard_support.certificate.pem using the buttons above. Don't worry if you get logged out, or lose the files. You can come back to this site later and re-download them. We'll assume that you downloaded these files into your Desktop folder,~/Desktop/.
Use the following comands to make a directory inside of /var/pkg to store the key and certificate, and copy the key and certificate into this directory. The key files are kept by reference, so if the files become inaccessible to the packaging system, you will encounter errors. Here is how to do it:
$ pfexec mkdir -m 0755 -p /var/pkg/ssl $ pfexec cp -i ~/Desktop/OpenSolaris_2008.11_standard_support.key.pem /var/pkg/ssl $ pfexec cp -i ~/Desktop/OpenSolaris_2008.11_standard_support.certificate.pem /var/pkg/sslAdd the publisher:
$ pfexec pkg set-authority \ -k /var/pkg/ssl/OpenSolaris_2008.11_standard_support.key.pem \ -c /var/pkg/ssl/OpenSolaris_2008.11_standard_support.certificate.pem \ -O https://pkg.sun.com/opensolaris/support/ opensolaris.orgTo see the packages supplied by this authority, try:
$ pkg list -a 'pkg://opensolaris.org/*'
If you use the Package Manager graphical application, you will be able to locate the newly discovered packages when you restart Package Manager.
How to Install this OpenSolaris extras Certificate
Download the provided key and certificate files, called OpenSolaris_extras.key.pem and OpenSolaris_extras.certificate.pem using the buttons above. Don't worry if you get logged out, or lose the files. You can come back to this site later and re-download them. We'll assume that you downloaded these files into your Desktop folder, ~/Desktop/.
Use the following comands to make a directory inside of /var/pkg to store the key and certificate, and copy the key and certificate into this directory. The key files are kept by reference, so if the files become inaccessible to the packaging system, you will encounter errors. Here is how to do it:
$ pfexec mkdir -m 0755 -p /var/pkg/ssl $ pfexec cp -i ~/Desktop/OpenSolaris_extras.key.pem /var/pkg/ssl $ pfexec cp -i ~/Desktop/OpenSolaris_extras.certificate.pem /var/pkg/sslAdd the publisher:
$ pfexec pkg set-authority \ -k /var/pkg/ssl/OpenSolaris_extras.key.pem \ -c /var/pkg/ssl/OpenSolaris_extras.certificate.pem \ -O https://pkg.sun.com/opensolaris/extra/ extraTo see the packages supplied by this authority, try:
$ pkg list -a 'pkg://extra/*'If you use the Package Manager graphical application, you will be able to locate the newly discovered packages when you restart Package Manager.
( May 01 2009, 12:27:21 PM BST ) Permalink Trackback
Monday April 20, 2009
Off to Newcastle for the mash up This is worse than being on a mobile, but I'm on the train o.k.?
Tomorrow I will be a the System Admin Mash up event at Newcastle. If you are going to be there I suggest you don't bother asking us about Sun/Oracle and instead go straight to www.oracle.com/sun then you will know as much as us.
( Apr 20 2009, 08:46:28 PM BST ) Permalink Trackback
Sunday April 19, 2009
User and group quotas for ZFS! This push will be very popular amoung those who are managing servers with thousands of users:
Repository: /export/onnv-gate Total changesets: 1 Changeset: f41cf682d0d3 Comments: PSARC/2009/204 ZFS user/group quotas & space accounting 6501037 want user/group quotas on ZFS 6830813 zfs list -t all fails assertion 6827260 assertion failed in arc_read(): hdr == pbuf->b_hdr 6815592 panic: No such hold X on refcount Y from zfs_znode_move 6759986 zfs list shows temporary %clone when doing online zfs recv
User quotas for zfs has been the feature I have been asked about most when talking to customers. This probably relfects that most customers are simply blown away by the other features of ZFS and the only missing feature was user quotas if you have a large user base.
( Apr 19 2009, 04:18:44 PM BST ) Permalink Trackback
Tuesday April 14, 2009
zfs list -d I've just pushed the changes for zfs list that give it a -d option to limit the depth to which recursive listings will go. This is of most use when you wish to list the snapshots of a given data set and only the snapshots of that data set.
PSARC
2009/171 zfs list -d and zfs get -d
6762432
zfs list --depth
Before this you could achieve this using a short pipe line which while it produced the correct results was horribly inefficient and very slow for datasets that had lots of descendents.
: v4u-1000c-gmp03.eu TS 6 $; zfs list -t snapshot rpool | grep '^rpool@' rpool@spam 0 - 64K - rpool@two 0 - 64K - : v4u-1000c-gmp03.eu TS 7 $; zfs list -d 1 -t snapshot NAME USED AVAIL REFER MOUNTPOINT rpool@spam 0 - 64K - rpool@two 0 - 64K - : v4u-1000c-gmp03.eu TS 8 $;
It will allow the zfs-snapshot service to be much more efficient when it needs to list snapshots. The change will be in build 113.
( Apr 14 2009, 05:52:07 PM BST ) Permalink
Friday April 10, 2009
Native solaris printing working The solution to my HP printer not working on Solaris is to use CUPS. Since I have a full install and solaris now has the packages installed all I had to do was switch the service over:
$ pfexec /usr/sbin/print-service -s cups
then configure the printer using http://localhosts:631 in the same way as I did with ubunto. Now I don't need the virtual machine running which is a bonus. I think cups may be a better fit for me at home since I don't have a nameservice running so the benefits of the lp system are very limited.
The bug is 6826086: printer drivers from package SUNWhpijs are not update with HPLIP
( Apr 10 2009, 11:16:19 AM BST ) Permalink Trackback
Saturday March 28, 2009
snapshot on unlink? This thread on OpenSolaris made me wonder how hard it would be to take a snapshot before any file is deleted. It turns out that using dtrace it is not hard at all. Using dtrace to monitor unlink and unlinkat calls and a short script to take the snapshots:
#!/bin/ksh93
function snapshot
{
eval $(print x=$2)
until [[ "$x" == "/" || -d "$x/.zfs/snapshot" ]]
do
x="${x%/*}"
done
if [[ "$x" == "/" || "$x" == "/tmp" ]]
then
return
fi
if [[ -d "$x/.zfs/snapshot" ]]
then
print mkdir "$x/.zfs/snapshot/unlink_$1"
pfexec mkdir "$x/.zfs/snapshot/unlink_$1"
fi
}
function parse
{
eval $(print x=$4)
if [[ "${x%%/*}" == "" ]]
then
snapshot $1 "$2$4"
else
snapshot $1 "$2$3/$4"
fi
}
pfexec dtrace -wqn 'syscall::fsat:entry /pid != '$$' && uid > 100 && arg0 == 5/ {
printf("%d %d \"%s\" \"%s\" \"%s\"\n",
pid, walltimestamp, root, cwd, copyinstr(arg2)); stop()
}
syscall::unlink:entry /pid != '$$' && uid > 100 / {
printf("%d %d \"%s\" \"%s\" \"%s\"\n",
pid, walltimestamp, root, cwd, copyinstr(arg0)); stop()
}' | while read pid timestamp root cwd file
do
print prun $pid
parse $timestamp $root $cwd $file
pfexec prun $pid
done
Now this is just a Saturday night proof of concept and it should be noted it has a significant performance impact and single threads all calls to unlink.
Also you end up with lots of snapshots:
cjg@brompton:~$ zfs list -t snapshot -o name,used | grep unlink rpool/export/home/cjg@unlink_1238270760978466613 11.9M rpool/export/home/cjg@unlink_1238275070771981963 59K rpool/export/home/cjg@unlink_1238275074501904526 59K rpool/export/home/cjg@unlink_1238275145860458143 34K rpool/export/home/cjg@unlink_1238275168440000379 197K rpool/export/home/cjg@unlink_1238275233978665556 197K rpool/export/home/cjg@unlink_1238275295387410635 197K rpool/export/home/cjg@unlink_1238275362536035217 197K rpool/export/home/cjg@unlink_1238275429554657197 136K rpool/export/home/cjg@unlink_1238275446884300017 350K rpool/export/home/cjg@unlink_1238275491543380576 197K rpool/export/home/cjg@unlink_1238275553842097361 197K rpool/export/home/cjg@unlink_1238275643490236001 63K rpool/export/home/cjg@unlink_1238275644670212158 63K rpool/export/home/cjg@unlink_1238275646030183268 0 rpool/export/home/cjg@unlink_1238275647010165407 0 rpool/export/home/cjg@unlink_1238275648040143427 54K rpool/export/home/cjg@unlink_1238275649030124929 54K rpool/export/home/cjg@unlink_1238275675679613928 197K rpool/export/home/cjg@unlink_1238275738608457151 198K rpool/export/home/cjg@unlink_1238275800827304353 57.5K rpool/export/home/cjg@unlink_1238275853116324001 32.5K rpool/export/home/cjg@unlink_1238275854186304490 53.5K rpool/export/home/cjg@unlink_1238275862146153573 196K rpool/export/home/cjg@unlink_1238275923255007891 55.5K rpool/export/home/cjg@unlink_1238275962114286151 35.5K rpool/export/home/cjg@unlink_1238275962994267852 56.5K rpool/export/home/cjg@unlink_1238275984723865944 55.5K rpool/export/home/cjg@unlink_1238275986483834569 29K rpool/export/home/cjg@unlink_1238276004103500867 49K rpool/export/home/cjg@unlink_1238276005213479906 49K rpool/export/home/cjg@unlink_1238276024853115037 50.5K rpool/export/home/cjg@unlink_1238276026423085669 52.5K rpool/export/home/cjg@unlink_1238276041792798946 50.5K rpool/export/home/cjg@unlink_1238276046332707732 55.5K rpool/export/home/cjg@unlink_1238276098621721894 66K rpool/export/home/cjg@unlink_1238276108811528303 69.5K rpool/export/home/cjg@unlink_1238276132861080236 56K rpool/export/home/cjg@unlink_1238276166070438484 49K rpool/export/home/cjg@unlink_1238276167190417567 49K rpool/export/home/cjg@unlink_1238276170930350786 57K rpool/export/home/cjg@unlink_1238276206569700134 30.5K rpool/export/home/cjg@unlink_1238276208519665843 58.5K rpool/export/home/cjg@unlink_1238276476484690821 54K rpool/export/home/cjg@unlink_1238276477974663478 54K rpool/export/home/cjg@unlink_1238276511584038137 60.5K rpool/export/home/cjg@unlink_1238276519053902818 71K rpool/export/home/cjg@unlink_1238276528213727766 62K rpool/export/home/cjg@unlink_1238276529883699491 47K rpool/export/home/cjg@unlink_1238276531683666535 3.33M rpool/export/home/cjg@unlink_1238276558063169299 35.5K rpool/export/home/cjg@unlink_1238276559223149116 62.5K rpool/export/home/cjg@unlink_1238276573552877191 35.5K rpool/export/home/cjg@unlink_1238276584602668975 35.5K rpool/export/home/cjg@unlink_1238276586002642752 53K rpool/export/home/cjg@unlink_1238276586522633206 51K rpool/export/home/cjg@unlink_1238276808718681998 216K rpool/export/home/cjg@unlink_1238276820958471430 77.5K rpool/export/home/cjg@unlink_1238276826718371992 51K rpool/export/home/cjg@unlink_1238276827908352138 51K rpool/export/home/cjg@unlink_1238276883227391747 198K rpool/export/home/cjg@unlink_1238276945366305295 58.5K rpool/export/home/cjg@unlink_1238276954766149887 32.5K rpool/export/home/cjg@unlink_1238276955946126421 54.5K rpool/export/home/cjg@unlink_1238276968985903108 52.5K rpool/export/home/cjg@unlink_1238276988865560952 31K rpool/export/home/cjg@unlink_1238277006915250722 57.5K rpool/export/home/cjg@unlink_1238277029624856958 51K rpool/export/home/cjg@unlink_1238277030754835625 51K rpool/export/home/cjg@unlink_1238277042004634457 51.5K rpool/export/home/cjg@unlink_1238277043934600972 52K rpool/export/home/cjg@unlink_1238277045124580763 51K rpool/export/home/cjg@unlink_1238277056554381122 51K rpool/export/home/cjg@unlink_1238277058274350998 51K rpool/export/home/cjg@unlink_1238277068944163541 59K rpool/export/home/cjg@unlink_1238277121423241127 32.5K rpool/export/home/cjg@unlink_1238277123353210283 53.5K rpool/export/home/cjg@unlink_1238277136532970668 52.5K rpool/export/home/cjg@unlink_1238277152942678490 0 rpool/export/home/cjg@unlink_1238277173482320586 0 rpool/export/home/cjg@unlink_1238277187222067194 49K rpool/export/home/cjg@unlink_1238277188902043005 49K rpool/export/home/cjg@unlink_1238277190362010483 56K rpool/export/home/cjg@unlink_1238277228691306147 30.5K rpool/export/home/cjg@unlink_1238277230021281988 51.5K rpool/export/home/cjg@unlink_1238277251960874811 57K rpool/export/home/cjg@unlink_1238277300159980679 30.5K rpool/export/home/cjg@unlink_1238277301769961639 50K rpool/export/home/cjg@unlink_1238277302279948212 49K rpool/export/home/cjg@unlink_1238277310639840621 28K rpool/export/home/cjg@unlink_1238277314109790784 55.5K rpool/export/home/cjg@unlink_1238277324429653135 49K rpool/export/home/cjg@unlink_1238277325639636996 49K rpool/export/home/cjg@unlink_1238277360029166691 356K rpool/export/home/cjg@unlink_1238277375738948709 55.5K rpool/export/home/cjg@unlink_1238277376798933629 29K rpool/export/home/cjg@unlink_1238277378458911557 50K rpool/export/home/cjg@unlink_1238277380098888676 49K rpool/export/home/cjg@unlink_1238277397738633771 48K rpool/export/home/cjg@unlink_1238277415098386055 49K rpool/export/home/cjg@unlink_1238277416258362893 49K rpool/export/home/cjg@unlink_1238277438388037804 57K rpool/export/home/cjg@unlink_1238277443337969269 30.5K rpool/export/home/cjg@unlink_1238277445587936426 51.5K rpool/export/home/cjg@unlink_1238277454527801430 50.5K rpool/export/home/cjg@unlink_1238277500967098623 196K rpool/export/home/cjg@unlink_1238277562866135282 55.5K rpool/export/home/cjg@unlink_1238277607205456578 49K rpool/export/home/cjg@unlink_1238277608135443640 49K rpool/export/home/cjg@unlink_1238277624875209357 57K rpool/export/home/cjg@unlink_1238277682774484369 30.5K rpool/export/home/cjg@unlink_1238277684324464523 50K rpool/export/home/cjg@unlink_1238277685634444004 49K rpool/export/home/cjg@unlink_1238277686834429223 75.5K rpool/export/home/cjg@unlink_1238277700074256500 48K rpool/export/home/cjg@unlink_1238277701924235244 48K rpool/export/home/cjg@unlink_1238277736473759068 49.5K rpool/export/home/cjg@unlink_1238277748313594650 55.5K rpool/export/home/cjg@unlink_1238277748413593612 28K rpool/export/home/cjg@unlink_1238277750343571890 48K rpool/export/home/cjg@unlink_1238277767513347930 49.5K rpool/export/home/cjg@unlink_1238277769183322087 50K rpool/export/home/cjg@unlink_1238277770343306935 48K rpool/export/home/cjg@unlink_1238277786193093885 48K rpool/export/home/cjg@unlink_1238277787293079433 48K rpool/export/home/cjg@unlink_1238277805362825259 49.5K rpool/export/home/cjg@unlink_1238277810602750426 195K rpool/export/home/cjg@unlink_1238277872911814531 195K rpool/export/home/cjg@unlink_1238277934680920214 195K rpool/export/home/cjg@unlink_1238277997220016825 195K rpool/export/home/cjg@unlink_1238278063868871589 54.5K rpool/export/home/cjg@unlink_1238278094728323253 61K rpool/export/home/cjg@unlink_1238278096268295499 63K rpool/export/home/cjg@unlink_1238278098518260168 52K rpool/export/home/cjg@unlink_1238278099658242516 56K rpool/export/home/cjg@unlink_1238278103948159937 57K rpool/export/home/cjg@unlink_1238278107688091854 54K rpool/export/home/cjg@unlink_1238278113907980286 62K rpool/export/home/cjg@unlink_1238278116267937390 64K rpool/export/home/cjg@unlink_1238278125757769238 196K rpool/export/home/cjg@unlink_1238278155387248061 136K rpool/export/home/cjg@unlink_1238278160547156524 229K rpool/export/home/cjg@unlink_1238278165047079863 351K rpool/export/home/cjg@unlink_1238278166797050407 197K rpool/export/home/cjg@unlink_1238278168907009714 55K rpool/export/home/cjg@unlink_1238278170666980686 341K rpool/export/home/cjg@unlink_1238278171616960684 54.5K rpool/export/home/cjg@unlink_1238278190336630319 777K rpool/export/home/cjg@unlink_1238278253245490904 329K rpool/export/home/cjg@unlink_1238278262235340449 362K rpool/export/home/cjg@unlink_1238278262915331213 362K rpool/export/home/cjg@unlink_1238278264915299508 285K rpool/export/home/cjg@unlink_1238278310694590970 87K rpool/export/home/cjg@unlink_1238278313294552482 66K rpool/export/home/cjg@unlink_1238278315014520386 31K rpool/export/home/cjg@unlink_1238278371773568934 258K rpool/export/home/cjg@unlink_1238278375673503109 198K rpool/export/home/cjg@unlink_1238278440802320314 138K rpool/export/home/cjg@unlink_1238278442492291542 55.5K rpool/export/home/cjg@unlink_1238278445312240229 2.38M rpool/export/home/cjg@unlink_1238278453582077088 198K rpool/export/home/cjg@unlink_1238278502461070222 256K rpool/export/home/cjg@unlink_1238278564359805760 256K rpool/export/home/cjg@unlink_1238278625738732194 63.5K rpool/export/home/cjg@unlink_1238278633428599541 61.5K rpool/export/home/cjg@unlink_1238278634568579678 137K rpool/export/home/cjg@unlink_1238278657838186760 288K rpool/export/home/cjg@unlink_1238278659768151784 223K rpool/export/home/cjg@unlink_1238278661518121640 159K rpool/export/home/cjg@unlink_1238278664378073421 136K rpool/export/home/cjg@unlink_1238278665908048641 138K rpool/export/home/cjg@unlink_1238278666968033048 136K rpool/export/home/cjg@unlink_1238278668887996115 281K rpool/export/home/cjg@unlink_1238278670307970765 227K rpool/export/home/cjg@unlink_1238278671897943665 162K rpool/export/home/cjg@unlink_1238278673197921775 164K rpool/export/home/cjg@unlink_1238278674027906895 164K rpool/export/home/cjg@unlink_1238278674657900961 165K rpool/export/home/cjg@unlink_1238278675657885128 165K rpool/export/home/cjg@unlink_1238278676647871187 241K rpool/export/home/cjg@unlink_1238278678347837775 136K rpool/export/home/cjg@unlink_1238278679597811093 199K rpool/export/home/cjg@unlink_1238278687297679327 197K rpool/export/home/cjg@unlink_1238278749616679679 197K rpool/export/home/cjg@unlink_1238278811875554411 56.5K cjg@brompton:~$
Good job that snapshots are cheap. I'm not going to be doing this all the time but it makes you think what could be done.
( Mar 28 2009, 10:52:35 PM GMT ) Permalink Trackback
Friday March 27, 2009
zfs list webrev I've just posted the webrev for review for an RFE to “zfs list”:
PSARC 2009/171 zfs list -d and zfs get
-d
6762432
zfs list --depth
This will allow you to limit the depth to which a recursive listing of zfs file systems will go. This is particularly useful if you only want to list the snapshots of the current file system.
The webrev is here:
http://cr.opensolaris.org/~cjg/zfs_list/zfs_list-d-2/
Comments welcome.
( Mar 27 2009, 06:05:34 PM GMT ) Permalink
Sunday March 15, 2009
Converting flac encoded audio to mp3. Solaris has the flac command which will happily decode flac encoded file and metaflac to read the flac meta data but you need to download lame either precompiled or from sourceforge and build it. Then it is a simple matter to convert your flac encoded files:
$ flac -c -d file.flac | lame - file.mp3
you can add various flags to change the data rates and put tags into the mp3. I feel sure someone should have written a script that would convert an entire library but I could not find one so here is the one I wrote to do this:
#!/bin/ksh93
umask 22
SRC="$1"
DST="$2"
BITRATE=128
export PATH=/opt/lame/bin:/usr/sbin:/usr/sfw/bin:/usr/bin
function flactags2lametags
{
typeset IFS="$IFS ="
typeset key value
metaflac --export-tags /dev/fd/1 "$1" | while read key value
do
print typeset "${key}=\"$(print $value| sed 's/["'\'']/\\&/g')\";"
done
}
function cleanup
{
if [[ "$FILE" != "" ]]
then
rm "$FILE"
fi
}
function convert_file
{
typeset -i ret=0
typeset f="${1%.flac}"
if [[ "$f" != "${1}" ]]
then
typeset out="${DST}/${f}.mp3"
if ! [[ -f "${out}" ]] || [[ "$1" -nt "$out" ]]
then
print $out
eval $(flactags2lametags "$1")
FILE="$out"
flac --silent -c -d "$1" | \
lame --quiet -b ${BITRATE} -h \
${ARTIST:+--ta }"${ARTIST}" \
${ALBUM:+--tl }"${ALBUM}" \
${TRACKNUMBER:+--tn }"${TRACKNUMBER}" \
${DATE:+--ty }"${DATE%%-*}" \
- "${out}"
ret=$?
FILE=""
fi
elif [[ $1 != "${1%.jpg}" ]]
then
[[ -f "${DST}/${1}" ]] || cp "$1" "${DST}"
fi
return ${ret}
}
function do_dir
{
typeset i
for i in "$1"/*
do
if [[ -f "$i" ]]
then
convert_file "$i" || return 1
elif [[ -d "$i" ]]
then
if test -d "${DST}/$i" || mkdir "${DST}/$i"
then
do_dir "$i" || return 1
fi
fi
done
return 0
}
function usage
{
print USAGE: $1 src_dir dest_dist >&2
exit 1
}
trap cleanup EXIT
(( $# == 2 )) || usage $0
if test -d "${DST}/$1" || mkdir "${DST}/$1"
then
do_dir "$1"
fi
exit $?
( Mar 15 2009, 06:48:24 PM GMT ) Permalink Trackback
Tuesday March 10, 2009
Update to scsi.d I've just released a new version of scsi.d that fixes a strange bug that only effected 10 but would result in scsi.d not starting:
# dtrace -qCs scsi.d-1.16 dtrace: failed to compile script scsi.d-1.16: "/usr/include/sys/scsi/scsi_pkt.h", line 33: incomplete struct/union/enum struct scsi_address: pkt_address #
The solution was simple even if I still don't really understand why the header included fine in OpenSolaris but not in 10.
( Mar 10 2009, 04:49:23 PM GMT ) Permalink
Sunday March 08, 2009
AV codecs for Solaris One of the big complaints about using OpenSolaris and Solaris is multi media support. The lack of codecs to allow you to play "common" if not "open" formats is irritatings.
I've been using the Fluendo mp3 codecs for a while as they have a compelling price point, being free. Fluendo also have a bundle of other codecs that they supply for OpenSolaris containing codecs for a number of other popular formats. I decided the €28 was would be worth it and have now installed the codecs on my home server. I now have gstreamer support for:
: pearson FSS 1 $; gst-inspect | grep -i flue fluasf: flurtspwms: Fluendo WMS RTSP Extension fluasf: fluasfcmdparse: Fluendo ASF Command Parser fluasf: fluasfdemux: Fluendo ASF Demuxer fluwmvdec: fluwmvdec: Fluendo WMV Decoder fluaacdec: fluaacdec: Fluendo AAC Decoder flumms: flummssrc: Fluendo MMS source fludivx3dec: fludivx3dec: Fluendo DivX 3.11 Decoder flulpcm: flulpcmdec: Fluendo LPCM decoder fluwma: fluwmsdec: Fluendo WMS Decoder fluwma: fluwmadec: Fluendo WMA Decoder (STD + PRO + LSL + WMS) fluh264dec: fluh264dec: Fluendo H264 Decoder flumpeg4vdec: flumpeg4vdec: Fluendo MPEG-4 ASP Video Decoder flumpeg2vdec: flumpeg2vdec: Fluendo MPEG-2 Video Decoder flump3dec: flump3dec: Fluendo MP3 Decoder (C build) : pearson FSS 2 $;
Alas on the Sun Rays the video performance is still less than great but on the console it is faultless and I am seeing the Totem is not quite as useless as I had previously thought.
It would be really cool if you could get OpenSolaris codecs via IPS repository.
The next thing I want is to be able to edit video.
( Mar 08 2009, 05:52:48 PM GMT ) Permalink Trackback
Saturday February 21, 2009
[Open]Solaris logfiles and ZFS root This week I had reason to want to see how often the script that controls the access hours of Sun Ray users actually did work so I went off to look in the messages files only to discover that there were only four and they only went back to January 11.
: pearson FSS 22 $; ls -l mess* -rw-r--r-- 1 root root 12396 Feb 8 23:58 messages -rw-r--r-- 1 root root 134777 Feb 8 02:59 messages.0 -rw-r--r-- 1 root root 53690 Feb 1 02:06 messages.1 -rw-r--r-- 1 root root 163116 Jan 25 02:01 messages.2 -rw-r--r-- 1 root root 83470 Jan 18 00:21 messages.3 : pearson FSS 23 $; head -1 messages.3 Jan 11 05:29:14 pearson pcplusmp: [ID 444295 kern.info] pcplusmp: ide (ata) instance #1 vector 0xf ioapic 0x2 intin 0xf is bound to cpu 1 : pearson FSS 24 $;
I am certain that the choice of only four log files was not a concious decision I have made but it did make me ponder whether logfile management should be revisted in the light of ZFS root. Since clearly if you have snapshots firing logs could go back a lot futher:
: pearson FSS 40 $; head -1 $(ls -t /.zfs/snapshot/*/var/adm/message*| tail -1) Dec 14 03:15:14 pearson time-slider-cleanup: [ID 702911 daemon.notice] No more daily snapshots left : pearson FSS 41 $;
It did not take long for this shell function to burst into life:
function search_log
{
typeset path
if [[ ${2#/} == $2 ]]
then
path=${PWD}/$2
else
path=$2
fi
cat $path /.zfs/snapshot/*$path | egrep $1 | sort -M | uniq
}
Not a generalized solution but one that works when you root filesystem contains all your logs and if you remember to escape any globbing on the command line will search all the log files:
: pearson FSS 46 $; search_log block /var/adm/messages\* | wc
51 688 4759
: pearson FSS 47 $; There are two ways to view this. Either it it great that the logs are kept and so I have all this historical data or it is a pain as getting red of log files becomes more of a chore, indeed this is encouraging me to move all the logfiles into their own file systems so that the management of those logfiles is more granular.
At the very least it seems to me that OpenSolaris should sort out where it's log files are going and end the messages going in /var/adm and move them to /var/log which then should be it's own file system.
( Feb 21 2009, 01:49:59 PM GMT ) Permalink
Tuesday February 17, 2009
VirtualBox pause and resume I have a VirtualBox running ubuntu that is acting as a print server for my HP Officejet J6410 and although that works well it chews a significant amount of CPU resource which given that the printer is not in use most of the time seems a bit of a waste, especially when the Sun Ray server can have 4 active sessions running on it. So now I am running this simple script:
#!/bin/ksh93 -p
logger -p daemon.notice -t ${0##*/}[$$] |&
exec >&p 2>&1
while :
do
echo resuming vbox
su - vbox -c "VBoxManage -nologo controlvm ubunto resume"
ssh print-server sudo ntpdate 3.uk.pool.ntp.org
while ping printer 2 > /dev/null 2>&1
do
sleep 60
done
echo pausing vbox
su - vbox -c "VBoxManage -nologo controlvm ubunto pause"
until ping printer 2 > /dev/null 2>&1
do
sleep 1
done
done
After the usual magic to redirect it's output to syslog this just loops forever pinging the printer. If the printer is alive the VrtualBox is resumed and if the printer is not alive the VirtualBox is paused. So as soon as the printer is turned on the virtual box is ready and then within 60 seconds of the printer being turned off the virtualbox is paused.
So that the clock on the VirtualBox is kept correct (the guest additions are, I am told, supposed to do this for free but in practice they do not for me) after resuming there is the ssh onto the virtual machine to set it's clock*, so I have a work around in place for VirtualBox which is itself a work around for a defect in Solaris and that work around is also working around another issue with Solaris.
Life on the bleeding edge is always fun!
*) I would normally use the address of my server to sync with but thanks to the issues with build 108 I currently don't have a working ntp server on the system.
( Feb 17 2009, 09:20:15 AM GMT ) Permalink Trackback
Sunday February 15, 2009
Build 108 I've managed to upgrade my home server to build 108 which is an important mile-stone for me as it has the fix for:
6763600: nautilus becomes unusable on a system with 39000 snapshots.
Which was rendering nautilus close to unusable for any users who moved out of their lofs automounted home directories. In partilcular any attempt to use it to manage the photo directories was painful.
However all was not smooth as again I hit this bug:
6795892: Sun Ray X servers (both Xsun and Xorg) suffer network problems in snv_106
but since I was expecting this I tried the workaround from bug 6799655 which is the same as the one for 6763600:
In /etc/sock2path change the following lines:
2 2 0 tcp
2 2 6 tcp
26 2 0 tcp
26 2 6 tcp
2 1 0 udp
2 1 17 udp
26 1 0 udp
26 1 17 udp
to:
2 2 0 /dev/tcp
2 2 6 /dev/tcp
26 2 0 /dev/tcp6
26 2 6 /dev/tcp6
2 1 0 /dev/udp
2 1 17 /dev/udp
26 1 0 /dev/udp6
26 1 17 /dev/udp6While this got the Sun Rays up it also stopped named from working, spewing errors like this:
Feb 15 15:10:39 pearson named[15558]: [ID 873579 daemon.error] 71/Protocol error Feb 15 15:10:39 pearson named[15558]: [ID 873579 daemon.error] socket.c:4315: unexpected error:
So have had to revert to some internal to Sun binaries that work around this while the underlying bug is fixed. It is slighly worring as I'm left wondering what other victims are out there. One I have already found is ntp, which is a known bug:
6796948: NTP completely broken by I_SETSIG semantics change in snv_106
I suspect that the system will have to revert to build 105 soon.
( Feb 15 2009, 05:48:57 PM GMT ) Permalink
Except where otherwise noted, this site is
licensed under a Creative Commons License 2.0
This is a personal weblog, I do not speak for my employer.
