Monday Nov 03, 2008
After some experimenting and looking at source I've determined that the
kadmind does have support for rotating its own log that is separate from the krb5kdc log (by default the kadmind logs to the log used by krb5kdc). To configure this, edit /etc/krb5/krb5.conf and add:
admin_server = FILE:/var/krb5/kadmin.log
admin_server_rotate = {
period = 1d
versions = 10
}
in the [logging] section. Unfortunately, this is not documented properly in the krb5.conf man page but it basically works the same as the kdc_rotate parameter which is documented in man krb5.conf.
Note, to configure both the kdc and kadmind logging behavior to log to
separate files, use something like:
[logging]
# commenting out default so kadmind will log to a separate file
# default = FILE:/var/krb5/kdc.log
kdc = FILE:/var/krb5/kdc.log
kdc_rotate = {
# How often to rotate kdc.log. Logs will get rotated no more
# often than the period, and less often if the KDC is not used
# frequently.
period = 1d
# how many versions of kdc.log to keep around (kdc.log.0, kdc.log.1, ...)
versions = 10
}
# controls kadmind logging
admin_server = FILE:/var/krb5/kadmin.log
admin_server_rotate = {
period = 1d
versions = 10
}
This is the supported way to rotate the krb5kdc and kadmind logs. Also note that the kdc.conf man page is in error regarding the logging section. Use krb5.conf to control KDC logging instead.
Wednesday Sep 05, 2007
A long time ago I wrote about my experiences playing with various memory debuggers in Solaris. One thing I mentioned was:
Note, a core dump is not necessary. Use "mdb -o nostop -p PID" where PID
is the proc. ID of the running process and then do the findleaks stuff:
echo '::findleaks' | mdb -o nostop -p $(pgrep gssd)
echo '000cb608::bufctl_audit' | mdb -o nostop -p $(pgrep gssd)
The "mdb -o nostop" trick does not work in the upcoming Solaris Nevada. Instead use:
gcore $(pgrep -x daemon_name)
to get a core dump of a running process then do:
echo "::findleaks" | mdb core_produced_by_gcore
and so on. You can read the rest of my previous blog entry
here.
Wednesday Nov 16, 2005
Using NFSsec with krb5i method of security protection provides the security of Kerberos authentication and integrity checking of each NFS message transfered between client and server. This is certainly useful in guaranteeing that NFS data is not maliciously manipulated. This integrity checking can also be useful when using NFS over unreliable networks where data corruption may not be caught by the CRC checksum used by TCP. This happened in Sun some time ago where a busy server was connected to the network via a faulty switch and there were problems with a code build as a result. If NFSsec with krb5i had been used at that time there would not have been a problem with data corruption.
Monday Oct 24, 2005
I created a dtrace script based on Brendan Gregg's dapptrace script that allows one to easily trace all the function calls of a GSS or Kerberos
application on Solaris. It can be run in a manner similar to apptrace and can be either given a command line to trace or a PID. The script, called krbdtrace is here. Note that this script can show the entire call path in the Kerberos and GSS-API libs. It shows return codes but does not handle the return types (they are all displayed as integers). Still, if there is an error that is hard to diagnose and krbdtrace shows a function returning a negative number that may provide a clue as to what is failing. Update: it was pointed out to me that truss supports -u mech_krb5:: which will show more detail in tracing user level Kerberos code. Still the krbdtrace script could be useful as a template for using dtrace to do things that truss can not such as reporting only functions that return negative integers and so on.
Technorati Tag: Solaris
Technorati Tag: Kerberos
Technorati Tag: dtrace
Technorati Tag: GSS-API
Friday Jul 29, 2005
I just became aware of another blog with scripts that integrate libumem memory debugging with dbx.
You can read more about the dbx module here:
http://blogs.sun.com/roller/page/quenelle?entry=umem_integration_with_dbx
http://blogs.sun.com/roller/resources/quenelle/umem.dbx
You can read more about libumem here:
http://access1.sun.com/techarticles/libumem.html
http://blogs.sun.com/roller/page/ahl/20040713#solaris_10_top_11_20
Thanks to Chris Quenelle for this information!
As an aside, I've found libumem to be very useful in doing QA testing. I love to pound on code with these settings exported in the environment:
LD_PRELOAD="libumem.so.1
UMEM_LOGGING='transaction=1M'
UMEM_DEBUG='audit,contents,guards,verbose'
One can find memory leaks, double free's, writes past allocated buffers, and more with libumem. And integrated with dbx and -g compiled source, finding the problem will be easier than ever.
Tuesday Jun 14, 2005
Playing with Solaris memory debuggers
Playing with Solaris memory debuggers
The following are notes that I've made for myself as I used various Solaris
memory debugging libraries. Given the following was originally for my
consumption I can not vouch for the correctness of the grammar.
====================================================================
UserSpace info:
I was playing with both watchmalloc.so.1, libumem.so.1 to see what they can do.
Here's what I observe:
Using environment variables:
LD_PRELOAD=watchmalloc.so.1 MALLOC_DEBUG=WATCH,RW
watchmalloc can find bugs like (using MALLOC_DEBUG=RW):
p=malloc(128);
free(p);
foo=*p; /* invalid read */
or:
p=malloc(128);
free(p);
*p=1; /* invalid write */
or:
p=malloc(1);
memset(p, 1, 10); /* write past buffer */
Note, watchmalloc will core dump when it detects an error. And it causes
programs to run MUCH slower.
===========================================================
libumem does have some memory guards to detect invalid writes but it requires
use of the mdb command ::umem_verify on the core. It does not catch bad reads
like watchmalloc but it does have guards with patterns like 0xdeadbeef and
0xbaddcafe. And it does not slow down program execution like watchmalloc.
Use LD_PRELOAD=libumem.so.1
With libumem one can do memory leak detection (using
UMEM_LOGGING=transaction UMEM_DEBUG=default):
p=malloc(128);
p=malloc(128);
abort();
then do 'echo ::findleaks|mdb core' to see:
CACHE LEAKED BUFCTL CALLER
000bb088 1 000cb608 main+8
----------------------------------------------------------------------
Total 1 buffer, 320 bytes
then use:
echo '000cb608::bufctl_audit' | mdb core
to see the stack trace where the leak allocation took place.
Note, LEAKED is the number of times a leak occurred.
Note, a core dump is not necessary. Use "mdb -o nostop -p PID" where PID
is the proc. ID of the running process and then do the findleaks stuff:
echo '::findleaks' | mdb -o nostop -p $(pgrep gssd)
echo '000cb608::bufctl_audit' | mdb -o nostop -p $(pgrep gssd)
This is good for daemons. Or use "gcore <PID>" to get a core dump of a
running process. This is useful to look for leaks in daemons like krb5kdc.
Also, to watch the memory size of a running daemon to see if it is growing over
time use:
prstat -p <PID of daemon> 300 > /tmp/prstat.out
---------------------------------------------------------------------------
Both watchmalloc and libumem will core dump on double free()'s.
Based on the above it seems like it would be good to use watchmalloc for
some memory corruption testing and use libumem for memory leak
detection.
Note use:
print ::umem_status|mdb core
to see umem status for user space core when debugging with libumem.
Use:
print ::umalog | mdb core
to see umem transaction log and stack traces.
It's also good do do:
print ::stack | mdb core
and look at the stack trace (note, the values in each function listed
are the input registers %i0-5).
Look at umalog to see if it's possible to determine if memory was
free'ed earlier. Use:
print "<address>::dis" | mdb core
to see the assembly around the stack function address to see where the
call was made (look for other call's).
===========================================================
Kernel memory debugging info:
In /etc/system do:
* kmem lite flag, must use independently of other kmem flags
* set kmem_flags = 0x100
* kmem flags: audit, test, redzone
set kmem_flags = 0x7
and reboot system.
Use:
echo "::dcmds" | mdb unix.0 vmcore.0
to see debugging commands (look for kmem stuff).
Note, the kernel kmem outputs debug messages to syslog.
Technorati Tag:
OpenSolaris
Technorati Tag:
Solaris
Technorati Tag:
mdb
Tuesday Jun 14, 2005
wx and You: your friend in managing workspace change
wx and You: your friend in managing workspace change
"Relieves OpenSolaris putback anxiety or your money back."
The following is a webpage I wrote to introduce Sun developers to a source
code workspace management tool,
wx,
that I extensively modified after getting burned when using it to
collapse a delta on a renamed file. Actually it was about a year after
that incident that I realized, after talking to different folks at Sun,
how wx could be modified to make my life easier (I had a hard time
remembering all the development rules and I hate having to do things
manually that can be automated). During my recovery from back surgery I
started hacking on wx and discovered there was a lot that could be done
to make the tool better. At the time I was not thinking that my version
would be official so I called it wwx and let some of my Kerberos team
members try it out. Within a month I started getting calls/e-mails from
other Sun developers with thanks, suggestions and bug reports on wwx.
Things snowballed from there and eventually my changes made it into the
official version of wx.
Note that the contents below are oriented towards internal
development in Sun using Teamware workspaces so some items may not be applicable to
OpenSolaris
development. It looks like we're moving to Mercurial controlled workspaces which
are not compatible with wx. Instead one should use the Cadmium extensions to Mercurial.
You can read more about this on the OpenSolaris site.
Contents
About wx
Examples
How I normally
use wx
Getting a
project gate/workspace ready for putback
Advanced Usage
How to skip
particular pbchk/nits checks
How to skip
files in webrevs
How to
edit the active list (and deal with new files that haven't been
"created")
Environment
Variables
Tips for use
with bugtraq
How to return
a file to its parent version
How to
initialize in a STC workspace (non-default src path)
How to debug wx
Credits
About wx
wx's reason for being is to help the developer keep track of file
changes in a workspace and follow the Solaris source gate rules. Over the last
several months I decided to enhance wx to be more functional and
robust. As a result
wx
now does the following:
Keeps track of changed and new
files in the active list.
Keeps track of renamed and deleted
files in the renamed list.
Keeps track of comments associated
with files in the active list.
Provides a wrapper around the
underlying SCCS and workspace commands used to manipulate files in a
workspace. This allows wx to update the active and renamed lists
automatically and also do some checking to help the developer
conform to the OpenSolaris gate rules.
The pbchk command does a number of
new checks on active files and comments to make sure they conform to
OpenSolaris rules. This includes looking for "sccs rmdel"
which causes Teamware problems, too many delta's, files that are
checked out that are not in the active list, poorly formed active
list comments, active list comments that don't match those in the
SCCS delta and whether the RTI is approved for the bugs listed in
the active list comments.
The webrev command generates
accurate webrevs that include renamed/delete info. A webrev is
is a set of HTML code diffs that allow easy review with a browser.
The backup and restore commands
allow easy backup and restore of files in the active and renamed
list to a directory in your home directory which is usually backed up and
thus safer than a workspace on a build system.
The putback/pb command allows the
developer to do a putback with more safety and convenience.
The reedit command is enhanced to
make it safer to use on files that have been renamed. Note, the
reedit command is used to collapse deltas of files in the active
list which is useful if the file was merged as a result of a resolve
or anything else that could cause the file to be checked out and in
more than once. Note, there is also a new “redelget”
command that collapses file histories but leaves the file checked
in. This is currently the only way to collapse new files.
Provides several informational
commands so the developer can see what files are renamed, deleted,
new, etc...
The mv and rm commands check for
cyclic renames which cause problems for Teamware.
Provides a number of command
aliases for ease of use.
See "wx help" for a list of all the commands,
aliases and flags.
wx issues that the new wx fixes
Old "wx reedit" treats renamed files as new files, losing
the file history in the process. New wx uses the Teamware
nametable hash info to determine if a file has been renamed or
is new. This is one reason new wx is slower than old wx but more
accurate in this regard. And this is why both the parent
workspace and local workspace nametables must be accurate. If
wx thinks a file is new it will warn and ask the user if it is
okay to proceed with the reedit (answer "no" if the file is not
new).
Old "wx reedit" used the
modification time of the SCCS delta files to determine if the
parent file contains a delta that the child doesn't. This is
risky because the child delta could be modified by a check in
and the result would be that the reedit would lose the parent's
delta code changes. New wx looks at the latest delta comment in
the parent and determines if this exists in the child's delta
history. If it doesn't then wx skips the reedit for that file
and warns the user that a bringover is required.
Old "wx new" lists renamed
files as new. New wx doesn't do that but it is slower as a
result of more checking.
Not really a bug but old wx uses the paradigm of either
editing the active list manually to add entries or using the update
command to search all the directories to find files that are checked out
(adding them to the active list). This means that the user must
remember to update the active list after they check out files using
"sccs edit". New wx can automatically update the
active list when a file is being checked out with "wx co file"
(edit/checkout/co are all aliases for the same command). This
requires the developer to do fewer steps and the active list is more
likely to stay in sync with the file changes in a workspace.
The best way to use new wx is to do all file manipulation in a
workspace with the wx commands (do not use SCCS commands). Doing so
consistently automatically updates the active, new and renamed lists
and thus does not require that the update command be run in order to
update these lists. Be aware that in order for wx to work properly
the active and renamed file lists must be accurate. So if you do use
SCCS commands or workspace filemv or filerm then you will need to use
the 'wx update' command. Note, if you decide to remove a newly
created file which is in your active list from your workspace, use
'wx rm file' to remove the file so the active list is
updated accurately.
New wx now keeps track of file renames and deletes in a renamed
list. This is separate from the active list since the active list
only stores info on files that have been edited or are newly created.
This separation allows wx commands like reedit to run only on active
files and not files that have only been renamed/deleted. Note, if a
file in the active list is renamed it will also appear in the renamed
list.
New wx assumes that the parent workspace associated with the
current child workspace contains the same set of files (except for
new files) that the child workspace contains. Be careful if you
change the parent of the child workspace since wx will assume that if
it cannot locate a file in the parent then the local file is new. If
this assumption is wrong, wx can output erroneous information or in
the case of the reedit command, delete the SCCS history. Use "wx new"
to see which files wx thinks are new.
Examples
Here's how I normally use wx:
Create local workspace then do "ws workspace"
and bringover files. Note, you do not have to do "ws workspace"
in order to use wx in a workspace. Instead, just cd to a workspace
and start using wx (note, wx will complain if you ws'ed to one
workspace and cd to another and then try to use wx).
"wx init". Initialize wx in a workspace (creates the wx subdir
and some files for tracking changes). If I haven't checked out any files
I use the "no update" option which creates a empty active list (fast).
If you have modified files you should use one of the other updated
options listed. Note, you can keep an active list sorted by default or
sort it manually by using the "wx update -q -s" command.
"wx co file_to_modify ...". (Note, co is an alias for
checkout/edit. This does a "sccs edit file_to_modify ..." and
adds an entry to the active list for each file on the command line. For
example I would "cd usr/src" then "wx co Makefile" to check out the
Makefile for editing which would also put an entry for usr/src/Makefile
in my active list.) Use "wx list" or "wx active" to list files in
the active list.
"wx create new_file ...". New
files need to be "created" to be recognized by the
Teamware tools. This command starts a new file history for the new
file and adds a entry in the active list. Use "wx new"
to list new files in the active list.
"wx rm file_to_delete ...". Use
this if you need to remove a file in the OpenSolaris approved way. wx will
actually move the file from usr/ to deleted_files/ and add an entry
in the renamed list. Note, for new files the files are not renamed
and wx will ask if you want to remove the files completely from the
workspace. Typically you would answer yes but make sure you have a
backup of the files just in case you need them. The renamed list
will not be updated in this case but entries in the active list will
be removed. Use "wx renamed" to list renamed files.
"wx mv file new_file".
Use this if you need to rename file to new_file. Will
add an entry in the renamed list. The active list will be updated
with the new name if file is in the active list. Use
"wx renamed" to list renamed files. See "wx help"
for more info.
"wx bu -t" (bu is alias for backup, -t only backup if needed).
Use this periodically to backup up files that wx are tracking. The
files are backed up in the wx.backup subdir in your home directory which is
useful since most build systems don't backup workspaces unlike your
home directory. Note, "wx backup -i" provides info about the backup dir and
files. There are other flags to control compression. See "wx help" for
more info.
When I have checked out and modified all the files required
for the bug fix I build and test. If the testing looks good I then
do:
"wx bu -t". Just to be safe.
"wx ci -c comment_file".
Check in all active files. The check in comments will be taken from
the active list comments which will be replaced by the comments in
the comment_file. If you do this you can skip editing the active
list comments manually.
"wx webrev". Generates HTML based code diffs
under the webrev/ directory at top of workspace. Have code reviewed using
a web browser.
Once the code review is done and all changes have been made I
file an RTI using the web RTI page. XXX Note, for OpenSolaris I am not
sure how an RTI is filed but I will add a pointer to that info when
I find it.
Login to the putback gate system,
"ws /net/path_to_your_workspace", and reparent your
workspace if necessary (See "man workspace").
"wx pbchk". Check files and comments for OpenSolaris
gate rules conformance. Note, nits runs a subset of pbchk checks and
is more useful for checking your files while you've got them checked
out.
If "wx pbchk" complains about multiple SCCS
deltas/file versions, use "wx redelget -m" to collapse
the deltas so there is only one version difference between the
local file and the parent. This will also reset the file history for
new files to 1.1. Note, reci is an alias for redelget.
"wx pb -n". Do a trial putback and check
for conflicts. If there are conflicts, run
"bringover files_in_conflict then "wx resolve"
to resolve the conflicts and collapse the deltas created by the
merge ("wx resolve" uses the reedit subcommand to
collapse the SCCS deltas). Note, pb is an alias for putback. You can
see the list of files that wx provides to the putback command by
doing "wx pblist". "wx pbcom" will
display the putback comments. If you resolved conflicts then repeat
steps 10-15
"wx pb". Do the real putback. Note, wx will do
some checking and will prompt the user before actually doing the
putback. It will pass the comments found in the active list to the
putback command and will display the putback comments before the
prompt to do the putback. You can also use "wx pbcom"
to see what the putback comments will be.
After the putback is done, if I intend to do more bug fix
work in this workspace I'll save the current wx directory (either tar it
or rename it), remove the current wx directory if I didn't rename it and
then do "wx init" to initialize wx with fresh state.
That way there is less chance of getting confused about the changes
made for the new bug fix.
Getting a project gate/workspace ready for putback to the OpenSolaris gate:
Note, the following describes the model that I use when working
on a project that will require a significant amount of time and code
change. In order to reduce the pain involved, my model uses a
project source gate created by copying the source from the official
OpenSolaris gate. Development occurs in workspaces that are
children of this project gate with intermediate putbacks from this
child workspaces going back to the project gate. This allows the
project to use it's own rules regarding number of deltas in a
putback and other issues without regard to the official OpenSolaris
gate rules. When the project is done and ready to be putback to the
Open Solaris gate these are the steps I use (featuring wx):
First, a caveat about using wx in a project gate. Never
use "wx reedit" or redelget (or any equivalent alias that collapses
a SCCS delta) in a project gate that is the parent of developer
workspaces. This will confuse the Teamware commands like putback
and bringover when used between a child workspace and the project
gate. Generally, it is best to avoid the wx reedit/redelget
commands until it is time to putback to the OpenSolaris gate.
Create a local copy of the official OpenSolaris gate
(I'll call it opensolaris_copy in this example). The parent of
opensolaris_copy should be the official OpenSolaris gate.
"putback usr/src delete_files" from the project gate to
opensolaris_copy. This leaves the project gate intact and any
modification (collapsing) of file SCCS delta histories for the
real OpenSolaris putback will be done in opensolaris_copy.
Note, if the putback reports conflicts these will need to be
resolved in the project gate. The resolve will change files in
the project gate but child workspaces will be able to properly
bringover those changes. Backup the project gate before doing a
resolve. Again, do NOT use wx reedit/redelget in a project workspace
if you want bringover to work in child workspaces.
"ws opensolaris_copy". Set the local ON
clone copy as the current workspace.
"wx init -ft". Initialize the
opensolaris_copy workspace using a thorough wx update. This will
create the active and renamed lists with all the files that were
changed in the project gate.
"wx list". Make sure active list looks okay.
"wx new". Make sure new list looks okay.
"wx renamed". Make sure renamed list looks
okay.
"wx ea". Update the active list comments. The comments
will be placed in the SCCS delta history for files in the active
list when the "wx redelget" step is done below and will also be
used as the putback comment.
"wx redelget". Collapse all the active list files SCCS
delta history so there will only be one SCCS delta for each file
in the active list when they are checked in. Note, recheckin
and reci are aliases for redelget.
Do a nightly build in the opensolaris_copy gate. If any
files need to be modified, do the modification in the project
gate and putback the changes to the local copy. Do a "wx
deltachk" to see if there are any delta problems.
Follow steps 13-16 from the "How I
normally use wx" list of wx steps. to putback the files from
the opensolaris_copy gate to the official OpenSolaris gate.
Advanced Usage
How to skip particular pbchk/nits checks
Some of the checks that the pbchk and nits commands run will skip
active files listed in a file called wx/command.NOT where
command is the name of the wx command doing the particular
check. For example if I want to skip cstyle checking of all my active
list files when I run "wx pbchk" I would first do
"wx list >wx/cstyle.NOT". Note, I only skip
checks for which I know I have an exemption from the gatekeepers. For
example for some open source code in Solaris there are exemptions
from the cstyle rules. Similarly for files containing non-Sun
copyrights one may want to list files that the user is sure there is
not a copyright problem in "wx/copyright.NOT".
How to skip file in webrevs
List files (one per line) in wx/webrev.NOT that you don't want to
include in webrevs generated by "wx webrev".
How to edit the active list (and deal
with new files that haven't been "created")
The original wx required the user to edit the active list to
associate comments with the active files (note, wx checkin now
provides a -c comment_file option which will automatically update the
comments in the active list as well as use them for the SCCS check-in
delta). Also, some people like to manually add new file entries to
the active list before they were 'sccs created' so they could use wx
for backups and nits checking, etc... The format of an entry in the
active list is:
[filepath]
#empty line
[one or more comment lines]
#empty line
where [filepath] is the file path of an existing or new file relative
to the top of the workspace. Here's an example entry with three
comment lines (note, each entry starts with the filepath, there is
not a empty line before in the example below):
usr/src/cmd/krb5/klist/klist.c
4772119 Enabling Kerberos's Triple DES/SHA-1 Enctype Support
PSARC/2002/178 Enabling Kerberos's Triple DES/SHA-1 Enctype Support
4705662 GSS/Kerberos clients requesting dec-cbc-crc before des-cbc-md5
Note, if you are adding new files to the active list you should run
"wx new -t" to populate the new list. New files that haven't been
created will be created when "wx delget" is run (note checkin and ci are
aliases for delget).
Environment Variables
PUTBACK: specifies the command to do
the putback. This is useful if you want to use something like Casper
Dik's turbo-dir.flp scripts as in this example: "export
PUTBACK='cm_env -g -o putback'"
WXDIFFCMD: specifies the diff command and args for the wx diffs
commands like diffs and pdiffs. This is similar to the CDIFFCMD and
UDIFFCMD environment variables that webrev uses. A good setting is:
"export WXDIFFCMD='diff -bw -U 5'"
WXWEBREV: specifies the webrev command and args used by the
"wx webrev" command.
WXDIR: specifies the directory where wx will keep it's state
files. The default location is in wx/ at the top of the workspace
however via WXDIR you can change this to point to a different path like
/tmp/my_wx. This is useful when you want to run wx in a workspace where
you don't have write permission.
Tips for use with bugtraq
How to return a file to its parent version
To undo all changes in a file that you've checked out and set it to the
current version in the parent workspace use: wx reset file
Note, this will bringover the file from the parent and will undo local file
renames.
To undo all changes in a file and return it to the version when it was
originally brought over from the parent do:
wx reedit file
wx unedit file
This does not bringover the file from the parent so the file contents will
be that of the last bringover of that file. It also does not undo a file
rename.
How to initialize in a STC workspace
(non-default src path)
Since the source in a STC workspace doesn't use the wx default
path of usr/src you'll need to initialize using "wx init usr/ontest"
in order for it to find the source. Note, if the source you're
working on isn't found under usr/src from the top of your workspace
then you'll need to initialize using the common path starting from
the top of your workspace.
How to debug wx
To turn on debug output in wx do "wx command -D command args".
This will send debug info to stderr so to page the output use 2>&1.
Credits
Thanks go to Jeff Bonwick who wrote wx, David
Robinson who had a great idea about using the nametable hashes, Brent Callaghan
who wrote webrev, John Beck (code review), Anup Sekhar (testing), Glenn Barry,
Arun Perinkolam, Wyllys Ingersoll, Chin-Long Shu, Alastair McDermott, Matt
Simmons, Valerie Anne Bubb, Bill Sommerfeld, Heiner Steven, Nikolay Molchanov,
Alan Burlison, Nico Williams, Ann-Marie Westgate, James Carlson, Jeff Smith, and
last but not least my mother for pointing out that my caching methodology could
be more generalized.
Technorati Tag:
OpenSolaris
Technorati Tag:
Solaris
|
|
|
|
