Matt Ingenthron's Stream of Consciousness

Monday December 22, 2008

Sun Web Stack 1.4 Release

Sun Web Stack 1.4 has now been released. You can download the release now if you plan to install on Solaris 10 or Red Hat Enterprise Linux. For OpenSolaris, the same software has been integrated into the package repository, so you can obtain the Sun Web Stack 1.4 release there by simply adding the packages you would like. There are some package clusters, like "amp" and "ruby" to simplify adding multiple packages, and there is a pretty slick, developer oriented GUI in the "webstack-ui" package. For more details on the packages in the OpenSolaris 2008.11 release, which aligns to Sun Web Stack 1.4, refer to the getting started guide.

I've been working with both Cool Stack and the OpenSolaris Web Stack projects for some time. Starting with OpenSolaris 2008.11 and Sun Web Stack 1.4 (the logical successor to Cool Stack 1.3.x), we've brought the two projects closer together. I'm sure this will raise some questions, and since I've been involved in both I think I'm in a position to help answer a few of them.

Why is the new release version "1.4"?

Astute observers have probably wondered why it's called Sun Web Stack 1.4 and not 1.0 or 2.0. Cool Stack users may recall that the last release was version 1.3.1. The choice of version 1.4 is a subtle way of saying this is the successor to Cool Stack 1.3.1. In fact, most of the components use the same optimizations (with a slightly different build process) and have been built and tested by the same engineers-- they're just updated components. The only exception is MySQL (more on that later).

Why did MySQL go from version 5.1 in Cool Stack 1.3.x to 5.0 in Sun Web Stack 1.4?

When we started work on Cool Stack 1.3, MySQL 5.1 was expected out any day, so it was updated to be current once Cool Stack 1.3 was released. Shanti and others in PAE did a lot of experimentation with MySQL 5.1 and found it to have a number of advantages. Problem was, the update didn't come out, and Cool Stack was ready to ship. We had to make a choice, so we decided to release Cool Stack with MySQL 5.1, with the thought to update it later once MySQL had a Community Server final release. The problem was, eventually we got to the point that Cool Stack was ready, but MySQL 5.1 was still undergoing development. So, release, or hold back?

Ultimately we decided to release. In retrospect, this may not have been the best thing since MySQL 5.1 was still evolving quickly. It's hard to say, since some people were using it. For others, it caused migration issues.

Regardless, for Sun Web Stack 1.4, we coordinated a bit further with our MySQL brethren and collectively decided the best thing was to ship MySQL 5.0. Yes, it's a version regression from what we had in Cool Stack 1.3.1. Cool Stack is still available though, and we don't have any plan to pull it. We'll also be looking to update things going into the next release. If you have any thoughts on this, it'd be great to hear feedback in the comments or on the Sun Web Stack forum

The paths seem different compared to Cool Stack... why were they changed?

For users who weren't using zones, the old approach worked. However, if you were using Solaris zones, or you decided to adopt Solaris Zones after installing into the global zone, then it didn't work so well. It was possible to just install into the non-global zones, but that did require some planning.

At the same time, I know there is a nice simple abstraction in having everything in a common directory. You look in /opt/coolstack and find it all there. The good news is that in the unbundled release, for the most part we kept this abstraction. The only difference is that this release uses /var/opt/ (variable files, like your htdocs) and /etc/opt (configuration files), which will work correctly with zones.

Having said that, I know that there are still those who prefer a zip or tarball style download which lives in a self-contained directory. There are some technical challenges in doing that correctly, but we are evaluating what it would take. If this is something you'd be interested in, we'd like to get some feedback.

Updates coming soon...

Wednesday December 10, 2008

restricting MySQL memory with Solaris resource controls and rcapd

Now that OpenSolaris 2008.11 is out, that means the next iteration from the OpenSolaris Web Stack project, a.k.a. Sun Web Stack, is available! I'll post another blog or two on this soon.

One of the underutilized features of Solaris (IMHO) is the resource management capabilities. Generally speaking, people know zones have resource controls but they may not be aware that the resource controls have been in Solaris for quite a while, since Solaris 9 updates. In other words, it's not required to use zones to use resource management.

Recently a customer encountered a memory leak bug with MySQL. We'll obviously fix that, but getting it into the patch cycle and tested will take a little while. In the interim, we needed a solution to keep MySQL from leaking so much that it affects the system. Enter Solaris Resource Management. With a bit of experimentation, as expected, I did find that I could effectively limit the max address space and limit the resident memory with both rcapd and generic resource controls.

Assuming the user mysql,which is the default in the version we ship in OpenSolaris Web Stack, here's what I set (I'm running OpenSolaris build 101, but expect similar behavior on S10U6):

# projadd user.mysql
# projmod -s -K "rcap.max-rss=100MB" -K "process.max-address-space=(priv,100MB,deny)" user.mysql
# rctladm -e syslog=WARNING process.max-address-space

This will limit the max-rss (resident memory, via rcapd, which needs to be enabled: svcadm enable rcap) and the process max address space to 100MB for the user mysql (by configuring the user's default project). In practical terms, prstat showed the memory usage a bit lower, probably due to how the accounting is done for shared objects.

The rctladm command will tell Solaris to syslog a warning (on a global basis) if the process tries to exceed the memory amount set out. Note that becuase I'm using deny on the max-address-space, it can't get any more memory anyway.... Solaris will just act like there's no more memory available when the process tries to allocate some. In this case, we may want to restart the MySQL service, through a cron job that looks to the syslog.... though we'd have to handle that carefully.

I experimented with mysql 5.0 and a small program to get mysql to try to use a lot of memory and show rcapd/resource controls doing their job. I ran rcapd in debug mode to verify it was doing the correct thing.

rcapd: collection types: 0x1
rcapd: vmusage sample flags 0x4
rcapd: getvmusage time: 170.14 milliseconds
rcapd: kernel nres 4
rcapd: vmusage_sample
rcapd: 0: id: 100, type: 0x4, rss_all: 108523520 (105980KB), swap: 2916417536
rcapd: 1: id: 3, type: 0x4, rss_all: 3014656 (2944KB), swap: 253952
rcapd: 2: id: 10, type: 0x4, rss_all: 1350504448 (1318852KB), swap: 1255268352
rcapd: 3: id: 0, type: 0x4, rss_all: 193441792 (188908KB), swap: 180629504
rcapd: project user.mysql rss/cap: 105980/102400, excess = 3580 kB
rcapd: any collection/project over cap = 1, 1
rcapd: enforcing caps
rcapd: project user.mysql scanner starting to scan, excess 3580k
rcapd: project user.mysql scanner resuming process 29686
rcapd: process 29686: 4/0kB rfd/mdfd since last read
rcapd: identified nonpageable schedctl mapping at fea04000
rcapd: identified nonpageable schedctl mapping at fea04000
rcapd: process 29686: 4/0kB rfd/mdfd since hand swept
rcapd: process 29686: 2857044/0kB scannable
rcapd: identified nonpageable schedctl mapping at fea04000
rcapd: project user.mysql scanner trying to resume from 0x33634000, next 0x33634000
rcapd: project user.mysql scanner paging out process 29686
rcapd: project user.mysql scanner paged out 0x33634000+0t(468/3580)kB
rcapd: project user.mysql scanner paged out 0x339b3000+0t(0/3112)kB
rcapd: project user.mysql scanner paged out 0x33cbd000+0t(0/3112)kB
rcapd: project user.mysql scanner paged out 0x33fc7000+0t(0/3112)kB
rcapd: project user.mysql scanner paged out 0x342d1000+0t(120/3112)kB
rcapd: project user.mysql scanner paged out 0x345db000+0t(0/2992)kB
rcapd: project user.mysql scanner paged out 0x348c7000+0t(0/2992)kB
rcapd: project user.mysql scanner paged out 0x34bb3000+0t(0/2992)kB
rcapd: project user.mysql scanner paged out 0x34e9f000+0t(0/2992)kB
rcapd: project user.mysql scanner paged out 0x3518b000+0t(0/2992)kB
rcapd: project user.mysql scanner paged out 0x35477000+0t(4/2992)kB
rcapd: project user.mysql scanner paged out 0x35763000+0t(0/2988)kB
rcapd: project user.mysql scanner paged out 0x35a4e000+0t(844/2988)kB
rcapd: project user.mysql scanner paged out 0x35d39000+0t(1732/2144)kB
rcapd: project user.mysql scanner paged out 0x35f51000+0t(412/412)kB
rcapd: project user.mysql scanner done, excess 0
rcapd: sleeping 0.71 seconds
rcapd: updating statistics...
rcapd: project user.mysql status: succeeded/attempted (k): 3580/42512, ineffective/scans/unenforced/samplings:  0/1/0/1, RSS min/max (k): 0/243872, cap 102400 kB, processes/thpt: 1/0, 1 scans over 907 ms
rcapd: sleeping 3.81 seconds
rcapd: collection types: 0x1
rcapd: vmusage sample flags 0x4
rcapd: getvmusage time: 3.34 microseconds
rcapd: kernel nres 4
rcapd: vmusage_sample
rcapd: 0: id: 100, type: 0x4, rss_all: 108523520 (105980KB), swap: 2916417536
rcapd: 1: id: 3, type: 0x4, rss_all: 3014656 (2944KB), swap: 253952
rcapd: 2: id: 10, type: 0x4, rss_all: 1350504448 (1318852KB), swap: 1255268352
rcapd: 3: id: 0, type: 0x4, rss_all: 193441792 (188908KB), swap: 180629504
rcapd: project user.mysql rss/cap: 105980/102400, excess = 3580 kB
rcapd: any collection/project over cap = 1, 1
rcapd: enforcing caps
rcapd: project user.mysql scanner starting to scan, excess 3580k
rcapd: project user.mysql scanner resuming process 29686
rcapd: process 29686: 4/0kB rfd/mdfd since last read
rcapd: identified nonpageable schedctl mapping at fea04000
rcapd: identified nonpageable schedctl mapping at fea04000
rcapd: process 29686: 4/0kB rfd/mdfd since hand swept
rcapd: process 29686: 2857044/0kB scannable
rcapd: identified nonpageable schedctl mapping at fea04000
rcapd: project user.mysql scanner trying to resume from 0x35fb8000, next 0x35fb8000
rcapd: project user.mysql scanner paging out process 29686
rcapd: project user.mysql scanner paged out 0x35fb8000+0t(3580/3580)kB
rcapd: project user.mysql scanner done, excess 0
rcapd: sleeping 0.88 seconds
rcapd: updating statistics...
rcapd: project user.mysql status: succeeded/attempted (k): 3580/3580, ineffective/scans/unenforced/samplings:  0/1/0/1, RSS min/max (k): 0/243872, cap 102400 kB, processes/thpt: 1/0, 1 scans over 299 ms
rcapd: sleeping 3.81 seconds

Setting the resident set and max memory to the same amount is probably not the right approach for most uses, it's just what I experimented with here. I'd think in most cases you'd set the max higher and set the resident set to something sane for the system. If you want to be sure the mysqld won't overflow to swap, you may want to actually set the max memory to something less than physical memory. Keep in mind, we were pretty coarse grained there by setting it up with the user mysql. You can use a project and the newtask(1) command instead if you want the resource controls to apply to particular processes owned by a user.

This is just a simple example of the kinds of things you can do. Have runaway processes or threads occasionally? You can catch them and kill them with resource controls. You can also use coreadm(1M) to be sure you're capturing the errant behavior to analyze and fix the issues in useful core files, not just droppings all over the filesystem. Have a look at resource_controls(5) and related documentation for details.