I got compiz (the compositing window manager) to run very nicely on Solaris x86 (build 55)
I've tried so far on an Ultra 20 and Ultra 40 with Nvidia cards.

Here is the obligatory screenshot :)

You can only see translucency and drop shadows effects from the picture but have a look at these videos to have an idea of what kind of animations an hardware accelerated OpenGL compositing window manager is capable of.

You'll find below how to compile it on Solaris :

IMPORTANT : Please be sure to backup all files that are going to be edited/replaced. There is no warranty that this will work on your particular computer :)

Prerequisites :

You need :

- a version of (Open) Solaris with Xorg 7.2 (build 59/60)
- a machine with an Nvidia Card with the latest drivers
    - In your xorg.conf you need to add the following settings to enable the Composite extension and to hardware accelerate the Render extension :
       (if you don't have an /etc/X11/xorg.conf file you can either run nvidia-xconfig to generate one or run /usr/X11/bin/Xorg -configure, then take the file generated /xorg.conf-new and move it to /etc/xorg.conf and then add the sections below.)

in Section "Screen":
Option "RenderAccel" "True"
Option "AllowGLXWithComposite" "True"
Option "AddARGBGLXVisuals" "True"
and :
Section "Extensions"
Option "Composite" "Enable"
EndSection

- JDS and it's development packages (select "install developer packages" during install)
- JDS Build environment

Setup :

As for the moment Xorg 7.2 on Solaris doesn't ship all the 7.2 version of its client libraries there are some missing pkg-config, headers files and libraries that needs to be added.
You'll find all the missing files in tarball called "missing-stuff" here.

You'll need to add the files from this tarball at the following locations :

- Xregion.h in /usr/include/X11
- replace /usr/openwin/share/include/X11/extensions/Xrender.h by the one provided in the tarball
- Add all the *.pc files in /usr/lib/pkg-config
- As libXcomposite shipped with Xorg 7.2 on Solaris requires libXfixes 4.0 and the one shipped isonly 3.0.2 you need  to either :
        - get Xfixes from fdo then compile and install it
            or
        - replace /usr/lib/X11/libXcomposite.so.1 by the precompiled x86 lib provided in the missing-stuff tarball
- Then you need to get version 0.9.2 of Xrender as some gradient functions are missing from the one available on Solaris again you can either :
        - get Xrender from fdo then compile and install it
            or
        - replace /usr/lib/X11/libXrender.so.1 by libXrender.so.1.3.0 provided in the missing-stuff tarball

- if you don't have Xcomposite.h, composite.h and compositeproto.h installed on your system used the one provided in the tarball and add them in  /usr/X11/include/X11/extensions

Patch, Compile, Install :

You can get the compiz source either :
    - from the official git repository on freedesktop.org (git clone git://git.freedesktop.org/git/xorg/app/compiz).
        - git isn't available by default on Solaris. You can get spec files ready to compile for git from the SFE repository from the pkgbuild project.  (make sure to compile the dependencies before compile SFEgit itself)

    OR
    -  you can get my already patched version of the compiz source :)

If you got compiz from the git repository you need to get and apply my patch to get compiz to compile on Solaris (in the compiz dir do : gpatch -p1 < <pathto>compiz-solaris-2-3-7.diff)

Now you're ready to compile compiz ! Use the following command :

LDFLAGS="-lX11 -lXext" ./autogen.sh --prefix=/usr

then make and as root make install

Settings and Running it :

Once compiz is installed you need to enable some plugins to get most eye candy from startup. Use this command to enable the plugins via gconf:

 gconftool-2 -s /apps/compiz/general/allscreens/options/active_plugins "[gconf,decoration,wobbly,fade,minimize,cube,switcher,move,resize,place,rotate,zoom,scale,dbus]" -t list --list-type=string

Then in a gnome session run :

gtk-window-decorator --replace & compiz --replace gconf &

Voila ! 

Compiz-settings (compiz preference/configuration manager)

Now that compiz is running you'll want to use a GUI to set the many properties that drive the numerous compiz plugins.
Compiz-settings is what you need. In order to get it to compile on Solaris with Sun's cc a good few changes need to be made
First get compiz-settings tarball or get from its git repository then apply my patch which fixes compilation problems them configure it with the following command line :

 LDFLAGS="-lX11 -lXext" ./autogen.sh --prefix=/usr

 then make and as root make install.