@(#)gpacket.txt 1.1 03/05/14 GPacket: A Generic Binary Packet Format Version 3.5 (350) 5/13/2003 Introduction ------------ This document specifies a simple, generic, binary packet format. The design of this format is driven by the following requirements: 1. Simple. Where possible the packet format is kept as simple as possible. 2. Compact. The format attempts to be relatively compact, but not at the sacrifice of simplicity. 3. Fast. The format is binary and designed for rapid marshalling and unmarshalling. 5. Flexible. The format is not application specific and supports generic name/value pairs, and an opaque payload, so that it can be used in a variety of applications, and can adapt for future needs. 6. Versioned. The format provides a degree of version control and ability to accommodate future format changes if needed. Overview -------- The a GPacket is divided into 3 sections: 1. A 32 byte fixed header 2. A variable sized properties section 3. A variable sized payload section For the sake of simplicity we limit the size of the fields to the following: 1 bit (in a group of 8 bits) 8 bits 16 bits 32 bits 64 bits This results in some waste, but we deem this an acceptable compromise between simplicity and space efficiency. All values must fall on byte boundaries. Additionally some portions of the packet (such as the fixed header) are defined so that the fields fall on the proper bit boundary (i.e. a 32 bit boundary for ints, 64 bit boundary for longs, etc). This reduces waste as well as makes it easier to map to C structs. All values are in big-endian byte orer (high-order byte first). This is called "network byte order" and is also the same as is defined by the Java programming language. Packet Format ------------- 0 1 2 3 |0 1 2 3 4 5 6 7|8 9 0 1 2 3 4 5|6 7 8 9 0 1 2 3|4 5 6 7 8 9 0 1| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | magic # | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | version | packet type | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | size | -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | property data size | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + timestamp + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + sequence + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 36 bytes|f|e|d|c|b|a|Z|Y|X|W|V|... 1 bit flags ...|K|J|I|H|G|F|E|D|C|B|A| --------+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + property data + | . . . | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + payload + | . . . | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ version: 16 bits A unique 16 bit value that changes with each incompatible change to the packet format. This specification describes version 3.5, so this value would be 350. packet type: 16 bits Type of packet. This is an application specified value that defines the type of the packet. It is typically used to determine how to handle the packet and interpret the data in the properties and payload section. size: 32 bits Size of entire packet (including first 12 bytes: magic, version, type and size). magic: 32 bits A unique 32 bit value that will never change from packet version to packet version. This value helps us to detect bogus data on the wire (although it certainly does not guarantee it). Note that 'version' also helps detect corrupted packets. Value should be: 2147476418 property data size: 32 bits Size of the property data in bytes timestamp: 64 bits Timestamp of when message was sent by original sender sequence number: 32 bits Sequence number generate by original sender. This number must be increased monotonically for every message sent by the sender. bit flags: 32 bits 1 bit flags for use by the application. property data Variable section of the packet that contains marshalled application defined name/value pairs. The format is: 0 1 2 3 |0 1 2 3 4 5 6 7|8 9 0 1 2 3 4 5|6 7 8 9 0 1 2 3|4 5 6 7 8 9 0 1| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | version # | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | size | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | property name length | property name (UTF-8) . . . +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | property value type | property value . . . +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ . . . +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Where version #: A version number. Current version is 1. size : number of properties marshalled These are followed by a series of name/value pairs. The name is marshalled as a length followed by the UTF-8 representation of the string. The value is marhsalled as a type followed by the value. The supported types and their corresponding values are as follow. Note that the exact data formats are as defined by the corresponding write methods for java.io.DataOutputStream: BOOLEAN=1 8 bit integer (1=true, 0=false) BYTE=2 8 bit integer (signed) SHORT=3 16 bit integer (signed) INTEGER=4 32 bit integer (signed) LONG=5 64 bit integer (signed) FLOAT=6 32 bit floating point as specified by the Java method Float.floatToIntBits() DOUBLE=7 64 bit floating point as specified by the Java method Float.floatToLongBits() STRING=8 Short length followed by UTF-8 encoded string OBJECT=9 Short length followed by serialized Java object payload Application specify payload. Use In S1MQ ----------- GPacket is basically a simplified and generalized version of the S1MQ client<->broker packet format (we'll call this JMQPacket). Current plans are to use GPacket for broker<->boker cluster communication. At some point in the future we could choose to port client<->broker communication to use it. The motivation for this would be to simplify our code base and standardize on one packet format. It's unclear if the benefits of doing this justify the cost. If we were to do this the approach would go something like this: 1. Any information in the fixed header of the JMQPacket that is not in the fixed header of the GPacket (like expiration) would move to the property section of GPacket (using terse property names). 2. Any information in the variable header portion of the JMQPacket would move to the property section of GPacket (using terse property names). 3. Properties on JMQPacket control messages would stay in the property section of GPacket. 4. When marshalling a JMS Message, the GPacket payload section would contain the marshalled JMS message properties as well as the message body.