cn=Directory Manager
All about Directory Server
All | Personal | Sun

20060518 Thursday May 18, 2006

A Quick Introduction to ASN.1 BER

Many network protocols are text-based, which has the advantages of being relatively easy to understand if you examine the network traffic, and in many cases you can even interact with the target server by simply telnetting to it and typing in the appropriate commands. However, there are disadvantages as well, including that they are generally more verbose and less efficient to parse than they need to be. On the other hand, other protocols use a binary encoding that is more compact and more efficient. LDAP falls into this category, and uses the ASN.1 (abstract syntax notation one) mechanism, and more specifically the BER (basic encoding rules) flavor of ASN.1. There are a number of other encoding rules (e.g., DER, PER, CER, etc.) that fall under the ASN.1 umbrella, but since LDAP uses BER that's what I'll focus on in this post. In general, when I talk about ASN.1, I mean BER.

I should first point out that this is a very cursory overview of ASN.1 and doesn't attempt to cover everything. I'm largely focusing on the subset of BER that is actually used by LDAP, and there are some obscure special cases that I'll not get into as well. For a much more in-depth reference, check out the excellent ASN.1 Complete reference book by John Larmouth which is freely available in PDF form (although you do have to fill out a form to be able to download it) or you can buy the book in "dead tree" form. I should also say that this discussion assumes that you have at least a basic understanding of binary and hexadecimal numbering systems. If you aren't familiar with that or need to brush up on it, then I'm sure you'll be able to find plenty of sites to help with that.

BER elements use a TLV structure, where TLV stands for "type, length, and value". That is, each BER element has one or more bytes (in all cases I'm aware of in LDAP, it's only a single byte) that indicates the data type for the element, one or more bytes that indicates the length of the value, and the encoded value itself (where the form of the encoded value depends on the data type) which can be zero or more bytes. I'll expand on each of these in the next sections.

The BER Type
The BER type indicates the data type for the value of the element. There are lots of different data types available, but the most commonly-used (at least in LDAP) include OCTET STRING (which can be either a text string or just some binary data), INTEGER, BOOLEAN, NULL, ENUMERATED (like an integer, but where each value has a special meaning), SEQUENCE (an ordered collection of other elements, kind of like an array), and SET (the same as a sequence, except that the order doesn't matter). There is also a CHOICE element, but most of the time it just means that you can have one of a few different kinds of elements.

As I mentioned above, the BER type is usually only a single byte, and this byte has data encoded in it. The two most significant bits (i.e., the two leftmost bits, since BER always uses big endian/network ordering) are used to indicate the class for the element. The possible class values are:
  • 00 -- This is the universal class. All of the "standard" BER elements have a universal type, so any time you see an element with a universal type you know what kind of data it holds. Examples of universal types include 0x01 (BOOLEAN), 0x02 (INTEGER), 0x04 (OCTET STRING), 0x05 (NULL), 0x0A (ENUMERATED), 0x30 (SEQUENCE), and 0x31 (SET). You'll notice that the binary encodings for all of those type values have the leftmost two bits set to zero.

  • 01 -- This is the application-specific class. This class is used to allow an "application" to define its own types that will be consistent throughout that application. In this context, LDAP is considered an application. For example, any time you see 0x42 in LDAP, you know that it indicates an unbind request protocol op because RFC 2251 section 4.3 states that the bind request protocol op has a type of "[APPLICATION 2]".

  • 10 -- This is the context-specific class. This class is used to indicate that the type is specific to a particular usage within a given application. The same type may be re-used in different contexts in the same application as long as there is enough other information for you to determine which context is applicable in a given situation. For example, in the context of the credentials in a bind request protocol op, the context-specific type 0x80 is used to hold the bind password, but in the context of an extended operation it would be used to hold the request OID.

  • 11 -- This is the private class. I'm not aware of any cases in which it is used in LDAP.

The next bit (the third from the left) is the primitive/constructed bit. If it is set to zero (i.e., "off"), then the element is considered primitive and therefore the value would be encoded in accordance with the rules of that data type. If it is set to one (i.e., "on"), then it means that the value is constructed from zero or more other ASN.1 elements that are concatenated together in their encoded forms. For example, if you look at the universal SEQUENCE type of 0x30, the binary encoding is "00110000" and the primitive/constructed bit is set to one indicating that the value of the sequence is constructed from zero or more encoded elements.

The final five bits of the BER type byte are used to specify the value of that type, and it's treated as a simple integer value (where "00000" is zero, "00001" is one, "00010" is two, "00011" is three, etc.). The only special value is "11111", which means that the type value is larger than can fit in the five bits allowed so multiple bytes will be required. Since this doesn't happen in LDAP we'll ignore it in this discussion.

The BER Length
The second component in the TLV structure of a BER element is the length. This specifies the size in bytes of the encoded value. For the most part, this uses a straightforward binary encoding of the integer value (e.g., so if the encoded value is five bytes long, then it would be encoded as 00000101 binary, or 0x05 hex), but if the value is longer than 127 bytes then it will be necessary to use multiple bytes to encode the length. In that case, the first byte has the leftmost bit set to one and the remaining seven bits are used to specify the number of bytes required to encode the full length. For example, if there are 500 bytes in the length (hex 0x01F4), then the encoded length will actually consist of three bytes: 82 01 F4.

Note that there is an alternate form for encoding the length called the indefinite form. In this mechanism, only a part of the length is given at a time, kind of like the chunked encoding that is available in HTTP 1.1. However, this form is not used in LDAP (as per RFC 2251 section 5.1), so it won't be discussed here any further.

The BER Value
The value is the heart of the BER element because it contains the actual data of the element. Because BER is a binary encoding, the encodings can take advantage of that to represent the data in a compact form. As such, each data type has its own encoded form. These encodings include:
  • NULL -- The NULL element never has a value, and therefore the length is always zero.

  • OCTET STRING -- The value of this element is simply encoded as a concatenation of the raw bytes of the data being represented. For example, to represent the string "Hello", the encoded value would be "48 65 6C 6C 6F". The value can have a length of zero bytes.

  • BOOLEAN -- The value of this element is always a single byte. If all the bits in that byte are set to zero (i.e., 0x00), then the value is "FALSE". If one or more of the bytes is set to one, then the value is "TRUE". This means that there are 255 different ways to encode a BOOLEAN value of "TRUE", although in practice it's generally encoded as 0xFF (that is, all the bits are set to one).

  • INTEGER -- The value of this element is encoded as a binary integer in two's complement form (see this article if you're not familiar with this representation). Although BER itself does not place a limit on the magnitude of the values that can be encoded, many software implementations have a cap of four or eight bytes (i.e., 32-bit or 64-bit integer values), and LDAP generally uses a maximum of 4 bytes (allowing you to encode values within the plus or minus 2 billion range). There will always be at least one byte in the value.

  • ENUMERATED -- The value of this element is encoded in exactly the same way as the value of an INTEGER element.

  • SEQUENCE -- The value of this element is simply a concatenation of the encoded BER elements contained in the sequence. For example, if I wanted to encode a sequence with two octet string elements encoding the text "Hello" and "there", then the encoded sequence value would be "04 05 48 65 6C 6C 6F 04 05 74 68 65 72 65". A sequence value can be zero bytes if there are no elements in the sequence.

  • SET -- The value of this element is encoded in exactly the same way as the value of a SEQUENCE element.

BER Encoding Examples
Now that we've covered the basics of encoding the type, length, and value components of a BER element, we can put together some examples. The example above for encoding a SEQUENCE value actually had two complete BER elements concatenated together: the OCTET STRING representations of the strings "Hello" and "there". They are:
04 05 48 65 6C 6C 6F
04 05 74 68 65 72 65

In both of these cases, the first byte is the type (0x04, which is the universal primitive OCTET STRING type), and the second is the length (0x05, indicating that there are five bytes in the value). The remaining five bytes are the encoded representations of the strings "Hello" and "there".

Another simple examle would be to encode the integer value 3. This time, though, let's use a context-specific type value of 5 rather than the universal INTEGER type. In this case, the encoding would be:
85 01 03

Now let's go for a little more involved (and more practical) example. Let's encode an LDAP bind request protocol op as defined in RFC 2251 section 4.2. A simplified BNF representation of this element is as follows:
BindRequest ::= [APPLICATION 0] SEQUENCE {
     version                    INTEGER (1 .. 127),
     name                       OCTET STRING,
     authentication             CHOICE {
          simple                [0] OCTET STRING,
          sasl                  [3] SEQUENCE {
               mechanism        OCTET STRING,
               credentials      OCTET STRING OPTIONAL } } }

In this case, we'll encode a bind request using simple authentication for the user "cn=test" with a password of "password". The complete encoding for this bind request protocol op is:
60 16 02 01 03 04 07 63 6E 3D 74 65 73 74 80 08 70 61 73 73 77 6F 72 64

That's a fairly long string of bytes, but let's break it down to make it simpler:
  • The first byte is 0x60 and it is the BER type for the bind request protocol op. It comes from the "[APPLICATION 0] SEQUENCE" portion of the definition. Because it's application-specific, then the class bytes will be 01, and because it's a SEQUENCE, then it will be constructed. If you put that together with a type value of zero, then the binary representation is "01100000", which is 0x60 hex.

  • The second byte is 0x16, which indicates the length of the bind request sequence. 0x16 hex is 22 decimal, and if you count the number of bytes after the 0x16 then you'll see that there are 22 of them.

  • Next comes "02 01 03", which is a universal INTEGER value of 3. It corresponds to the "version" component of the bind request sequence, and it indicates that this is an LDAPv3 bind request.

  • Next comes "04 07 63 6E 3D 74 65 73 74", which is a universal OCTET STRING containing the text "cn=test". It corresponds to the "name" component of the bind request sequence.

  • The last component is "80 08 70 61 73 73 77 6F 72 64", which is an element with a type of "context-specific primitive 0" and a length of eight bytes. From the definition of the bind request protocol op above, we can see that "context-specific" maps to the simple authentication type and that it should be treated as an OCTET STRING, and upon closer examination we can see that those eight bytes in the value do represent the encoded string "password".

I realize that was a pretty significant jump in complexity between my examples. However, hopefully if you can follow the explanation of the encoding of the bind request element, then you're well on your way to being able to debug LDAP protocol communication. For additional help, check out the LDAPDecoder tool provided as part of SLAMD (if you use the "-b" option, it will show you the raw bytes for the communication along with the decoded human-readable representation. You can also check out the code in the com.sun.slamd.asn1 package in the SLAMD source code for a Java implementation of a simple BER encoder/decoder (it's what the LDAPDecoder uses behind the scenes to translate between raw bytes and BER elements).

Posted by cn_equals_directory_manager ( May 18 2006, 08:03:47 AM CDT ) Permalink Comments [2]

Comments:

in the below paragraph I think there is a typo. "Next comes "04 07 63 6E 3D 74 65 73 74", which is a universal OCTET STRING containing the text "cn=manager"." shouldn't that be cn=test?

Posted by Ashok Nair on May 22, 2006 at 01:28 PM CDT #

You are correct. It should be "cn=test". I have updated the post to include the correct information. Thanks for pointing this out.

Posted by Neil Wilson on May 22, 2006 at 01:35 PM CDT #

Post a Comment:

Comments are closed for this entry.

Archives
Language
Links
Referrers