Friday September 17, 2004

Detecting XML charset encoding, again

I've got some comments (and corrections) on my previous posting Detecting XML charset encoding, getting it right (or at least trying to). Based on the received feedback I've rewritten the algorithms (they are simpler now, and -hopefully- correct). They are already implemented in Rome. Currently they are in CVS and they will be part of the upcoming release.

As before, comments are welcome.

Raw XML charset encoding detection

Detecting charset encoding of a XML document without external information (i.e. reading a XML document from a file):

BOMEnc     : byte order mark. Possible values: 'UTF-8', 'UTF-16BE', 
             'UTF-16LE' or NULL
XMLGuessEnc: best guess using the byte representation of the first 
             bytes of XML declaration ('<?xml...?>') if present. 
             Possible values: 'UTF-8', 'UTF-16BE', 'UTF-16LE' or NULL
XMLEnc     : encoding in the XML declaration ('<?xml encoding="..."?>'). 
             Possible values: anything or NULL

if BOMEnc is NULL
  if XMLGuessEnc is NULL or XMLEnc is NULL
    encoding is 'UTF-8'                                                               [1.0]
  else
  if XMLEnc is 'UTF-16' and (XMLGuessEnc is 'UTF-16BE' or XMLGuessEnc is 'UTF-16LE')
    encoding is XMLGuessEnc                                                           [1.1]
  else
    encoding is XMLEnc                                                                [1.2]
else
if BOMEnc is 'UTF-8'
  if XMLGuessEnc is not NULL and XMLGuessEnc is not 'UTF-8'
    ERROR, encoding mismatch                                                          [1.3]
  if XMLEnc is not NULL and XMLEnc is not 'UTF-8'
    ERROR, encoding mismatch                                                          [1.4]
  encoding is 'UTF-8'
else
if BOMEnc is 'UTF-16BE' or BOMEnc is 'UTF-16LE'
  if XMLGuessEnc is not NULL and XMLGuessEnc is not BOMEnc
    ERROR, encoding mismatch                                                          [1.5]
  if XMLEnc is not NULL and XMLEnc is not 'UTF-16' and XMLEnc is not BOMEnc
    ERROR, encoding mismatch                                                          [1.6]
  encoding is BOMEnc
else
  ERROR, cannot happen given BOMEnc possible values (see above)                       [1.7]

Byte Order Mark encoding and XML guessed encoding detection rules are clearly explained in the XML 1.0 Third Edition Appendix F.1. Note that in this algorithm BOMEnc and XMLGuessEnc are restricted to UTF-8 and UTF-16* encodings.

#1.0. There is no BOM, an encoding or encoding family cannot be guessed from the first bytes in the stream, defaulting to UTF-8.

#1.1. Strictly following XML 1.0 Third Edition Section 4.3.3 Charset Encoding in Entities (2nd paragraph) no BOM and UTF-16 XML declaration encoding is an error. The BOM is required to identify if the encoding is BE or LE. But if XMLEnc was read it means that the encoding byte order of the stream was guessed from the first bytes in the stream, note that this is possible only if the document starts with a XML declaration. The logic verifies that there is no conflicting encoding information and relaxes the BOM requirement if a XML declaration (that allows guessing the byte order) is present.

#1.2. XMLEnc is present, it is not UTF-16 (although it can be UTF-16BE or UTF-16LE), the guessed encoding is used to read the XML declaration encoding. Detecting encoding mismatches here it would require awareness of charset families, instead the algorithm relies on the charset encoding routines that will process the stream to discover and report mismatches. IMPORTANT: To handle other encodings (i.e. USC-4 or EBCDIC encoding families) the algorithm should be extended here.

#1.3, #1.4, #1.5, #1.6 There is an explicit encoding mismatch.

#1.7. Given the currently assumed BOMEnc values this case cannot happen. IMPORTANT: To handle other encodings (i.e. USC-4 or EBCDIC encoding families) the algorithm should be extended here.

HTTP XML charset encoding detection

Detecting charset encoding of a XML document with external information provided by HTTP:

ContentType: Content-Type HTTP header
CTMime     : MIME type defined in the ContentType
CTEnc      : charset encoding defined in the ContentType, 
             NULL otherwise
BOMEnc     : byte order mark. Possible values: 'UTF-8', 'UTF-16BE', 
             'UTF-16LE' or NULL
XMLGuessEnc: best guess using the byte representation of the first 
             bytes of XML declaration ('<?xml...?>') if present. 
             Possible values: 'UTF-8', 'UTF-16BE', 'UTF-16LE' or NULL
XMLEnc     : encoding in the XML declaration ('<?xml encoding="..."?>'). 
             Possible values: anything or NULL
APP-XML    : RFC 3023 defined 'application/*xml' types
TEXT-XML   : RFC 3023 defined 'text/*xml' types

if CTMime is APP-XML or CTMime is TEXT-XML
  if CTEnc is NULL
    if CTMime is APP-XML
      encoding is determined using the Raw XML charset encoding detection algorithm   [2.0]
    else
    if CTMime is TEXT-XML
      encoding is 'US-ASCII'                                                          [2.1]
  else
  if (CTEnc is 'UTF-16BE' or CTEnc is 'UTF-16LE') and BOMEnc is not NULL
    ERROR, RFC 3023 explicitly forbids this                                           [2.2]
  else
  if CTEnc is 'UTF-16'
    if BOMEnc is 'UTF-16BE' or BOMEnc is 'UTF-16LE' 
      encoding is BOMEnc                                                              [2.3]
    else
      ERROR, missing BOM or encoding mismatch                                         [2.4]
  else
    encoding is CTEnc                                                                 [2.5]
else
  ERROR, handling for other MIME types is undefined                                   [2.6]

Byte Order Mark encoding and XML guessed encoding detection rules are clearly explained in the XML 1.0 Third Edition Appendix F.1. Note that in this algorithm BOMEnc and XMLGuessEnc are restricted to UTF-8 and UTF-16* encodings.

#2.0. HTTP content type declares a MIME of application type and XML sub-type. There is not HTTP Content-type charset encoding information. The charset encoding is determined using the Raw XML charset encoding detection algorithm. Refer to RFC 3023 Section 3.2. Application/xml Registration.

#2.1. HTTP content type declares a MIME of text type and XML sub-type. There is not HTTP Content-type charset encoding information. The charset encoding is US-ASCII. Refer to RFC 3023 Section 3.1. Text/xml Registration.

#2.2. For text-XML and application-XML MIME types if the HTTP content type declares 'UTF-16BE' or 'UTF-16LE' as the charset encoding, a BOM is prohibited. Refer to RFC 3023 Section 4. The Byte Order Mark (BOM) and Conversions to/from the UTF-16 Charset.

#2.3. For text-XML and application-XML MIME types if the HTTP content type declares 'UTF-16' as the charset encoding, a BOM is required and it must be used to determine the byte order. Refer to RFC 3023 Section 4. The Byte Order Mark (BOM) and Conversions to/from the UTF-16 Charset.

#2.4. For text-XML and application-XML MIME types if the HTTP content type declares 'UTF-16' as the charset encoding, a BOM must be present and it must be 'UTF-16BE' or 'UTF-16LE'. Refer to RFC 3023 Section 4. The Byte Order Mark (BOM) and Conversions to/from the UTF-16 Charset.

#2.5. For text-XML and application-XML MIME types if the HTTP content type declares a charset encoding other than the UTF-16 variants, that charset encoding is used, no other special handling is required.

#2.6. There is not defined logic to determine the charset encoding based on the MIME type given by the HTTP content type.