XSLT transformation in ASP: non-standard XML encoding

If you want to generate documents with non-standard encodings with server-side XSLT transformation in IIS/ASP environment, you should:

• Set the Response.Charset property to the desired character set;
• Set the Response.Codepage property to the desired code page (or use the <%@ LANGUAGE="VBScript" CODEPAGE="codepage" %> directive in your ASP script).
• Omit XML declaration from the translated text (using omit-xml-declaration=”yes” attribute in xsl:output element) and prepend the desired XML declaration in front of the translated text.

For example, the following program generates XML (or XHTML) document encoded in windows-1250 character set (codepage: 1250) …

<%
Const DOMClass = "MSXML2.DOMDocument"
Set XSLT = Server.CreateObject(DOMClass)
Set XDoc = Server.CreateObject(DOMClass)
XDoc.loadXML("<root greek='&#946;' ee='č' />")
XSLT.load(Server.MapPath("SampleXSLT.xsl"))
Response.Clear
Response.Charset = "windows-1250"
Response.Codepage = 1250
Response.ContentType = "text/xml"
Response.Write "<?xml version='1.0' encoding='windows-1250'?>"
Response.Write XDoc.transformNode(XSLT)
%>

… when using the following XSL transformation:

<?xml version="1.0"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" omit-xml-declaration="yes" />
… rest deleted …

XSLT transformation in ASP: XML with MSXML6

Similar to the XML-to-HTML transformation with MSXML6, the results of the transformNode function do not include the encoding information; the XML text produced by the sample program with the XSL transformation described in the previous post is a perfect XML document:

<?xml version="1.0"?>
<output>
  Greek letter: β 
  EE: č
</output>

An XML document without the encoding attribute in the xml declaration is assumed to be encoded in UTF-8. The results of the transformNode function in MSXML6 are thus absolutely correct if you use UTF-8 output encoding.

XSLT transformation in ASP: XML with MSXML3

You might consider server-side XML-to-XML transformations rare, but you have to use <xsl:output method=”xml”> if you want to generate valid XHTML code from your XML data. To shorten the printouts, we’ll use a simple (non-XHTML) XSL transformation to generate the test results:

<?xml version="1.0" encoding="utf-8" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    
<xsl:output method="xml" encoding="utf-8" />

<xsl:template match="root">
  <output>
    Greek letter: <xsl:value-of select="@greek" /> 
    EE: <xsl:value-of select="@ee" />
  </output>
</xsl:template> 

</xsl:stylesheet>

As with HTML, MSXML3 interferes with the output, inserting improper UTF-16 encoding directive, resulting in the following XML text:

<?xml version="1.0" encoding="UTF-16"?>
<output>
  Greek letter: β 
  EE: č
</output>

You could bypass this bug by setting the omit-xml-declaration attribute of the xsl:output element to yes …

<?xml version="1.0" encoding="utf-8" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    
<xsl:output method="xml" encoding="utf-8" omit-xml-declaration="yes" />

… rest deleted …

… resulting in the following transformation output:

<output>
  Greek letter: β 
  EE: č
</output>

However, if you want to retain XML declaration in the XML document, you have to replace UTF-16 in the output string with UTF-8, like we did in the HTML transformation case. The following modified test program produces perfect XML document when used with the original XSLT transformation (without the omit-xml-declaration attribute):

<%
Const DOMClass = "MSXML2.DOMDocument"

Set XSLT = Server.CreateObject(DOMClass)
Set XDoc = Server.CreateObject(DOMClass)

XDoc.loadXML("<root greek='β' ee='č' />")
XSLT.load(Server.MapPath("SampleXSLT.xsl"))

Response.Clear
Response.Charset = "utf-8"
Response.Codepage = 65001
Response.Write Replace(XDoc.transformNode(XSLT), _
  "encoding=""UTF-16""","encoding=""utf-8""")
%>

XSLT transformation in ASP: HTML with MSXML6

If your IIS platform includes MSXML6, you should use MSXML6 to transform XML into HTML instead of MSXML3 (see the Using the right version of MSXML article for proper fallback process). MSXML6 still generates the META tag (the MSXML3-related post describes problems caused by the META tag), but does not include the charset parameter in it, resulting in perfect HTML output. The sample ASP program using the simple XSLT transformation from the “XSLT transformation in ASP: HTML with MSXML3” post produces the following output when using MSXML6 (MSXML2.DomDocument.6.0 ProgID):

<html>
<head>
<META http-equiv="Content-Type" content="text/html">
<title>Sample XSLT server-side transformation</title>
</head>
<body><node type="test">Greek letter: β EE: č</node></body>
</html>

If you want to use complex XSLT transformations with MSXML6, you might have to set additional second-level DOM properties, for example AllowDocumentFunction. You might also want to set ValidateOnParse to false if the validity of your XML document is not a major concern.

XSLT transformation in ASP: HTML with MSXML3

Most commonly, you’ll use server-side XSLT transformation in ASP to transform XML data into HTML using MSXML3 (available on almost all IIS platforms). The following XSLT stylesheet is used in the HTML test:

<?xml version="1.0" encoding="utf-8" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    
<xsl:output method="html" encoding="utf-8" />

<xsl:template match="root">
  <html>
    <head>
      <title>Sample XSLT server-side transformation</title>
    </head>
    <body>
      <node type="test">
        Greek letter: <xsl:value-of select="@greek" /> 
        EE: <xsl:value-of select="@ee" />
      </node>
    </body>
  </html>
</xsl:template> 

</xsl:stylesheet>

With MSXML3, the transformNode function inserts a META directive in the output stream, resulting in the following transformation result:

<html>
<head>
<META http-equiv="Content-Type" content="text/html; charset=UTF-16">
<title>Sample XSLT server-side transformation</title>
</head>
<body><node type="test">
Greek letter: β 
EE: č
</node>
</body>
</html>

On the other hand, the HTTP headers generated by the test ASP script claim the content is UTF-8 encoded (and the raw data dump performed with Fiddler confirms that). The response headers were taken from LiveHTTPHeaders Firefox extension:

HTTP/1.x 200 OK
Server: Microsoft-IIS/5.1
Date: Mon, 15 Sep 2008 11:20:56 GMT
X-Powered-By: ASP.NET
Content-Length: 222
Content-Type: text/html; Charset=utf-8
Cache-Control: private

Most browsers (including IE and Firefox) take the Content-type from HTTP headers, using the META directive as a fallback. They thus render the page as intended. Some browsers (including the TextView tab of Fiddler) prefer the META directive and produce garbage.

The solution

To match the META header with the HTTP headers, replace the charset=UTF-16 string in the text returned by the transformNode function with charset=UTF-8. The modified sample ASP script is included below:

<%
Const DOMClass = "MSXML2.DOMDocument"

Set XSLT = Server.CreateObject(DOMClass)
Set XDoc = Server.CreateObject(DOMClass)

XDoc.loadXML("<root greek='β' ee='č' />")
XSLT.load(Server.MapPath("SampleXSLT.xsl"))

Response.Clear
Response.Charset = "utf-8"
Response.Codepage = 65001
Response.Write Replace(XDoc.transformNode(XSLT),"charset=UTF-16","charset=UTF-8")
%>

XSLT transformation in ASP: The testbed

As I’ve mentioned in the previous post, MSXML functions called from ASP believe they work in UTF-16 environment. The transformNode function might insert this information in the output string based on several parameters, one of them being the version of the MSXML ActiveX control. To test behavior of various versions of MSXML, we’ll use the following test program:

<%
Const DOMClass = "MSXML2.DOMDocument"

Set XSLT = Server.CreateObject(DOMClass)
Set XDoc = Server.CreateObject(DOMClass)

XDoc.loadXML("<root greek='β' ee='č' />")
XSLT.load(Server.MapPath("SampleXSLT.xsl"))

Response.Clear
Response.Charset = "utf-8"
Response.Codepage = 65001
Response.Write XDoc.transformNode(XSLT)
%>

And a stylesheet similar to this one:

<?xml version="1.0" encoding="utf-8" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    
<xsl:output method="xml" encoding="utf-8" />

<xsl:template match="root">
  <output>
    <node type="test">
      Greek letter: <xsl:value-of select="@greek" /> 
      EE: <xsl:value-of select="@ee" />
    </node>
  </output>
</xsl:template> 

</xsl:stylesheet>

We’ll test two MSXML versions: MSXML3 (default with Windows XP) and MSXML6 (default with Vista, also available on Windows XP). MSXML3 should be available on almost all web servers, you might not get MSXML6 everywhere (my hosting provider did not offer it when it mattered most to me).

The test results will be published in the next few posts.

Further reading:

• Using the right version of MSXML

XSLT transformation in ASP: the principles

If you want to perform server-based XSLT transformations on IIS using classic ASP, you should keep in mind the following facts:

• ASP uses 16-bit Unicode characters internally. MSXML ActiveX modules called from ASP therefore think they work in UTF-16 environment.
• Due to the impression that the XSLT transformations happen in UTF-16 environment, MSXML procedures might prepend UTF-16 specific headers in transformed text (more about that in follow-up posts).
• The transformed text returned to ASP from transformNode function is UTF-16 encoded.
• When the ASP script sends the text to the browser (or writes it into a text file), the codepage set with Response.Codepage is used to transform UTF-16 into target character set.

Further reading:

• Output XML data from ASP
• How do you write utf-8 data from ASP?
• UTF-8 text file generated from ASP
• Optimized Presentation of XML Content
• Server-Side XSLT Transformation
• AJAX ASP library
• Building XML Response in a Server-Side Script

XPATH expressions in MSXML selectNodes() function are evaluated on the whole DOM tree

The selectNodes() function available in the Microsoft XML (MSXML) API evaluates the XPATH expressions supplied as the argument in the context of the whole DOM tree to which the DOM node belongs, not just the node on which the selectNodes function was executed. For example, if you use the following code …

set firstDiv = document.selectSingleNode("//div")
set paraList = firstDiv.selectNodes("//p")

… the paraList will contain the list of all paragraphs in the whole document, not just the list of paragraphs in the DIV on which the selectNodes call was executed.

Client-side development environment

Scenario: XSLT deployment in a browser/server scenario with XSLT transformations performed on the browser (JavaScript or with <?xsl-stylesheet ?>)

Potential tools:

• Saxon is great for off-line testing (primarily while developing small bits of code or trying new programming techniques). It cannot fetch the information from the server, so you have to provide test data in local XML files
• Firefox is the best tool if you want to test client-server integration and use <?xsl-stylesheet ?> approach as it can recover from XSLT errors (which IE can't). However, it doesn't detect all errors, so the stylesheets working with Firefox might break in IE (but not due to an IE bug)
• IE is a nightmare when using <?xsl-stylesheet ?> directive. Whenever something breaks (error in stylesheet or even server-side error resulting in HTML response instead of XML response), its cache gets messed up and you have to restart the browser to continue the tests. I therefore use IE only in the final tests to verify that my XSLT has no weird errors not detected by Firefox.
• The best testing mechanism is probable the manual (JavaScript) client-side transformation. You can do it in IE or Firefox; in this case I would prefer IE, as it has better XSLT error reporting (MSXML is not that bad :).

MSXML is still only DOM-1 compliant

If you use DOM calls to manipulate XML documents returned with XMLHttpRequest object (and who doesn't), you'll be sooner or later highly upset when the code you've tested on Firefox or Opera fails in IE (including IE7). The reason - Microsoft still did not implement Node.getElementById function. So here's the "replacement":

function getDomById(e,id) {
  if (e.getElementById) { return e.getElementById(id); }
  if (e.documentElement) e = e.documentElement;
  
  var cn = e.childNodes;
  for (var i = 0 ; i != cn.length; i++) {
    var n = cn[i]; 
    if (n.nodeType == 1) {
      if (n.getAttribute("id") == id) return n ;
    }
  }
  for (var i = 0 ; i != cn.length; i++) {
    var n = cn[i]; 
    if (n.nodeType == 1) {
      var result = getDomById(n,id); if (result) return result;
    }
  }
  return false;
}

Note: The procedure is doing breadth-first search as I was looking for IDs that were pretty high in the hierarchy of trees with many branches.