3 Parsing DocBook Documents

$Revision: 1.2 $

$Date: 2002/03/23 20:57:55 $

A key feature of SGML and XML markup is that you validate it. The DocBook DTD is a precise description of valid nesting, the order of elements, and their content. All DocBook documents must conform to this description or they are not DocBook documents (by definition).

A validating parser is a program that can read the DTD and a particular document and determine whether the exact nesting and order of elements in the document is valid according to the DTD.

If you are not using a structured editor that can enforce the markup as you type, validation with an external parser is a particularly important step in the document creation process. You cannot expect to get rational results from subsequent processing (such as document publishing) if your documents are not valid.

The most popular free SGML parser is SP by James Clark, available at http://www.jclark.com/.

SP includes nsgmls, a fast command-line parser. In the world of free validating XML parsers, James Clark's xp is a popular choice.

Note

Not all XML parsers are validating, and although a non-validating parser may have many uses, it cannot ensure that your documents are valid according to the DTD.

Validating Your Documents

The exact way in which the parser is executed varies according to the parser in use, naturally. For information about your particular parser, consult the documentation that came with it.

Using nsgmls

The nsgmls command from SP is a validating SGML parser. The options used in the example below suppress the normal output (-s), except for error messages, print the version number (-v), and specify the catalog file that should be used to map public identifiers to system identifiers. Printing the version number guarantees that you always get some output, so that you know the command ran successfully:

[n:\dbtdg] nsgmls -sv -c \share\sgml\catalog test.sgm
m:\jade\nsgmls.exe:I: SP version "1.3.2"

Because no error messages were printed, we know our document is valid. If you're working with a document that you discover has many errors, the -f option offers a handy way to direct the errors to a file so they don't all scroll off your screen.

If you want to validate an XML document with SP, you must make sure that SP uses the correct declaration. An XML declaration called xml.dcl is included with SP.

The easiest way to make sure that SP uses xml.dcl is to include the declaration explicitly on the command line when you run nsgmls (or Jade, or other SP tools):

[n:\dbtdg] nsgmls -sv -c \share\sgml\catalog m:\jade\xml.dcl test.xml
m:\jade\nsgmls.exe:I: SP version "1.3.2"

Using xp

The xp distribution includes several sample programs. One of these programs, Time, performs a validating parse of the document and prints the amount of time required to parse the DTD and the document. This program makes an excellent validity checker:

java com.jclark.xml.apps.Time examples\simple.xml
6.639

The result states that it took 6.639 seconds to parse the DTD and the document. This indicates that the document is valid. If the document is invalid, additional error messages are displayed.

Understanding Parse Errors

Every parser produces slightly different error messages, but most indicate exactly (at least technically)^[12] what is wrong and where the error occurred. With a little experience, this information is all you'll need to quickly identify what's wrong.

In the rest of this section, we'll look at a number of common errors and the messages they produce in SP. We've chosen SP for the rest of these examples because that is the same parser used by Jade, which we'll be discussing further in Chapter 4, Publishing DocBook Documents.

DTD Cannot Be Found

The telltale sign that SP could not find the DTD, or some module of the DTD, is the error message: "cannot generate system identifier for public text …". Generally, the errors that occur after this are spurious; if SP couldn't find some part of the DTD, it's likely to think that everything is wrong.

Careful examination of the following document will show that we've introduced a simple typographic error into the public identifier (the word “DocBook” is misspelled with a lowercase “b”):

<!DOCTYPE chapter PUBLIC "-//OASIS//DTD Docbook XML V4.1.2//EN"
                  "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd">
<chapter><title>Test Chapter</title>
<para>
This is a paragraph in the test chapter. It is unremarkable in
every regard. This is a paragraph in the test chapter. It is
unremarkable in every regard. This is a paragraph in the test
chapter. It is unremarkable in every regard.
</para>
<para>
<emphasis role="bold">This</emphasis> paragraph contains
<emphasis>some <emphasis>emphasized</emphasis> text</emphasis>
and a <superscript>super</superscript>script
and a <subscript>sub</subscript>script.
</para>
<para>
This is a paragraph in the test chapter. It is unremarkable in
every regard. This is a paragraph in the test chapter. It is
unremarkable in every regard. This is a paragraph in the test
chapter. It is unremarkable in every regard.
</para>
</chapter>

SP responds dramatically to this error:

hermes:/documents/books/tdg/examples/errs$ nsgmls -sv -c cat1 /usr/lib/sgml/declaration/xml.dcl nodtd.sgm
nsgmls:I: SP version "1.3.4"
nsgmls:nodtd.sgm:2:76:E: could not resolve host "www.oasis-open.org" (try again later)
nsgmls:nodtd.sgm:2:76:E: DTD did not contain element declaration for document type name
nsgmls:nodtd.sgm:3:8:E: element "chapter" undefined
nsgmls:nodtd.sgm:3:15:E: element "title" undefined
nsgmls:nodtd.sgm:4:5:E: element "para" undefined
nsgmls:nodtd.sgm:10:5:E: element "para" undefined
nsgmls:nodtd.sgm:11:15:E: there is no attribute "role"
nsgmls:nodtd.sgm:11:21:E: element "emphasis" undefined
nsgmls:nodtd.sgm:12:9:E: element "emphasis" undefined
nsgmls:nodtd.sgm:12:24:E: element "emphasis" undefined
nsgmls:nodtd.sgm:13:18:E: element "superscript" undefined
nsgmls:nodtd.sgm:14:16:E: element "subscript" undefined
nsgmls:nodtd.sgm:16:5:E: element "para" undefined

Other things to look for, if you haven't misspelled the public identifier, are typos in the catalog or failure to specify a catalog that resolves the public identifier that can't be found.

ISO Entity Set Missing

A missing entity set is another example of either a misspelled public identifier, or a missing catalog or catalog entry.

In this case, there's nothing wrong with the document, but the catalog that's been specified is missing the public identifiers for the ISO entity sets:

[n:\dbtdg]nsgmls -sv -c examples\errs\cat2 examples\simple.sgm
m:\jade\nsgmls.exe:I: SP version "1.3.2"
m:\jade\nsgmls.exe:n:/share/sgml/docbook/3.1/dbcent.mod:53:65:W: cannot generate system identifier for public text "ISO 8879:1986//ENTITIES Added Math Symbols:Arrow Relations//EN"
m:\jade\nsgmls.exe:n:/share/sgml/docbook/3.1/dbcent.mod:54:8:E: reference to entity "ISOamsa" for which no system identifier could be generated
m:\jade\nsgmls.exe:n:/share/sgml/docbook/3.1/dbcent.mod:52:0: entity was defined here
m:\jade\nsgmls.exe:n:/share/sgml/docbook/3.1/dbcent.mod:60:66:W: cannot generate system identifier for public text "ISO 8879:1986//ENTITIES Added Math Symbols:Binary Operators//EN"
m:\jade\nsgmls.exe:n:/share/sgml/docbook/3.1/dbcent.mod:61:8:E: reference to entity "ISOamsb" for which no system identifier could be generated
m:\jade\nsgmls.exe:n:/share/sgml/docbook/3.1/dbcent.mod:59:0: entity was defined here
m:\jade\nsgmls.exe:n:/share/sgml/docbook/3.1/dbcent.mod:67:60:W: cannot generate system identifier for public text "ISO 8879:1986//ENTITIES Added Math Symbols:Delimiters//EN"
m:\jade\nsgmls.exe:n:/share/sgml/docbook/3.1/dbcent.mod:68:8:E: reference to entity "ISOamsc" for which no system identifier could be generated
m:\jade\nsgmls.exe:n:/share/sgml/docbook/3.1/dbcent.mod:66:0: entity was defined here
m:\jade\nsgmls.exe:n:/share/sgml/docbook/3.1/dbcent.mod:74:67:W: cannot generate system identifier for public text "ISO 8879:1986//ENTITIES Added Math Symbols:Negated Relations//EN"
...

The ISO entity sets are required by the DocBook DTD, but they are not distributed with it. That's because they aren't maintained by OASIS.^[13]

Character Data Not Allowed Here

Out of context character data is frequently caused by a missing start tag, but sometimes it's just the result of typing in the wrong place!

<!DOCTYPE chapter PUBLIC "-//Davenport//DTD DocBook V3.0//EN">
<chapter><title>Test Chapter</title>
<para>
This is a paragraph in the test chapter. It is unremarkable in
every regard. This is a paragraph in the test chapter. It is
unremarkable in every regard. This is a paragraph in the test
chapter. It is unremarkable in every regard.
</para>
You can't put character data here.
<para>
<emphasis role=bold>This</emphasis> paragraph contains
<emphasis>some <emphasis>emphasized</emphasis> text</emphasis>
and a <superscript>super</superscript>script
and a <subscript>sub</subscript>script.
</para>
<para>
This is a paragraph in the test chapter. It is unremarkable in
every regard. This is a paragraph in the test chapter. It is
unremarkable in every regard. This is a paragraph in the test
chapter. It is unremarkable in every regard.
</para>
</chapter>

[n:\dbtdg] nsgmls -sv -c \share\sgml\catalog examples\errs\badpcdata.sgm
m:\jade\nsgmls.exe:I: SP version "1.3.2"
m:\jade\nsgmls.exe:examples\errs\badpcdata.sgm:9:0:E: character data is not allowed here

Chapters aren't allowed to contain character data directly. Here, a wrapper element, such as Para, is missing around the sentence between the first two paragraphs.

Misspelled Start Tag

If you spell it wrong, the parser gets confused.

<!DOCTYPE chapter PUBLIC "-//Davenport//DTD DocBook V3.0//EN">
<chapter><title>Test Chapter</title>
<para>
This is a paragraph in the test chapter. It is unremarkable in
every regard. This is a paragraph in the test chapter. It is
unremarkable in every regard. This is a paragraph in the test
chapter. It is unremarkable in every regard.
</para>
<paar>
<emphasis role=bold>This</emphasis> paragraph contains
<emphasis>some <emphasis>emphasized</emphasis> text</emphasis>
and a <superscript>super</superscript>script
and a <subscript>sub</subscript>script.
</para>
<para>
This is a paragraph in the test chapter. It is unremarkable in
every regard. This is a paragraph in the test chapter. It is
unremarkable in every regard. This is a paragraph in the test
chapter. It is unremarkable in every regard.
</para>
</chapter>

[n:\documents\books\dbtdg]nsgmls -sv -c \share\sgml\catalog examples\errs\misspe
ll.sgm
m:\jade\nsgmls.exe:I: SP version "1.3.2"
m:\jade\nsgmls.exe:examples\errs\misspell.sgm:9:5:E: element "PAAR" undefined
m:\jade\nsgmls.exe:examples\errs\misspell.sgm:14:6:E: end tag for element "PARA" which is not open
m:\jade\nsgmls.exe:examples\errs\misspell.sgm:21:9:E: end tag for "PAAR" omitted, but OMITTAG NO was specified
m:\jade\nsgmls.exe:examples\errs\misspell.sgm:9:0: start tag was here

Luckily, these are pretty easy to spot, unless you accidentally spell the name of another element. In that case, your error might appear to be out of context.

Misspelled End Tag

Spelling the end tag wrong is just as confusing.

<!DOCTYPE chapter PUBLIC "-//Davenport//DTD DocBook V3.0//EN">
<chapter><title>Test Chapter</titel>
<para>
This is a paragraph in the test chapter. It is unremarkable in
every regard. This is a paragraph in the test chapter. It is
unremarkable in every regard. This is a paragraph in the test
chapter. It is unremarkable in every regard.
</para>
<para>
<emphasis role=bold>This</emphasis> paragraph contains
<emphasis>some <emphasis>emphasized</emphasis> text</emphasis>
and a <superscript>super</superscript>script
and a <subscript>sub</subscript>script.
</para>
<para>
This is a paragraph in the test chapter. It is unremarkable in
every regard. This is a paragraph in the test chapter. It is
unremarkable in every regard. This is a paragraph in the test
chapter. It is unremarkable in every regard.
</para>
</chapter>

[n:\dbtdg]nsgmls -sv -c \share\sgml\catalog examples\errs\misspell2.sgm
m:\jade\nsgmls.exe:I: SP version "1.3.2"
m:\jade\nsgmls.exe:examples\errs\misspell2.sgm:2:35:E: end tag for element "TITEL" which is not open
m:\jade\nsgmls.exe:examples\errs\misspell2.sgm:3:5:E: document type does not allow element "PARA" here; missing one of "FOOTNOTE", "MSGTEXT" start-tag
m:\jade\nsgmls.exe:examples\errs\misspell2.sgm:9:5:E: document type does not allow element "PARA" here; missing one of "FOOTNOTE", "MSGTEXT" start-tag
m:\jade\nsgmls.exe:examples\errs\misspell2.sgm:15:5:E: document type does not allow element "PARA" here; missing one of "FOOTNOTE", "MSGTEXT" start-tag
m:\jade\nsgmls.exe:examples\errs\misspell2.sgm:21:9:E: end tag for "TITLE" omitted, but OMITTAG NO was specified
m:\jade\nsgmls.exe:examples\errs\misspell2.sgm:2:9: start tag was here
m:\jade\nsgmls.exe:examples\errs\misspell2.sgm:21:9:E: end tag for "CHAPTER" which is not finished

These are pretty easy to spot as well, but look at how confused the parser became. From the parser's point of view, failure to close the open Title element means that all the following elements appear out of context.

Out of Context Start Tag

Sometimes the problem isn't spelling, but placing a tag in the wrong context. When this happens, the parser tries to figure out what it can add to your document to make it valid. Then it proceeds as if it had seen what was added in order to recover from the error seen, which can cause future errors.

<!DOCTYPE chapter PUBLIC "-//Davenport//DTD DocBook V3.0//EN">
<chapter><title>Test Chapter</title>
<para>
This is a paragraph in the test chapter. It is unremarkable in
every regard. This is a paragraph in the test chapter. It is
unremarkable in every regard. This is a paragraph in the test
chapter. It is unremarkable in every regard.
</para>
<para><title>Paragraph With Inlines</title>
<emphasis role=bold>This</emphasis> paragraph contains
<emphasis>some <emphasis>emphasized</emphasis> text</emphasis>
and a <superscript>super</superscript>script
and a <subscript>sub</subscript>script.
</para>
<para>
This is a paragraph in the test chapter. It is unremarkable in
every regard. This is a paragraph in the test chapter. It is
unremarkable in every regard. This is a paragraph in the test
chapter. It is unremarkable in every regard.
</para>
</chapter>

[n:\dbtdg]nsgmls -sv -c \share\sgml\catalog examples\errs\badstarttag.sgm
m:\jade\nsgmls.exe:I: SP version "1.3.2"
m:\jade\nsgmls.exe:examples\errs\badstarttag.sgm:9:12:E: document type does not allow element "TITLE" here; missing one of "CALLOUTLIST", "SEGMENTEDLIST", "VARIABLELIST", "CAUTION", "IMPORTANT", "NOTE", "TIP", "WARNING", "BLOCKQUOTE", "EQUATION", "EXAMPLE", "FIGURE", "TABLE" start-tag

In this example, we probably wanted a FormalPara, so that we could have a title on the paragraph. But note that the parser didn't suggest this alternative. The parser only tries to add additional elements, rather than rename elements that it's already seen.

Missing End Tag

Leaving out an end tag is a lot like an out of context start tag. In fact, they're really the same error. The problem is never caused by the missing end tag per se, rather it's caused by the fact that something following it is now out of context.

<!DOCTYPE chapter PUBLIC "-//Davenport//DTD DocBook V3.0//EN">
<chapter><title>Test Chapter</title>
<para>
This is a paragraph in the test chapter. It is unremarkable in
every regard. This is a paragraph in the test chapter. It is
unremarkable in every regard. This is a paragraph in the test
chapter. It is unremarkable in every regard.
</para>
<para>
<emphasis role=bold>This</emphasis> paragraph contains
<emphasis>some <emphasis>emphasized</emphasis> text</emphasis>
and a <superscript>super</superscript>script
and a <subscript>sub</subscript>script.
<para>
This is a paragraph in the test chapter. It is unremarkable in
every regard. This is a paragraph in the test chapter. It is
unremarkable in every regard. This is a paragraph in the test
chapter. It is unremarkable in every regard.
</para>
</chapter>

[n:\dbtdg]nsgmls -sv -c \share\sgml\catalog examples\errs\noendtag.sgm
m:\jade\nsgmls.exe:I: SP version "1.3.2"
m:\jade\nsgmls.exe:examples\errs\noendtag.sgm:14:5:E: document type does not allow element "PARA" here; missing one of "FOOTNOTE", "MSGTEXT", "CAUTION", "IMPORTANT", "NOTE", "TIP", "WARNING", "BLOCKQUOTE", "INFORMALEXAMPLE" start-tag
m:\jade\nsgmls.exe:examples\errs\noendtag.sgm:20:9:E: end tag for "PARA" omitted, but OMITTAG NO was specified
m:\jade\nsgmls.exe:examples\errs\noendtag.sgm:9:0: start tag was here

In this case, the parser figured out that the best thing it could do is end the paragraph.

Bad Entity Reference

If you spell an entity name wrong, the parser will catch it.

<!DOCTYPE chapter PUBLIC "-//Davenport//DTD DocBook V3.0//EN">
<chapter><title>Test Chapter</title>
<para>
This is a paragraph in the test chapter. It is unremarkable in
every regard. This is a paragraph in the test chapter. It is
unremarkable in every regard. This is a paragraph in the test
chapter. It is unremarkable in every regard.
</para>
<para>
There's no entity called &xyzzy; defined in this document.
</para>
<para>
<emphasis role=bold>This</emphasis> paragraph contains
<emphasis>some <emphasis>emphasized</emphasis> text</emphasis>
and a <superscript>super</superscript>script
and a <subscript>sub</subscript>script.
</para>
<para>
This is a paragraph in the test chapter. It is unremarkable in
every regard. This is a paragraph in the test chapter. It is
unremarkable in every regard. This is a paragraph in the test
chapter. It is unremarkable in every regard.
</para>
</chapter>

[n:\dbtdg]nsgmls -sv -c \share\sgml\catalog examples\errs\badent.sgm
m:\jade\nsgmls.exe:I: SP version "1.3.2"
m:\jade\nsgmls.exe:examples\errs\badent.sgm:10:26:E: general entity "xyzzy" not defined and no default entity

More often than not, you'll see this when you misspell a character entity name. For example, this happens when you type &ldqou; instead of “.

Invalid 8-Bit Character

In XML, the entire range of Unicode characters is available to you, but in SGML, the declaration indicates what characters are valid. The distributed DocBook declaration doesn't allow a bunch of fairly common 8-bit characters.

<!DOCTYPE chapter PUBLIC "-//Davenport//DTD DocBook V3.0//EN">
<chapter><title>Test Chapter</title>
<para>
This is a paragraph in the test chapter. It is unremarkable in
every regard. This is a paragraph in the test chapter. It is
unremarkable in every regard. This is a paragraph in the test
chapter. It is unremarkable in every regard.
</para>
<para>
The DocBook declaration in use doesn't allow 8 bit characters
like �this�.
</para>
<para>
<emphasis role=bold>This</emphasis> paragraph contains
<emphasis>some <emphasis>emphasized</emphasis> text</emphasis>
and a <superscript>super</superscript>script
and a <subscript>sub</subscript>script.
</para>
<para>
This is a paragraph in the test chapter. It is unremarkable in
every regard. This is a paragraph in the test chapter. It is
unremarkable in every regard. This is a paragraph in the test
chapter. It is unremarkable in every regard.
</para>
</chapter>

[n:\dbtdg]nsgmls -sv -c \share\sgml\catalog examples\errs\badchar.sgm
m:\jade\nsgmls.exe:I: SP version "1.3.2"
m:\jade\nsgmls.exe:examples\errs\badchar.sgm:11:0:E: non SGML character number 147
m:\jade\nsgmls.exe:examples\errs\badchar.sgm:11:5:E: non SGML character number 148

In this example, the Windows code page values for curly left and right quotes have been used, but they aren't in the declared character set. Fix this by converting them to character entities.

You can also fix them by changing the declaration, but if you do that, make sure all your interchange partners are aware of, and have a copy of, the modified declaration. See Appendix F, Interchanging DocBook Documents.

Considering Other Schema Languages

Historically, DTDs were the only way to describe the valid stricture of SGML and XML documents, but that is no longer the case. At the time of this writing (January, 2001), DocBook is experimentally available in three other schema languages:

XML Schema: The schema language being defined by the W3C as the successor to DTDs for describing the structure of XML. XML Schema are likely to become a W3C Recommendation in 2001.
RELAX: RELAX, the Regular Language description for XML) is a less complex alternative to XML Schemas. The RELAX Core module is defined by ISO in ISO/IEC DTR 22250-1, Document Description and Processing Languages -- Regular Language Description for XML (RELAX) -- Part 1: RELAX Core, 2000. The RELAX Namespaces module is currently under development.
TREX: TREX, Tree Regular Expressions for XML, is another less complex alternative to XML Schemas. It is concise, powerful, and datatype neutral.

Parsing and Validation

Before we look closer at these new schema languages, there's one significant difference between DTDs and all of them that we should get out of the way: XML parsers (which may understand DTDs) build an XML information set out of a stream of characters, all of these other schema languages begin with an information set and perform validation on it.

What I mean by that is that an XML parser reads a stream of bytes:

"<" "?" "x" "m" "l" " " "v" "e" ...
"<" "!" "D" "O" "C" "T" "Y" "P" "E" " " "b" "o" "o" "k" ...
"<" "b" "o" "o" "k" " " "i" "d" "=" "'" "f" "o" "o" "'" ">"
...
"<" "/" "b" "o" "o" "k" ">"

interprets them as a stream of characters (which may change the interpretation of some sequences of bytes) and constructs some representation of the XML document. This representation is the set of all the XML information items encountered: the information set of the document. The W3C XML Core Working Group is in the process of defining what an XML Information Set contains.

The other schema languages are defined not in terms of the sequence of characters in the file but in terms of the information set of the XML document. They have to work this way because the XML Recommendation says what an XML document is and they all want to work on top of XML.

So what, you might ask? Well, it turns out that this has at least one very significant implication: there's no way for these languages to provide support for entity declarations.

An entity, like “&ora;” as a shortcut for “O'Reilly & Associates” or “é” as a mnemonic for “é”, is a feature of the character stream seen by the XML parser, it doesn't exist in the information set of valid XML documents. More importantly, this means that even if the schema language had a syntax for declaring entities, it wouldn't help the XML parser that needs to know the definitions long before the schema language processor comes into play.

There are a couple of other XML features that are impacted, though not necessarily as significantly: notations and default attribute values. One use for notations is on external entity declarations, and as we've already seen, the schema language is too late to be useful for anything entity related. Default attribute values are also problematic since you would like them to be in the information set produced by the parser so that the schema language sees them.

A Coarse Comparison of Three XML Schema Languages

FIXME: write a short synopsis of how these languages compare.

^[12] It is often the case that you can correct an error in the document in several ways. The parser suggests one possible fix, but this is not always the right fix. For example, the parser may suggest that you can correct out of context data by adding another element, when in fact it's “obvious” to human eyes that the problem is a missing end tag.

^[13] If you need to locate the entity sets, consult https://www.oasis-open.org/cover/topics.html#entities.