SGML, still a cutting-edge technology?		Table of contents		Indexes		Cost Justifying Your SGML Project

Gutentag Eduardo

Suttor Jeff

The Evolution of Sun's AnswerBooks

A Case Study: Report From the Trenches

Abstract:

Sun is 3 years into a large project that radically revamps its online document creation, management and delivery system, from a proprietary product to an open SGML system designed to meet all current needs, including XML and Java, as well as whatever the future might bring.

The goals are deceptively simple: take the writers' documents and, with a minimum of processing, package them and make them available to all users; support links not only within the books but also between books, independently of where the books are ultimately located; execute context driven searches; support user manipulation of contents of book collections and installation wherever they wish; enable viewing books outside of the Solaris environment - and we want, finally, for anybody to be able to publish their own books within this environment.

A decision was made, very early in the process, to migrate to an SGML-based delivery process, and, following consultations with others in the industry and our own research, we also decided to migrate our authoring environment and tools to SGML, rather than continue using a third party proprietary product, so as to avoid the painful problems associated with multiple conversions.

Three processes were put in place: conversion/authoring, production, and delivery vehicles.

This case study will concentrate on

how we arrived at our DTD

how SunSoft decided on an editing tool

how inter-book linking in the absence of standard URNs is accomplished

how books are delivered over the Net, including multiple HTML formats

print-on-demand

I18N/L10N

what is the relationship between all of the above and the so-called "production process", which controls both the automatic production of collections, and the repository/database where all meta- information is maintained.

Introduction

Sun is 3 years into a large project that radically revamps its online document creation, management and delivery system, from a proprietary product to an Sun is 3 years into a large project that radically revamps its online document creation, management and delivery system, from a proprietary product to an SGML system designed to meet all current needs, including XML and Java, as well as whatever the future might bring.

A decision was made, very early in the process, to migrate to an A decision was made, very early in the process, to migrate to an SGML-based delivery process, and, following consultations with others in the industry and our own research, we also decided to migrate our authoring environment and tools to SGML-based delivery process, and, following consultations with others in the industry and our own research, we also decided to migrate our authoring environment and tools to SGML, rather than continue using a third party proprietary product, so as to avoid the painful problems associated with multiple conversions.

Three processes were put in place: conversion/authoring, production, and delivery vehicles.

This case study will concentrate on

how we arrived at our DTD
how SunSoft decided on an editing tool
how inter-book linking in the absence of standard URNs is accomplished
how books are delivered over the Net, including multiple HTML formats, print-on-demand, I18N, and
what is the relationship between all of the above and the so-called "production process", which controls both the automatic production of collections, and the repository/database where all meta-information is maintained.

The DTD

We all knew from the beginning that in order to succeed in our efforts we had to obtain management support from the earliest stages. And one of the key factors for obtaining this support was demonstrating that what we wanted was not simply to replace one method with another, but to replace one method with a standards-based one.

Of course, choosing Of course, choosing SGML as the base for the new system went a long ways in the right direction, but it was not enough, as it did not ensure document interchange, except at the theoretical level. Fortunately, by that time we were already involved in the Docbook effort through our participation in the Davenport group, so the solution to that question was a no-brainer.

On the other hand, it soon become evident that we could not use the Docbook DTD "as is": Docbook is a generic DTD, meant to support much more than what we really needed, certainly much more than what our writers could cope with in terms of tag availability.

So two things became clear very soon:

We had to subset the Docbook DTD

This was not a job for one person

With some help from our friends in managment we put together a team to deal with the DTD. This so called "Tag Team" was composed of writers and editors from the various writing organizations within Sun's planets.

Practically no one in the team knew the first thing about SGML, so the first order of business was to teach them how to interpret the DTD's content models. The team met for two hours every week, and everybody in it received special dispensation from their managers to participate and to work on whatever issues arose above and beyond those two weekly hours. Special monetary incentives were also put in place — that is, they got small bonuses.

Once the members of the Tag Team could talk intelligently about the DTD, we concentrated on deciding what we wanted to do with it. And it soon became clear that what we had to do was constrain it as much as possible in some areas, and eliminate freedom of choice as much as possible in other areas.

Many elements were dropped, both block (e.g. Article) and inline (e.g. wordasword). The operative questions we asked ourselves while examining all the elements were: "Do we have any use for this?" and "Does anybody understand what this is supposed to mean"? If no one could come up with a reasonable use for an element, or an explanation that made immediate obvious sense, the element was dropped.

Many content models were modified to accomodate our current practices and editorial style. Being able to enforce what until now had only been toothless editorial policies was seen by many as one of the main benefits of using SGML.

And finally, almost all attributes whose value was defined as CDATA were modified to have predefined values. This, again, caused content model changes, since attributes that are ubiquitous in Docbook (such as Role) had to be redefined on an element-by-element basis, and if no predefined values could be found for an attribute in the context of a particular element, the attribute was dropped.

At the end of the year we had what became known as Solbook DTD version 1.0, which is a true subset of the Docbook DTD, designed for authoring, but also designed to ensure that whatever documents are written with it are readable as Docbook 2.4.1 documents.

Choosing an authoring tool

Another big factor in our ability to "sell" SGML to managment was the idea of breaking the chains of tool dependency. We would not be dependent any more on one particular tool, and if the tool supplier went belly up we would not end up with unsupported legacy documentation.

One other important issue was the fact that different writing organizations within Sun would be able to choose their own SGML tools, independently of each other, following their own criteria, without having to worry (too much) about document interchange.

So, without having to consult with any other organization within Sun, we at SunSoft proceeded with the task of choosing an authoring tool, while the DTD was still being worked on.

At the time there were only three serious contenders within the US. What we wanted to know, really, was which of the tools offered the most in terms of writer comfort, in ease of customization and in maintainability. To test them in terms of writer comfort, we set up a Usability Study.

The participants in the Usability Study were writers who had mostly never ever seen an SGML tool, and who had no idea what SGML was, other than a buzz word. For the purpose of the study, all writers tested the three tools, but after testing one they were promoted from "naive" user to "expert" user.

The participants were given some simple tasks to perform with a given tool (such as inserting elements, deleting elements, finding elements, modifying text, cutting and pasting, etc.); in most cases the tasks were performed in front of a terminal. In some cases the tasks were performed in a mock environment, with printed menus and mockups.

All the participants' actions and reactions were carefully videotaped and recorded, as were the comments of the person conducting the tests.

One of the most surprising results of the tests was that although one of the tools was very similar to what they were currently using, performing tasks in that tool was not as overwhelmingly easier for the writers as one would have expected, and in most cases the learning curve for the other tools was so fast that the initial differences between the tools leveled out very fast.

In other words, once a user achieved familiarity with the tool, the differences seemed to disappear. The following table shows that although one of the tools had a higher subjective satisfaction rating, the overall difference was minimal:

Tool	Novice Performance	Expert Performance	Subjective Satisfaction	Overall Score
A	112	115	170	124
B	105	100	136	107
C	100	115	100	109
Tool A's Overall Advantage	14%

Three further observations reinforced these results:

Novice task time decreased with repetitions of the same tasks, thus the order of use was important

Tool A had an initial time advantage, but that was quickly lost (it was calculated that the initial advantage might be equivalent to about 1/2 day training)

Experts completed tasks significantly faster than novices.

This finding was crucial: with this information in hand we could then pay attention to other factors, such as the level of commitment to SGML that the tool provider had, the stability of the product, its history in the marketplace, its ease of customization and, of course, price.

Finally, one other factor came into play, that we had not considered initially but which proved to be the clinching factor: what plans, if any, did the tool companies have for multi-byte authoring? Only one company had anything remotely resembling concrete plans, and that was the company that SunSoft chose in the end.

By now you may be thinking that we never did anything without looking over our shoulders to see where managment was at any give time. And if you're thinking that, you're right. We were determined not to let happen to this project what had happened in many other occassions, and in many other companies, to incipient projects: we did not want the project to be killed through managment indifference or lack of understanding.

We were very lucky in that our managment was with us from day one.

It is not easy for writers used to 100% WYSIWYG tools to do the necessary brain rotation needed to become effective SGML writers. No matter what tool is given to them, if there is no managment support for the project, the project will die.

Authoring Links

Once we chose the DTD and we chose the authoring tool we declared ourselves open for business.

But while the conversion of our legacy documents proceeded, and our writers attended classes and broke their first teeth against this new beast, in the background we were trying to figure out how to implement links.

By virtue of using a subset of the DocBook DTD we had four essential linking mechanisms:

```
xref
```
```
link
```
```
ulink
```
```
olink
```

xref

, of course, was assigned to autogenerated linking duty within a book,

link

was assigned for use as authored linking, and

ulink

was put aside for URL linking.

That left olink for the purpose of inter book linking. Of course the issue of doing any kind of effective linking between two discrete documents is not trivial, otherwise we would already have an URN mechanism in place. Unfortunately, there was no URN mechanism then, nor is there one yet. So this is what we came up with in order to solve the problem that writers would not know the location of a book being linked to at authoring time, and the system would not know the location of a book at link resolution time.

In the original AnswerBook architecture, every book has, associated with it, an abstract name that uniquely identifies that book through its lifetime of versions and revisions. For example, a book dealing with the system administration of the Solaris operating system would have the abstract name SYSADMIN associated with it, whether it dealt with version 2.1 or with version 2.6 of the operating system.

It was, therefore, quite an easy conceptual leap to extend this model into the SGML world, and associate an FPI to each book, where the FPI would be composed of the regular FPI fields, for example:

–//Sun::SunSoft//DOCUMENT SYSADMIN Version 1.0//EN

Now that's the easy part. The hard part is: How do you link to it, and how do you link to a region within it?

First you have to declare an ENTITY in your document, of the form:

<!ENTITY SYSADMIN PUBLIC "–//Sun::SunSoft//DOCUMENT SYSADMIN Version 1.0//EN" NDATA sgml

Then you write <olink targetdocent="SYSADMIN" type="V-ONLY" localinfo="somevalidID"> (where localinfo's value is the ID of the region you want to link directly to, and where type specifies whether you want to link only to the version specified in the entity, or whether other choices are applicable if that version is not available locally.

And then you start asking yourself: "How am I ever going to explain this to my writers?"

So you don't. Instead you customize your tools so that it all happens automagically. And while you're at it, you also make xref and link authoring part of the same magical process.

After some tinkering, then, this is how the current Link Editor panel looks:

All the writer has to do is double click on one of the titles being shown, and then double click where she wants the xref to be inserted

The "Switch Modes" Menu allows the writer to switch from xrefs to links to ulinks and, finally, to olinks:

If the writer double clicks on any of the collections presented at this time, it expands to show the books that compose it:

Double clicking on a book name and then double clicking at the target insertion point creates the correct olink; double clicking on the square box beside the collection name and then double clicking at the target insertion point creates a title citation; double clicking on the collection name collapses the collection.

The next generation of the link editor will allow writers to expand books to show their contents and link to internal targets in the books.

And how are those olinks resolved at the time the user clicks on them? Good question. We'll answer that a bit later.

Using SGML On The Internet

If I see so far, it is because I stand on the shoulders of giants.

If I can do so much with information, it is because I stand on the shoulders of SGML.

Many of the interesting techniques discussed are only possible because we are starting from a base of highly structured information that is marked up in an explicit and well understood way. Although it is possible to start with legacy information, proprietary information, almost-kind-of structured information and add value to it, you can only pull so much magic out of a hat. On the other hand, if you start with SGML, you can do today's Internet better that most, and tomorrow's far better. While SGML users update their style sheets to rapidly take advantage of new Internet technologies, users with repositories of HTML + proprietary vendor elements + embedded JavaScript will be forced to throw it all away and start over.

Good SGML == Good Internet Delivery

I am starting to think that HTML is good enough. (clueless comment from an unnamed SGML consulting CEO)

It is easy to create valid HTML 3.2 documents from SGML. Why treat HTML as SGML, i.e. valid to an HTML 3.2 DTD, structure v. presentation markup? Universal access is a key reason.

Valid HTML has a far better chance of being processed and displayed correctly by the widest variety of browsers. Note that many "browsers" are hidden in word processing, email and many productivity applications. HTML engines everywhere and devices like Web-TV and Java cell phones are only the beginning. Universal access includes not only planning for heterogeneous software/hardware applications, but most importantly, heterogeneous user populations. Valid structure based HTML supports meaningful user access to the information with large print, voice synthesis, Braille and output to other adaptive technologies designed for the specific needs of the individual user. It is against the very tenants of SGML to tell the user

you must use this specific application/release/patch/platform because I've embeddedSPAM (specific proprietary application markup) into the information stream

you cannot use large print, voice synthesis or Braille in a meaningful way because I've used presentation based markup, e.g. large print is unusable because <Br> v. <P> was used for "formatting"

you cannot post-process the information in a meaningful way, e.g. a dynamic table of contents will show what's been marked up to be big and bold v. structural headings

Now, let's look at some specific HTML issues from an SGML perspective. Linking, style sheet based transformations and insuring that our information lives into the future.

FPI —> SOI on the WWW ?!?

No matter where you go, there you are. (user comment on using the WWW and following links)

AnswerBook is being developed and deployed in the most interesting of times. During its lifecycle the technologies of distributed network information systems will evolve in unknowable ways. In order to thrive in this environment, AnswerBook's hyperlinking model will be based on open standards that are platform and application independent. An infocentric approach will place the most value on the future of the information itself and interface to current platforms and applications by down-transformations, style sheets/templates and external configuration and mapping files. This allows the information to be quickly deployed on new platforms and applications without having to diddle the data. In brief, interdocument hyperlinks will use an abstract persistent public name that will be resolved through use of external configuration and processing. This approach is in stark contrast to the current practice of hard coding hyperlinks as a pointer to a specific host, port, path and other system dependent information.

So why is the world's largest and most successful hyperlinking system, WWW, using hard coded system specific information? Because it's easy to hack the initial implementation. Now imagine infobases in the large. Will host names change? Will location paths change? Of course they will! Many users are already suffering as they attempt to navigate through cobwebs of dead links. Many information providers are already suffering as they attempt to maintain hyperlinks and expend more resources on maintenance than creation of new content and applications.

System information independent naming and resolution of hyperlinks will solve these problems and enable resource efficient adoption of new technologies.

Objectives

Installation and modification of Sun publications in a worldwide multiserver environment.

Transparent resolution of links between Sun publications in a worldwide multiserver environment.

Transparent navigation of Sun publications regardless of location.

Preservation of links between documents in an environment where physical copies are frequently being created, moved, or destroyed.

Support for bookmarks that remain valid regardless of a book's physical location.

Support for the future creation of advanced user-defined abstract data management structures such as topic maps and directed navigation paths.

Reasonable performance in navigating and linking distributed collections.

Support for the specification of user/administrator preference among multiple copies of the same publication in order to optimize performance in a Web environment.

The notation or data format of a document -- AB1, SGML, vendor2X, vendor3X, etc. -- is considered to be a characteristic of a particular physical realization of the document and is therefore not specified in an FPI but rather in some external binding of an FPI with a particular copy.

Standards-based solutions will provide maximum flexibility in meeting future needs. In this paper, the SGML Formal Public Identifier (FPI) is suggested as a standards-based method for the implementation of persistent names.

User Scenarios

I decide to move all the Developer books to a server in another building. After the move, everything (including bookmarks, etc.) still works.

I am reading in French, and I want to see French books if I follow a link. If a French version isn't available but a Spanish version is, then I want to see the Spanish version.

I have an on-line ENP book that wants to link to the latest configuration instructions for setting up modems in Solaris. For me, "latest" means Solaris 2.5, because I have not upgraded to 2.6. Therefore, I want to see the latest version of this information for me... in a Solaris 2.5 book. Specifically, I don't want to have the system try to link to a newer version of the book on the net.

I have SunSolve collections installed locally for speed, but occasionally I want to go out to the 'net and get the very latest info (which is updated nightly). There is a graceful way to do this.

SOCats

The standard way to associate FPIs with physical locations (either file paths or URLs) is thesocat (SGML Open Catalog) . In AnswerBook, the data needed to locate every publication available to an AnswerBook server is stored in an extendedsocat that may contain entries for local resources, remote resources and delegated resources.

The localsocat entries contain FPI/location data for all of the books actually installed on the server and is typically created or updated automatically (or under manual control) by an administration program when books are installed. In future implementations, the location will be any formal system identifier, not just a file path or URL.

The remotesocat entries contain FPI/location data for known books installed on remote servers and is typically created or updated automatically via caching (or under manual control) by an administration configuration program. In future implementations, the location will be any formal system identifier, not just a URL.

Thesocat can also contain DELEGATE directives. Each DELEGATE entry points to the localsocat of another server where Sun documentation might be installed, with the last DELEGATE entry pointing to the localsocat of Sun's own documentation server. DELEGATE entries are added by the system administrator, and by implication their order indicates an order of preference among alternative repositories, presumably based on response time, trustworthiness, cost, or other criteria.

When the virtualsocat is built for a server, the combination of local and remote entries on that server will contain PUBLIC entries for all the books accessible in the network environment defined by the system administrator's choice of entries and user driven caching, and if a book exists on more than one server in that environment, the copy that appears first in a server'ssocat will be the one in the most desirable location as determined by the order of the entries in that server'ssocat . Any duplicate entries for a given FPI are purposely retained as a way to provide maximum immunity from link failure.

In the typical case where the server is connected to the Internet, the presence of the default DELEGATE entry for Sun's own server at the bottom of thesocat guarantees that ultimately, an FPI/location entry can be resolved for every book published by Sun. Thus, under ordinary circumstances, a link to, or navigational access of, any particular book will find an FPI match in thesocat either in the local or remote entries; in the case of Sun publications it should rarely be necessary for the server to fall through to the entries for DELEGATE.

Late Binding of the FPI

The longer a link can be kept as an FPI, the more value it has. Once an FPI has been resolved to an SOI, all of the benefits of using FPIs vanish. Imagine that a user bookmarks a link or emails a link to a friend. The link must be expressed as an FPI in the bookmark or in the content of the email. Embedding FPIs into URLs is the solution that was chosen. In Sun's implementation, the FPI is added to the URL as a parameter. This decision was driven by the underlying WWW toolkit and the philosophy that it was good programming practice to treat the resolution mechanism as black-box function that was passed an FPI and returned an SOI. This has worked very well and was reasonable to implement. At is most basic, FPI —> SOI resolution is a glorified table lookup with a few simple precedence rules. One draw back of this approach is that some browsers, proxies and servers cannot effectively cache URLs that contain parameters. Unfortunately this behavior appears to be vendor/application/version/platform depended. It is strongly recommended that implementors test their local software to determine its behavior in this regard. FPIs contain characters that must be URL-encoded in order for it to be interpreted correctly. e.g. spaces must be represented as "%20".

An abstract example of a URL that uses an FPI:

http://server:port/<path>/<FPI2SOI_resolver>;FPI=<URL-encoded_FPI>

The "stuck out there" Problem

One complication of the mechanism just described arises when a book has been resolved to a server distant from the user's local network (our Sun server, for example) because the book does not exist in the local installation, and then a link is executed from that remotely installed book to another book that, in fact, does happen to be installed in the local network. The server at this point is the remote server, and so the book will be resolved according to the remote server'ssocat rather than a localsocat . Once "out there" at the remote server, the user will continue to be served books from the less desirable remote location even when they are available locally. There are probably a number of solutions to this problem, but the easiest ones all assume that the browser is smarter than current browsers actually are. The trick is how to solve this within the limitations of current stateless HTML browser technology. One possible (though not particularly graceful) approach is to use cookie(s) that refer all links back to the original server. The key concept here is that of a "home server" -- that is, the default doc server that a user typically accesses at the beginning of a AnswerBook session and the one that presumably carries within itssocat the information about preferred servers from which the "homesocat " has been generated.

Style Sheets

Sure, we can do that, it's a mere style sheet issue (the favorite SGML geek reply to all questions).

Style sheets are such a good idea that everyone should have one (the favorite vendor reply to all implementations).

What do you mean,YASSL (Yet Another Style Sheet Language) (the favorite management reply when the number of different style sheets is equal to the number of different applications).

Dynamic style sheet based transformation of SGML to HTML is at the heart of on-line delivery of AnswerBooks. Although on-line AnswerBooks are the focus of this paper, they are only one output format for our single SGML source multiple output via style sheet based transformation strategy. Although this approach is successful, it can only truly achieve its potential with a common style language such as DSSSL. There are already three different style sheet languages being used by AnswerBook. This is an unnecessarily complex and expensive situation. If you are a user, it is recommend that you put DSSSL support in all of your requests for information, proposals and contracts. If you are a vendor and are not actively developing DSSSL, do not expect much business once the first DSSSL enabled application in your product category appears. When two or more applications in different product categories supporting DSSSL appear, the economics alone will drive the market to DSSSL products. To be more blunt, this should be the last release ofYASSL products. DSSSL or die.

Structural Based Presentation and Navigation

Most of today's HTML is living in a flat information space with a possible attempt at hierarchy to be provided by the physical file system. An application may strive to present this information space to the user in a rational fashion through the hand crafting of links, cgi-bin trickery or other limited error prone methods. When you start with SGML, it is relatively easy to use style sheets to

dynamically present a meaningful table of contents

dynamically present meaningfully sized fragments, chunks, of information

dynamically present navigation aids that really understand what "next" and "previous" unit of information mean

dynamically present search results based on the context of the query and the context of the results

The word "dynamically" was intentionally used multiple times. Just as late binding is useful in preserving the value of FPI —> SOI resolutions, it is useful to use late binding of the style sheet to the SGML source. AnswerBook dynamically generates every single page of HTML it serves. This supports conditional style sheets that can present alternate views driven by user defined preferences.

Internationalization / Localization (I18N/L10N)

This is theWORLD Wide Web. If you are embedding the equivalent of autogenerated text or user interface components directly into the HTML, what will happen when an HTML client from a different locale accesses it? Just as style sheets have been used to generate text for print output, the same techniques can be applied to dynamic HTML from SGML delivery. AnswerBook determines the locale of the client on a per request basis. All generated text and user interface components are then transparently added to the HTML in the locale of the end user. This is of increasing importance as the WWW literally unites the world into a global information space. Imagine what non-SGML + style sheet users have to resort to. Multiple serves with multiple source documents creating a information management quagmire.

Print on Demand

Print will never go way, period. So how do most HTML applications force you get hard-copy? Click on menu bar, click on print this page, click to follow next link, repeat too many times. What do you end up with? A bunch of pages v. a coherent whole. And what is the quality of the print style? Now image a system with SGML at its roots. Since it knows the structure of the document, it can allow the user to selectively print any portion of the document up to and including the entire document with a single interaction. AnswerBook uses this technique to dynamically apply a style sheet specifically designed for print quality. The result of the SGML + style sheet is Postscript™ being returned to the end user's client for printing, saving to disk, viewing or any further user manipulation or purpose.

Future

Noooooooo fuuuuuuture! Noooooooo fuuuuuuture! The early Sid Vicious describing closed, proprietary or limited systems of information exchange.

If native HTML is being used as a repository format, there is no future for that information.

If SGML is being used as a repository format, there is an unlimited future for that information.

Nobody predicted the WWW.

Nobody can predict the future anymore.

Most people's information will need to live past the next information technology shift.

Any questions? :) Or why would anyone that cared about information and its usage do anything less than enable the future by using SGML?

SGML and Behavior (Java as a Mere Style Sheet Issue)

Parsing an SGML document yields a dynamic custom Java Applet/Application.

At the time a document instance is served, custom JAVA applet(s) could be delivered. The applet would contain the specific objects needed to provide the behaviors and methods for the specific data content types, nothing more, nothing less. The document itself would drive the assembly of a totally tight dynamic applet that would be the ultimate fit with the content.

content is king/queen/court jester

intimate user interactions with deep structure

maintenance/management

functionality beyond any generic tool, no trade-offs

Accessibility (ICADD) as an example.

An idea is to create a user.access.ICADD class that could be unleashed on the Internet to transform any structured content into Braille, large print, and/or voice synthesis. This would be a dramatic event for accessibility- a cross platform network based applet that could turn most any WWW page into living information for the disabled community. The real challenge here would be to create tolerant parsing algorithms to deal with the diverse content that people are creating.

XML

When AnswerBook transforms SGML to HTML for delivery, it is aDOWN transformation. Most but not all of the document's structure must be discarded to fit into HTML. Now image passing all of that structure through to the end user. We are actively supporting the delivery of XML today and here come the applications.

Do you think that Microsoft and Netscape own the WWW market? Think again. XML is your opportunity to create the next Mosaic.

XML is!


SGML, still a cutting-edge technology?		Table of contents		Indexes		Cost Justifying Your SGML Project