Daniel calls OpenDocument "our best chance to fight vendor lock-in associated with proprietary formats." It's currently being considered by the EU Commission as a candidate for an official format. Note that you can also download this article in various formats, including, naturally, OpenDocument, from links at the end of the article. In some browsers, you may need to Save As, instead of just clicking on the links.

Daniel Carrera is a mathematics PhD student at the University of Maryland, as well as an active OpenOffice.org volunteer. He serves in the OOo community council as the "community representative". Daniel also facilitates the OOoAuthors project, where most English documentation is developed. He will be speaking at the OOoRegiCon in San Diego on Feb 9, the day before and leading into the " Desktop Summit 2005", a conference about the present and future of open source and desktop Linux technologies being held in San Diego, CA February 9-11. He is speaking opening day on the subject, "INREACH, Building an Active Community."

The complete list of speakers is here, and I must say, it sounds like fun. Mitch Kapor and Doc Searls are two of the keynote speakers at the Desktop conference. Sun's Simon Phipps kicks off OOoRegicon, and I've heard he is an excellent speaker. You can find out more here. It's limited seating, and the page says the price goes up after February 5.

*******************************

The Future is Open: What OpenDocument Is and Why You Should Care,
~ by Daniel Carrera

• What is OpenDocument?
  • Executive summary
  • Timeline
• The European Union and the Valoris report
  • Selection criteria
  • Finalists
  • OpenDocument vs MS XML
• Reactions to the report
  • Microsoft responds to the Valoris report
  • Sun responds to the Valoris report
  • The EU TAC makes recommendations
  • Microsoft responds to the TAC recommendations
  • Sun responds to the TAC recommendations
  • IBM responds to the TAC recommendations
• Where we are, where we need to go
  • This is a battle we can win
  • What OpenDocument needs to succeed
  • What you can do to help
• Resources

What Is OpenDocument?

Executive summary

Quiz: Complete the sentence,"OpenDocument is ..."

(a) An open, XML-based file format.
(b) An open standard, supported by the OASIS and ISO standards groups.
(c)The default file format for the upcoming OpenOffice.org 2.0 and KOffice 1.4.
(d) A top prospect for an official format for the European Commission.
(e) Our best chance to fight vendor lock-in associated with proprietary formats.
(f) All of the above.

The correct answer is (f) All of the above.

Timeline

This article links to reports written over a period of several months. To make things easier to follow (in particular, the name of the format), I include an approximate timeline.

1. OpenOffice.org creates an open file format called "OpenOffice.org 1.0 format."

2. The European Union commissions Valoris to report on open file formats.

3. The OpenOffice.org 1.0 format is submitted for OASIS standardization. KDE and Corel join the OASIS Technical Committee and expand the format to cover a wider range of applications.

4. The new OASIS format is called "Open Office XML". OpenOffice.org and KOffice both commit to making the format their primary/native format.

5. The Valoris report is published. Microsoft and Sun respond to it. The European Union TAC makes recommendations.

6. The format is submitted for ISO standardization and changes its name to the OpenDocument format.

In order to avoid promoting an outdated name, I will always refer to the format as OpenDocument.

The European Union and the Valoris report

The key player here is the European Union's Telematics between Administrations Committee (TAC). The TAC hired the Valoris consulting group to explore the possibility of using Open Standards to create a more competitive marketplace.

The Valoris report was well researched and very thorough. They recommended the adoption of an open XML format backed by a recognized open standards group.

The Valoris report is a very detailed and most compelling document describing how open XML technologies are impacting government information systems and requirements policies.

It's difficult to summarize a 78-page document. I encourage you to read it yourself [PDF]. But over all, it is well-researched and rather encouraging of open source, OpenDocument, and EU hopes for an open marketplace.

The Valoris group closely monitored the development of the OpenDocument format, and by November 2004 every one of the concerns and issues they raised were directly addressed by OASIS TC. As a case in how governments and organizations go about setting information system requirements policy, the Valoris effort provides an unparalleled insight.

For the report, Valoris compiled an exhaustive list of existing file formats. These were compared against a series of requirements. Only two formats performed well, and these were selected for further analysis.

Being familiar with these criteria is important for us. It teaches us what a government requires before it will adopt an open XML format. I divide these into three groups:

1. Open: At this stage of analysis Valoris only looked for openness in the sense that a public, royalty-free specification is available. For example, PDF and MS XML both met this requirement.

2. Non-binary: Binary formats get in the way of neutrality. A format that depends on Windows components will be hard to support on GNU/Linux and Mac OS.

3. Cross-platform: An obvious requirement for neutrality.

1. Preserve format fidelity: This refers to both presentation and structure. For many applications, format fidelity is an absolute imperative.

2. Modifiable: This excludes formats like PDF.

3. Support current word processor features: An open format is no use if it can't represent your data. This list included Unicode support, bi-directional (Hebrew), and scripting, among others. Abiword and KOffice failed this requirement.

4. Support emerging requirements: Digital signatures, access rights, version control, etc. Almost every format failed this requirement.

This sounds odd. Why is this important? Shouldn't we pick the best format regardless?

The answer is simple: Have you heard of the ODA? I didn't think so.

In the 80's Europe made an attempt to define an open standard format called Open Document Architecture (ODA). It received a lot of support from European institutions including the European Commission. It was also an ECMA standard and an ISO standard.

ODA failed miserably.

ODA had no backing from the industry. The format was complex, and the companies preferred to support more pragmatic standards such as SGML and RTF.

Conclusion: Industry support matters. This is why the W3C never accepts a standard unless it has at least one working implementation.

Note: "widely adopted" does not mean "dominant". It means that there is enough industry adoption to sustain the format. For example, OpenDocument meets this requirement. Unlike the ODA effort, OpenDocument enjoys the support of two office suites as well as major players like Sun, IBM, HP, Novell and even Adobe.

No format met all the criteria perfectly well. But two formats stood out above the rest:

• OpenDocument - main failing: lacked some support for emerging requirements.
• Microsoft XML - main failing: medium fidelity, unsure about cross-platform status.

The only apparent advantage of MS XML over the OpenDocument format was the presence of custom-designed schemas.On the other hand, OpenDocument had several advantages:

• More open: No legal constraints, and support from OASIS.
• Reuse of existing open standards when possible (SVG, Dublin Core, MathML, etc).
• Higher format fidelity.
• Friendly to XSLT and other XML-based tools.

Reactions To The Report

Microsoft's response [PDF] essentially comes down to trying to convince the reader that custom designed schemas are really really important. This isn't surprising since it seems to be the only card they could play.

Sun's response [PDF] expresses several concerns about Microsoft's approach to XML. I recommend reading the full response for details, but here are a few:

• MS XML doesn't cover all of MS Office (for example, it doesn't cover PowerPoint).

• MS XML doesn't support some of the advanced MS Office features.

• MS XML can contain binary objects that depend on MS Office and Windows (e.g. OLE and VBA) and those lack complete documentation.

• MS did not commit to make future changes to MS XML available to the public, only the current one.

After getting responses, the EU TAC wrote their final recommendations. Those too are very encouraging. They stood firm on the importance of file formats:

"...the public sector should avoid any format that does not safeguard equal opportunities to market actors to implement format-processing applications, especially where this might impose product selection on the side of citizens or businesses..."

Their response to Microsoft's argument on custom-defined schemas was interesting. Instead of arguing whether the feature is really valuable, they chose to challenge OASIS to add custom-defined schemas to the format to match Microsoft's argument. The TAC also had some suggestions for Microsoft, finding that the MS XML and MS XML Reference License were simply not open enough.

Recommendations for OASIS:

• Add custom-defined schemas to the format.
• Submit the format to ISO for a more official seal of approval.

Recommendations for Microsoft:

• A public commitment to keeping future versions of MS XML public.
• Submit the format to an international standards body of their choice.
• Remove non-XML components from the format.

Recommendations for other industry players:

• Participate in the OpenDocument standardization process to encourage wider industry consensus around the format.
• Include filters to support both OASIS OpenDocument and MS XML.
• Provide tools to help the public sector migrate its documents to XML formats.

Recommendations for the general public:

• Provide your documents in multiple formats, or alternatively, in an open format with industry consensus and adoption.

Microsoft's response [PDF] essentially comes down to:

• They agree to publish future versions of MS XML under non-discriminatory terms.

• They say they will "vigorously" work on documenting the non-XML elements of the format, though they disagree that they should exclude all non-XML elements. They claim that OpenDocument also has non-XML elements (such as images) but fail to point out that in OpenDocument those are in a separate directory and in MS XML they are embedded throughout the XML tags.

• They also say that supporting various formats is important, but fail to say that they'll support OpenDocument.

• They insist that their licensing scheme is just fine, and that "royalty-free licensing programs have a role to play alongside formal standards". Translation: No, they won't submit it to a standards body.

Sun's response [PDF] reiterates the importance of formats that are truly open, in every sense. They like the idea of ISO support and are confident that the format will become an ISO standard.

IBM's response [PDF] was also positive. There was a tiny bit that concerned me:

... we believe [that standard] technologies should be published without restriction (other than reasonable royalties for essential patents)...

Well, I don't really know much about this, so I'll let other people decide if the stuff in brackets is significant. But other than that, IBM informs the EU that:

• They will join the the OASIS Technical Committee.
• They already offer products (Workplace) that conform with OpenDocument.

This last bit isn't surprising, since Workplace is based on OpenOffice.org code, but it's a good reminder.

Where We Are, Where We Need To Go

The OpenDocument format is arguably the single most important step for the FOSS movement after software patents. It is unique in that:

• It cuts right to the core of the fight: Vendor lock-in.

• This is a battle we can win.

The OpenDocument format is in a unique position because it has a real chance of succeeding:

• Technical merit: With the inclusion of custom schemas, OpenDocument can meet and exceed every technical hurdle that can be thrown at it.

• Adoption: OpenOffice.org, KOffice, IBM Workplace, StarOffice. OpenDocument is not a dream. It is a real format with enough support to present a real alternative.

  OpenOffice.org is cross platform, easy to download and install, and is preparing to ship version 2.0. Sun, IBM, HP, Red Hat, Novell,and even Adobe were able to announce full support of the EU open XML standards requirements based on the inclusion of OpenOffice.org components in their products and services. That both OpenOffice.org and KOffice are open source with open component frameworks enables the entire marketplace of vendors to easily comply if they choose to do so.

• Open standard: OpenDocument is more than an open format, it is a open standard . That is, it is backed by standards groups, ISO and OASIS. It is not controlled by any company or product. Not by Sun or IBM. Not by OpenOffice.org or KDE. Is Microsoft willing to match that?

• Pushing OpenDocument as a Standard : The European Union is pushing for a policy of open standards. So is the state of Massachusetts, USA. The time is right to push for OpenDocument as the standard for government documents. People are listening.

What OpenDocument Needs To Succeed

The Valoris report makes it clear that any open format that is to succeed as an official standard for government documents needs four things:

• Technical merit: Most open formats that were evaluated did not make it to the final round because they lacked important features. Any technical shortcoming, no matter how theoretical, gives Microsoft a lobbying opportunity.

• Adoption: No adoption, no success. Learn from ODA.

• Open standard: Why is this so important? Why not just any open format like the old OpenOffice.org 1.0 format or KOffice? Because Microsoft would argue unfair treatment. Yes, I know, ha ha. But still, Microsoft could effectively lobby against any format controlled by one application supplier.

  This is why OpenOffice.org and KOffice are both switching to OpenDocument as their primary format (in versions 2.0 and 1.4 respectively).

• The Fight for Open Standards, OpenDocument As a Standard: The fight for open standards will not be easy. It will take will power from politicians, and the public who elects them, to come through. Open XML technologies are the future of collaborative computing, but the proprietary efforts to compromise their implementations, or lock them up with insidious patents and restrictive licenses, continues.

What You Can Do To Help

Keep your focus on the four items above. Let's go through each and see what we need.

Gary Edwards, from the OASIS TC, reviewed this article for technical accuracy. Jean Hollis Weber edited the final draft. Any errors or omissions are entirely my own.

Read the Valoris report, along with TAC recommendations and all responses.

This article is licensed under the same Creative Commons License as PJ's articles. Daniel Carrera can be reached at dcarrera at openoffice.org

You can download this article in OpenDocument format as well as OpenOffice.org 1.0 and PDF.