Take a look at this and tell me what you think.

There are two options with different benefits that I have used.  One is to build a actual file format that holds your canonical data.  Another is to use a database to act as your canonical.  Both of these have trade offs.

File based Canonical:

To make a canonical file format you will need to pick the type of file that you want all of your applications to be receiving either directly or through an adapter of some kind.  Building from scratch a flat file or XML file is the most flexible, but requires you to do a bunch of planning.  And after it is done, this format must be maintained.

Using a file based canonical does allow you a fairly easy way to find the state of a failed step, as you can look at the file and identify what is wrong.  You can also correct the data there and allow the process to continue.  You can also make copies of these files for your monitoring so that tracking your data and transactions becomes easy, and you performance metrics become rich with data.

Database based Canonical:

Sometimes people, DBAs especially, get excited when we talk about doing this.  They are visualizing one massive Canonical Database that holds all of the transactions and is accessed by all of the applications.  And there are many products that work this way internally.  But this is not the only approach.

The one database to rule all canonical, or as I like to call it “Lord of the Databases”, requires a DBA to pay attention to optimizing, backing up, and all of the other care and feeding tasks that go along with having a database that you maintain for the long haul.  This is efficient in that you can get all of your performance data from one place, and monitoring is one connection.  However, some times applications have limitations in how they talk to a database that is not theirs, and this can make implementation complex.

Another way to use a Database based Canonical model is to use disposable databases.  In the Disposable Database implementation, you create a database that is small, only contains the structures and tables for the one transaction, and gets destroyed at the end of the transaction life cycle.  Using the Disposable Database, you don’t ever have to optimize them, back them up, or any of the other care and feeding tasks that are part of the LOTDB implementation.

Comparative implementations:

I want to examine and compair the File vs DB canonical implementations in more detail in another article.  If you have another Canonical implementation that I haven’t seen, please let me know.  I would love to examine that as well.

What is Canonical Data?

Canonical Data is the data that you or your business needs to complete the job you have to do with it.  If we are talking about order processing, then the Canonical Data will comprise the data that is used to complete the order.  This is not the minimal data set, this is the whole data set.  This is not necessarily the EDI enveloping data, it may be Identifiers that relate to the TPIDs, and it may not contain control numbers and other things that may only be found in the EDI file.  But it will contain information that you need for the order, PO Number, Line Item data, and so forth.  All of this data will need to be passed from your integration, translation, and or communication software, to your order processing systems.  And it may be passed around between systems as the order is filled and processed or what ever else you are doing with it.

What this data is, becomes defined based on the requirements of your internal systems, software and data needs.  One company may take shipping data and stick it into a data base outside of the canonical data.  It gets retrieved later, but the internal applications don’t have to deal with and preserve it.  Others may include it and leverage it on some internal aspects of their order fulfilment.  But the Canonical Dataset that your business needs and uses will be defined by the internal systems so that they can do what you need them to do.

What form should it take?

This is largely a choice that is up to you and you organization.  There will probably be some formats that are native to some applications.  As long as these fill the requirements of encompassing the data that you need to process on, these are fine choices.  Also, looking at working with these internally through upgrades to 3rd party applications, and possible inclusion of new applications, a flexible format that can be updated is important.

For me, in my experience I have seen flat file and XML used with a good deal of success.  EDI may have a comprehensive set of data, but can become really clumsy when working with internal applications that need to be extended by adding data to the canonical file.  So I would discourage EDI or a canonized XML like cXML to be used at the canonical

Who sees the Canonical Documents?

Please keep your Canonical Documents for internal consumption only.  It is not that they contain secret information, (they may have confidential data)  but they are designed just for you and  your applications.  I worked with a supply chain company once that developed their internal XML format, and then published it out to a couple of trading partners.  These partners integrated to this.  Eighteen months later, when the company was doing some growing, they were stuck supporting this one-off format that was no longer used as the internal canonical.

Developers, and those supporting the applications are going to see and deal with the canonical data and data files.  Monitoring and reporting systems may represent the canonical data, but it is rare that you need to invest time training and supporting bringing people outside of this group up to a familiar speed on your canonical file format.

What have you seen done?

Does your company have a formal canonical dataset?  Or what have you seen done that works to fill the need of canonical data?