If you have used a standards editor, you probably know what an SEF file is, or have used it.  Some applications and EAI even use SEF files as part of their document creation and validation processes.  It becomes useful to describe briefly what SEF files look like, and what type of information they contain.

What is in there?

If you are familiar with EDI, and have cracked open an SEF file with your standard text editor of choice, you will already know what I am about to say.  The contents of the SEF file don’t look that complex to someone who is used to and familiar with how various EDI files look.  For those that are not so brave, I will explain this painlessly.

First, the SEF file is really just a text file.  It can be edited by hand, but I don’t really recommend it.  (not because you can’t, but because it is tedious.)

Second, it contains its data in sections.  There is a section for doc types, segments, elements, and encoded data.  There is also a few housekeeping sections like version and name etc.

What is an SEF file for?

SEF stands for Standards Exchange Format.  This is literally a file format that was designed to contain information about EDI standards.  It was created so that a computer application could understand an EDI file.  It is used by some application like Standard Repositories and Standards Editors to allow us humans to deal with a standard like it was a text document.  While at the same time keeping a repository that can be used by an integration application to form and validate an EDI file.

If you are familiar with the concept of metadata then an SEF file is the EDI standard’s metadata.  If you are not familiar with metadata, then skip this part.

Do I need to know about SEF files and EDI?

Nope.  SEF is only useful if you are exchanging standards using the format.  If your standard and usage is in the form of a PDF or a spreadsheet, that is fine.  Many people do just that.

For me, I need to know this because I am trying to build some Standard Repository tools, and want to use the SEF format to manage them.

And for those that want a deeper look, I will be following this article with some deeper and more detailed articles dealing with the internals of the SEF file.

EDI Specifications

In the larger sense, the EDI specification is the set of rules that define each document type, the segments they contain, and the size and type of data in the elements.  When we talk about the EDI Standard Specification, we are talking about the whole set of valid EDI document types.  If something is valid EDI, then it complies with the EDI Standard Specification.  However, this large, all encompassing specification is not useful in coordinating the exchange of documents between two trading partners.

What goes in an EDI Usage Specification?

So we take EDI Standard Specification and we reduce it.  We remove the unused document types and the unused segments within the documents we will use.  And we even remove the unused elements and encoded values within the segments we will use.  At this point, we have created an EDI Usage Specification.  We can also add to the specification by including values that we need, like designating the an optional value or ID as required and specifying what type is should be.

How is an EDI specification Used?

Now that we have a usage specification we can use it to do to basic things.  First, it is the source of information for us to set-up our integration for our EDI document exchange.  And Second it is the format that we validate our EDI documents with to determine a valid transaction.

EDI Usage Specifications can be a source of data and integration documentation.  This can become extremely valuable when the choices and information about how your EDI interface works is recorded into the notes of the Usage Specification.

Beyond a visual validation of looking at the EDI file and comparing it with the specification, many times a specification can be found in the form of an SEF file.  The SEF file can be used in an EDI validation application.  This allows a potentially large and unwieldy EDI file to be scanned for compliance and accuracy.

There may be other uses, but these are the main two, integration data repository, tool for EDI validation.

EDI Specification Examples:

To walk through how a usage specification works, and is used, check out this review of BookNet Canada’s specification.

Then you can take a look at the EDI specifications published by Macy’sNet that they keep online to assist their trading partners in making and keeping effective integrations.

Sometimes we only use the EDI specification, they way we have always used them.  It can be instructive to browse the usage specifications of other companies.  We see what they remove, what they include, what comments they make, and what rules they document.  And seeing how others do things can help us to be more effective in how we use EDI specification documents.

I have used Foresights EDISIM, but I wonder what others use.  I have found a few links to SEF software, and will list them below.

Software I have found or used:

• Foresight EDISIM  (Used several version of their software and liked them all.)
• EDIdev has an SEF Manager  (haven’t used it but the screen shots look like files have have received from trading partners in the past.  Also, looks a bit dated, but it may just be me.)
• Stylus Studio (claims to handle SEF and EDI, but is very XML focused on their sight.  Trying out their demo now, and will update this page when I am done.)

What do you use/like?

This is not a comprehensive list.  I know there are people out there that use something else.  Maybe even better.  If you do, please post it in the comments.  Particularly, I am interested in any online, SEF or EDI standards tools.  I haven’t found any, and have been thinking about writing one.  So if you know of an online EDI Standard or SEF tool, please let me know.

Here is the standard, if you want to look at what they have published.  I will crop out a few pieces to talk about.  You might want to look over the full published standard.

ISA

The ISA segment contains some essential information that your trading partners will need.  Integrations of EDI rest on accurate trading partner values in the ISA values.  Sometimes even trading partners with existing connections will need to verify the ISA information and trading partner IDs.  For new trading partners, having this information available makes it possible for proactive partners to get a jump on integrations.

One thing of note here is their use of a sample segment to illustrate what the ISA segment will look like.  This can be very useful especially when dealing with trading partners that lack EDI experience.  Also useful when dealing with unusual segments or difficult to describe structures.

However, there is one caution.  I once had a trading partner hard code all of the sample segments into their map.  Thus we had an N1 with their ID and other information, and then an N1 with the same qualifiers but with the sample data from our specification.  This caused some problems for both of us.  Admittedly this only happened the once, but I was careful after that.  In any case, be sure that your examples are clearly marked as such to avoid repeating this mistake.

BIG

Every document has a beginning segment that is specific to that document type.  This segment will hold specific information that relates to that document type.

There are two things of note here.  First, is that they have listed the “not used” segments in their specification.  I don’t always to this myself but it can be done, and in this case illustrates one of the aspects of EDI good form practices. In the example segment they have none of the trailing elements that are not used, but they do have the one filed that is not used, but comes before a used element.  Thus illustrating the “no trailing delimiters” best practices rule.

The Second thing of note is their use of “Notes” to leave indication of a precedence rule in their mapping.  You can see here that they have a rule that uses the BIG_04 as the purchase order number only when there is no superseding PO number on the IT1 segment.  This type of notation is really helpful to trading partners and others in your own company down the road when problems happened and people need to discover what went wrong.

N1

N1 segments and N1 groups are important segments and can contain valuable data.  N1 segments are found in every EDI document that I have dealt with.

You’ll notice that there are three N1 segments listed in this Standards Document.  We know that the N1 can repeat, but this is not the way that is indicated.  This is one of the ways that the Standards Specification and the Usage Specification differ.  In the Standard, we talk about what may happen, or what may occur.  But in the Usage Specification, we talk about what should happen or what will occur.

All of the looping or repeating segment values have not been shown in this document.  The looping of the IT1 and N1 still happen, but the looping is not pertinent to the Usage Specification here. (at least to BookNet)  They have also included a short list of the acceptable code qualifiers to be used in the N1_03.  And you will note that they have defined them so that trading partners won’t have to look them up, or guess, and have indicated their preference.

Standards

Crafting a Standard and Usage Specification can make a real difference in the ease of boarding trading partners.  It can make future trouble shooting easier for your, and for others both at your company, and at the trading partner’s.  And it can illustrate the look and feel of EDI to less EDI sophisticated individuals.

Now that you have looked over the BookNet specification, is there anything that you think they should have changed to add clarity?  Or are there any thing that you have learned that you want to add to your own Usage Specification documentation?

There is a temptation to use this same validation in the production integration. But this would be a mistake. I’m not saying, “Don’t validate.” I highly encourage validation on both standard compliance and required data validation in mapping and integration. But to use the usage specification has a side effect that I witnessed once. (Only once.)

Validating Transactional EDI to Usage Specification

We had an integration requirement once that required that we only allow EDI elements that were in our usage specification. This was outside of our normal process and we all soon learned why.

When we validated EDI files based on the standard, we would error out documents that were corrupted, or in violation of the full EDi specification. (having mangled segments, or invalid UOMs and such.) But this one time, we thought it would be a good idea to also fail documents that had any extra data that we weren’t going to use. So we pulled all of the segments and elements out so that we were just down to what was in our usage specification.

This worked just like we told it to. As when the next file came in that was in bad form with extra trailing delimiters on the line item, it failed. And then the next with an extra segment that we didn’t care about? Well it failed too. As so on.

We soon were buried under failed EDI files that were valid EDI, and did contain the data we needed, but simply had something extra. And we were calling them all exceptions.

At first we added in a few of the common segments that were being sent. This relieved the lion’s share of the errors. But it still cropped up from time to time. Each time we would have failures for this reason we had to explain why. This grew tedious and embarrassing.

Eventually we re-mapped the doc-type using the complete standard in our validation. Magically all of these errors went away, and so did the embarrassing explanations.

Here is the way the process should work, generalized.

1. EDI envelop is validated against EDI standard.
2. EDI document is validated against EDI standard
3. Mission critical values are validated against Usage Specification.
4. File is processed.

What this does is to establish an inherent hierarchy of exceptions.  Transactions that fail on step 1 are all falling because the envelope is corrupt or  in error.  Also, Transactions that fail on step 2 are failing for a general EDI reason.  Both of these steps will have exceptions that can be handled by anyone with EDI knowledge.  Operational and Application rules don’t necessarily need to be known to understand how to handle errors at these levels.

Step 3 and 4 are exactly the opposite.  Errors that happen because of mission critical data that is missing or in error must be handled by someone that knows what the mission critical form is supposed to look like.  And files that pass validation but fail later in processing may need to be corrected and the validation process updated.  Or they may point to an error in the application itself.  In any case, much more knowledge of the whole process is required at these stages.

Take Away

First, by sharing this experience I hope that everyone can appreciate that sometimes requirements make us to funny things.  But I also wanted to share the approach to validation that I have found most helpful.  I have seen implementation that don’t do any EDI or Enveloping validation.  These are just as dangerous and frustrating as my experience.  But now all of the errors are pushed down to step 3 and 4.

I’d also like to invite you to subscribe to my newsletter.  It comes out once a month, starting at the end of this month.  I’ve got some great ideas for it, and think it will be terrific.

One of the things that in invaluable is the ability to share your standard documentation both internally and externally.  EDISIM is a tool that has allowed me to do that.  They produce a very professional looking document for external consumption and the standard is exportable so that everyone using EDISIM can share the same standards file in a format called SEF.

Features of a Good Standards Reference

Navigate Standards (multiple versions, doctypes, segments, etc)

A Standards Reference needs to offer a navigation interface of some kind.  You need to be able to see the versions,  standards and document types.  You will also want to drill in and view the reference outside of the document types, and  browse the segments or data element definitions.  But at the core, you need to be able to see a graphical representation of the standard, view the format and see the structure of how data will be represented in the EDI format and document.

Edit Standards

Just browsing the standard is not going to get the job done.  You need to be able to make alteration so that you don’t just have the high level X12 standard, like the book on my desk, but can make a standard that is your usage specification.  This means you will need to be able to do two operations on the standard.

First you will need to make changes to what elements and segments are displayed, and make changes to the encoded values that these elements support.  Editing these structures will aid you when you use the standard to validate and check for compliance based on your usage specification, and when creating documents to share and publish.

Second you will be able to add some of your business rules documentation to clarify the default definition of the standard.  This is especially useful if you need to pass some data that uses a mutually defined qualifier.  This lets you define what the “mutually defined” value should be.  This again helps when you publish and share your standard.

Export and Import (Read SEF)

All of the standards references that I have used had the ability to import and export a standard in an SEF file format.  SEF or “Standard Exchange Format” is a file that holds the standard dictionary and format for EDI files.  This is a nice way to  not only transfer and store the standard your company uses so that everyone has a copy of the same usage implementation.  But if no proprietary information is stored in the notes, it can be sent to trading partners out side of the company to aid in their integration efforts.  Any other export and import is a bonus, but SEF is a requirement..

Print (export to doc or pdf, or printer)

In spite of all this paperlessness, some times we still need to print things out.  Printing to a pdf or printing to paper is useful when sharing the standard and usage specification in a non-editable format so that others can view it, and make use of the information.  So for managers, or clients, or other parties that only need a viewable format to follow, this feature is a must have.

Save as editable file (RTF or other.)

Ok, just like printing, sometimes companies want all of their documentation to look the same.  To facilitate this, it is nice to be able to save the specification out to a rtf, or doc, and then add the company header, logo, watermark, etc.  This allows you to take the standards reference and really own the edits and notes that you have added.  For companies that have extensive documentation processes, this feature is a must have.

Other Good Reference Features

Validation

Validation is not a feature of an EDI standards reference, but it is a good tool that sometimes comes with them.  Validation tools read and EDI file, and compare the contents with the standard.  They should then identify all of the errors they can see and report on them.

In my experience validations may need to be run repeatedly as some errors in an EDI file will mask following errors.  For instance, if  the N1 segment is wrong, the validator may not be able to tell if all of the other lines in the N1 loop are valid.  So when I validate an EDI file, I correct the errors seen on the first pass and re-validate.  This has saved me from telling a trading partner to fix an issue, only to have to tell them about another issue later.  After all, we do want our integrations to go quickly.

Test Data

Sometimes looking at the standard is just not enough to show us how things will look when formed into the document.  Seeing sample data is essential, and one of the first things I request at the beginning of any integration project.  A Standards Reference will sometimes have tools that will create a test data EDI file based on the standard.  This can be useful if you are working with a new document type that you haven’t encountered before and need to see how the standard looks in the formed document.

Evaluation of Standards Reference Technologies

Is there a feature that I have missed?  Please let me know.  Also,what EDI standards reference tools have you used?  What do you like or dislike about them?  How about any non-EDI specific tools?  Please let me know what you have used.  I would like to compile a list of tools and run some comparisons of what is out there.