This introduction aims at addressing the basic questions of convention and implementation of the EDI technology.  I have a list of topics, and hope that over time I can cover all of them.  I also hope that people reading here may have questions that I haven’t thought to write about.  Please let  me know what your questions are and I will see what I can do about providing answers.

• What is EDI?
• Parts of EDI
• Segments
• Delimiters
• Conventions of EDI
• Why are there more than one version of EDI?
• Standards and Usage
• Encoded Values
• EDI in Good Form
• Element Notation
• Wrapped and Unwrapped EDI
• Enveloping Part One
• Enveloping Part Two – ISA
  
• Enveloping Part Three – GS
• Enveloping Part Four – ST
• Loops in EDI
• Hierarchical Levels

Of course understanding how EDI presents and represents data is the first step.  From here we may want to go to our Data Mapping Guide, or if we are familiar with mapping there are some examples in our mapping case studies of EDI to and from other formats.  Again, if you have an interesting problem or solution, I would love to here about it.  Please send it to
roy@TheIntegrationEngineer.com and I will see what I can do.

With this as the abstract version of our goal. We break the task into three sections. Data types that we will exchange both outbound and inbound. What transport or connectivity solutions we will used to ensure a robust and reliable transmission and reception of the data. And what data translation, storage and processing solutions we will employ.

Data Types

Many people don’t think about this part very much because they have only dealt with EDI systems that are already in service or in the process of being launched.  By this point the types of data being passed between trading partners have already been decided.  So to begin this process we ask what type of business we are conducting.

If we are doing e-commerce, we are probably going to exchange orders, confirmations, invoices and ship notices.  If we are a retailer, we may be receiving catalog data, where if we are a vendor we will be transmitting it.  And so forth.

We can find out what document types to use in two easy ways.  First, any of our trading partners currently exchange EDI data with other parties, (like integrating with a vendor that has other customers doing EDI), we can ask for if we haven’t already received the EDI specifications that they use.  If we are setting up a new integration with a new trading partner with no EDI history, we will then need to look at the list of EDI document types and find the ones that will fill our needs of data exchange.  (All of the EDI document specifications have a name as well as a number.  For this exercise we will need to look at their name.)

Transformation, Processing and Storage

For data coming into the process for our trading partners is what I call “inbound” data.  And so of course data going out to a trading partner is called “outbound” data.  In either case we start the Transformation, processing and storage by identifying what our destination format, and location will be for the data.  Inbound data has the destination of whatever process or storage we are going to place it, and in the format that we need it in to process or store.  And outbound data has the destination of our connectivity solution to our trading partner and the format of the EDI document type that we have agreed to use.

This is called Target Based planning and it goes hand in hand with Target Based Mapping.  We plan by starting with where we want to end up, in the transformation, processing and storage of the data.  Then we work backword from that goal and create a way to get the data from where it is, to where we need it to be.

For instance if we have the destination of an EDI 850 on our AS2 connection, we start with a standards format of the 850 the trading partner can accept and the location or API that our AS2 connection needs us to use.  After we have that target defined, we map and plan out how to get our order data to that location, coming out of our database, ERP, e-commerce system or some file that we are processing.  (Possibly from a combination of these.)

And we do the opposite if we are processing an inbound 850, we start with the format and interface that we need to input an order into our system, and then build the process to get the information we need out of the 850 and trading partner database etc.

Connectivity

This is a critically important issue, that we will not go into detail here.  But you will want a connectivity solution that is reliable and provides you with a positive confirmation that a file has been received by the other partner and establishes accountability for the transaction, and a process for remediation if there is any dispute.  I will have to talk more about connectivity another time when I can go into greater detail.

Summary

Breaking big tasks like rolling out an EDI strategy can become very complex.  There are two secrets to being successful.  First, break your big tasks into three pieces; Inputs, Outputs and Processes.  The second secret is to begin with your end, or goal, or target.  And then work back from there.  Putting these two pieces of advice into use will make your EDI strategy more manageable and come of more smoothly.

Early in my integration career I was working on a medical supply chain.  Accuracy was important as you can imagine.  And there was a good deal of stress that we shared as we brought our products on-line and let them encounter real data.

We tried to ensure that the data that we sent to suppliers was received, and when things when wrong we were on the phone quickly asking for confirmation that orders were received.  As time when on we became better both at reliably getting data to the correct place, and in determining when problems where really problems.

“There was one Vendor that sold capital equipments.  So they received few orders, but when they did, their orders were for a high dollar items.  They also kept shutting down the system that we sent orders to.  I think years went by where every order was accompanied by a phone call to their systems administrator asking that they re-boot their server.”

The EDI way

The easiest and most EDI like way to establish accountability of EDI transactions is the use of a 997 transaction.  Basically a 997 works as an acknowledgment.  The name of this document actually is Functional Acknowledgment.  So we are acknowledging that a specific functional EDI component was received, and possibly how it was received.

If you are versed in the EDI enveloping levels you have already realized the  link of between the 997 and the GS level of the EDI envelope.  Also you may remember that you can request a 997 be returned for your transaction in the ISA segment in ISA_14.  But merely setting up your ISA to request a 997 does not actually force your trading partner to comply.   Your trading partner may not be able to, or may not wish to return a 997 for either business or technology reasons.

Assuming that you are able to get a 997 back.  Just receiving the 997 does not mean that all is well with your transaction.  The 997 is an EDI transaction.  It can contain data indicating just the success of receiving the data, or a few validation levels indicating data corruption in transit or rejection for business process reasons.

The Transport Technology way

Another way is to place the accountability on the transport of the message.  Using AS2 or other transport technologies that attempt to guarantee delivery of messages, it is possible to extend our reliable message solution to transactions of multiple types.  AS2 and other transport solutions return an MDN or message disposition notification letting the transporting party know that its message has been received.

The BPM way

In spite of the facility of EDI to process and utilize 997s and of AS2 to utilize MDNs, the most exciting method of assuring message delivery and accountability is utilizing business intelligence.  For me, I remember a system that we developed to help us with the health care supply chain.  We simple began tracking the response time that was typical for each of our supplier trading partners.  After gathering trending data, we applied alerts when responses were overdue by a specified degree of variance from the norm.

Since this process relied on the past 6 months of performance as the baseline, it was adaptive to changes in the vendor.  We could also manually change the amount and degree if we started getting false alerts.  In this way we had a way to react to failed transaction without requiring agreement and arrangements with the trading partner to confirm receipt or processing of transactions.

The Business Agreement

Even in cases where you have implemented a BPM process to alert to failed transactions, you have to have an agreement and arrangement with your trading partners that allow you to deal with failures in transmission and processing of your e-commerce transactions.  At a minimum this needs to include contact information for support teams on each side of the transaction.  It does no good to know there is a problem with a transaction but to be unable to solve the problem because the other party is unavailable.

In the end, it is what you have agreed with our trading partner that make the sender or receiver responsible for processing the transaction.

What is a line item?

A line item is a group of data and its structure that is or can be repeated multiple times in its document.  It holds the data that is the focus of the document, and a part of a document or transaction that contains distinct part that can stand alone.  For instance, an order that has two line items could be split into two orders that would be able to be filled separately.  It does not necessarily contain information that can be applied to a group of items, although sometimes this information is included.  This feeds into what type of data that a line item contains.

What type of data does a line item contain contain?

In a purchase order, or other commerce related document these pieces of data will be found in the line item:

1. Purpose of line
2. Product or Service ID
3. Quantity
4. Unit of Measure
5. Price or Fee
6. Taxes
7. Special Handling
8. Descriptions

Some additional data can be aggregated to the document level or published at the line item level;

1. Ship to or Deliver to data.
2. Discounts
3. Shipping methods or tracking information

Where in EDI does this data go?

For the most part, the PO1 segment and group contain the line item data.  There are a few transactions that use another segment.  The invoice for instance uses the IT1 segment.  Depending on the scope of our definition, an HL group could be a line item in some documents.

Line items are at the center of our transactions.  Purchase Orders, Invoices, and Catalogs would not be worth creating without the line items they contain.

So it goes like this:

850 is a Purchase order, and is sent to the Vendor

855 is an Order Response, and is sent from the Vendor.  It confirms, updates or rejects each line, or the PO as a whole.

856 is the Ship Notice. and is sent from the Vendor.  It may contain one or more lines from one or more orders and can indicate shipment of partial lines from a PO.

810 is the Invoice, and is sent from the Vendor.  It contains one or more lines from one or more POs, but generally people keep these to only one PO.

820 is Remittance, and is sent to the Vendor.  It contains information relating to payment.

If you have used a standards editor, you probably know what an SEF file is, or have used it.  Some applications and EAI even use SEF files as part of their document creation and validation processes.  It becomes useful to describe briefly what SEF files look like, and what type of information they contain.

What is in there?

If you are familiar with EDI, and have cracked open an SEF file with your standard text editor of choice, you will already know what I am about to say.  The contents of the SEF file don’t look that complex to someone who is used to and familiar with how various EDI files look.  For those that are not so brave, I will explain this painlessly.

First, the SEF file is really just a text file.  It can be edited by hand, but I don’t really recommend it.  (not because you can’t, but because it is tedious.)

Second, it contains its data in sections.  There is a section for doc types, segments, elements, and encoded data.  There is also a few housekeeping sections like version and name etc.

What is an SEF file for?

SEF stands for Standards Exchange Format.  This is literally a file format that was designed to contain information about EDI standards.  It was created so that a computer application could understand an EDI file.  It is used by some application like Standard Repositories and Standards Editors to allow us humans to deal with a standard like it was a text document.  While at the same time keeping a repository that can be used by an integration application to form and validate an EDI file.

If you are familiar with the concept of metadata then an SEF file is the EDI standard’s metadata.  If you are not familiar with metadata, then skip this part.

Do I need to know about SEF files and EDI?

Nope.  SEF is only useful if you are exchanging standards using the format.  If your standard and usage is in the form of a PDF or a spreadsheet, that is fine.  Many people do just that.

For me, I need to know this because I am trying to build some Standard Repository tools, and want to use the SEF format to manage them.

And for those that want a deeper look, I will be following this article with some deeper and more detailed articles dealing with the internals of the SEF file.

EDI Specifications

In the larger sense, the EDI specification is the set of rules that define each document type, the segments they contain, and the size and type of data in the elements.  When we talk about the EDI Standard Specification, we are talking about the whole set of valid EDI document types.  If something is valid EDI, then it complies with the EDI Standard Specification.  However, this large, all encompassing specification is not useful in coordinating the exchange of documents between two trading partners.

What goes in an EDI Usage Specification?

So we take EDI Standard Specification and we reduce it.  We remove the unused document types and the unused segments within the documents we will use.  And we even remove the unused elements and encoded values within the segments we will use.  At this point, we have created an EDI Usage Specification.  We can also add to the specification by including values that we need, like designating the an optional value or ID as required and specifying what type is should be.

How is an EDI specification Used?

Now that we have a usage specification we can use it to do to basic things.  First, it is the source of information for us to set-up our integration for our EDI document exchange.  And Second it is the format that we validate our EDI documents with to determine a valid transaction.

EDI Usage Specifications can be a source of data and integration documentation.  This can become extremely valuable when the choices and information about how your EDI interface works is recorded into the notes of the Usage Specification.

Beyond a visual validation of looking at the EDI file and comparing it with the specification, many times a specification can be found in the form of an SEF file.  The SEF file can be used in an EDI validation application.  This allows a potentially large and unwieldy EDI file to be scanned for compliance and accuracy.

There may be other uses, but these are the main two, integration data repository, tool for EDI validation.

EDI Specification Examples:

To walk through how a usage specification works, and is used, check out this review of BookNet Canada’s specification.

Then you can take a look at the EDI specifications published by Macy’sNet that they keep online to assist their trading partners in making and keeping effective integrations.

Sometimes we only use the EDI specification, they way we have always used them.  It can be instructive to browse the usage specifications of other companies.  We see what they remove, what they include, what comments they make, and what rules they document.  And seeing how others do things can help us to be more effective in how we use EDI specification documents.

Parts of ACK

Here is a diagram of a sample ACK segment:

Like any other segment in EDI, the ACK segment is composed of elements.  Here is a list, element by element and what it contains.

Best Practices

With the ACK the best practices are more in how to implement and what to expect.  So here are some things to remember when implementing this segment.

1. An ACK is part of a PO1 line for a response to another PO1 line transaction.  This segment supplies a status, and can modify the line item contents.
2. There can be more than one ACK segment for a line item.  (4010 855s allow 104 repetitions of the ACK)
3. When an ACK explicitly changes the line item it can only change that one type.
4. There is a status called Accepted With Changes.  “AC”  For this, you will need to parse the ACK and see if any of the values have changed.

Parts of IT1

Here is a diagram of a sample IT1 segment.

Like any other segment in EDI, the IT1 segment is composed of elements.  Here is a list, element by element and what it contains.

I wrote a section on Line Item Good Form and Best Practices in the article about PO1 Line Items.  Instead of repeating that I will just let you hop over there are read it.

IT1 Loop Segments

As we said in the beginning, the IT1 is a segment, and a loop/group. And as you may have noticed, there is data that you may need that is not covered in the elements available in the IT1 Segment that belong to the line item.  Values like Tax information, found in the TAX and TXI segments.  These and other values are found on the line item, but are not on the IT1 segment.

Parts of the PO1

Here is a diagram of a sample PO1 segment:

Like any other segment in EDI, the PO1 segment is composed of elements.  Here is a list, element by element and what it contains.

Good Form on PO1 line

EDI in Good Form is both style and best practice tips to avoid common exceptions.  Paying attention to good form tips and best practices helps you create an integration solution that is not only complies to the standard, but functions to help you and your trading partner have an unnecessary support burden due to the avoidable unexpected.

1.  It is bad from to have PO1 Product and Service pairs that are empty.  This relates to not having any trailing element delimiters.  It also goes along with not having empty pairs in the line with populated Product and Service element pairs following.  Thus it is best to build your mapping so that optional data doesn’t create empty elements when it opts out.

2.  When reading a PO1 line, it is best to assume that an expected Product and Service value could be in any pair.  So the map needs to walk down the pairs of elements until it finds the qualifier that indicates the value you are looking for.  This allows you to use the same input map with multiple EDI producing trading partners that may have interpreted the priority of data in different ways, and avoid exceptions.

3.  When producing EDI with a PO1 line, it is best to be as consistent as possible with your placement of the Product and Service data.  Create your logic and mapping so that mandatory data occurs early in the PO1 line, and optional data is moved to the end.  This will keep your required data from hopping around positionally.  This will help your trading partners that may not have had the foresight to follow Good Form Tip number 2 from having exceptions that slow down you process.

PO1 Loop Segments

As we said in the beginning, the PO1 is a segment, and a loop/group. And as you may have noticed, there is data that you may need that is not covered in the elements available in the PO1 Segment that belong to the line item.  Values like Item Description, found in the PID segment. Or Tax information, found in the TAX and TXI segments.  These and other values are found on the line item, but are not on the PO1 segment.  We will have talk about some of these other segments and where to find other common data on another day.

I didn’t, and a search of the web didn’t reveal anything that I could, in any confidence, recommend.  So I bring this question to you.  Do you know of a DOT NET framework that can be used for generating and or parsing EDI data?  If  you do, please comment below and let everyone know.  I will also pass it on to my friend who is looking for this.

Development Frameworks

The reader that I mentioned asked about Dot Net, but there are other times and environments where companies need to setup and EDI interface for their implementation.  I have spent most of my career working with enterprise level applications to handle the EDI transportation and transformation.  But I have seen a few times when it would have been useful to code an interface for and EDI file onto a stand alone application.  But I am not a developer in that sense.  Hoping that some of you may be, what are some EDI frameworks or modules etc that you have used or know about?  Java, C++, VB, or even perl, python, tcl, php, etc.  Please leave a comment with the framework or tool that you know of.  You can say a few words of eveluation if you can, or just leave a link to it and I will look them over and try to post about them over the next few weeks.

EDI and B2B tools

This brings up another point.  I have been trying to add to my Toolbox page as I have gone along.  I have posted a few tables and spreadsheets.  I also had a post about tools that an effective integration engineer should use.  I plan on posting more tools in my Toolbox too.  And I have a reall love of free tools.  But that is not the only kind of tool that I use, or will recomend.  If you know of a tool or technology that you think others should know about, or might help them, post a link, or drop me a line and I will add it to my Toolbox or write a post about it.  (or both)

So bring me your suggestions and I will share them with others and write an article about them when I can.

Thanks

Here is the standard, if you want to look at what they have published.  I will crop out a few pieces to talk about.  You might want to look over the full published standard.

ISA

The ISA segment contains some essential information that your trading partners will need.  Integrations of EDI rest on accurate trading partner values in the ISA values.  Sometimes even trading partners with existing connections will need to verify the ISA information and trading partner IDs.  For new trading partners, having this information available makes it possible for proactive partners to get a jump on integrations.

One thing of note here is their use of a sample segment to illustrate what the ISA segment will look like.  This can be very useful especially when dealing with trading partners that lack EDI experience.  Also useful when dealing with unusual segments or difficult to describe structures.

However, there is one caution.  I once had a trading partner hard code all of the sample segments into their map.  Thus we had an N1 with their ID and other information, and then an N1 with the same qualifiers but with the sample data from our specification.  This caused some problems for both of us.  Admittedly this only happened the once, but I was careful after that.  In any case, be sure that your examples are clearly marked as such to avoid repeating this mistake.

BIG

Every document has a beginning segment that is specific to that document type.  This segment will hold specific information that relates to that document type.

There are two things of note here.  First, is that they have listed the “not used” segments in their specification.  I don’t always to this myself but it can be done, and in this case illustrates one of the aspects of EDI good form practices. In the example segment they have none of the trailing elements that are not used, but they do have the one filed that is not used, but comes before a used element.  Thus illustrating the “no trailing delimiters” best practices rule.

The Second thing of note is their use of “Notes” to leave indication of a precedence rule in their mapping.  You can see here that they have a rule that uses the BIG_04 as the purchase order number only when there is no superseding PO number on the IT1 segment.  This type of notation is really helpful to trading partners and others in your own company down the road when problems happened and people need to discover what went wrong.

N1

N1 segments and N1 groups are important segments and can contain valuable data.  N1 segments are found in every EDI document that I have dealt with.

You’ll notice that there are three N1 segments listed in this Standards Document.  We know that the N1 can repeat, but this is not the way that is indicated.  This is one of the ways that the Standards Specification and the Usage Specification differ.  In the Standard, we talk about what may happen, or what may occur.  But in the Usage Specification, we talk about what should happen or what will occur.

All of the looping or repeating segment values have not been shown in this document.  The looping of the IT1 and N1 still happen, but the looping is not pertinent to the Usage Specification here. (at least to BookNet)  They have also included a short list of the acceptable code qualifiers to be used in the N1_03.  And you will note that they have defined them so that trading partners won’t have to look them up, or guess, and have indicated their preference.

Standards

Crafting a Standard and Usage Specification can make a real difference in the ease of boarding trading partners.  It can make future trouble shooting easier for your, and for others both at your company, and at the trading partner’s.  And it can illustrate the look and feel of EDI to less EDI sophisticated individuals.

Now that you have looked over the BookNet specification, is there anything that you think they should have changed to add clarity?  Or are there any thing that you have learned that you want to add to your own Usage Specification documentation?

One of the things that in invaluable is the ability to share your standard documentation both internally and externally.  EDISIM is a tool that has allowed me to do that.  They produce a very professional looking document for external consumption and the standard is exportable so that everyone using EDISIM can share the same standards file in a format called SEF.

Features of a Good Standards Reference

Navigate Standards (multiple versions, doctypes, segments, etc)

A Standards Reference needs to offer a navigation interface of some kind.  You need to be able to see the versions,  standards and document types.  You will also want to drill in and view the reference outside of the document types, and  browse the segments or data element definitions.  But at the core, you need to be able to see a graphical representation of the standard, view the format and see the structure of how data will be represented in the EDI format and document.

Edit Standards

Just browsing the standard is not going to get the job done.  You need to be able to make alteration so that you don’t just have the high level X12 standard, like the book on my desk, but can make a standard that is your usage specification.  This means you will need to be able to do two operations on the standard.

First you will need to make changes to what elements and segments are displayed, and make changes to the encoded values that these elements support.  Editing these structures will aid you when you use the standard to validate and check for compliance based on your usage specification, and when creating documents to share and publish.

Second you will be able to add some of your business rules documentation to clarify the default definition of the standard.  This is especially useful if you need to pass some data that uses a mutually defined qualifier.  This lets you define what the “mutually defined” value should be.  This again helps when you publish and share your standard.

Export and Import (Read SEF)

All of the standards references that I have used had the ability to import and export a standard in an SEF file format.  SEF or “Standard Exchange Format” is a file that holds the standard dictionary and format for EDI files.  This is a nice way to  not only transfer and store the standard your company uses so that everyone has a copy of the same usage implementation.  But if no proprietary information is stored in the notes, it can be sent to trading partners out side of the company to aid in their integration efforts.  Any other export and import is a bonus, but SEF is a requirement..

Print (export to doc or pdf, or printer)

In spite of all this paperlessness, some times we still need to print things out.  Printing to a pdf or printing to paper is useful when sharing the standard and usage specification in a non-editable format so that others can view it, and make use of the information.  So for managers, or clients, or other parties that only need a viewable format to follow, this feature is a must have.

Save as editable file (RTF or other.)

Ok, just like printing, sometimes companies want all of their documentation to look the same.  To facilitate this, it is nice to be able to save the specification out to a rtf, or doc, and then add the company header, logo, watermark, etc.  This allows you to take the standards reference and really own the edits and notes that you have added.  For companies that have extensive documentation processes, this feature is a must have.

Other Good Reference Features

Validation

Validation is not a feature of an EDI standards reference, but it is a good tool that sometimes comes with them.  Validation tools read and EDI file, and compare the contents with the standard.  They should then identify all of the errors they can see and report on them.

In my experience validations may need to be run repeatedly as some errors in an EDI file will mask following errors.  For instance, if  the N1 segment is wrong, the validator may not be able to tell if all of the other lines in the N1 loop are valid.  So when I validate an EDI file, I correct the errors seen on the first pass and re-validate.  This has saved me from telling a trading partner to fix an issue, only to have to tell them about another issue later.  After all, we do want our integrations to go quickly.

Test Data

Sometimes looking at the standard is just not enough to show us how things will look when formed into the document.  Seeing sample data is essential, and one of the first things I request at the beginning of any integration project.  A Standards Reference will sometimes have tools that will create a test data EDI file based on the standard.  This can be useful if you are working with a new document type that you haven’t encountered before and need to see how the standard looks in the formed document.

Evaluation of Standards Reference Technologies

Is there a feature that I have missed?  Please let me know.  Also,what EDI standards reference tools have you used?  What do you like or dislike about them?  How about any non-EDI specific tools?  Please let me know what you have used.  I would like to compile a list of tools and run some comparisons of what is out there.

Structure

Two important concepts in EDI are not exactly parts.  They are the structures of Enveloping and Segments:

• Enveloping, as we discuss in another post, is a way for an EDI file to be self determining.  In the first segment of an EDI file, the ISA segment, all of the information can be found to allow a system designed to interpret EDI to know where it is going, where it came from, what version it is, and what characters it is using as delimiters.
• Segments following this first segment are not fixed position.  White space is not used unless it is part of the data in the element.  Each segment has a header element that indicates the definition used for the rest of the data elements in that segment.  A segment may not need to use all of the possible elements in its definition.  When this is the case, the segment terminates with the segment delimiter or terminator.

The parts:

• Standard is a concept that has more than one meaning, There is the standard, that is the overall X12 compliance rules that regulate what is and may be an EDI document.  But there is also the concept of a specific use case, or usage specification.  Both of these are referred to as ‘Standards’ but in this description of the parts of EDI, the standard is the first concept.  This is that when an EDI document is parsed, it must comply with this standard of segments, delimiters, and format, or it is rejected as being corrupt.
• Version is a sub-component of the standard.  It is declared in the ISA, and lets the parser know what version of the EDI standard will be contained inside the envelope.  This is  important as to format and structure of the data that has changed as technology and needs have matured.
• Doctype is not declared in the ISA, but is declared in the ST.  We may also know when we see the functional declaration in the GS.  The doctype determines which segments and in what order and structure may be present.  Not all EDI documents will need all segment types, so the doctype is very important when parsing the EDI file.  This is covered more in the ST Enveloping section.
• Segments are the most basic part that makes and EDI file and EDI file is the segment.  There is a more detailed discussion of the nuances of segments here.  Segments are defined by their type, the characters that are located before the first delimiter in the segment.  And segments are terminated by a special delimiter called either the segment delimiter or segment terminator.
• Element exist inside the segment.  Elements hold the data.  Elements are separated by element delimiters.  On rare occasions some elements can have sub-elements.  There is a more detailed discussion of elements here.
• Envelope Segments There are three pairs of special segments that must be found, and can only be found, in the enveloping structure.  Enveloping contains the origin and destination identification, as well as content definition identification values.  A more detailed discussion of enveloping can be found here.
• Files are not really a part of EDI, but when EDI is formed, it is generally saved in a file.  The file can contain any mix of EDI document types and going to any number of destination.  Generally, for sanity’s sake, this is not done.  But there is nothing in the name of the EDI file that matters to the contents.  There is no really expected extention like “.edi” for instance, and there is not need for white space removal from the begining or end of an EDI file.
• Documents are another matter.  An EDI document is the sebments that reside within the ST envelope.  As such, they are a set of data that has a single format, (EDI standard), a single sender and single receiver.  Thus when an EDI doscument is discussed, we are really discussing the payload of the EDI envelopes.

–

Looking for something else relating to EDI?  Check out the EDI Primer post

Next in our series is a more in-depth look into Segments.

–

The Target

Open Office tables are divided into two tables. Invoice and Invoice Details. This can be mapped in two ways; The first way is to map the data into one common format and rely on whatever ETL tool is importing the data to catch and split the data. Second, a key can be acquired or constructed in the transformation and then the the data can be divided in to matching input formats. Then when these inputs are moved into the database, they will relate to each other on this key.

The choice of how you will do this will depend on your environment. Questions like, “Will I have enough data to provide a unique key?”, or “is there a way to get a key with an API call or database query?” The answers to these questions will determine what course you will take.

If the system ultimately receiving the data is asynchronous to the transformation, and you need to send the invoice and invoice details data separately, some care needs to be taken to ensure that the data can be related after it is separated.

So what data in the invoice can be used to tie the invoice to the invoice details? The first answer might be, “The Invoice Number.” But this number is not guaranteed to be unique across multiple vendors.  In EDI and cXML there are document unique identifiers.  Since we are using EDI, we can use a combination of the ISA Sender, Receiver, and Control number.  We will also want to use the GS control number, and ST control number in the event that more than one invoice is sent in a single envelope.  And we might as well tack on the actual invoice number from the BIG_02.

Database Tables

Invoice Table:	Invoice Details Table:

The Paper Map

Now that we know what the target looks like, we fill out the target side of the paper map.  Since we will create two “files” in our output.  We are creating the Invoice and the InvoiceDetails file, but we can use one paper map for both, and will distinguish this with a bar between the two “files”.  (I am saying files, but this could be a queue, or a post, or an insert over odbc, etc)

What’s Next

Today we went through the process of identifying the target, and creating a paper map with the target format identified.  We talked about some of the strategy that we use in deciding what to map and how to map it.  Next time we will identify the source, and begin mapping data from the source.

EDI is not a flat file, but it does have repeated lines or segments.  Some times this repetition comes in groups and sometimes its just a line that get repeated in part of the document.  Repeating structured data is fundamental to EDI, but can become difficult to interpret at times.  Understanding the types of repeating structures and when and where they may occur can save a lot of time.

Repetition

There are three kinds of loops/repeated segments; First there is a simple segment that can appear more than once.  Lets call this a Repeating Segment.  Second there is a group of segments that appear together, and may repeat.  Lets call this a Repeating Group.  And finally there is a segment or group that repeats within a beginning and ending loop envelope.  Lets call this a Repeating Envelop.

Repeating Segments

A repeating segment is pretty much just what it sounds like, a segment that may occur more than one time.  A good example of this would be the AMT segment.  The AMT is a segment that contains a monetary value.  The AMT_01 declares what type of value it is, and the AMT_02 has the amount of money.  In an EDI document like an invoice, we may need to have many values, subtotals, taxes, base price, and so on.  Having a repeating segment lets us put all of these values on their own independent segment to be interpreted and included as best fits the Trading Partners and the nature of the transaction.

Here is an example of an unwrapped N1 group where the N3 is repeated:

N1*AC*ARTHUR JONES*1*9012345918341*11*SE~
N2*GEORGE WASHINGTON~
N3*157 WEST 57TH STREET~
N3*Suite 2001*5th Floor~
N4*CINCINNATI*OH*43017~

When segments are repeatable, they are indicated with a “Max Use” value of greater than one in the standards and usage specification.  In the N1 Group, the N2 and N3 segments may have a max use of 2.

Repeating Groups

A repeating group is also what it sounds like.  It is a group of segments that work together and get repeated.  This loop is generally referred to by the primary segment.  This segment is required in this group.  This means that if this group occurs, this segment must also occur.

Most Repeating Groups are contextual.  This means that they are defined by the segments that surround them, and are terminated by the first segment that is not in the group, or starting the group again.  So a Repeating Group may repeat, but as soon as we move out of the segments in this group, we are done repeating it and move on to the next segments or groups defined in the standard.

The N1 is a good example.  The N1 group contains name, address and contact related information.  The only required segment is the N1.  Subordinate to the N1 (that contains name and such) is the N2 (that contains secondary name or attention type data.) Then comes the N3 (with the street address) and then the N4 (with the city, state, country, zip).

Not all groups have the same letters, like ‘N’ and the N1 group is no different.  It also contains the PER segment that contains a persons name, and contact info like phone number or email.  Any PER that happens under an N1 is for that location.

In a document like an order you may want to have the address and contact for the shipper, the receiver, a biller, and a payer.  There are many possibilities, and these are all repeated in this group. Here is an example of an unwrapped N1 group where the that is repeated:

N1*ST*MRS JONES*1*123456789*11*SE~
N2*BEVERLY SIMPSON~
N3*157 WEST 57TH STREET~
N4*CINCINNATI*OH*43017~
N1*SF*MR JONES*1*9012345918341*11*SE~
N2*MAGGIE MCGILLICUTTY~
N3*1001 MAPLE STREET~
N4*PITTSBURGH*PA*86753~

Segments that are in a repeating group are listed as being in a Loop.  They may be repeated whenever the loop is repeated as specified in the standards and usage specification.  Thus you may have a max use of the N3 as 2, and an N1 Loop that has a max use of 1000.  So really you could see 2000 N3 segments, (2 for each of the 1000 possible N1 loops).

Repeating Envelope

A repeating envelope makes reference to EDI Enveloping.  Basically, the ISA-IEA envelope can contain one or more GS-GE envelopes.  And each GS-GE can contain more than one ST-SE envelope.  These levels of enveloping are covered in greater detail on the EDI Enveloping page.  But it doesn’t have to stop there.  In the 3050 850 document there is a loop called LS-LE that contains terms.  This section happens inside the ST-SE.  And there is nothing to prevent other sub envelops from happening where there is a need.

What is fundamental to these types of loops is that they have a designated beginning, and ending.  They must have both.  Repeating Groups end when a beginning of a new iteration of the loop is found, or a segment that is not in the group is found.  This can leave some ambiguity if the standard is not followed closely.  The Repeating Envelope leaves nothing to chance and declares that it is done.

A note about Required segments

You may have noticed in the examples above or in the specification that we referenced that there are required segments in loops that are optional.  This is not a contradiction and I will explain this here.  A loop may be optional.  If it is, then it may be ommitted entirely from a document while still compliant with the standard.  If the loop is included, then the required and optional segments within that loop apply.  And they apply in each instance of the loop.  (We can’t have an second N1 loop unless we include the required N1 segment.)

Basically this is like scope in XML.  The requirements of mandatory or optional only apply within the scope of the loop where they are defines. in the same fashion as sub nodes in and XML hierarchy.

Basic Fundamentals to Remember

1. Some segments can occur more than once, or can be repeated.
2. Some groups of segments can and often do occur more than once.  These are contextual and are terminated by segments after the group in the standard.
3. Some segments are encapsulated in beginning and ending segments like enveloping.  These segments can repeat, but only in another encapsulation or envelope.
4. The standard will indicate when repetition is allowed.

–

Looking for something else relating to EDI?  Check out the EDI Primer post

–

Document Identifier

The document Identifier is not encoded.  That is to say, for an 850 the ST document ID is “850″.  For an 810, the Document Identifier is “810″ and so on.  Thus if you open an EDI file, and don’t recognize the GS_01, skip to the ST_01 and it will tell you what standard you will need to find to read this document.

Good form would dictate that the GS_01 and the ST_01 would specify the same type of document.  The GS would specify it on a functional level, and the ST on the standard level.  Thus they would not be identical strings but would indeed convey the same message about what the document(s) would be.  Some EDI translator or processing software will validate this relationship and some may not.  But in the end consistency is the best policy.

Control Number

The control number is different for the ST than we saw in both the ISA and GS.  The ST_02 is a 3 to 9 digit number.  This means that it will start with 001 not 000000001.  Of course you can still use a 000000001 but don’t be surprised when you see less than 9 digits in the ST-SE control number.  Whichever you chose, you must do the same for both the ST and SE.

(Using 000 or 000000000 may be allowed, but is not recommended as any application that trims the zeroes will get an empty string and produce an error.  Be nice and only use values greater than zero.)

Segment Counter

The SE segment counter is the part of the ST-SE that gives people hand editing an EDI document the most trouble.  If the counter is inaccurate it indicates that the document is corrupted.  This should fail in all good translators as processing a corrupt EDI file is saying, “Its okay if we get bad data.”  This counter represents the number of segment between the ST and before the SE.

And it is just that simple.  Remember that you can have more than one GS-GE envelope inside an ISA-IEA, and you can have more than one ST-SE inside a GS-GE, but you can only have one document of one type in side a ST-SE envelope.

–

For an overview of Enveloping go here.

Looking for the ISA enveloping page?

Looking for the  GS enveloping page?

Looking for something else relating to EDI?  Check out the EDI Primer post

Wrapped

As I said wrapped EDI has nothing between the segment terminator and the next segment, in this format;

SEG|||~SEG||||||~

and so on.  Wrapped EDI looks like this:

ISA*00*          *00*          *01*9012345720000  *01*9088877320000  *090508*1459*U*00200*000000001*0*T*:~GS*PO*901234572000*908887732000*090508*1459*1*T*004010~ST*850*0001~BEG*01*BK*99AKDF9DAL393*39483920193843*20090508*AN3920943*AC*IBM*02*AE*02*BA~DTM*002*20090508*1615*CT*CM*DATE TIME PERIOD~N1*AC*ARTHUR JONES*1*9012345918341*11*SE~N2*MAGGIE MCGILLICUTTY*BEVERLY SIMPSON~N3*PO BOX 123*157 WEST 57TH STREET~N4*CINCINNATI*OH*43017*US*IP*02535~PER*AC*ARTHUR JONES*TE*(614)555-1212*TE*(614)555-1212*TE*(614)555-1212*ADDL CONTACT DESC~PO1*AAA849*100*DP*100.00*PD*AB*ASSEMBLY*BL*BRAND OR LABEL*ON*CUST12345*AB*ASSEMBLY*BL*BRAND OR LABEL*ON*CUST12345*AB*ASSEMBLY*BL*BRAND OR LABEL*ON*CUST12345*AB*ASSEMBLY~PID*F*BC*AX*UNSCENTED*GENERAL PURPOSE*BS*SUBQUALIFIER*N*ENG~CTT*10*855*12345678*HB*40*QD*GENERAL PURPOSE~SE*12*0001~GE*1*1~IEA*1*000000001~

Unwrapped

When we unwrap the EDI, we don’t just place any character between the segment terminator and the next segment, we place a carriage return and line feed.  This makes all of the segments appear to be justified like separate lines of a text file.  Thus the format is changed in this manner;

SEG|||~

SEG||||||~

and so on.  Unwrapped EDI looks like this:

ISA*00*          *00*          *01*9012345720000  *01*9088877320000  *090508*1459*U*00200*000000001*0*T*:~
GS*PO*901234572000*908887732000*090508*1459*1*T*004010~
ST*850*0001~
BEG*01*BK*99AKDF9DAL393*39483920193843*20090508*AN3920943*AC*IBM*02*AE*02*BA~
DTM*002*20090508*1615*CT*CM*DATE TIME PERIOD~
N1*AC*ARTHUR JONES*1*9012345918341*11*SE~
N2*MAGGIE MCGILLICUTTY*BEVERLY SIMPSON~
N3*PO BOX 123*157 WEST 57TH STREET~
N4*CINCINNATI*OH*43017*US*IP*02535~
PER*AC*ARTHUR JONES*TE*(614)555-1212*TE*(614)555-1212*TE*(614)555-1212*ADDL CONTACT DESC~
PO1*AAA849*100*DP*100.00*PD*AB*ASSEMBLY*BL*BRAND OR LABEL*ON*CUST12345*AB*ASSEMBLY*BL*BRAND OR LABEL*ON*CUST12345*AB*ASSEMBLY*BL*BRAND OR LABEL*ON*CUST12345*AB*ASSEMBLY~
PID*F*BC*AX*UNSCENTED*GENERAL PURPOSE*BS*SUBQUALIFIER*N*ENG~
CTT*10*855*12345678*HB*40*QD*GENERAL PURPOSE~
SE*12*0001~
GE*1*1~
IEA*1*000000001~

Converting from Wrapped to Unwrapped and back

When you need to do this, you need to do this fast, not manually.  Sure the manual way works with small files, but if you are trying to edit a big invoice or order, you will want to do something snappier than walking through the file segment at a time.  Here are two ways to do it with common tools.

VI

To unwrap:

1. open the file.  Type “vi [file name]“
2. Hit “:” to get the command prompt
3. Type “%” to do a every line replace. (may not be needed as this is wrapped, but it doesn’t hurt.)
4. Type “s” for a substitution.
5. Type “/” to begin the string to find.  (you may need to escape characters like ~ and | with a \.  I will show this in the example)
6. Type the delimiter to be replaced.
7. Type “/” to begin the replaced string (again with the possible need to escape characters.)
8. Type the delimiter again, as we don’t want to lose it, then follow it will a [crtl]+v and [crtl]+m
9. Type “/” to end the substitution
10. Type “g” to make it global or indicating all of the instances not just the first one.

It looks like this:  “%s/[delimiter]/[delimiter]^M/g”

Put another way, here is what I typed for our example:  “%s/\~/\~^M/g”  (notice that I am escaping the ~ with a \)

To re-wrap:

1. open the file.  Type “vi [file name]“
2. Hit “:” to get the command prompt
3. Type “%” to do a every line replace. (may not be needed as this is wrapped, but it doesn’t hurt.)
4. Type “s” for a substitution.
5. Type “/” to begin the string to find.  (you may need to escape characters like ~ and | with a \.  I will show this in the example)
6. Type “\n” to designate the newlines.
7. Type “/” to begin the replaced string (again with the possible need to escape characters.)
8. Type “//” to begin and end the substitution.
9. Type “g” so that it will get any multiples of the newline.

It looks like this: “%s/\n//g”

MS Word

To Unwrap:

1. Open file.
2. Select Edit>Replace or press [ctrl]+h
3. In the “Find what” field insert the delimiter.
4. In the “Replace with” field insert the delimiter, and then click on the “More” button on the bottom of the control.
5. In more, there is another button called “Special”  Click on this button and select “Manual Line Break”
6. Click replace all and the EDI will unwrap.

To re-wrap:

1. Open file.
2. Select Edit>Replace or press [ctrl]+h
3. In the “Find what” field click on the “More” button on the bottom of the control.
4. In more, there is another button called “Special”  Click on this button and select “Manual Line Break”
5. Don’t put anything in the  “Replace with” field.
6. Click replace all and the EDI will re-wrap.

Other Tools:

There are lots of other tools that can do the search and replace with newlines, these are only two ways.  If you have one that you would like to share, please let me know.

—

Looking for something else relating to EDI?  Try here.

]]> http://www.theintegrationengineer.com/edi-wrapped-and-unwrapped/feed/ 0 http://www.theintegrationengineer.com/edi-elements/ http://www.theintegrationengineer.com/edi-elements/#comments Tue, 13 Jan 2009 16:44:18 +0000 Roy http://www.theintegrationengineer.com/?p=59 What is an Element?

In the context of delimited data files, an element is the contents between two delimiters.  Like in a spreadsheet, the element is the contents of a cell.  In EDI, we have delimited segments, and just like a delimited file, the contents between the delimiters is the element.

So in this segment “PO1*1*8*EA*10**SW*H11510.REF*CT*089480-00-B-0336~”, the 5th element or PO1_04 is “10″, while the 6th element or PO1_05 is empty.

Empty elements are not necessarily null.  Calling something null when the data type is know like an empty element can cause some data theory debates.  I don’t want to start one here, but may write about that philosophical discussion at a later time.

What types of elements are there?

What we are really talking about is the data type.  Each element in an EDI file has a specific data type defined.  Part Numbers are Alpha-Numeric which means that they may contain letters in the alphabet, and number in any mixture.  Where a price is not just a number, but is a currency.  Letters are not acceptable in the price field, so one can’t populate it with “N/A” in any case.

I won’t describe all of the data types here.  That is a discussion for some other time.  Most of the time these are fairly easy to work with, especially when you have samples to work from.  But there is another type of element.  This is the encoded value element.  This element does still contain a value, but it is in a code.

UOM or Unit Of Measure is the easiest to relate to, as it deals with something physical.  In the PO1 segment above, we find the UOM in the PO1_03 element.  In this case it means “each”  and by putting “EA” in this position we don’t have to type out “each” every time we just us EA as the abbreviation for this.

Why do we do this?  Well as I said the UOM is the easiest encoded element.  Some of the other encoded elements that are available define larger concepts.  Also, by having an encoded element, the EDI standard constrains the data to a rational type of data.

For instance, if you have an order response, we don’t want to have to process a free form response, we want to know if the product is accepted, back ordered or canceled.  And I want to know this in the same way for everyone.

What is Element Notation?

Elements are contained in segments.  The segment is referred to by the characters that preceded the first delimiter.  Each element in the segment has the name of the segment followed by its position in the segment.  Sometimes these are not separated, or they are separated by a space or underscore.  The position of the element is 0 based counting.  This means that the first position, the position preceding the first delimiter where the name of the segment is found, is position 0.

Thus in this segment, “N1^ST^Bill’s Grill^01^123456789~”, the N1 is N1_00.  I prefer the underscore to the space or no delimiter format for clarity.  But they are all used, so this would be the same as N101 and N1 01.  We use a two digit position designator even for positions less than 10 for clarity as well.  If there are any segments that develop more than 99 positions, there won’t be many.  And even fewer will ever be used.  And we don’t reduce it to just one digit, as N11 might actually exist, and that could cause confusion.

Now back to our example;  N1_01 is “ST”.  N1_02 is “Bill’s Grill” as so forth.

An example of how this is used is first found in our mapping rules that we need to write down.  And then in our notes and documentation.  If you wanted to say, “Whenever we have a ship to address for ‘Bill’s Grill’ we need to used the Duns number of ‘123456789′.”  We could say , “When N1_01 = ‘ST’ and N1_02 = ‘Bill’s Grill’ then N1_03 = ‘01′ and N1_04 = ‘123456789′.”  This way we are not leaving anything open to interpretation and clarity is our friend when writing rules for mapping and forming data.

You will see this used extensively in the mapping exercises here, and once you get the hang of using Element Notation, you will never write it any other way.

–

Looking for something else relating to EDI?  Check out the EDI Primer post

–

Files are called “Flat Files” when they contain a single data structure.  Generally this structure is the column and row structure like a spreadsheet or table, but a file in binary or encrypted with a single encryption key could also be called a flat file.  Files that are not flat; marked up files like XML or HTML, EDI files, other formats like HL7 or SEF files and others.  Here I am going to briefly discuss two flat file types; Delimited Files, and Fixed Width Files.

What is a Delimited File?

Ok, to describe it briefly, a delimited file is a file where the data is organized in rows and columns.  Each row has a set of data, and each column has a type of data.  If it sounds like I am describing a spreadsheet, you are right on the money.  To make the column, each row has the columns separated with a character called a delimiter.  See the example below.

Tables of data and spreadsheets are both similar to a delimited file in the way they organize data.  In the delimited file all of the empty space, or white space is removed.  What we see here is a classic example of exporting a spreadsheet table as a comma delimited file.  In theory, this data can be imported by any other application that can read a delimited file.

Believe it of not, a space is a character, and takes up space in a file.  Back in the day people went out of their way to save space so that files could be send over slow modem connections.

What is a Fixed Width File?

There is another type of file, is is called a Fixed Width or Fixed Position file.  It is different from a delimited file in that the data fields are defined by the character position.  See the example below.

In a fixed width file, the delimiter characters are eliminated.  If the data is formulated such that the data fields are the same size, this format can be more compact than a delimited file. You can see here that we know the size of the Birthdate data, so we eliminate all the spaces between the Bdate and Department fields.  If all of the data was formatted for size like this, we could really make this file small, so that it only contains the data.

We also eliminate the pesky problem of delimiters found in data.  The issue of a comma delimited file containing a field that has a comma in the data.  How does the parser know that this comma is not really a delimiter, but is part of the data?  Anyway, that problem is eliminated in a fixed width file.

Comparison

This is not a contest of which format is superior.  Both file architectures are useful and both are used commonly enough that you need to be at ease working with both.  Delimited files are really easy to work with as long as your data is clean of the delimiter character.  Doing quick integration of data common in ETL tasks, delimited files are far more common that Fixed Width.  Continuous operations of data integration and importation many times find that Fixed Width or Position files are more reliable for the unattended operation, even ETL if it is unattended.

As with many things in integration work, we want to pick the best option.  Knowing and working with both fixed and delimited files will help you determine which is the right choice for the task you have before you.