The Integration Engineer » EDI

Document Choreography of an EDI Purchase

Roy — Wed, 10 Feb 2010 15:36:20 +0000

So it goes like this:

850 is a Purchase order, and is sent to the Vendor

855 is an Order Response, and is sent from the Vendor. It confirms, updates or rejects each line, or the PO as a whole.

856 is the Ship Notice. and is sent from the Vendor. It may contain one or more lines from one or more orders and can indicate shipment of partial lines from a PO.

810 is the Invoice, and is sent from the Vendor. It contains one or more lines from one or more POs, but generally people keep these to only one PO.

820 is Remittance, and is sent to the Vendor. It contains information relating to payment.

Basics of EDI

EvilRobot — Tue, 18 Oct 2011 12:39:09 +0000

Today the Evil Robot has found a EDI training blast from the past. So if you want to experience the cutting edge of EDI training that people received back in the 90′s when power-point was all the rage and managers stopped being able to communicate without them, then this video if for you. But if you can get past the power-point nature of the presentation, this is a good basic over view of what EDI does, how it is used and covers many of the commonly asked questions by people first encountering EDI.

Of course for those who may be EDI vets, this video may be a challenge to sit through. But if you are like most people today, you came at using EDI sideways. That is to say, you inherited a system that used EDI, and are now maintaining it. So you didn’t start with the training or foundational knowledge of how EDI is put together. In this case, this is a good 11 minutes to spend to cover those foundation pieces and fill in the gaps.

Of course this is not a comprehensive video. If you have specific questions about EDI or its implementation, I would be happy to answer your questions directly, and I may even make a video for you.

What is SEF?

Roy — Thu, 19 Nov 2009 14:35:54 +0000

Starting off work on my On-line Status Repository, one of the things that I will be starting with is uploading and downloading SEF file from a data repository. SEF stand for Standards Exchange Format. SEF files are repositories of standards information that can then be exchanged between people and applications to define the format of EDI documents.

If you have used a standards editor, you probably know what an SEF file is, or have used it. Some applications and EAI even use SEF files as part of their document creation and validation processes. It becomes useful to describe briefly what SEF files look like, and what type of information they contain.

What is in there?

If you are familiar with EDI, and have cracked open an SEF file with your standard text editor of choice, you will already know what I am about to say. The contents of the SEF file don’t look that complex to someone who is used to and familiar with how various EDI files look. For those that are not so brave, I will explain this painlessly.

First, the SEF file is really just a text file. It can be edited by hand, but I don’t really recommend it. (not because you can’t, but because it is tedious.)

Second, it contains its data in sections. There is a section for doc types, segments, elements, and encoded data. There is also a few housekeeping sections like version and name etc.

What is an SEF file for?

SEF stands for Standards Exchange Format. This is literally a file format that was designed to contain information about EDI standards. It was created so that a computer application could understand an EDI file. It is used by some application like Standard Repositories and Standards Editors to allow us humans to deal with a standard like it was a text document. While at the same time keeping a repository that can be used by an integration application to form and validate an EDI file.

If you are familiar with the concept of metadata then an SEF file is the EDI standard’s metadata. If you are not familiar with metadata, then skip this part.

Do I need to know about SEF files and EDI?

Nope. SEF is only useful if you are exchanging standards using the format. If your standard and usage is in the form of a PDF or a spreadsheet, that is fine. Many people do just that.

For me, I need to know this because I am trying to build some Standard Repository tools, and want to use the SEF format to manage them.

And for those that want a deeper look, I will be following this article with some deeper and more detailed articles dealing with the internals of the SEF file.

Usage of EDI specifications

Roy — Thu, 05 Nov 2009 15:54:48 +0000

When two trading partners agree to send each other electronic documents. And they begin to describe what EDI documents they will exchange and how the documents will flow, they should also exchange EDI specification documents. EDI usage or specification documents describe what fields and what segments a trading partner will send or expect to convey the information necessary to complete a transaction. It doesn’t matter if we are ordering widgets, or invoicing, or transmitting catalog data, or checking insurance claims eligibility, the EDI needs to contain the data that the two parties need to communicate. To explain this, and document it to that both trading partners know what is expected, we create an EDI usage specification.

EDI Specifications

In the larger sense, the EDI specification is the set of rules that define each document type, the segments they contain, and the size and type of data in the elements. When we talk about the EDI Standard Specification, we are talking about the whole set of valid EDI document types. If something is valid EDI, then it complies with the EDI Standard Specification. However, this large, all encompassing specification is not useful in coordinating the exchange of documents between two trading partners.

What goes in an EDI Usage Specification?

So we take EDI Standard Specification and we reduce it. We remove the unused document types and the unused segments within the documents we will use. And we even remove the unused elements and encoded values within the segments we will use. At this point, we have created an EDI Usage Specification. We can also add to the specification by including values that we need, like designating the an optional value or ID as required and specifying what type is should be.

How is an EDI specification Used?

Now that we have a usage specification we can use it to do to basic things. First, it is the source of information for us to set-up our integration for our EDI document exchange. And Second it is the format that we validate our EDI documents with to determine a valid transaction.

EDI Usage Specifications can be a source of data and integration documentation. This can become extremely valuable when the choices and information about how your EDI interface works is recorded into the notes of the Usage Specification.

Beyond a visual validation of looking at the EDI file and comparing it with the specification, many times a specification can be found in the form of an SEF file. The SEF file can be used in an EDI validation application. This allows a potentially large and unwieldy EDI file to be scanned for compliance and accuracy.

There may be other uses, but these are the main two, integration data repository, tool for EDI validation.

EDI Specification Examples:

To walk through how a usage specification works, and is used, check out this review of BookNet Canada’s specification.

Then you can take a look at the EDI specifications published by Macy’sNet that they keep online to assist their trading partners in making and keeping effective integrations.

Sometimes we only use the EDI specification, they way we have always used them. It can be instructive to browse the usage specifications of other companies. We see what they remove, what they include, what comments they make, and what rules they document. And seeing how others do things can help us to be more effective in how we use EDI specification documents.

Line Item Part Three, The ACK

Roy — Fri, 23 Oct 2009 17:03:57 +0000

ACK is an Acknowledgment segment. It is commonly found on response transactions to Purchase Orders or 855. It is one of the lines found in the PO1 loop or group. As this line modifies the PO1 line data from the Purchase Order, what it can contain and how that data is handled is important for the stability of the supply chain integration. For clarity, we are going to use the 4010 x12 standard.

Parts of ACK

Here is a diagram of a sample ACK segment:

Like any other segment in EDI, the ACK segment is composed of elements. Here is a list, element by element and what it contains.

ACK_01	This is the Line Item Status Code. This code can be the entire purpose of the ACK line if it indicates that the line is cancelled, or it can indicate that something has changed. The Diagram above has a Status Code of “IP” indicating that the price from the Order has changed. The ACK should then reflect the changed price here. In the case of an 855 that indicates the part number has changed or substituted the status will be “IS” and the new part number will be listed in the Product or Service ID element. The PO1 line that this ACK is a part of should contain the original part number, price, quantity, or whatever was on the order, and the ACK will contain the modification.
ACK_02	This is the quantity modified. This is not required, but is very important. If the order was for 10 items, and there is a change, substitution or delay in 5 of them, a PO1 group should contain 2 ACK lines one with the accepted status on the 5 that are ok, and another for the 5 that are
ACK_03	This is the Unit of Measure. It is a 2 character, Alpha-Numeric value. It is also an encoded value. Being “encoded” means that there is a list of acceptable values for this element, any value outside of that set is invalid and will cause an exception.When the ACK_02 and ACK_03 are used together, we can tell how much of something is being modified or acknowledged
ACK_04	This is the date Qualifier. This is not required, even if ACK_05 is present, this element may be empty. When it is present, it is an encoded value that identifies what type of data is in the ACK_05. In the case of our example, it is the PO date.
ACK_05	This is a Date element. It is required if ACK_04 is populated. The format is CCYYMMDD. This element is always 8 characters long when it is populated.
ACK_06	This is a reference number. I have never seen this used, but if you have a use for it, go ahead.
ACK_07	This is a Product or Service Qualifier. This element is a 2 character, encoded value. All of the “Qualifier” defined elements are encoded values. This, again, means that there is a list of acceptable values for this element, any value outside of that set is invalid and will cause an exception.This qualifier defines what type of value is found in the next element, the ACK_08. So if the ACK_07 has a “VC”, then the ACK_08 contains a Vendor’s Catalog Number. (This means it is a part number as found in the Vendor Catalog.)
ACK_08	This is a Product or Service ID. It is Alpha-Numeric, and can be from 1 to 48 characters in length. It is conditional with the ACK_07, if one is present, the other must be present. This value is an identifier like a part number or other type of line item data.
ACK_09 to ACK_26	This pattern continues all way to ACK_25 and ACK_26. All of the segment pairs from ACK_07 to ACK_26 that have the same potential values for each pair.
ACK_27	This is Agency Qualifier Code. It is an 2 character encoded value to designate what agency, state or entity is being referenced. It is not required and I have never seen this used.
ACK_28	This is Source SubQualifier. It is an 15 character Alpha-Numeric value that references the source table being used. It is not required and I have never seen this used.
ACK_29	This is Industry Code. It is an 30 character Alpha-Numeric value that references an industry code list. It is not required and I have never seen this used.

Best Practices

With the ACK the best practices are more in how to implement and what to expect. So here are some things to remember when implementing this segment.

An ACK is part of a PO1 line for a response to another PO1 line transaction. This segment supplies a status, and can modify the line item contents.
There can be more than one ACK segment for a line item. (4010 855s allow 104 repetitions of the ACK)
When an ACK explicitly changes the line item it can only change that one type.
There is a status called Accepted With Changes. “AC” For this, you will need to parse the ACK and see if any of the values have changed.

Line Item part two, IT1

Roy — Thu, 15 Oct 2009 14:56:23 +0000

Like the PO1 segment, IT1 is another line item segment used in the supply chain. This is the line item segment found in invoices. And since invoices make up an important part of a supply chain, I wanted to spend another article on this segment. (IT1 is a group or loop so it is more than just a segment.) In this article we will discuss what data is found on the IT1 segment, and how it should be handled on both inbound and outbound document. For clarity, we are going to use the 4010 x12 standard.

Parts of IT1

Here is a diagram of a sample IT1 segment.

Like any other segment in EDI, the IT1 segment is composed of elements. Here is a list, element by element and what it contains.

IT1_01	This is the Line Identifier, or Line Number. This is a value that must be unique among the lines or IT1 segments of a Invoice. There is no rule that these must be incrementing integers starting at 1. But it is normal for these to be Numeric even if the type is Alpha-Numeric. And it is general practice to have them reflect the Line Number of the Purchase Order they are invoicing. Because this is some times not possible, the IT101 is optional. All effort should be made to include it, but for lines that have no number from the PO associated with them, it is sometimes a good practice to just leave this blank. When handling IT1 lines in invoices produced by others, it is essential that you verify the line item with a secondary value other than just the line number. Using a part number or “Product or Service ID Value” to match against the line is a good best practice.
IT1_02	This is the quantity invoiced. Most of the time people will just call this the quantity, but there are places and times, (like in an invoices) that quantity ordered and other quantities have a distinction. The quantity is Numeric integers only. Invoicing fractions of something is not supported. The IT1_02 is conditional with the IT1_03 and IT1_04, if one exists all others are required.
IT1_03	This is the Unit of Measure. It is a 2 character, Alpha-Numeric value. It is also an encoded value. Being “encoded” means that there is a list of acceptable values for this element, any value outside of that set is invalid and will cause an exception.When the IT1_02 and IT1_03 are used together, we can tell how much of something is being Invoiced. 1 CA (1 case) is not the same as Invoicing 1 EA (1 each) of an item.
IT1_04	This is the Price. This is a Numeric, but can be decimal it is not strictly currency formatted, so it is possible to have item prices precise past $0.01. This is the price of a single item, so we would multiply the IT1_02 and the IT1_04 to arrive at the cost of the line item. (We might also have a additional charges or tax on a line item in another segment within the IT1 loop. If this is the case, that cost would also be included in the cost total of the line item but not multiplied by the quantity.)
IT1_05	This is the Price Basis Code. This is optional, and most of the time is not used. If it is used, it denotes where the IT1_04 price was derived from.
IT1_06	Is a Product or Service Qualifier. This element is a 2 character, encoded value. All of the “Qualifier” defined elements are encoded values. This, again, means that there is a list of acceptable values for this element, any value outside of that set is invalid and will cause an exception.This qualifier defines what type of value is found in the next element, the IT1_07. So if the IT1_06 has a “VC”, then the IT1_07 contains a Vendor’s Catalog Number. (This means it is a part number as found in the Vendor Catalog.)
IT1_07	This is a Product or Service ID. It is Alpha-Numeric, and can be from 1 to 48 characters in length. It is conditional with the IT1_06, if one is present, the other must be present. This value is an identifier like a part number or other type of line item data.
IT1_08	The IT1_08 has the exact same definition as the IT1_06 with the exception that it defines the type of value is found in the IT1_09.
IT1_09	The IT1_09 has the exact same definition as the IT1_07. This is a Product or Service ID. It is Alpha-Numeric, and can be from 1 to 48 characters in length. It is conditional with the IT1_08, if one is present, the other must be present. This value is an identifier like a part number or other type of line item data.
IT1_06 to IT1_25	This pattern continues all way to IT1_24 and IT1_25. All of the segment pairs from IT1_06 to IT1_25 that have the same potential values for each pair. In earlier version of EDI, like 3010, the IT1 segment only had 23 elements. Or in other words it was one Product or Service pair shorter that what we have in 4010.

I wrote a section on Line Item Good Form and Best Practices in the article about PO1 Line Items. Instead of repeating that I will just let you hop over there are read it.

IT1 Loop Segments

As we said in the beginning, the IT1 is a segment, and a loop/group. And as you may have noticed, there is data that you may need that is not covered in the elements available in the IT1 Segment that belong to the line item. Values like Tax information, found in the TAX and TXI segments. These and other values are found on the line item, but are not on the IT1 segment.

Line Item Part one, PO1

Roy — Fri, 02 Oct 2009 16:40:28 +0000

When we are dealing with supply chain or e-commerce related EDI, we probably will deal with line item data. On a Purchase Order, (PO) or 850, the line item data is contained on the PO1 segment or the PO1 group. (PO1 is a group or loop so it is more than just a segment.) In this article we will discuss what data is found on the PO1 segment, and how it should be handled on both inbound and outbound document. For clarity, we are going to use the 4010 x12 standard. The PO1 hasn’t changed in content for a while, but it has gotten longer, (you will see how this works when we talk about the PO1 elements).

Parts of the PO1

Here is a diagram of a sample PO1 segment:

Like any other segment in EDI, the PO1 segment is composed of elements. Here is a list, element by element and what it contains.

PO1_01	This is the Line Identifier, or Line Number. This is a value that must be unique among the lines or PO1 segments of a purchase order. There is no rule that these must be incrementing integers starting at 1. But it is normal for these to be Numeric even if the type is Alpha-Numeric. And it is general practice to have them increment in some fashion. Thus having the first line item number as 1. The second as 2, and so forth. I have also seen the first line item being 10 , the second 20 and so forth. As well as 001 and 002. In any case, I would discourage using random numbers. And I would discourage using Alpha-Numerics. These may confuse your trading partners and cause them to have exceptions that there is no need to have. On the other hand, when planning on handling line numbers from other parties, you should expect that they might not be using nice incrementing numbers. This will prevent getting exceptions in your system when your trading partner does something odd.
PO1_02	This is the quantity ordered. Most of the time people will just call this the quantity, but there are places and times, (like in order responses, invoices and shipping notices) that quantity ordered and other quantities have a distinction. The quantity is Numeric integers only. Ordering fractions of something is not supported. The PO1_02 is conditional with the PO1_03, if one exists the other is required.
PO1_03	This is the Unit of Measure. It is a 2 character, Alpha-Numeric value. It is also an encoded value. Being “encoded” means that there is a list of acceptable values for this element, any value outside of that set is invalid and will cause an exception. When the PO1_02 and PO1_03 are used together, we can tell how much of something is being ordered. 1 CA (1 case) is not the same as ordering 1 EA (1 each) of an item. Generally this is not a problem. The data in the catalog is accurate. When there is a problem, it is important to go back to the PO to find out exactly how many items were ordered and in what UOM.
PO1_04	This is the Price. It is optional, but most of the time is present. It is required if the PO1_05 is present. This is a Numeric, but can be decimal it is not strictly currency formatted, so it is possible to have item prices that precise past .01. This is the price of a single item, so we would multiply the PO1_02 and the PO1_04 to arrive at the cost of the line item.
PO1_05	This is the Price Basis Code. This is optional, and most of the time is not used. If it is used, it denotes where the PO1_04 price was derived from.
PO1_06	Is a Product or Service Qualifier. This element is a 2 character, encoded value. All of the “Qualifier” defined elements are encoded values. This, again, means that there is a list of acceptable values for this element, any value outside of that set is invalid and will cause an exception. This qualifier defines what type of value is found in the next element, the PO1_07. So if the PO1_06 has a “VC”, then the PO1_07 contains a Vendor’s Catalog Number. (This means it is a part number as found in the Vendor Catalog.)
PO1_07	This is a Product or Service ID. It is Alpha-Numeric, and can be from 1 to 48 characters in length. It is conditional with the PO1_06, if one is present, the other must be present. This value is an identifier like a part number or other type of line item data.
PO1_08	The PO1_08 has the exact same definition as the PO1_06 with the exception that it defines the type of value is found in the PO1_09.
PO1_09	The PO1_09 has the exact same definition as the PO1_07. This is a Product or Service ID. It is Alpha-Numeric, and can be from 1 to 48 characters in length. It is conditional with the PO1_08, if one is present, the other must be present. This value is an identifier like a part number or other type of line item data.
PO1_06 to PO1_25	This pattern continues all way to PO1_24 and PO1_25. All of the segment pairs from PO1_06 t PO1_25 that have the same potential values for each pair. In earlier version of EDI, like 3010, the PO1 segment only had 23 elements. Or in other words it was one Product or Service pair shorter that what we have in 4010.

Good Form on PO1 line

EDI in Good Form is both style and best practice tips to avoid common exceptions. Paying attention to good form tips and best practices helps you create an integration solution that is not only complies to the standard, but functions to help you and your trading partner have an unnecessary support burden due to the avoidable unexpected.

1. It is bad from to have PO1 Product and Service pairs that are empty. This relates to not having any trailing element delimiters. It also goes along with not having empty pairs in the line with populated Product and Service element pairs following. Thus it is best to build your mapping so that optional data doesn’t create empty elements when it opts out.

2. When reading a PO1 line, it is best to assume that an expected Product and Service value could be in any pair. So the map needs to walk down the pairs of elements until it finds the qualifier that indicates the value you are looking for. This allows you to use the same input map with multiple EDI producing trading partners that may have interpreted the priority of data in different ways, and avoid exceptions.

3. When producing EDI with a PO1 line, it is best to be as consistent as possible with your placement of the Product and Service data. Create your logic and mapping so that mandatory data occurs early in the PO1 line, and optional data is moved to the end. This will keep your required data from hopping around positionally. This will help your trading partners that may not have had the foresight to follow Good Form Tip number 2 from having exceptions that slow down you process.

PO1 Loop Segments

As we said in the beginning, the PO1 is a segment, and a loop/group. And as you may have noticed, there is data that you may need that is not covered in the elements available in the PO1 Segment that belong to the line item. Values like Item Description, found in the PID segment. Or Tax information, found in the TAX and TXI segments. These and other values are found on the line item, but are not on the PO1 segment. We will have talk about some of these other segments and where to find other common data on another day.

EDI Standards Software

Roy — Thu, 24 Sep 2009 13:27:28 +0000

I have been studying some on the SEF format. SEF stands for Standards Exchange Format. This is a file that defines the EDI standard so that you can use a validation or standard and usage document editor to create nice, and clean specifications for you and your trading partners.

I have used Foresights EDISIM, but I wonder what others use. I have found a few links to SEF software, and will list them below.

Software I have found or used:

Foresight EDISIM (Used several version of their software and liked them all.)
EDIdev has an SEF Manager (haven’t used it but the screen shots look like files have have received from trading partners in the past. Also, looks a bit dated, but it may just be me.)
Stylus Studio (claims to handle SEF and EDI, but is very XML focused on their sight. Trying out their demo now, and will update this page when I am done.)

What do you use/like?

This is not a comprehensive list. I know there are people out there that use something else. Maybe even better. If you do, please post it in the comments. Particularly, I am interested in any online, SEF or EDI standards tools. I haven’t found any, and have been thinking about writing one. So if you know of an online EDI Standard or SEF tool, please let me know.

Dot Net EDI framework

Roy — Thu, 10 Sep 2009 14:05:33 +0000

A reader asked me a question, and I want to pass it along: “Do you know of any DOT NET frameworks for working with EDI?”

I didn’t, and a search of the web didn’t reveal anything that I could, in any confidence, recommend. So I bring this question to you. Do you know of a DOT NET framework that can be used for generating and or parsing EDI data? If you do, please comment below and let everyone know. I will also pass it on to my friend who is looking for this.

Development Frameworks

The reader that I mentioned asked about Dot Net, but there are other times and environments where companies need to setup and EDI interface for their implementation. I have spent most of my career working with enterprise level applications to handle the EDI transportation and transformation. But I have seen a few times when it would have been useful to code an interface for and EDI file onto a stand alone application. But I am not a developer in that sense. Hoping that some of you may be, what are some EDI frameworks or modules etc that you have used or know about? Java, C++, VB, or even perl, python, tcl, php, etc. Please leave a comment with the framework or tool that you know of. You can say a few words of eveluation if you can, or just leave a link to it and I will look them over and try to post about them over the next few weeks.

EDI and B2B tools

This brings up another point. I have been trying to add to my Toolbox page as I have gone along. I have posted a few tables and spreadsheets. I also had a post about tools that an effective integration engineer should use. I plan on posting more tools in my Toolbox too. And I have a reall love of free tools. But that is not the only kind of tool that I use, or will recomend. If you know of a tool or technology that you think others should know about, or might help them, post a link, or drop me a line and I will add it to my Toolbox or write a post about it. (or both)

So bring me your suggestions and I will share them with others and write an article about them when I can.

Thanks

BooknetCanada: Real Life Standards and Usage Spec

Roy — Tue, 25 Aug 2009 14:32:31 +0000

From time to time I am asked if I know where such and such standard can be found. This is more than idle curiosity. People are trying to integrate, or update integrations, and the standard documentation is not available or has been lost. From the perspective of an integration engineer, such standards are more precious than gold. I keep a collection of the standards that I have used over the years, (something that I recommend) but what I really encourage is for companies to publish their standards in a publicly accessible location. Booknet Canada is an example of a company that has done this. Lets take a look and see what it says….

Here is the standard, if you want to look at what they have published. I will crop out a few pieces to talk about. You might want to look over the full published standard.

ISA

The ISA segment contains some essential information that your trading partners will need. Integrations of EDI rest on accurate trading partner values in the ISA values. Sometimes even trading partners with existing connections will need to verify the ISA information and trading partner IDs. For new trading partners, having this information available makes it possible for proactive partners to get a jump on integrations.

One thing of note here is their use of a sample segment to illustrate what the ISA segment will look like. This can be very useful especially when dealing with trading partners that lack EDI experience. Also useful when dealing with unusual segments or difficult to describe structures.

However, there is one caution. I once had a trading partner hard code all of the sample segments into their map. Thus we had an N1 with their ID and other information, and then an N1 with the same qualifiers but with the sample data from our specification. This caused some problems for both of us. Admittedly this only happened the once, but I was careful after that. In any case, be sure that your examples are clearly marked as such to avoid repeating this mistake.

BIG

Every document has a beginning segment that is specific to that document type. This segment will hold specific information that relates to that document type.

There are two things of note here. First, is that they have listed the “not used” segments in their specification. I don’t always to this myself but it can be done, and in this case illustrates one of the aspects of EDI good form practices. In the example segment they have none of the trailing elements that are not used, but they do have the one filed that is not used, but comes before a used element. Thus illustrating the “no trailing delimiters” best practices rule.

The Second thing of note is their use of “Notes” to leave indication of a precedence rule in their mapping. You can see here that they have a rule that uses the BIG_04 as the purchase order number only when there is no superseding PO number on the IT1 segment. This type of notation is really helpful to trading partners and others in your own company down the road when problems happened and people need to discover what went wrong.

N1 segments and N1 groups are important segments and can contain valuable data. N1 segments are found in every EDI document that I have dealt with.

You’ll notice that there are three N1 segments listed in this Standards Document. We know that the N1 can repeat, but this is not the way that is indicated. This is one of the ways that the Standards Specification and the Usage Specification differ. In the Standard, we talk about what may happen, or what may occur. But in the Usage Specification, we talk about what should happen or what will occur.

All of the looping or repeating segment values have not been shown in this document. The looping of the IT1 and N1 still happen, but the looping is not pertinent to the Usage Specification here. (at least to BookNet) They have also included a short list of the acceptable code qualifiers to be used in the N1_03. And you will note that they have defined them so that trading partners won’t have to look them up, or guess, and have indicated their preference.

Standards

Crafting a Standard and Usage Specification can make a real difference in the ease of boarding trading partners. It can make future trouble shooting easier for your, and for others both at your company, and at the trading partner’s. And it can illustrate the look and feel of EDI to less EDI sophisticated individuals.

Now that you have looked over the BookNet specification, is there anything that you think they should have changed to add clarity? Or are there any thing that you have learned that you want to add to your own Usage Specification documentation?

Validate to Standard, not to spec

Roy — Thu, 20 Aug 2009 13:31:08 +0000

After you have created your usage specification, it can be useful to use a validation tool to check to verify that new trading partners comply during your boarding process. I have used Foresight’s EDI Analyzer for this many times, and it lets be quickly see where the EDI file departs from the specification.

There is a temptation to use this same validation in the production integration. But this would be a mistake. I’m not saying, “Don’t validate.” I highly encourage validation on both standard compliance and required data validation in mapping and integration. But to use the usage specification has a side effect that I witnessed once. (Only once.)

Validating Transactional EDI to Usage Specification

We had an integration requirement once that required that we only allow EDI elements that were in our usage specification. This was outside of our normal process and we all soon learned why.

When we validated EDI files based on the standard, we would error out documents that were corrupted, or in violation of the full EDi specification. (having mangled segments, or invalid UOMs and such.) But this one time, we thought it would be a good idea to also fail documents that had any extra data that we weren’t going to use. So we pulled all of the segments and elements out so that we were just down to what was in our usage specification.

This worked just like we told it to. As when the next file came in that was in bad form with extra trailing delimiters on the line item, it failed. And then the next with an extra segment that we didn’t care about? Well it failed too. As so on.

We soon were buried under failed EDI files that were valid EDI, and did contain the data we needed, but simply had something extra. And we were calling them all exceptions.

At first we added in a few of the common segments that were being sent. This relieved the lion’s share of the errors. But it still cropped up from time to time. Each time we would have failures for this reason we had to explain why. This grew tedious and embarrassing.

Eventually we re-mapped the doc-type using the complete standard in our validation. Magically all of these errors went away, and so did the embarrassing explanations.

Here is the way the process should work, generalized.

1. EDI envelop is validated against EDI standard.
2. EDI document is validated against EDI standard
3. Mission critical values are validated against Usage Specification.
4. File is processed.

What this does is to establish an inherent hierarchy of exceptions. Transactions that fail on step 1 are all falling because the envelope is corrupt or in error. Also, Transactions that fail on step 2 are failing for a general EDI reason. Both of these steps will have exceptions that can be handled by anyone with EDI knowledge. Operational and Application rules don’t necessarily need to be known to understand how to handle errors at these levels.

Step 3 and 4 are exactly the opposite. Errors that happen because of mission critical data that is missing or in error must be handled by someone that knows what the mission critical form is supposed to look like. And files that pass validation but fail later in processing may need to be corrected and the validation process updated. Or they may point to an error in the application itself. In any case, much more knowledge of the whole process is required at these stages.

Take Away

First, by sharing this experience I hope that everyone can appreciate that sometimes requirements make us to funny things. But I also wanted to share the approach to validation that I have found most helpful. I have seen implementation that don’t do any EDI or Enveloping validation. These are just as dangerous and frustrating as my experience. But now all of the errors are pushed down to step 3 and 4.

I’d also like to invite you to subscribe to my newsletter. It comes out once a month, starting at the end of this month. I’ve got some great ideas for it, and think it will be terrific.

EDI Standards Reference

Roy — Tue, 11 Aug 2009 16:25:23 +0000

I have sitting on my desk, a very expensive book published by the ASC that contain the guidelines for the X12 3010 EDI standard. I have rarely used them. The EDI standards tool that I have used the most and can highly recommend is EDISIM from Foresight. The last version that I used was version 5.0 and they are on version 6.8 as I write this post. But for a long time, Foresight has nailed the conceptualization of working with EDI standards.

One of the things that in invaluable is the ability to share your standard documentation both internally and externally. EDISIM is a tool that has allowed me to do that. They produce a very professional looking document for external consumption and the standard is exportable so that everyone using EDISIM can share the same standards file in a format called SEF.

Features of a Good Standards Reference

Navigate Standards (multiple versions, doctypes, segments, etc)

A Standards Reference needs to offer a navigation interface of some kind. You need to be able to see the versions, standards and document types. You will also want to drill in and view the reference outside of the document types, and browse the segments or data element definitions. But at the core, you need to be able to see a graphical representation of the standard, view the format and see the structure of how data will be represented in the EDI format and document.

Edit Standards

Just browsing the standard is not going to get the job done. You need to be able to make alteration so that you don’t just have the high level X12 standard, like the book on my desk, but can make a standard that is your usage specification. This means you will need to be able to do two operations on the standard.

First you will need to make changes to what elements and segments are displayed, and make changes to the encoded values that these elements support. Editing these structures will aid you when you use the standard to validate and check for compliance based on your usage specification, and when creating documents to share and publish.

Second you will be able to add some of your business rules documentation to clarify the default definition of the standard. This is especially useful if you need to pass some data that uses a mutually defined qualifier. This lets you define what the “mutually defined” value should be. This again helps when you publish and share your standard.

Export and Import (Read SEF)

All of the standards references that I have used had the ability to import and export a standard in an SEF file format. SEF or “Standard Exchange Format” is a file that holds the standard dictionary and format for EDI files. This is a nice way to not only transfer and store the standard your company uses so that everyone has a copy of the same usage implementation. But if no proprietary information is stored in the notes, it can be sent to trading partners out side of the company to aid in their integration efforts. Any other export and import is a bonus, but SEF is a requirement..

Print (export to doc or pdf, or printer)

In spite of all this paperlessness, some times we still need to print things out. Printing to a pdf or printing to paper is useful when sharing the standard and usage specification in a non-editable format so that others can view it, and make use of the information. So for managers, or clients, or other parties that only need a viewable format to follow, this feature is a must have.

Save as editable file (RTF or other.)

Ok, just like printing, sometimes companies want all of their documentation to look the same. To facilitate this, it is nice to be able to save the specification out to a rtf, or doc, and then add the company header, logo, watermark, etc. This allows you to take the standards reference and really own the edits and notes that you have added. For companies that have extensive documentation processes, this feature is a must have.

Other Good Reference Features

Validation

Validation is not a feature of an EDI standards reference, but it is a good tool that sometimes comes with them. Validation tools read and EDI file, and compare the contents with the standard. They should then identify all of the errors they can see and report on them.

In my experience validations may need to be run repeatedly as some errors in an EDI file will mask following errors. For instance, if the N1 segment is wrong, the validator may not be able to tell if all of the other lines in the N1 loop are valid. So when I validate an EDI file, I correct the errors seen on the first pass and re-validate. This has saved me from telling a trading partner to fix an issue, only to have to tell them about another issue later. After all, we do want our integrations to go quickly.

Test Data

Sometimes looking at the standard is just not enough to show us how things will look when formed into the document. Seeing sample data is essential, and one of the first things I request at the beginning of any integration project. A Standards Reference will sometimes have tools that will create a test data EDI file based on the standard. This can be useful if you are working with a new document type that you haven’t encountered before and need to see how the standard looks in the formed document.

Evaluation of Standards Reference Technologies

Is there a feature that I have missed? Please let me know. Also,what EDI standards reference tools have you used? What do you like or dislike about them? How about any non-EDI specific tools? Please let me know what you have used. I would like to compile a list of tools and run some comparisons of what is out there.

The Parts of EDI

Roy — Thu, 06 Aug 2009 16:12:40 +0000

As part of our series “EDI Primer” we are going to go over the various parts of EDI. EDI is neither truly delimited, nor fixed width. EDI has some fixed position components, and uses delimiters in a different way than a truly delimited file. This post it going to overview some facts about EDI and its parts in an introductory fashion. If you are unfamiliar with EDI, you might want to start with the post, “What is EDI?”

Structure

Two important concepts in EDI are not exactly parts. They are the structures of Enveloping and Segments:

Enveloping, as we discuss in another post, is a way for an EDI file to be self determining. In the first segment of an EDI file, the ISA segment, all of the information can be found to allow a system designed to interpret EDI to know where it is going, where it came from, what version it is, and what characters it is using as delimiters.
Segments following this first segment are not fixed position. White space is not used unless it is part of the data in the element. Each segment has a header element that indicates the definition used for the rest of the data elements in that segment. A segment may not need to use all of the possible elements in its definition. When this is the case, the segment terminates with the segment delimiter or terminator.

The parts:

Standard is a concept that has more than one meaning, There is the standard, that is the overall X12 compliance rules that regulate what is and may be an EDI document. But there is also the concept of a specific use case, or usage specification. Both of these are referred to as ‘Standards’ but in this description of the parts of EDI, the standard is the first concept. This is that when an EDI document is parsed, it must comply with this standard of segments, delimiters, and format, or it is rejected as being corrupt.
Version is a sub-component of the standard. It is declared in the ISA, and lets the parser know what version of the EDI standard will be contained inside the envelope. This is important as to format and structure of the data that has changed as technology and needs have matured.
Doctype is not declared in the ISA, but is declared in the ST. We may also know when we see the functional declaration in the GS. The doctype determines which segments and in what order and structure may be present. Not all EDI documents will need all segment types, so the doctype is very important when parsing the EDI file. This is covered more in the ST Enveloping section.
Segments are the most basic part that makes and EDI file and EDI file is the segment. There is a more detailed discussion of the nuances of segments here. Segments are defined by their type, the characters that are located before the first delimiter in the segment. And segments are terminated by a special delimiter called either the segment delimiter or segment terminator.
Element exist inside the segment. Elements hold the data. Elements are separated by element delimiters. On rare occasions some elements can have sub-elements. There is a more detailed discussion of elements here.
Envelope Segments There are three pairs of special segments that must be found, and can only be found, in the enveloping structure. Enveloping contains the origin and destination identification, as well as content definition identification values. A more detailed discussion of enveloping can be found here.
Files are not really a part of EDI, but when EDI is formed, it is generally saved in a file. The file can contain any mix of EDI document types and going to any number of destination. Generally, for sanity’s sake, this is not done. But there is nothing in the name of the EDI file that matters to the contents. There is no really expected extention like “.edi” for instance, and there is not need for white space removal from the begining or end of an EDI file.
Documents are another matter. An EDI document is the sebments that reside within the ST envelope. As such, they are a set of data that has a single format, (EDI standard), a single sender and single receiver. Thus when an EDI doscument is discussed, we are really discussing the payload of the EDI envelopes.

–

Looking for something else relating to EDI? Check out the EDI Primer post

Next in our series is a more in-depth look into Segments.

–

EDI Repeated Segments

Roy — Sun, 28 Jun 2009 17:13:59 +0000

In EDI there is a construct called a Loop. But this is not like you think of a loop using an while statement. EDI is a file format, not a programming language. When we speak of Loops in EDI, what we are really talking about is repeated structures, or segments.

EDI is not a flat file, but it does have repeated lines or segments. Some times this repetition comes in groups and sometimes its just a line that get repeated in part of the document. Repeating structured data is fundamental to EDI, but can become difficult to interpret at times. Understanding the types of repeating structures and when and where they may occur can save a lot of time.

Repetition

There are three kinds of loops/repeated segments; First there is a simple segment that can appear more than once. Lets call this a Repeating Segment. Second there is a group of segments that appear together, and may repeat. Lets call this a Repeating Group. And finally there is a segment or group that repeats within a beginning and ending loop envelope. Lets call this a Repeating Envelop.

Repeating Segments

A repeating segment is pretty much just what it sounds like, a segment that may occur more than one time. A good example of this would be the AMT segment. The AMT is a segment that contains a monetary value. The AMT_01 declares what type of value it is, and the AMT_02 has the amount of money. In an EDI document like an invoice, we may need to have many values, subtotals, taxes, base price, and so on. Having a repeating segment lets us put all of these values on their own independent segment to be interpreted and included as best fits the Trading Partners and the nature of the transaction.

Here is an example of an unwrapped N1 group where the N3 is repeated:

N1*AC*ARTHUR JONES*1*9012345918341*11*SE~
N2*GEORGE WASHINGTON~
N3*157 WEST 57TH STREET~
N3*Suite 2001*5th Floor~
N4*CINCINNATI*OH*43017~

When segments are repeatable, they are indicated with a “Max Use” value of greater than one in the standards and usage specification. In the N1 Group, the N2 and N3 segments may have a max use of 2.

Repeating Groups

A repeating group is also what it sounds like. It is a group of segments that work together and get repeated. This loop is generally referred to by the primary segment. This segment is required in this group. This means that if this group occurs, this segment must also occur.

Most Repeating Groups are contextual. This means that they are defined by the segments that surround them, and are terminated by the first segment that is not in the group, or starting the group again. So a Repeating Group may repeat, but as soon as we move out of the segments in this group, we are done repeating it and move on to the next segments or groups defined in the standard.

The N1 is a good example. The N1 group contains name, address and contact related information. The only required segment is the N1. Subordinate to the N1 (that contains name and such) is the N2 (that contains secondary name or attention type data.) Then comes the N3 (with the street address) and then the N4 (with the city, state, country, zip).

Not all groups have the same letters, like ‘N’ and the N1 group is no different. It also contains the PER segment that contains a persons name, and contact info like phone number or email. Any PER that happens under an N1 is for that location.

In a document like an order you may want to have the address and contact for the shipper, the receiver, a biller, and a payer. There are many possibilities, and these are all repeated in this group. Here is an example of an unwrapped N1 group where the that is repeated:

N1*ST*MRS JONES*1*123456789*11*SE~
N2*BEVERLY SIMPSON~
N3*157 WEST 57TH STREET~
N4*CINCINNATI*OH*43017~
N1*SF*MR JONES*1*9012345918341*11*SE~
N2*MAGGIE MCGILLICUTTY~
N3*1001 MAPLE STREET~
N4*PITTSBURGH*PA*86753~

Segments that are in a repeating group are listed as being in a Loop. They may be repeated whenever the loop is repeated as specified in the standards and usage specification. Thus you may have a max use of the N3 as 2, and an N1 Loop that has a max use of 1000. So really you could see 2000 N3 segments, (2 for each of the 1000 possible N1 loops).

Repeating Envelope

A repeating envelope makes reference to EDI Enveloping. Basically, the ISA-IEA envelope can contain one or more GS-GE envelopes. And each GS-GE can contain more than one ST-SE envelope. These levels of enveloping are covered in greater detail on the EDI Enveloping page. But it doesn’t have to stop there. In the 3050 850 document there is a loop called LS-LE that contains terms. This section happens inside the ST-SE. And there is nothing to prevent other sub envelops from happening where there is a need.

What is fundamental to these types of loops is that they have a designated beginning, and ending. They must have both. Repeating Groups end when a beginning of a new iteration of the loop is found, or a segment that is not in the group is found. This can leave some ambiguity if the standard is not followed closely. The Repeating Envelope leaves nothing to chance and declares that it is done.

A note about Required segments

You may have noticed in the examples above or in the specification that we referenced that there are required segments in loops that are optional. This is not a contradiction and I will explain this here. A loop may be optional. If it is, then it may be ommitted entirely from a document while still compliant with the standard. If the loop is included, then the required and optional segments within that loop apply. And they apply in each instance of the loop. (We can’t have an second N1 loop unless we include the required N1 segment.)

Basically this is like scope in XML. The requirements of mandatory or optional only apply within the scope of the loop where they are defines. in the same fashion as sub nodes in and XML hierarchy.

Basic Fundamentals to Remember

Some segments can occur more than once, or can be repeated.
Some groups of segments can and often do occur more than once. These are contextual and are terminated by segments after the group in the standard.
Some segments are encapsulated in beginning and ending segments like enveloping. These segments can repeat, but only in another encapsulation or envelope.
The standard will indicate when repetition is allowed.

–

Looking for something else relating to EDI? Check out the EDI Primer post

–

EDI Enveloping Part Four (The ST)

Roy — Tue, 26 May 2009 16:00:44 +0000

The 3rd layer of the EDI envelope is the ST segment. The ST segment contains a Document Identifier, and a control number. The ST and its partner the SE segment define the beginning and ending of the Document. The SE segment contains a counter of segments within the document, and the corresponding control number to the ST. Relatively simple compared to the ISA and the GS.

Document Identifier

The document Identifier is not encoded. That is to say, for an 850 the ST document ID is “850″. For an 810, the Document Identifier is “810″ and so on. Thus if you open an EDI file, and don’t recognize the GS_01, skip to the ST_01 and it will tell you what standard you will need to find to read this document.

Good form would dictate that the GS_01 and the ST_01 would specify the same type of document. The GS would specify it on a functional level, and the ST on the standard level. Thus they would not be identical strings but would indeed convey the same message about what the document(s) would be. Some EDI translator or processing software will validate this relationship and some may not. But in the end consistency is the best policy.

Control Number

The control number is different for the ST than we saw in both the ISA and GS. The ST_02 is a 3 to 9 digit number. This means that it will start with 001 not 000000001. Of course you can still use a 000000001 but don’t be surprised when you see less than 9 digits in the ST-SE control number. Whichever you chose, you must do the same for both the ST and SE.

(Using 000 or 000000000 may be allowed, but is not recommended as any application that trims the zeroes will get an empty string and produce an error. Be nice and only use values greater than zero.)

Segment Counter

The SE segment counter is the part of the ST-SE that gives people hand editing an EDI document the most trouble. If the counter is inaccurate it indicates that the document is corrupted. This should fail in all good translators as processing a corrupt EDI file is saying, “Its okay if we get bad data.” This counter represents the number of segment between the ST and before the SE.

And it is just that simple. Remember that you can have more than one GS-GE envelope inside an ISA-IEA, and you can have more than one ST-SE inside a GS-GE, but you can only have one document of one type in side a ST-SE envelope.

–

For an overview of Enveloping go here.

Looking for the ISA enveloping page?

Looking for the GS enveloping page?

Looking for something else relating to EDI? Check out the EDI Primer post

Amazon.com Widgets

eCommerce, EDI and CRM Integration-How It Works

EvilRobot — Fri, 27 May 2011 14:36:50 +0000

I wish I could say this was a gripping video of e-commerce and EDI integration that would keep you on the edge of your seat. But as Mr. Brown says at the begining of the video. ”The sequence of operations once you are integrated is not dissimilar to how you did it before.”

Huh? Right, well basically the integrated process is like the pre-integrated process, just faster, and without some of the human error components.

Colin Brown is the CEO of eBridge Software. In this video he explains the typical sequence of operations for eCommerce, EDI, CRM integration and how it works once it is integrated. Additionally, Mr. Brown discusses why trading partners make their customers do EDI.

This concept of replication and automation of process to achieve supply chain efficiencies applies to all forms, EDI, XML, or others. The first step is not to replace the process, but to automate it and make it faster.

EDI: Wrapped and Unwrapped

Roy — Mon, 18 May 2009 16:00:14 +0000

EDI in its proper form is wrapped. This means that there is nothing between the segment terminator and the beginning of the next segment. Sometimes this is hard to read, for humans anyway. And at those times people will unwrap the EDI so that it looks more like a flat file. This is not hard to do, and is useful for debugging and trouble shooting processes. A good parser or validator should be able to read either file.

Wrapped

As I said wrapped EDI has nothing between the segment terminator and the next segment, in this format;

SEG|||~SEG||||||~

and so on. Wrapped EDI looks like this:

ISA*00* *00* *01*9012345720000 *01*9088877320000 *090508*1459*U*00200*000000001*0*T*:~GS*PO*901234572000*908887732000*090508*1459*1*T*004010~ST*850*0001~BEG*01*BK*99AKDF9DAL393*39483920193843*20090508*AN* 3920943*AC*IBM*02*AE*02*BA~DTM*002*20090508*1615*CT*CM*DATE TIME PERIOD~N1*AC*ARTHUR JONES*1*9012345918341*11*SE~N2*MAGGIE MCGILLICUTTY*BEVERLY SIMPSON~N3*PO BOX 123*157 WEST 57TH STREET~N4*CINCINNATI*OH*43017*US*IP*02535~PER*AC*ARTHUR JONES*TE*(614)555-1212*TE*(614)555-1212*TE*(614)555-1212*ADDL CONTACT DESC~PO1*AAA849*100*DP*100.00*PD*AB*ASSEMBLY*BL*BRAND OR LABEL*ON*CUST12345*AB*ASSEMBLY*BL*BRAND OR LABEL*ON*CUST12345*AB*ASSEMBLY*BL*BRAND OR LABEL*ON*CUST12345*AB*ASSEMBLY~PID*F*BC*AX*UNSCENTED*GENERAL PURPOSE*BS*SUBQUALIFIER*N*ENG~CTT*10*855*12345678*HB*40*QD*GENERAL PURPOSE~SE*12*0001~GE*1*1~IEA*1*000000001~

Unwrapped

When we unwrap the EDI, we don’t just place any character between the segment terminator and the next segment, we place a carriage return and line feed. This makes all of the segments appear to be justified like separate lines of a text file. Thus the format is changed in this manner;

SEG|||~

SEG||||||~

and so on. Unwrapped EDI looks like this:

ISA*00* *00* *01*9012345720000 *01*9088877320000 *090508*1459*U*00200*000000001*0*T*:~
GS*PO*901234572000*908887732000*090508*1459*1*T*004010~
ST*850*0001~
BEG*01*BK*99AKDF9DAL393*39483920193843*20090508*AN3920943*AC*IBM*02*AE*02*BA~
DTM*002*20090508*1615*CT*CM*DATE TIME PERIOD~
N1*AC*ARTHUR JONES*1*9012345918341*11*SE~
N2*MAGGIE MCGILLICUTTY*BEVERLY SIMPSON~
N3*PO BOX 123*157 WEST 57TH STREET~
N4*CINCINNATI*OH*43017*US*IP*02535~
PER*AC*ARTHUR JONES*TE*(614)555-1212*TE*(614)555-1212*TE*(614)555-1212*ADDL CONTACT DESC~
PO1*AAA849*100*DP*100.00*PD*AB*ASSEMBLY*BL*BRAND OR LABEL*ON*CUST12345*AB*ASSEMBLY*BL*BRAND OR LABEL*ON*CUST12345*AB*ASSEMBLY*BL*BRAND OR LABEL*ON*CUST12345*AB*ASSEMBLY~
PID*F*BC*AX*UNSCENTED*GENERAL PURPOSE*BS*SUBQUALIFIER*N*ENG~
CTT*10*855*12345678*HB*40*QD*GENERAL PURPOSE~
SE*12*0001~
GE*1*1~
IEA*1*000000001~

Converting from Wrapped to Unwrapped and back

When you need to do this, you need to do this fast, not manually. Sure the manual way works with small files, but if you are trying to edit a big invoice or order, you will want to do something snappier than walking through the file segment at a time. Here are two ways to do it with common tools.

To unwrap:

open the file. Type “vi [file name]“
Hit “:” to get the command prompt
Type “%” to do a every line replace. (may not be needed as this is wrapped, but it doesn’t hurt.)
Type “s” for a substitution.
Type “/” to begin the string to find. (you may need to escape characters like ~ and | with a \. I will show this in the example)
Type the delimiter to be replaced.
Type “/” to begin the replaced string (again with the possible need to escape characters.)
Type the delimiter again, as we don’t want to lose it, then follow it will a [crtl]+v and [crtl]+m
Type “/” to end the substitution
Type “g” to make it global or indicating all of the instances not just the first one.

It looks like this: “%s/[delimiter]/[delimiter]^M/g”

Put another way, here is what I typed for our example: “%s/\~/\~^M/g” (notice that I am escaping the ~ with a \)

To re-wrap:

open the file. Type “vi [file name]“
Hit “:” to get the command prompt
Type “%” to do a every line replace. (may not be needed as this is wrapped, but it doesn’t hurt.)
Type “s” for a substitution.
Type “/” to begin the string to find. (you may need to escape characters like ~ and | with a \. I will show this in the example)
Type a backslach (\) followed by “n” to designate the newlines.
Type “/” to begin the replaced string (again with the possible need to escape characters.)
Type “//” to begin and end the substitution.
Type “g” so that it will get any multiples of the newline.

It looks like this: “%s/\n//g”

MS Word

To Unwrap:

Open file.
Select Edit>Replace or press [ctrl]+h
In the “Find what” field insert the delimiter.
In the “Replace with” field insert the delimiter, and then click on the “More” button on the bottom of the control.
In more, there is another button called “Special” Click on this button and select “Manual Line Break”
Click replace all and the EDI will unwrap.

To re-wrap:

Open file.
Select Edit>Replace or press [ctrl]+h
In the “Find what” field click on the “More” button on the bottom of the control.
In more, there is another button called “Special” Click on this button and select “Manual Line Break”
Don’t put anything in the “Replace with” field.
Click replace all and the EDI will re-wrap.

Other Tools:

There are lots of other tools that can do the search and replace with newlines, these are only two ways. If you have one that you would like to share, please let me know.

—

Looking for something else relating to EDI? Try here.

EDI Elements

Roy — Tue, 13 Jan 2009 16:44:18 +0000

What is an Element?

In the context of delimited data files, an element is the contents between two delimiters. Like in a spreadsheet, the element is the contents of a cell. In EDI, we have delimited segments, and just like a delimited file, the contents between the delimiters is the element.

So in this segment “PO1*1*8*EA*10**SW*H11510.REF*CT*089480-00-B-0336~”, the 5th element or PO1_04 is “10″, while the 6th element or PO1_05 is empty.

Empty elements are not necessarily null. Calling something null when the data type is know like an empty element can cause some data theory debates. I don’t want to start one here, but may write about that philosophical discussion at a later time.

What types of elements are there?

What we are really talking about is the data type. Each element in an EDI file has a specific data type defined. Part Numbers are Alpha-Numeric which means that they may contain letters in the alphabet, and number in any mixture. Where a price is not just a number, but is a currency. Letters are not acceptable in the price field, so one can’t populate it with “N/A” in any case.

I won’t describe all of the data types here. That is a discussion for some other time. Most of the time these are fairly easy to work with, especially when you have samples to work from. But there is another type of element. This is the encoded value element. This element does still contain a value, but it is in a code.

UOM or Unit Of Measure is the easiest to relate to, as it deals with something physical. In the PO1 segment above, we find the UOM in the PO1_03 element. In this case it means “each” and by putting “EA” in this position we don’t have to type out “each” every time we just us EA as the abbreviation for this.

Why do we do this? Well as I said the UOM is the easiest encoded element. Some of the other encoded elements that are available define larger concepts. Also, by having an encoded element, the EDI standard constrains the data to a rational type of data.

For instance, if you have an order response, we don’t want to have to process a free form response, we want to know if the product is accepted, back ordered or canceled. And I want to know this in the same way for everyone.

What is Element Notation?

Elements are contained in segments. The segment is referred to by the characters that preceded the first delimiter. Each element in the segment has the name of the segment followed by its position in the segment. Sometimes these are not separated, or they are separated by a space or underscore. The position of the element is 0 based counting. This means that the first position, the position preceding the first delimiter where the name of the segment is found, is position 0.

Thus in this segment, “N1^ST^Bill’s Grill^01^123456789~”, the N1 is N1_00. I prefer the underscore to the space or no delimiter format for clarity. But they are all used, so this would be the same as N101 and N1 01. We use a two digit position designator even for positions less than 10 for clarity as well. If there are any segments that develop more than 99 positions, there won’t be many. And even fewer will ever be used. And we don’t reduce it to just one digit, as N11 might actually exist, and that could cause confusion.

Now back to our example; N1_01 is “ST”. N1_02 is “Bill’s Grill” as so forth.

An example of how this is used is first found in our mapping rules that we need to write down. And then in our notes and documentation. If you wanted to say, “Whenever we have a ship to address for ‘Bill’s Grill’ we need to used the Duns number of ’123456789′.” We could say , “When N1_01 = ‘ST’ and N1_02 = ‘Bill’s Grill’ then N1_03 = ’01′ and N1_04 = ’123456789′.” This way we are not leaving anything open to interpretation and clarity is our friend when writing rules for mapping and forming data.

You will see this used extensively in the mapping exercises here, and once you get the hang of using Element Notation, you will never write it any other way.

–

Looking for something else relating to EDI? Check out the EDI Primer post

–

Mapping Excersizes: EDI Invoice to Open Office Tables (part One)

Roy — Tue, 07 Jul 2009 17:30:24 +0000

This is a mapping exercise that will go through the process of creating a paper map, or mapping document. We will start with an empty paper map that you can get here. And we will end with a completed paper map document that documents what data from the source goes into what fields on the target. This process will take more than one post, and I will link them together so that you can follow from one to the next. Along the way, we will discuss the things that we are doing so that you can apply this technique in your mapping using the target and source in your own mapping tasks.

The Target

Open Office tables are divided into two tables. Invoice and Invoice Details. This can be mapped in two ways; The first way is to map the data into one common format and rely on whatever ETL tool is importing the data to catch and split the data. Second, a key can be acquired or constructed in the transformation and then the the data can be divided in to matching input formats. Then when these inputs are moved into the database, they will relate to each other on this key.

The choice of how you will do this will depend on your environment. Questions like, “Will I have enough data to provide a unique key?”, or “is there a way to get a key with an API call or database query?” The answers to these questions will determine what course you will take.

If the system ultimately receiving the data is asynchronous to the transformation, and you need to send the invoice and invoice details data separately, some care needs to be taken to ensure that the data can be related after it is separated.

So what data in the invoice can be used to tie the invoice to the invoice details? The first answer might be, “The Invoice Number.” But this number is not guaranteed to be unique across multiple vendors. In EDI and cXML there are document unique identifiers. Since we are using EDI, we can use a combination of the ISA Sender, Receiver, and Control number. We will also want to use the GS control number, and ST control number in the event that more than one invoice is sent in a single envelope. And we might as well tack on the actual invoice number from the BIG_02.

Database Tables

Invoice Table:	Invoice Details Table:

The Paper Map

Now that we know what the target looks like, we fill out the target side of the paper map. Since we will create two “files” in our output. We are creating the Invoice and the InvoiceDetails file, but we can use one paper map for both, and will distinguish this with a bar between the two “files”. (I am saying files, but this could be a queue, or a post, or an insert over odbc, etc)

What’s Next

Today we went through the process of identifying the target, and creating a paper map with the target format identified. We talked about some of the strategy that we use in deciding what to map and how to map it. Next time we will identify the source, and begin mapping data from the source.

Why are there more than one version of EDI?

Roy — Mon, 17 Nov 2008 17:39:21 +0000

When EDI was born, (in the 1970s), the committee did its best adopt a forward thinking plan and architecture. But there was really no way for people living before the birth of e-mail to make place for what we use today to do business. So the early versions of EDI had no place for e-mail addresses, and other things that we use in e-commerce today.

The goal of EDI was to provide a way to support communications between dissimilar computer systems. EDI was intended as a light weight, reliable means to transmit data. By today’s standards, EDI may not seem that easy, open and light weight. But if you realize that at the time it was being developed some of the things we take for granted, like IP addresses, and Domain Name Systems were on the drawing board, or not really understood or used.

Data Dictionary

One of the ways that EDI saved space and enforced consistency was to use encoded values. Using the UOM list is easy to understand. And it really hasn’t changed that much. But there were also encoded values for things like the type of address or the type of financial transaction. Basically there are more variations of these today, than there were at the time of EDI’s birth.

Now this is not to say you can’t used something that the standard builders didn’t think of. There is a common wild card value for almost all of the encoded value fields. That is ZZ (note that if the length is 1 or 4 or any length, then it is the number of ‘Z’ characters to fill the required space). ZZ is a value for ‘Mutually Defined’. Meaning, “the standard doesn’t have this, but I expect the receiver to understand what this is.”

But just using that all of the time would reduce the value of the standard. So you may have used ZZ to list an address type that the 3030 standard doesn’t support, but the 4010 standard has it now, and we should used that when we build a 4010 document. The creation of a Data Dictionary not only reduced the size of the EDI file, but also helped enforce the usage of common concepts.

Syntax

Both the size and order of the segments has changed from the earliest versions of EDI. Segments were only created as long as was thought needed to contain the pertinent data. Segments that related to contact information defined in the era where you might have a fax number in addition to your office number, just doesn’t have the space for your cell phone. And the line item definition may not have enough space now that you might want to include the UNSPC number that didn’t exist then. So the segments have grown, and some orders have changed to allow for better representation of the data.

Objects Being Represented.

Back in the beginning, EDI was representing paper objects. So when people mapped from EDI, or talked about EDI, their brains didn’t go far from the idea of what the paper order or invoice, or status would look like. But very quickly, younger minds started to represent data as just data. No longer tied to the paper order form, or other physical media, data began to take on characteristics of the real data, and real objects that the paper was describing.

EDI has been slow follow this completely. Backward computability is a good excuse, but mostly it seems that the established syntax was flexible enough that new things can be represented. This is a double edged discussion, and we will leave it with this. As the nature of e-commerce evolved, EDI formats changed to follow the evolution.

Summary

EDI is a collection of standards and versions. As time has moved forward, the nature of our e-commerce needs have changed. EDI has changed too, but instead of letting working standards become obsolete, a new version of the standard is published. These standards implemented incremental changes in the Data Dictionary, Syntax and Objective Representation of the E-Commerce transactions.

–

Looking for something else relating to EDI? Check out the EDI Primer post

–