First you will need to have a Google Analytics account.  If you don’t have one, you might want to check it out.  Its free and you can sign up at http://www.google.com/analytics

As soon as you are ready with you Google Analytics account.  And you have told it that you want to monitor your site.  It will generate some code for you to place on every page that you want to monitor.  But don’t worry, Magento doesn’t need you to do much to accomplish this.  If you look at your accounts page, you will see something like this:

I have blurred out my domain and account, but what you will see in the red box, is the number that you will need to put into Magento.

So copy it to a text file or just leave the page up and go back to it when you need this number.

Now, go to your Admin page on your Magento installation.  You will need to go to System>configuration as shown below.

Once in configuration, you should find a set of navigation links on the left hand of the page.  In the “Sales” section you will find a “GoogleAPI” selection.

This will take you to the section with a Google Analytics control.

Expand the Google Analytics control and set Enabled to Yes.

Now here is where you type or past in your account number from the Google Analytics report page we saw at the top of this article.

Once you have typed it in, click on the “Save Config” button.

When you see this message, you are ready to go.

You might have to wait a few hours before the green check mark appears on the Google Analytics site.

This introduction aims at addressing the basic questions of convention and implementation of the EDI technology.  I have a list of topics, and hope that over time I can cover all of them.  I also hope that people reading here may have questions that I haven’t thought to write about.  Please let  me know what your questions are and I will see what I can do about providing answers.

• What is EDI?
• Parts of EDI
• Segments
• Delimiters
• Conventions of EDI
• Why are there more than one version of EDI?
• Standards and Usage
• Encoded Values
• EDI in Good Form
• Element Notation
• Wrapped and Unwrapped EDI
• Enveloping Part One
• Enveloping Part Two – ISA
  
• Enveloping Part Three – GS
• Enveloping Part Four – ST
• Loops in EDI
• Hierarchical Levels

Of course understanding how EDI presents and represents data is the first step.  From here we may want to go to our Data Mapping Guide, or if we are familiar with mapping there are some examples in our mapping case studies of EDI to and from other formats.  Again, if you have an interesting problem or solution, I would love to here about it.  Please send it to
roy@TheIntegrationEngineer.com and I will see what I can do.

With this as the abstract version of our goal. We break the task into three sections. Data types that we will exchange both outbound and inbound. What transport or connectivity solutions we will used to ensure a robust and reliable transmission and reception of the data. And what data translation, storage and processing solutions we will employ.

Data Types

Many people don’t think about this part very much because they have only dealt with EDI systems that are already in service or in the process of being launched.  By this point the types of data being passed between trading partners have already been decided.  So to begin this process we ask what type of business we are conducting.

If we are doing e-commerce, we are probably going to exchange orders, confirmations, invoices and ship notices.  If we are a retailer, we may be receiving catalog data, where if we are a vendor we will be transmitting it.  And so forth.

We can find out what document types to use in two easy ways.  First, any of our trading partners currently exchange EDI data with other parties, (like integrating with a vendor that has other customers doing EDI), we can ask for if we haven’t already received the EDI specifications that they use.  If we are setting up a new integration with a new trading partner with no EDI history, we will then need to look at the list of EDI document types and find the ones that will fill our needs of data exchange.  (All of the EDI document specifications have a name as well as a number.  For this exercise we will need to look at their name.)

Transformation, Processing and Storage

For data coming into the process for our trading partners is what I call “inbound” data.  And so of course data going out to a trading partner is called “outbound” data.  In either case we start the Transformation, processing and storage by identifying what our destination format, and location will be for the data.  Inbound data has the destination of whatever process or storage we are going to place it, and in the format that we need it in to process or store.  And outbound data has the destination of our connectivity solution to our trading partner and the format of the EDI document type that we have agreed to use.

This is called Target Based planning and it goes hand in hand with Target Based Mapping.  We plan by starting with where we want to end up, in the transformation, processing and storage of the data.  Then we work backword from that goal and create a way to get the data from where it is, to where we need it to be.

For instance if we have the destination of an EDI 850 on our AS2 connection, we start with a standards format of the 850 the trading partner can accept and the location or API that our AS2 connection needs us to use.  After we have that target defined, we map and plan out how to get our order data to that location, coming out of our database, ERP, e-commerce system or some file that we are processing.  (Possibly from a combination of these.)

And we do the opposite if we are processing an inbound 850, we start with the format and interface that we need to input an order into our system, and then build the process to get the information we need out of the 850 and trading partner database etc.

Connectivity

This is a critically important issue, that we will not go into detail here.  But you will want a connectivity solution that is reliable and provides you with a positive confirmation that a file has been received by the other partner and establishes accountability for the transaction, and a process for remediation if there is any dispute.  I will have to talk more about connectivity another time when I can go into greater detail.

Summary

Breaking big tasks like rolling out an EDI strategy can become very complex.  There are two secrets to being successful.  First, break your big tasks into three pieces; Inputs, Outputs and Processes.  The second secret is to begin with your end, or goal, or target.  And then work back from there.  Putting these two pieces of advice into use will make your EDI strategy more manageable and come of more smoothly.

Early in my integration career I was working on a medical supply chain.  Accuracy was important as you can imagine.  And there was a good deal of stress that we shared as we brought our products on-line and let them encounter real data.

We tried to ensure that the data that we sent to suppliers was received, and when things when wrong we were on the phone quickly asking for confirmation that orders were received.  As time when on we became better both at reliably getting data to the correct place, and in determining when problems where really problems.

“There was one Vendor that sold capital equipments.  So they received few orders, but when they did, their orders were for a high dollar items.  They also kept shutting down the system that we sent orders to.  I think years went by where every order was accompanied by a phone call to their systems administrator asking that they re-boot their server.”

The EDI way

The easiest and most EDI like way to establish accountability of EDI transactions is the use of a 997 transaction.  Basically a 997 works as an acknowledgment.  The name of this document actually is Functional Acknowledgment.  So we are acknowledging that a specific functional EDI component was received, and possibly how it was received.

If you are versed in the EDI enveloping levels you have already realized the  link of between the 997 and the GS level of the EDI envelope.  Also you may remember that you can request a 997 be returned for your transaction in the ISA segment in ISA_14.  But merely setting up your ISA to request a 997 does not actually force your trading partner to comply.   Your trading partner may not be able to, or may not wish to return a 997 for either business or technology reasons.

Assuming that you are able to get a 997 back.  Just receiving the 997 does not mean that all is well with your transaction.  The 997 is an EDI transaction.  It can contain data indicating just the success of receiving the data, or a few validation levels indicating data corruption in transit or rejection for business process reasons.

The Transport Technology way

Another way is to place the accountability on the transport of the message.  Using AS2 or other transport technologies that attempt to guarantee delivery of messages, it is possible to extend our reliable message solution to transactions of multiple types.  AS2 and other transport solutions return an MDN or message disposition notification letting the transporting party know that its message has been received.

The BPM way

In spite of the facility of EDI to process and utilize 997s and of AS2 to utilize MDNs, the most exciting method of assuring message delivery and accountability is utilizing business intelligence.  For me, I remember a system that we developed to help us with the health care supply chain.  We simple began tracking the response time that was typical for each of our supplier trading partners.  After gathering trending data, we applied alerts when responses were overdue by a specified degree of variance from the norm.

Since this process relied on the past 6 months of performance as the baseline, it was adaptive to changes in the vendor.  We could also manually change the amount and degree if we started getting false alerts.  In this way we had a way to react to failed transaction without requiring agreement and arrangements with the trading partner to confirm receipt or processing of transactions.

The Business Agreement

Even in cases where you have implemented a BPM process to alert to failed transactions, you have to have an agreement and arrangement with your trading partners that allow you to deal with failures in transmission and processing of your e-commerce transactions.  At a minimum this needs to include contact information for support teams on each side of the transaction.  It does no good to know there is a problem with a transaction but to be unable to solve the problem because the other party is unavailable.

In the end, it is what you have agreed with our trading partner that make the sender or receiver responsible for processing the transaction.

What is a line item?

A line item is a group of data and its structure that is or can be repeated multiple times in its document.  It holds the data that is the focus of the document, and a part of a document or transaction that contains distinct part that can stand alone.  For instance, an order that has two line items could be split into two orders that would be able to be filled separately.  It does not necessarily contain information that can be applied to a group of items, although sometimes this information is included.  This feeds into what type of data that a line item contains.

What type of data does a line item contain contain?

In a purchase order, or other commerce related document these pieces of data will be found in the line item:

1. Purpose of line
2. Product or Service ID
3. Quantity
4. Unit of Measure
5. Price or Fee
6. Taxes
7. Special Handling
8. Descriptions

Some additional data can be aggregated to the document level or published at the line item level;

1. Ship to or Deliver to data.
2. Discounts
3. Shipping methods or tracking information

Where in EDI does this data go?

For the most part, the PO1 segment and group contain the line item data.  There are a few transactions that use another segment.  The invoice for instance uses the IT1 segment.  Depending on the scope of our definition, an HL group could be a line item in some documents.

Line items are at the center of our transactions.  Purchase Orders, Invoices, and Catalogs would not be worth creating without the line items they contain.

So it goes like this:

850 is a Purchase order, and is sent to the Vendor

855 is an Order Response, and is sent from the Vendor.  It confirms, updates or rejects each line, or the PO as a whole.

856 is the Ship Notice. and is sent from the Vendor.  It may contain one or more lines from one or more orders and can indicate shipment of partial lines from a PO.

810 is the Invoice, and is sent from the Vendor.  It contains one or more lines from one or more POs, but generally people keep these to only one PO.

820 is Remittance, and is sent to the Vendor.  It contains information relating to payment.

If you have used a standards editor, you probably know what an SEF file is, or have used it.  Some applications and EAI even use SEF files as part of their document creation and validation processes.  It becomes useful to describe briefly what SEF files look like, and what type of information they contain.

What is in there?

If you are familiar with EDI, and have cracked open an SEF file with your standard text editor of choice, you will already know what I am about to say.  The contents of the SEF file don’t look that complex to someone who is used to and familiar with how various EDI files look.  For those that are not so brave, I will explain this painlessly.

First, the SEF file is really just a text file.  It can be edited by hand, but I don’t really recommend it.  (not because you can’t, but because it is tedious.)

Second, it contains its data in sections.  There is a section for doc types, segments, elements, and encoded data.  There is also a few housekeeping sections like version and name etc.

What is an SEF file for?

SEF stands for Standards Exchange Format.  This is literally a file format that was designed to contain information about EDI standards.  It was created so that a computer application could understand an EDI file.  It is used by some application like Standard Repositories and Standards Editors to allow us humans to deal with a standard like it was a text document.  While at the same time keeping a repository that can be used by an integration application to form and validate an EDI file.

If you are familiar with the concept of metadata then an SEF file is the EDI standard’s metadata.  If you are not familiar with metadata, then skip this part.

Do I need to know about SEF files and EDI?

Nope.  SEF is only useful if you are exchanging standards using the format.  If your standard and usage is in the form of a PDF or a spreadsheet, that is fine.  Many people do just that.

For me, I need to know this because I am trying to build some Standard Repository tools, and want to use the SEF format to manage them.

And for those that want a deeper look, I will be following this article with some deeper and more detailed articles dealing with the internals of the SEF file.

EDI Specifications

In the larger sense, the EDI specification is the set of rules that define each document type, the segments they contain, and the size and type of data in the elements.  When we talk about the EDI Standard Specification, we are talking about the whole set of valid EDI document types.  If something is valid EDI, then it complies with the EDI Standard Specification.  However, this large, all encompassing specification is not useful in coordinating the exchange of documents between two trading partners.

What goes in an EDI Usage Specification?

So we take EDI Standard Specification and we reduce it.  We remove the unused document types and the unused segments within the documents we will use.  And we even remove the unused elements and encoded values within the segments we will use.  At this point, we have created an EDI Usage Specification.  We can also add to the specification by including values that we need, like designating the an optional value or ID as required and specifying what type is should be.

How is an EDI specification Used?

Now that we have a usage specification we can use it to do to basic things.  First, it is the source of information for us to set-up our integration for our EDI document exchange.  And Second it is the format that we validate our EDI documents with to determine a valid transaction.

EDI Usage Specifications can be a source of data and integration documentation.  This can become extremely valuable when the choices and information about how your EDI interface works is recorded into the notes of the Usage Specification.

Beyond a visual validation of looking at the EDI file and comparing it with the specification, many times a specification can be found in the form of an SEF file.  The SEF file can be used in an EDI validation application.  This allows a potentially large and unwieldy EDI file to be scanned for compliance and accuracy.

There may be other uses, but these are the main two, integration data repository, tool for EDI validation.

EDI Specification Examples:

To walk through how a usage specification works, and is used, check out this review of BookNet Canada’s specification.

Then you can take a look at the EDI specifications published by Macy’sNet that they keep online to assist their trading partners in making and keeping effective integrations.

Sometimes we only use the EDI specification, they way we have always used them.  It can be instructive to browse the usage specifications of other companies.  We see what they remove, what they include, what comments they make, and what rules they document.  And seeing how others do things can help us to be more effective in how we use EDI specification documents.

1.  Identify the Trading Partners

This may seem really simple.  And it is.  But it is not too simple for us.

We need to know who we will integrate with.  Are these new contacts that have never worked with our company before?  Or are they existing customers that will be moving to this new integrated connections?  Both groups have their own sets of challenges and benefits.

Is this integration a one time event, or will we be integrating multiple trading partners using this same process?  This is also important and will make a difference on how trading partner specific we want to make it.

What is the technical sophistication of the proposed trading partners?  Are they going to be able to comply with our needs or will we have to do everything?

2.  Identify the Data

Again, this is basic but not too basic.

We need to know if we are dealing with just orders, or if there will be invoicing and catalogs.  We need to know if what type of performance we are looking for.  Is this a once a month catalog load that can take hours and gets a couple or retry windows, or is this real time price comparison that need to respond in seconds.

The data and how it needs to flow will be a big piece of what we need to know.

3.  Identify the Target

What is the target?

This can be taking some of your data and sending it to your Trading Partner.  In this case your target is the format and communication protocol that your partner can receive.

Or it can be an internal Canonical Data Format that your systems use that you are creating from transactions received from your partner sends you.

Or it could be something else.  Whatever it is, you will need to know what it is, and know all about it before you can really get started.

4.  Identify the Tools and Technology

Are you using EDI?  Or XML?  What EAI tools are you and your partner using?  SAP?  WebMethods?  GXS?  Other?

How do complete the task will rely to a great degree on what tools and technologies you are working with.  If you are an employee, you may already know this, but if you are a consultant you probably don’t.  You need to find out.

5.  Identify Support needs.

What happens when things go wrong?  Not necessarily “horribly wrong” but just wrong.  Is an error produced when the translation fails?  How about creating a 850 with no line items?  (no one wants to receive this.)  Or if the https post or SFTP send fails, can it retry, or does it die quietly?  These are questions that you need to ask and keep asking.  (Please include all of the answers in your documentation.)

6.  Identify the Time Frame

How long do you have to complete the integration?  Is there a plan to find and fix bugs after implementation?  How long is the integration planning to be used?  (short term during the merger of two companies, or for as long as Walmart will buy your product?)

All of the aspects of time, time to complete, time to live and time to revisit and fix problems are good things to know, or at least have an idea about.

7.  Get started, and Document your work.

Getting started, and starting the Documentation of a project should happen together.  Almost at the same moment.  Okay, really, at the same moment.

Documenting the answers to the questions above along with others is really the start of the integration project.  And the documentation process will help you to focus your work on getting the answers that you need to finish the project.

When you know what you should be doing, and how you should do it, many tasks become easy.  One of the illustrations of this is shown in my Mapping Excersizes.

Parts of ACK

Here is a diagram of a sample ACK segment:

Like any other segment in EDI, the ACK segment is composed of elements.  Here is a list, element by element and what it contains.

Best Practices

With the ACK the best practices are more in how to implement and what to expect.  So here are some things to remember when implementing this segment.

1. An ACK is part of a PO1 line for a response to another PO1 line transaction.  This segment supplies a status, and can modify the line item contents.
2. There can be more than one ACK segment for a line item.  (4010 855s allow 104 repetitions of the ACK)
3. When an ACK explicitly changes the line item it can only change that one type.
4. There is a status called Accepted With Changes.  “AC”  For this, you will need to parse the ACK and see if any of the values have changed.

Parts of IT1

Here is a diagram of a sample IT1 segment.

Like any other segment in EDI, the IT1 segment is composed of elements.  Here is a list, element by element and what it contains.

I wrote a section on Line Item Good Form and Best Practices in the article about PO1 Line Items.  Instead of repeating that I will just let you hop over there are read it.

IT1 Loop Segments

As we said in the beginning, the IT1 is a segment, and a loop/group. And as you may have noticed, there is data that you may need that is not covered in the elements available in the IT1 Segment that belong to the line item.  Values like Tax information, found in the TAX and TXI segments.  These and other values are found on the line item, but are not on the IT1 segment.

Parts of the PO1

Here is a diagram of a sample PO1 segment:

Like any other segment in EDI, the PO1 segment is composed of elements.  Here is a list, element by element and what it contains.

Good Form on PO1 line

EDI in Good Form is both style and best practice tips to avoid common exceptions.  Paying attention to good form tips and best practices helps you create an integration solution that is not only complies to the standard, but functions to help you and your trading partner have an unnecessary support burden due to the avoidable unexpected.

1.  It is bad from to have PO1 Product and Service pairs that are empty.  This relates to not having any trailing element delimiters.  It also goes along with not having empty pairs in the line with populated Product and Service element pairs following.  Thus it is best to build your mapping so that optional data doesn’t create empty elements when it opts out.

2.  When reading a PO1 line, it is best to assume that an expected Product and Service value could be in any pair.  So the map needs to walk down the pairs of elements until it finds the qualifier that indicates the value you are looking for.  This allows you to use the same input map with multiple EDI producing trading partners that may have interpreted the priority of data in different ways, and avoid exceptions.

3.  When producing EDI with a PO1 line, it is best to be as consistent as possible with your placement of the Product and Service data.  Create your logic and mapping so that mandatory data occurs early in the PO1 line, and optional data is moved to the end.  This will keep your required data from hopping around positionally.  This will help your trading partners that may not have had the foresight to follow Good Form Tip number 2 from having exceptions that slow down you process.

PO1 Loop Segments

As we said in the beginning, the PO1 is a segment, and a loop/group. And as you may have noticed, there is data that you may need that is not covered in the elements available in the PO1 Segment that belong to the line item.  Values like Item Description, found in the PID segment. Or Tax information, found in the TAX and TXI segments.  These and other values are found on the line item, but are not on the PO1 segment.  We will have talk about some of these other segments and where to find other common data on another day.

There is a temptation to use this same validation in the production integration. But this would be a mistake. I’m not saying, “Don’t validate.” I highly encourage validation on both standard compliance and required data validation in mapping and integration. But to use the usage specification has a side effect that I witnessed once. (Only once.)

Validating Transactional EDI to Usage Specification

We had an integration requirement once that required that we only allow EDI elements that were in our usage specification. This was outside of our normal process and we all soon learned why.

When we validated EDI files based on the standard, we would error out documents that were corrupted, or in violation of the full EDi specification. (having mangled segments, or invalid UOMs and such.) But this one time, we thought it would be a good idea to also fail documents that had any extra data that we weren’t going to use. So we pulled all of the segments and elements out so that we were just down to what was in our usage specification.

This worked just like we told it to. As when the next file came in that was in bad form with extra trailing delimiters on the line item, it failed. And then the next with an extra segment that we didn’t care about? Well it failed too. As so on.

We soon were buried under failed EDI files that were valid EDI, and did contain the data we needed, but simply had something extra. And we were calling them all exceptions.

At first we added in a few of the common segments that were being sent. This relieved the lion’s share of the errors. But it still cropped up from time to time. Each time we would have failures for this reason we had to explain why. This grew tedious and embarrassing.

Eventually we re-mapped the doc-type using the complete standard in our validation. Magically all of these errors went away, and so did the embarrassing explanations.

Here is the way the process should work, generalized.

1. EDI envelop is validated against EDI standard.
2. EDI document is validated against EDI standard
3. Mission critical values are validated against Usage Specification.
4. File is processed.

What this does is to establish an inherent hierarchy of exceptions.  Transactions that fail on step 1 are all falling because the envelope is corrupt or  in error.  Also, Transactions that fail on step 2 are failing for a general EDI reason.  Both of these steps will have exceptions that can be handled by anyone with EDI knowledge.  Operational and Application rules don’t necessarily need to be known to understand how to handle errors at these levels.

Step 3 and 4 are exactly the opposite.  Errors that happen because of mission critical data that is missing or in error must be handled by someone that knows what the mission critical form is supposed to look like.  And files that pass validation but fail later in processing may need to be corrected and the validation process updated.  Or they may point to an error in the application itself.  In any case, much more knowledge of the whole process is required at these stages.

Take Away

First, by sharing this experience I hope that everyone can appreciate that sometimes requirements make us to funny things.  But I also wanted to share the approach to validation that I have found most helpful.  I have seen implementation that don’t do any EDI or Enveloping validation.  These are just as dangerous and frustrating as my experience.  But now all of the errors are pushed down to step 3 and 4.

I’d also like to invite you to subscribe to my newsletter.  It comes out once a month, starting at the end of this month.  I’ve got some great ideas for it, and think it will be terrific.

Definition:  SKU stands for “Stock keeping Unit” initially used for identifying items that a company keeps in stock so that they can track how many they have and such.  Now also used for services, and for contracts or warranties.

Usage:

The SKU is assigned to each thing.  And it is unique to the company that issues it.  You might think this is the Vendor Catalog Number, but this is not always the case.  The SKU does not just refer to the type of item, say a plate, but also how it is packaged.  So a plate, and a box of plates would have different SKU numbers.  This is one reason that the VCN will not always be the same.  If you order a case, but the vendor is out of cases, but has the same unit quantity of boxes, they may respond and ship the 10 boxes that comprise a case.

Sometimes this is not a great solution.  I had a customer once that ordered a case of tape refills.  The vendor was out of cases, but had the individual refills.  So they shipped 500 tape refills.  The customer then had to inventory and stock the individuals, they were not having a good day.

Sometimes the SKU contains some of the UPC number.  But again these numbers are not the same.  The UPC is about identifying the Unique item type, the SKU is about identifying the stocking unit.

Looking in my handy EDI Standards Reference, I see that on 4010 PO1 line items, the SKU number is qualified by the “SK” qualifier.  If you are using a number you think is  a SKU when you order, but you are qualifying it with a “VC” then you are really using the Vendor Catalog Number.

EDI Example:

PO1*AAA849*100*DP*100.00**SK*123ABC******************~

As you can see, the PO1_06 is “SK”.  This indicates that the PO1_07 is the SKU number.

cXML Example:

<ProductRequest itemNumber=”1”>

<SupplierPartNumber>123ABC</SupplierPartNumber>

<UnitOfMeasure quantity=”1″ unit=”DP”/>

<Quantity>100</Quantity>

</ProductRequest>

In cXML it is less specific.  The SKU number is found in the contents of the “SupplierPartNumber” tag.  This may not always be used for a SKU Number.  However, I don’t know that <stockKeepingUnit> or <SKU> are valid tags in cXML form.

And thus we see why the confusion comes in as to what a SKU is.  Some standards like cXML are loose in their definition of these item identifying numbers and some materials systems as well.  And then others want to know what the SKU is so that they know if a pallet or a envelop are being confirmed on the ship notice.

Rationalizing these systems becomes an important task.  And paying attention to when the type of part number changes can help a supply chain integration alert people in advance of 500 individual tape refills showing up on your table.

Question?

What terms or peices of jargon have you heard people missusing?  Or have you done this yourself?  Let me know in the comments and I may post about some of them as well.

One of the things that in invaluable is the ability to share your standard documentation both internally and externally.  EDISIM is a tool that has allowed me to do that.  They produce a very professional looking document for external consumption and the standard is exportable so that everyone using EDISIM can share the same standards file in a format called SEF.

Features of a Good Standards Reference

Navigate Standards (multiple versions, doctypes, segments, etc)

A Standards Reference needs to offer a navigation interface of some kind.  You need to be able to see the versions,  standards and document types.  You will also want to drill in and view the reference outside of the document types, and  browse the segments or data element definitions.  But at the core, you need to be able to see a graphical representation of the standard, view the format and see the structure of how data will be represented in the EDI format and document.

Edit Standards

Just browsing the standard is not going to get the job done.  You need to be able to make alteration so that you don’t just have the high level X12 standard, like the book on my desk, but can make a standard that is your usage specification.  This means you will need to be able to do two operations on the standard.

First you will need to make changes to what elements and segments are displayed, and make changes to the encoded values that these elements support.  Editing these structures will aid you when you use the standard to validate and check for compliance based on your usage specification, and when creating documents to share and publish.

Second you will be able to add some of your business rules documentation to clarify the default definition of the standard.  This is especially useful if you need to pass some data that uses a mutually defined qualifier.  This lets you define what the “mutually defined” value should be.  This again helps when you publish and share your standard.

Export and Import (Read SEF)

All of the standards references that I have used had the ability to import and export a standard in an SEF file format.  SEF or “Standard Exchange Format” is a file that holds the standard dictionary and format for EDI files.  This is a nice way to  not only transfer and store the standard your company uses so that everyone has a copy of the same usage implementation.  But if no proprietary information is stored in the notes, it can be sent to trading partners out side of the company to aid in their integration efforts.  Any other export and import is a bonus, but SEF is a requirement..

Print (export to doc or pdf, or printer)

In spite of all this paperlessness, some times we still need to print things out.  Printing to a pdf or printing to paper is useful when sharing the standard and usage specification in a non-editable format so that others can view it, and make use of the information.  So for managers, or clients, or other parties that only need a viewable format to follow, this feature is a must have.

Save as editable file (RTF or other.)

Ok, just like printing, sometimes companies want all of their documentation to look the same.  To facilitate this, it is nice to be able to save the specification out to a rtf, or doc, and then add the company header, logo, watermark, etc.  This allows you to take the standards reference and really own the edits and notes that you have added.  For companies that have extensive documentation processes, this feature is a must have.

Other Good Reference Features

Validation

Validation is not a feature of an EDI standards reference, but it is a good tool that sometimes comes with them.  Validation tools read and EDI file, and compare the contents with the standard.  They should then identify all of the errors they can see and report on them.

In my experience validations may need to be run repeatedly as some errors in an EDI file will mask following errors.  For instance, if  the N1 segment is wrong, the validator may not be able to tell if all of the other lines in the N1 loop are valid.  So when I validate an EDI file, I correct the errors seen on the first pass and re-validate.  This has saved me from telling a trading partner to fix an issue, only to have to tell them about another issue later.  After all, we do want our integrations to go quickly.

Test Data

Sometimes looking at the standard is just not enough to show us how things will look when formed into the document.  Seeing sample data is essential, and one of the first things I request at the beginning of any integration project.  A Standards Reference will sometimes have tools that will create a test data EDI file based on the standard.  This can be useful if you are working with a new document type that you haven’t encountered before and need to see how the standard looks in the formed document.

Evaluation of Standards Reference Technologies

Is there a feature that I have missed?  Please let me know.  Also,what EDI standards reference tools have you used?  What do you like or dislike about them?  How about any non-EDI specific tools?  Please let me know what you have used.  I would like to compile a list of tools and run some comparisons of what is out there.

Initial Idea for B2T

First, I need to explain the background of the idea.  I know that not everyone on twitter is, er, human.  Yep, we have “people” that are not real flesh and blood on twitter, tweeting away about what they are doing and why you should visit their site, or buy their product.  Shocking I know, but some of them have quite a large following.  (other twitter bots I think)

So then I thought, (again, in context to the airline example that Jason was blogging about), “Hey, what if I could follow a flight on twitter?  Then I could know when it was due to arrive! When it had landed! And where to meet my buddy in baggage claim!”  (Yes, I think with lots of exclamation marks.)  But I still think this is a cool idea.  Each tweet could have a short statement about the flight status and a link to the airline’s page for people that needed to look up more about it.

Okay, then I had my second thought about how to use this.  You see so far I was still thinking, “customer service by managing the twiterverse”  which is fine, I am a customer expecting good service as well.  But then I put this together with another connection.  “What about Order notifications?”

B2T Expanded to Supply Chain

This is where Jason mentions a supplier monitoring their reputation and customer attitudes by following Southwest’s example.  But I thought, “What if Dell, let me follow my order on twitter? I could know when it was shipped!  When it would arrive!  And even know updates like delays, or arriving early!”  (Again, I think with lots of exclamations.)

Have worked on supply chain systems that sent out many email notifications.  Some users liked these, and some didn’t.  Using a twitter model, a user/customer could follow the orders or processes that were important to them on a case by case basis.  BPM could be followed in the same fashion.  And so forth, just by clicking on the follow like on the checkout confirmation page or equivalent.

What do you think?

What are other ways that you can integrate twitter with e-commerce, B2B, or Supply Chains?  I want to know your ideas.  And also, what do we call this?  (“B2T”, “B2Tweet”, “BtweeB”, etc)

When Defaults Attack

Once upon a time there was a supply chain consultant. (ok, so this was me.) And this supply chain consultant setup and integration between a vendor and a service provider. This hosted integration worked really well for years. Then suddenly we received a delivery at the location of the supply chain consulting office.

Needless to say that our office was not setup to receive the large pallet of materials that were being ordered.

Yes you guessed it. It was an order that had been processed from the service provider but sent to our location as the Ship To destination. Well, we returned it and began a remediation process to find out why our address had been used instead of the correct customer.  In the mean time, the Vendor wanted to charge restocking fees. The Provider wanted their product. And we just wanted to figure out what was going wrong.

Second Order

While we were in these conversations, a second order showed up.  Yep, to our office.  Now we had everyone’s attention.  We walked the order through the process and finally found the error.  Default shipping location in the Vendors system was listing us, the supply chain consultant, as the default location.

The service provider had added a facility to their office.  But it had the same shipping location as the parent organization.  So when they were adding this with the Vendor, they indicated to use the Default Shipping Location.  They assumed that this would be the corporate office.  The Vendor didn’t know this was a problem.  But during the initial setup we didn’t have all of the shipping info, and we used our in the test environment.  Once the went live everything worked great.  No one realized that the vendor had retained our test shipping location.

Resolution

Well, being that we, the supply chain consultants figured out what was wrong, confidence resumed in the integration.  The Vendor removed us from their shipping information, and the Service Provider began to get their shipments for the new facility.  You would think that everyone would be happy.

But there were some shipping and restocking fees that were now on the books.  What a mess.  I think that we eventually spit up these fees and we had to pay a portion of them to clear the books.

Lesson

Find out what happens when things go wrong, before it happens.  Because it will happen.

EDI is not a flat file, but it does have repeated lines or segments.  Some times this repetition comes in groups and sometimes its just a line that get repeated in part of the document.  Repeating structured data is fundamental to EDI, but can become difficult to interpret at times.  Understanding the types of repeating structures and when and where they may occur can save a lot of time.

Repetition

There are three kinds of loops/repeated segments; First there is a simple segment that can appear more than once.  Lets call this a Repeating Segment.  Second there is a group of segments that appear together, and may repeat.  Lets call this a Repeating Group.  And finally there is a segment or group that repeats within a beginning and ending loop envelope.  Lets call this a Repeating Envelop.

Repeating Segments

A repeating segment is pretty much just what it sounds like, a segment that may occur more than one time.  A good example of this would be the AMT segment.  The AMT is a segment that contains a monetary value.  The AMT_01 declares what type of value it is, and the AMT_02 has the amount of money.  In an EDI document like an invoice, we may need to have many values, subtotals, taxes, base price, and so on.  Having a repeating segment lets us put all of these values on their own independent segment to be interpreted and included as best fits the Trading Partners and the nature of the transaction.

Here is an example of an unwrapped N1 group where the N3 is repeated:

N1*AC*ARTHUR JONES*1*9012345918341*11*SE~
N2*GEORGE WASHINGTON~
N3*157 WEST 57TH STREET~
N3*Suite 2001*5th Floor~
N4*CINCINNATI*OH*43017~

When segments are repeatable, they are indicated with a “Max Use” value of greater than one in the standards and usage specification.  In the N1 Group, the N2 and N3 segments may have a max use of 2.

Repeating Groups

A repeating group is also what it sounds like.  It is a group of segments that work together and get repeated.  This loop is generally referred to by the primary segment.  This segment is required in this group.  This means that if this group occurs, this segment must also occur.

Most Repeating Groups are contextual.  This means that they are defined by the segments that surround them, and are terminated by the first segment that is not in the group, or starting the group again.  So a Repeating Group may repeat, but as soon as we move out of the segments in this group, we are done repeating it and move on to the next segments or groups defined in the standard.

The N1 is a good example.  The N1 group contains name, address and contact related information.  The only required segment is the N1.  Subordinate to the N1 (that contains name and such) is the N2 (that contains secondary name or attention type data.) Then comes the N3 (with the street address) and then the N4 (with the city, state, country, zip).

Not all groups have the same letters, like ‘N’ and the N1 group is no different.  It also contains the PER segment that contains a persons name, and contact info like phone number or email.  Any PER that happens under an N1 is for that location.

In a document like an order you may want to have the address and contact for the shipper, the receiver, a biller, and a payer.  There are many possibilities, and these are all repeated in this group. Here is an example of an unwrapped N1 group where the that is repeated:

N1*ST*MRS JONES*1*123456789*11*SE~
N2*BEVERLY SIMPSON~
N3*157 WEST 57TH STREET~
N4*CINCINNATI*OH*43017~
N1*SF*MR JONES*1*9012345918341*11*SE~
N2*MAGGIE MCGILLICUTTY~
N3*1001 MAPLE STREET~
N4*PITTSBURGH*PA*86753~

Segments that are in a repeating group are listed as being in a Loop.  They may be repeated whenever the loop is repeated as specified in the standards and usage specification.  Thus you may have a max use of the N3 as 2, and an N1 Loop that has a max use of 1000.  So really you could see 2000 N3 segments, (2 for each of the 1000 possible N1 loops).

Repeating Envelope

A repeating envelope makes reference to EDI Enveloping.  Basically, the ISA-IEA envelope can contain one or more GS-GE envelopes.  And each GS-GE can contain more than one ST-SE envelope.  These levels of enveloping are covered in greater detail on the EDI Enveloping page.  But it doesn’t have to stop there.  In the 3050 850 document there is a loop called LS-LE that contains terms.  This section happens inside the ST-SE.  And there is nothing to prevent other sub envelops from happening where there is a need.

What is fundamental to these types of loops is that they have a designated beginning, and ending.  They must have both.  Repeating Groups end when a beginning of a new iteration of the loop is found, or a segment that is not in the group is found.  This can leave some ambiguity if the standard is not followed closely.  The Repeating Envelope leaves nothing to chance and declares that it is done.

A note about Required segments

You may have noticed in the examples above or in the specification that we referenced that there are required segments in loops that are optional.  This is not a contradiction and I will explain this here.  A loop may be optional.  If it is, then it may be ommitted entirely from a document while still compliant with the standard.  If the loop is included, then the required and optional segments within that loop apply.  And they apply in each instance of the loop.  (We can’t have an second N1 loop unless we include the required N1 segment.)

Basically this is like scope in XML.  The requirements of mandatory or optional only apply within the scope of the loop where they are defines. in the same fashion as sub nodes in and XML hierarchy.

Basic Fundamentals to Remember

1. Some segments can occur more than once, or can be repeated.
2. Some groups of segments can and often do occur more than once.  These are contextual and are terminated by segments after the group in the standard.
3. Some segments are encapsulated in beginning and ending segments like enveloping.  These segments can repeat, but only in another encapsulation or envelope.
4. The standard will indicate when repetition is allowed.

–

Looking for something else relating to EDI?  Check out the EDI Primer post

–

Document Identifier

The document Identifier is not encoded.  That is to say, for an 850 the ST document ID is “850″.  For an 810, the Document Identifier is “810″ and so on.  Thus if you open an EDI file, and don’t recognize the GS_01, skip to the ST_01 and it will tell you what standard you will need to find to read this document.

Good form would dictate that the GS_01 and the ST_01 would specify the same type of document.  The GS would specify it on a functional level, and the ST on the standard level.  Thus they would not be identical strings but would indeed convey the same message about what the document(s) would be.  Some EDI translator or processing software will validate this relationship and some may not.  But in the end consistency is the best policy.

Control Number

The control number is different for the ST than we saw in both the ISA and GS.  The ST_02 is a 3 to 9 digit number.  This means that it will start with 001 not 000000001.  Of course you can still use a 000000001 but don’t be surprised when you see less than 9 digits in the ST-SE control number.  Whichever you chose, you must do the same for both the ST and SE.

(Using 000 or 000000000 may be allowed, but is not recommended as any application that trims the zeroes will get an empty string and produce an error.  Be nice and only use values greater than zero.)

Segment Counter

The SE segment counter is the part of the ST-SE that gives people hand editing an EDI document the most trouble.  If the counter is inaccurate it indicates that the document is corrupted.  This should fail in all good translators as processing a corrupt EDI file is saying, “Its okay if we get bad data.”  This counter represents the number of segment between the ST and before the SE.

And it is just that simple.  Remember that you can have more than one GS-GE envelope inside an ISA-IEA, and you can have more than one ST-SE inside a GS-GE, but you can only have one document of one type in side a ST-SE envelope.

–

For an overview of Enveloping go here.

Looking for the ISA enveloping page?

Looking for the  GS enveloping page?

Looking for something else relating to EDI?  Check out the EDI Primer post

Wrapped

As I said wrapped EDI has nothing between the segment terminator and the next segment, in this format;

SEG|||~SEG||||||~

and so on.  Wrapped EDI looks like this:

ISA*00*          *00*          *01*9012345720000  *01*9088877320000  *090508*1459*U*00200*000000001*0*T*:~GS*PO*901234572000*908887732000*090508*1459*1*T*004010~ST*850*0001~BEG*01*BK*99AKDF9DAL393*39483920193843*20090508*AN3920943*AC*IBM*02*AE*02*BA~DTM*002*20090508*1615*CT*CM*DATE TIME PERIOD~N1*AC*ARTHUR JONES*1*9012345918341*11*SE~N2*MAGGIE MCGILLICUTTY*BEVERLY SIMPSON~N3*PO BOX 123*157 WEST 57TH STREET~N4*CINCINNATI*OH*43017*US*IP*02535~PER*AC*ARTHUR JONES*TE*(614)555-1212*TE*(614)555-1212*TE*(614)555-1212*ADDL CONTACT DESC~PO1*AAA849*100*DP*100.00*PD*AB*ASSEMBLY*BL*BRAND OR LABEL*ON*CUST12345*AB*ASSEMBLY*BL*BRAND OR LABEL*ON*CUST12345*AB*ASSEMBLY*BL*BRAND OR LABEL*ON*CUST12345*AB*ASSEMBLY~PID*F*BC*AX*UNSCENTED*GENERAL PURPOSE*BS*SUBQUALIFIER*N*ENG~CTT*10*855*12345678*HB*40*QD*GENERAL PURPOSE~SE*12*0001~GE*1*1~IEA*1*000000001~

Unwrapped

When we unwrap the EDI, we don’t just place any character between the segment terminator and the next segment, we place a carriage return and line feed.  This makes all of the segments appear to be justified like separate lines of a text file.  Thus the format is changed in this manner;

SEG|||~

SEG||||||~

and so on.  Unwrapped EDI looks like this:

ISA*00*          *00*          *01*9012345720000  *01*9088877320000  *090508*1459*U*00200*000000001*0*T*:~
GS*PO*901234572000*908887732000*090508*1459*1*T*004010~
ST*850*0001~
BEG*01*BK*99AKDF9DAL393*39483920193843*20090508*AN3920943*AC*IBM*02*AE*02*BA~
DTM*002*20090508*1615*CT*CM*DATE TIME PERIOD~
N1*AC*ARTHUR JONES*1*9012345918341*11*SE~
N2*MAGGIE MCGILLICUTTY*BEVERLY SIMPSON~
N3*PO BOX 123*157 WEST 57TH STREET~
N4*CINCINNATI*OH*43017*US*IP*02535~
PER*AC*ARTHUR JONES*TE*(614)555-1212*TE*(614)555-1212*TE*(614)555-1212*ADDL CONTACT DESC~
PO1*AAA849*100*DP*100.00*PD*AB*ASSEMBLY*BL*BRAND OR LABEL*ON*CUST12345*AB*ASSEMBLY*BL*BRAND OR LABEL*ON*CUST12345*AB*ASSEMBLY*BL*BRAND OR LABEL*ON*CUST12345*AB*ASSEMBLY~
PID*F*BC*AX*UNSCENTED*GENERAL PURPOSE*BS*SUBQUALIFIER*N*ENG~
CTT*10*855*12345678*HB*40*QD*GENERAL PURPOSE~
SE*12*0001~
GE*1*1~
IEA*1*000000001~

Converting from Wrapped to Unwrapped and back

When you need to do this, you need to do this fast, not manually.  Sure the manual way works with small files, but if you are trying to edit a big invoice or order, you will want to do something snappier than walking through the file segment at a time.  Here are two ways to do it with common tools.

VI

To unwrap:

1. open the file.  Type “vi [file name]“
2. Hit “:” to get the command prompt
3. Type “%” to do a every line replace. (may not be needed as this is wrapped, but it doesn’t hurt.)
4. Type “s” for a substitution.
5. Type “/” to begin the string to find.  (you may need to escape characters like ~ and | with a \.  I will show this in the example)
6. Type the delimiter to be replaced.
7. Type “/” to begin the replaced string (again with the possible need to escape characters.)
8. Type the delimiter again, as we don’t want to lose it, then follow it will a [crtl]+v and [crtl]+m
9. Type “/” to end the substitution
10. Type “g” to make it global or indicating all of the instances not just the first one.

It looks like this:  “%s/[delimiter]/[delimiter]^M/g”

Put another way, here is what I typed for our example:  “%s/\~/\~^M/g”  (notice that I am escaping the ~ with a \)

To re-wrap:

1. open the file.  Type “vi [file name]“
2. Hit “:” to get the command prompt
3. Type “%” to do a every line replace. (may not be needed as this is wrapped, but it doesn’t hurt.)
4. Type “s” for a substitution.
5. Type “/” to begin the string to find.  (you may need to escape characters like ~ and | with a \.  I will show this in the example)
6. Type “\n” to designate the newlines.
7. Type “/” to begin the replaced string (again with the possible need to escape characters.)
8. Type “//” to begin and end the substitution.
9. Type “g” so that it will get any multiples of the newline.

It looks like this: “%s/\n//g”

MS Word

To Unwrap:

1. Open file.
2. Select Edit>Replace or press [ctrl]+h
3. In the “Find what” field insert the delimiter.
4. In the “Replace with” field insert the delimiter, and then click on the “More” button on the bottom of the control.
5. In more, there is another button called “Special”  Click on this button and select “Manual Line Break”
6. Click replace all and the EDI will unwrap.

To re-wrap:

1. Open file.
2. Select Edit>Replace or press [ctrl]+h
3. In the “Find what” field click on the “More” button on the bottom of the control.
4. In more, there is another button called “Special”  Click on this button and select “Manual Line Break”
5. Don’t put anything in the  “Replace with” field.
6. Click replace all and the EDI will re-wrap.

Other Tools:

There are lots of other tools that can do the search and replace with newlines, these are only two ways.  If you have one that you would like to share, please let me know.

—

Looking for something else relating to EDI?  Try here.

]]> http://www.theintegrationengineer.com/edi-wrapped-and-unwrapped/feed/ 0