Files are called “Flat Files” when they contain a single data structure.  Generally this structure is the column and row structure like a spreadsheet or table, but a file in binary or encrypted with a single encryption key could also be called a flat file.  Files that are not flat; marked up files like XML or HTML, EDI files, other formats like HL7 or SEF files and others.  Here I am going to briefly discuss two flat file types; Delimited Files, and Fixed Width Files.

What is a Delimited File?

Ok, to describe it briefly, a delimited file is a file where the data is organized in rows and columns.  Each row has a set of data, and each column has a type of data.  If it sounds like I am describing a spreadsheet, you are right on the money.  To make the column, each row has the columns separated with a character called a delimiter.  See the example below.

Tables of data and spreadsheets are both similar to a delimited file in the way they organize data.  In the delimited file all of the empty space, or white space is removed.  What we see here is a classic example of exporting a spreadsheet table as a comma delimited file.  In theory, this data can be imported by any other application that can read a delimited file.

Believe it of not, a space is a character, and takes up space in a file.  Back in the day people went out of their way to save space so that files could be send over slow modem connections.

What is a Fixed Width File?

There is another type of file, is is called a Fixed Width or Fixed Position file.  It is different from a delimited file in that the data fields are defined by the character position.  See the example below.

In a fixed width file, the delimiter characters are eliminated.  If the data is formulated such that the data fields are the same size, this format can be more compact than a delimited file. You can see here that we know the size of the Birthdate data, so we eliminate all the spaces between the Bdate and Department fields.  If all of the data was formatted for size like this, we could really make this file small, so that it only contains the data.

We also eliminate the pesky problem of delimiters found in data.  The issue of a comma delimited file containing a field that has a comma in the data.  How does the parser know that this comma is not really a delimiter, but is part of the data?  Anyway, that problem is eliminated in a fixed width file.

Comparison

This is not a contest of which format is superior.  Both file architectures are useful and both are used commonly enough that you need to be at ease working with both.  Delimited files are really easy to work with as long as your data is clean of the delimiter character.  Doing quick integration of data common in ETL tasks, delimited files are far more common that Fixed Width.  Continuous operations of data integration and importation many times find that Fixed Width or Position files are more reliable for the unattended operation, even ETL if it is unattended.

As with many things in integration work, we want to pick the best option.  Knowing and working with both fixed and delimited files will help you determine which is the right choice for the task you have before you.

In short, delimiters are characters that separate the elements of data so that one piece of data can be distinguished from another.  EDI files don’t have externally set delimiters.  This means in a pure sense, that an EDI parser may not know what the delimiters will be until it has begun to parse the file.  This may sound chaotic for someone familiar with strict delimited files.  But we need to remember that in and EDI file, the first segment is fixed position.  This is how EDI addresses the problem.

In EDI there are three delimiters, each with a separate purpose and usage.  First, there is a segment delimiter.  This is the last character of every segment.  It is sometimes referred to as the segment terminator.  Next, there is the Element delimiter.  This character separates each data element from the neighbouring data elements.  This character will most likely be the most common character in any given EDI file.  The least commonly used character is the sub-element delimiter.  Some elements have sub elements defined within them.  In my experience these location asre used rarely.  You will probably find that this character only appears once in most of your EDI files, and that is in the ISA segment where it is defined.

Where do you find the delimiters in the EDI file?

Delimiters are essential in deciphering the contents of an EDI file and getting data out of it.  Because the parser needs to know the identity of these character before it can do meaningful parsing.  It must know where to find them.  In every EDI file the first segment is the ISA segment.  This segment is always 106 characters long, counting the first character “I” as 1.  This means that if our parser grabs the 106th character it shows the segment delimiter.  And the two characters that preceded it are the element and sub-element.

Element Delimiter = Character 104

Sub-Element Delimiter = Character 105

Segment Delimiter = Character 106

What good are delimiters?

First lets say that this list of criteria will be incomplete as it is mostly based on my experiences.  Second, though I am gifted I doubt that I can imagine every possible way to create confusion.  Now here are the rules:

1.  Delimiters must be unique.  Even if you don’t use the sub-element, you still have to use a different character than used in the other delimiters.

2.  Delimiters must be single characters.  (we will talk about the exception on the Good Form section)

3.  Delimiters must be printable ASCII characters, excluding the alpha and numbered characters.  Also excluding the common punctuation.  (having the letter “B” as a delimiter would make a file unreadable.

4.  There are characters that are forbidden like the back tick.

5.  There are characters that are allowed like “*” but that are appear commonly in text fields and descriptions, so they should be avoided.

Included with my EDI Usage Specification are a set of recommended delimiters.  I set my transformer to use them as default, and encourage my trading partners to do the same.

What’s the big deal?

Delimiters are critical to having your integration function smoothly.  Selecting good delimiters and following good practices for using them help keep things from facing constant revision.  Being consistent helps your trading partners.

–

Looking for something else relating to EDI?  Check out the EDI Primer post

Eliminate trailing delimiters.

When EDI was first created, modems were much slower, and high speed internet and Wi-Fi didn’t exist.  Moving large, verbose files over a network took time, and e-commerce solutions charged by the kilo-character.  This means that having unneeded characters cost money.

EDI, in good form tries to eliminate extra characters.

The rule is to keep your segments as compact as possible by eliminating optional trailing data elements.

Choosing Effective Delimiters

It is illegal to have a delimiter as part of the data in any data field.  There is no escape character that will allow it.  That being said, many systems will allow EDI that breaks this rule to be formed, as it does not harm the system sending the message.

However, it becomes impossible for the receiver to read the message properly.  (I know that some of you have encountered and scripted solutions for this type of thing, but this is about good form, not work-a-rounds for bad form.)  If you are lucky, this violation only errors out of translation.  If you are not so lucky, the translation succeeds and you have corrupt  data in your system.

The rule is to select delimiters that will never appear in your output data.

Avoid Unnecessary Repetition

Some segments can be found in multiple locations in an EDI document.  This allows for a flexible and context sensitive usage of segments and data.

Lets take the N1 segment and group for example.  In a shipping notice you can use the N1 in the header section before any of the line items, but you can also use it in the line level.  This gives you power to specify specific shipping locations for the line.

This is great if each line is going to a different locations, but if most of the time this means that the N1 group repeats the same information over and over, then we are wasting space and are not in good form.  It would be best if we place as much information where it will only be displayed once if it is the same.  And in our shipping example, we can specify that N1 only appears in the line if and when there is an overriding shipping locations.  (this is where having effective usage specs comes in handy.)

The rule is to avoid repetition of any data that can be represented in a non-repeating structure.

Remove Excess White Space

Just like any other character spaces take up room in the file.  Outside of the ISA segment, no EDI fields should contain leading or trailing white space.

The rule is to trim or remove any whites space in an EDI field.

Send EDI in Wrapped form

When EDI is wrapped it looks like one continuous line of text characters.  When EDI is unwrapped each line appears on its own line.  Normally people do this by inserting carriage returns and line feeds after each segment delimiter.  Humans have an easier time reading an unwrapped EDI files.  So in most of the examples of EDI that I use here are unwrapped EDI.

Wrapped and unwrapped EDI are both EDI and most parsers don’t have a problem with either one.  But line feeds and carriage returns are characters and take up space.  And because in good form EDI eliminates excess characters, these should be eliminated as well.

The rule is, always transmit wrapped EDI.

Some people try to have the best of both worlds and try to use a line feed as a segment delimiters.  This is a bad idea.  First, delimiters are supposed to be a single character, second, different operation systems treat these characters differently.  And some systems pre process files and remove carriage returns and line feeds.  Basically, its not a nice way to do it and will eventually cause someone trouble.

What all these rules have in common?

What all of these rules have in common is saving space.  Creating EDI that uses extra or unnecessary characters is bad form.

–

Looking for something else relating to EDI?  Check out the EDI Primer post

–

This introduction aims at addressing the basic questions of convention and implementation of the EDI technology.  I have a list of topics, and hope that over time I can cover all of them.  I also hope that people reading here may have questions that I haven’t thought to write about.  Please let  me know what your questions are and I will see what I can do about providing answers.

• What is EDI?
• Parts of EDI
• Segments
• Delimiters
• Conventions of EDI
• Why are there more than one version of EDI?
• Standards and Usage
• Encoded Values
• EDI in Good Form
• Element Notation
• Wrapped and Unwrapped EDI
• Enveloping Part One
• Enveloping Part Two – ISA
  
• Enveloping Part Three – GS
• Enveloping Part Four – ST
• Loops in EDI
• Hierarchical Levels

Of course understanding how EDI presents and represents data is the first step.  From here we may want to go to our Data Mapping Guide, or if we are familiar with mapping there are some examples in our mapping case studies of EDI to and from other formats.  Again, if you have an interesting problem or solution, I would love to here about it.  Please send it to
roy@TheIntegrationEngineer.com and I will see what I can do.