One of the basic tasks Integration Engineers do is to compare files that we use or receive.  There are some interesting and useful tools that people can get out there to DIFF files.  But on Linux and Unix machines around the world there is a native tool that is almost always present.  Amazingly it is called DIFF.

Like some other command-line tools, its interface is not really intuitive.  Lets walk through the basics of how to get use out of this handy file comparing tool.  (If you are working with and comparing EDI files, you might want to look at the post on how to unwrap your EDI file so that our line by line comparison is more meaningful.)

How to used “diff”

You can get the real basics by executing “diff –help” to get the basic help and options for this application.  But in short, here is the thumb nail. “diff” is followed by some options.  Options are designated by a “-” and then a letter indicating the option.  Any options are then followed by the two file names that are being compared.  Lets look at an example.

Example:

We have two files, file1.txt and file2.txt

When we issue this command:  “diff file1.txt file2.txt” we get this result.

2c2
< And this is the first line of the first file.
—
> And this is the first line of the second file.
3a4
> Again.

• The first thing we see is “2c2″  This is line 2 of the first file, compared or changed to line 2 of second file.
• Next we have a < indicating the first file, and the line echoed.
• Following this we have a “—” as a separator between the line compared.
• Next we have > indicating the second file, and then that line is echoed.
• This is a comparison between to lines that were found to be different.
• For the next line that is shown, we have “3a4″ that indicates that there is a line added to the file.
• Finally, > indicates the second file followed by the line being echoed.

If we were to compare them in the other order, we end with these two lines:

4d3
< Again.

• Here, “4d3″ that the 4th line on the first file is deleted from the second file.
• Following this is < indicating the first file, and echoing the line.

Regular Options

Here are the list of options that “–help” gives you, with maybe some more explanation.

diff [-b] [-i] [-t] [-w] [-c] [-C] [-e] [-f] [-h] [-n] [-D string] [-l] [-r] [-s] [-S name] [fileone filetwo ] [directoryone directorytwo]

For comparing file, the first four options (-b, -i, -t, -w) are the most useful.  I don’t start with any options and add them as I need them to reduce the amount of change noise reported in the result set.

diff is your friend

Like many basic tools, “diff” is almost always there. And if you know how to use it effectively, it can really save time and frustration. Sure there are other cool file comparison tools. Some are even embedded into other products. But knowing how to use the basic tools that are always there will be a life saver in a crisis situation. And the only way to know how to use them is to actually use them sometimes.

Do you use “diff” with a set of options that does a specific task for you? If so, what are they, please share. And what other basic tools do you use?

1.  A big Whiteboard

This is probably my number one requirement.  When I am thinking, I like to draw it out.  I haven’t found an application that gives me the same creative release and adaptability as my whiteboard.  After starting to do some of my work from home, I went out and acquired a 10 X 4 whiteboard to put in my home office.  Having the extra room is essential.  I can get a call, and walk over to the whiteboard and start drawing out what needs to be built to solve the problems while I am still on the call.

You may not need a white board as big as mine.  And you may have an electronic solution that you like better.  If so, please let me know, I like to try out gadgets.

2.  Spreadsheet

Now this may not sound earth shattering, but we are not just talking about the basics.  You will need to learn to write filters, import data, link cells and perform calculations.  If you think that a spreadsheet is like a ledger, then you are missing the power of a spreadsheet.  If you don’t think you have the skills you need, here are some links to tutorials for the two most popular spreadsheet software.

• Very Basic Open Office Tutorial http://www.tutorialsforopenoffice.org/
• More specific/advanced tutorials http://openoffice.blogs.com/openoffice/
• Some basic and advanced help for MS Excel http://www.internet4classrooms.com

3.  Text editor.

Familiarity with more than one is needed as you will find yourself on Windows and Unix servers and they will have different sets of tools.  One of the things that is the most frustrating is to not know how to use the native editor of the system you are on.  So get familiar with Notepad, and then get familiar with VI.  You can add in other tools like UltraEdit, TextPad, and more, but you should know the native ones first, if not best.

• Ultra Edit is a popular tool in some circles.  http://www.ultraedit.com/
• Text Pad is also popular.  http://www.textpad.com/
• Believe it or not, notepad has some tutorials.  http://bink.nu/news
• And here is a cheat sheet for VI commands.  http://downloads.techrepublic.com

4.  File compare.

One task that will need to be done is to compare a file with another file to detect changes or differences.  This happens in both new integrations, and in trouble shooting or investigating existing ones.  Some systems have native comparison tools, others don’t.  And there is much variety in how they work.  Here are some of the tools I have seen and used.

• Wiki has a great list.  I don’t know that I can improve on.

5.  DB query

Much of the time, integration involves a database somewhere sometime.  To work on an integration and not have to have a DBA sitting in your lap, you need to be competent to query the database.  And this involves using some tool.  Sometimes systems will have native tools, other times you will need to connect your own.  Here is a list of tutorials for DB Query tools that I have seen and used.

• Squirrel SQL http://squirrel-sql.sourceforge.net/
• SQL Worksheet
• TOAD
  

Summary

This is by no means attempting to be a comprehensive list of tools.  Such a list would be long, if it were possible.  And I don’t think that it is.  There are however some tools/skills that help us to be more effective as Integration Engineers.  Seeing what others use is helpful especially when we find that the tool we used to use is not longer around.

What tools, applications, or skills do you find you are falling back on often, on more than one project?  Or what new tools have you found that you think you will use often in the future?

A character set is a collection or library of characters, (letters and symbols), and their identifying number.  Included with the printable characters, (letters and punctuation) are some unprintable yet important characters.  Characters are used to form messages.

Characters are not fonts.  Characters exist under the font that represent the definition of the character the  font is attempting to display.  When you change the font on a document the A is changed to an A, but the underlying character that identifies its meaning remains the same.  The font identifies how the character is displayed.  You can even convert to Wing Dings and the underlying character remains the same.

We can imagine that if I wrote this post using a character set that I created myself.  And then you came and tried you read it, without knowing what my character set was, you would see a bunch of garbage on the screen like if you go to a foreign language web page without the correct fonts loaded.  Even worse would be if you used the same characters, but had different Identifiers for them.  If an A is 001 (my set starts at A and moves on numerically) and you try to read it, (but in your character set you numbered the vowels after the consonants) and 001 to you is B.  Now all of the letters will be wrong.  And we get garbage.

Fortunately, some people got together early on and created a standard for characters.  The American Standard Code for Information Interchange created the character set we call ASCII.  The Extended Binary Coded Decimal Information Code was created by IBM, but they use ASCII now as well.

What is ASCII

ASCII is the acronym for American Standard Code for Information Interchange, and is a collection of characters defined from 0 to 127.  These definitions represent all of the standard English characters, numbers and symbols.  A number of other, unprintable, characters are also included.  You use one or two of these each time you hit the “Enter” key on your keyboard.  Depending on your operating systems, this sends the “carriage return” and or “line feed”

A “carriage return” comes from a printer where the head would move back and forth on the roller.  CR would tell the printer to move the head all the way to the left of its printing area.  A “line feed” is also from a printer perspective.  This tells the printer to roll the paper so that the head will be writing on the next line.  These are both examples of unprintable characters.  You can probably think of others.  For a complete list of ASCII characters, you can check out this table in my toolbox.

What is EPSIDIC

EPSIDIC  is the acronym for Extended Binary Coded Decimal Information Code.  This was created by IBM back in the day.  IBM now uses ASCII just like everyone else, but there are legacies that are still with us.  Old terminals like VT100 and some legacy communications equipment still expect messages using the EPSIDIC character set.

What is the big deal

As I said, some systems still want to use some of the characters in the EPSIDIC system.  Even fancy new systems producing XML will sometimes fall into this trap and cause problems.  The one that I have run into is the use of | called ‘pipe’  ASCII and EPSIDIC use different character IDs for this character.  And I have seen e-commerce systems, that are using ASCII for everything else, throw in an EPSIDIC pipe as a control character.  When this happens, other systems will choke on it.

When you find yourself getting an invalid character message, but the characters look fine.  Remember that there are some twists that may exist in the underlying character set.  If you can, manually replace the character with the character that it looks like, (in my case the EPSIDIC | with a ASCII | )  and see if the parser likes the file now.  If it does, you have encountered the character set problem as I have.  This can be a difficult problem to solve if you have never encountered it.

If the character that is causing problems is not a pipe, you may want to look at IBM’s ASCII to EPSIDIC conversion table.  This can be difficult to communicate with others that have never encountered it, so using the ACSII and EPSIDIC identifier designation can help explain what we are saying in email and documentation when we are trying to correct the issue.

Files are called “Flat Files” when they contain a single data structure.  Generally this structure is the column and row structure like a spreadsheet or table, but a file in binary or encrypted with a single encryption key could also be called a flat file.  Files that are not flat; marked up files like XML or HTML, EDI files, other formats like HL7 or SEF files and others.  Here I am going to briefly discuss two flat file types; Delimited Files, and Fixed Width Files.

What is a Delimited File?

Ok, to describe it briefly, a delimited file is a file where the data is organized in rows and columns.  Each row has a set of data, and each column has a type of data.  If it sounds like I am describing a spreadsheet, you are right on the money.  To make the column, each row has the columns separated with a character called a delimiter.  See the example below.

Tables of data and spreadsheets are both similar to a delimited file in the way they organize data.  In the delimited file all of the empty space, or white space is removed.  What we see here is a classic example of exporting a spreadsheet table as a comma delimited file.  In theory, this data can be imported by any other application that can read a delimited file.

Believe it of not, a space is a character, and takes up space in a file.  Back in the day people went out of their way to save space so that files could be send over slow modem connections.

What is a Fixed Width File?

There is another type of file, is is called a Fixed Width or Fixed Position file.  It is different from a delimited file in that the data fields are defined by the character position.  See the example below.

In a fixed width file, the delimiter characters are eliminated.  If the data is formulated such that the data fields are the same size, this format can be more compact than a delimited file. You can see here that we know the size of the Birthdate data, so we eliminate all the spaces between the Bdate and Department fields.  If all of the data was formatted for size like this, we could really make this file small, so that it only contains the data.

We also eliminate the pesky problem of delimiters found in data.  The issue of a comma delimited file containing a field that has a comma in the data.  How does the parser know that this comma is not really a delimiter, but is part of the data?  Anyway, that problem is eliminated in a fixed width file.

Comparison

This is not a contest of which format is superior.  Both file architectures are useful and both are used commonly enough that you need to be at ease working with both.  Delimited files are really easy to work with as long as your data is clean of the delimiter character.  Doing quick integration of data common in ETL tasks, delimited files are far more common that Fixed Width.  Continuous operations of data integration and importation many times find that Fixed Width or Position files are more reliable for the unattended operation, even ETL if it is unattended.

As with many things in integration work, we want to pick the best option.  Knowing and working with both fixed and delimited files will help you determine which is the right choice for the task you have before you.