The Integration Engineer
When it just has to work.

One of the basic tasks Integration Engineers do is to compare files that we use or receive.  There are some interesting and useful tools that people can get out there to DIFF files.  But on Linux and Unix machines around the world there is a native tool that is almost always present.  Amazingly it is called DIFF.

Like some other command-line tools, its interface is not really intuitive.  Lets walk through the basics of how to get use out of this handy file comparing tool.  (If you are working with and comparing EDI files, you might want to look at the post on how to unwrap your EDI file so that our line by line comparison is more meaningful.)

(more…)

Sometimes a picture can bring more clarity to a concept. For Canonical Data, an animation is what is called for. I found this animation of canonical data and its implementation.  I think the first minute and a half paint a very good picture of how canonical data is implemented and can be leveraged.  Later in the animation they start to describe a global vision of implementation.  Unfortunately I must disagree with this vision.  I don’t think that having a global canonical form of data will ever truly be a solution that works.

(more…)

Like the exchange of data between Sellers and Suppliers, the exchange of data within the company is vital to successful processing of transactions.  This could be the Seller or the Supplier, it doesn’t really matter.  As data passes from the processing application, to the external interface.  We have a concept that is called a Canonical form of data.  Canonical data is the data that is required and essential to completing your business.  I have written about this in passing before, but today I want to talk about what goes in it, and what it looks like. (more…)

This is a mapping exercise that will go through the process of creating a paper map, or mapping document.  We will start with an empty paper map that you can get here.  And we will end with a completed paper map document that documents what data from the source goes into what fields on the target.  This process will take more than one post, and I will link them together so that you can follow from one to the next.  Along the way, we will discuss the things that we are doing so that you can apply this technique in your mapping using the target and source in your own mapping tasks. (more…)

There are job or task specific tools that will have a high importance to each integration task.  When working on an SAP system, your SAP tools will be very important.  But there are tools and skills that are also important regardless of the systems and technologies that you are working on.  For me, these are the top 5 tools that an Integration Engineer should be able to use proficiently.  Do you use any of these?  Do you have others? (more…)

Definition:  A database is a structured collection of records or data.

Many moons ago, I was shown an database that was constructed back in the 50s.  It was hand held and consisted of a stack of cards that contained information about plant biology.  There was a series of holes that wrapped around the cards, some of the holes were notches, and others were not.

To query the database, one placed a pin through one of the wholes and let the cards that had a notch fall out of the stack.  You could continue this process by removing and adding pins to select different combinations of data.

Once you had narrowed your search, you could read the cards, and retrieve the data.

This was a very manual process, and I doubt that anyone today would take the time to learn to use such a tool, let alone update or add records to it. (more…)

I found this video on YouTube.

RFID is exciting technology for integration engineers.  All of this data coming in will have to be aggregated and integrated by someone.  Also, one of the focuses of integration is to provide more and better information to people and systems.  With more and better information people, businesses and systems can make better choices and decisions. (more…)

What is a “Character Set?”

A character set is a collection or library of characters, (letters and symbols), and their identifying number.  Included with the printable characters, (letters and punctuation) are some unprintable yet important characters.  Characters are used to form messages.

Characters are not fonts.  Characters exist under the font that represent the definition of the character the  font is attempting to display.  When you change the font on a document the A is changed to an A, but the underlying character that identifies its meaning remains the same.  The font identifies how the character is displayed.  You can even convert to Wing Dings and the underlying character remains the same. (more…)

A circular file is not a nickname for the waste can. Circular files, sometimes called round files, are useful in some applications and support tasks.  With a normal log file or repository, the log grows as logged events are added to the log.  The obvious danger is that if the space where the log is located becomes saturated when the log grows to fill this space.  Many applications will shut down and refuse to restart if this happens.  For some applications, having the log write to a circular file is the answer.

(more…)

What is a flat file?

Files are called “Flat Files” when they contain a single data structure.  Generally this structure is the column and row structure like a spreadsheet or table, but a file in binary or encrypted with a single encryption key could also be called a flat file.  Files that are not flat; marked up files like XML or HTML, EDI files, other formats like HL7 or SEF files and others.  Here I am going to briefly discuss two flat file types; Delimited Files, and Fixed Width Files. (more…)

“Is not the beginning of wisdom the words: ‘I do not know’?”

– Data, Star Trek: Next Generation: “Where Silence Has Lease“

If the beginning of wisdom is to realize what it is that we do not know. NULL, by its definition is this not knowing.  We do not know what NULL is, this is why it is NULL. (more…)