The Integration Engineer
When it just has to work.

I know it may sound cleche to state that a picture is worth a thousand words.  But some things are just really difficult to communicate in text.  You may have noticed a few diagrams showing up in my posts here.  Well, I have found that there are times in a conversation with other programmers, and more often in conversations with non-technical collegues that I am saying, “I need a whiteboard.”

Because of this propensity to illustrate my explanations, people have jokingly said that I couldn’t talk if they took my dry erase markers away.  At the same time, being able to communicate with others that are not versed in programming, databases, chemestry, or any other disapline that you find yourself will be a useful skill that will help you with your immediate project, and can advance your career.

When the definition for a data field or element is undefined or otherwise open to the whim of the user, we are asking for trouble.  Not in the sense that this causes trouble.  But how it is then used that causes trouble.  And this is not to pick on the CDATA tag in XML or ZZ qualified elements in EDI.  It seems that every data format of any size of adoption commits this sin of categorisation.

My general recommendation is to avoid using undefined or user defined data.  Relying on a user defined field that we use to solve a problem by working around some limitation (or perceived limitation) of the file data definition, may give us a feeling of power.  But over the long haul turns the implementation into a proprietary implementation that is not reusable with more than one trading partner.

And having each integration using a slightly unique implementation leads to chaos.  Both in suppport, and in the applicaiton of updates.

Undefined data exists.  People will use it.  You will have to deal with it as some time, but whenever possible. avaid making it a critical part of the implementation, and always provide it with a healty dose of error handling.

Just like synergy, a projects time-line is generally greater than the sum of its tasks.  Taking the time to complete tasks in a project together, there will need to be time between tasks.  Some of this time will just be spent updating people, and coordination of project participants.  There are also other needs, like water and bio-breaks.

Just like productivity during the day generally gives people a 15 minute break every 3 hrs or so.  When we plan our projects, we need to include not just the time that it takes to perform the collective tasks, but other activities that will be happening during the course of the project.

Odd errors can occur when you get data that doesn’t fit the format, but matches close enough not to fail outright.  This type of error can be really hard to unravel. Preventing these messy errors, it is a good practice to implement a validation process before processing the data.  Some integration technologies come with validation, if yours doesn’t you should pre-process inbound data for validation.

In addition to this, pre validation can be used as a way to catch and handle other errors early.  Leveraging pre-processing with validation and error handling can give you a one stop place to catch, and report or correct systemic problems.

The accuracy of the timeline will only be as acurate as the accuracy of the least acurate task estimate.

The time estimate part of project planning is a tricky thing.  Generally there is some guessing involved.  To keep us from falling short a good practice is to pad estimates to account for the unforeseeable.  Project planners can’t just pad and pad.  There is pressure from the other side to keep your estimates accurate and keep your timeline short.  Time is money after all.

So how do we put enough time in the timeline to not fall short, and keep the timeline from getting too long and expensive?  Well that is a dance that takes some practice to learn.

Just remember, when you pad your estimate to your project leader or get and estimate from someone on your project, that padding is necessary, and you should discuss it openly.

Learn the basic tools.  Learn them first, and learn them well.  Sure, you can find some fancy XSLT tools or text editors.  And you can carry them around on a thumb drive and install them on all of you computers, workstations, laptops, and servers.  But sometime, some day, you will need to make a change, or fix a problem and you won’t have the fancy tool.  My experience says that this will also happen in the middle of the night when you are under pressure.  Knowing the basic tools like VI and Notepad, and any other tools that are a standard part of your systems will allow you to get your integration back up and running quickly and get you back to bed so you can go in and answer the questions about what happened with just that much more sleep.

This is like the Covey proverb, “Begin with the end in mind.”  In an integration project, the first step is always to define the target of where your data is going to end up.  This is true for a document conversion like XSLT or a EDI or Flat File transformation to XML or another version.  It is also true for ETL tasks where we are getting data in and out of a Database or repository.

Starting the work of transforming data or files without a well defined target for the data will cause you to do extra and unnecessary work, and to repeat some steps when the true destination is known.  It may seem obvious, but many many people fall into this trap, and then don’t understand why their projects keep exceeding their time budgets.

Keeping a copy of all of the documentation you create is a pretty general benifit.  It helps you in three major ways;

1. Having a personal copy means that if the systems that have the public copies become unavilable, you will still have access to them.
2. Some times projects that get shelved, lose their documentation.  If you have a personal copy, when the project comes back to life, you will not be starting over.
3. And you never know what future project you will be working on that will spark the memory, “Hey we solved a problem like this on this other project…”  And having the documentation for it will help you.

I have never regretted keeping a personal copy of documentation.  But I have always regretted knowing that I didn’t keep one when I could have used it.