The Integration Engineer
When it just has to work.

Just like synergy, a projects time-line is generally greater than the sum of its tasks.  Taking the time to complete tasks in a project together, there will need to be time between tasks.  Some of this time will just be spent updating people, and coordination of project participants.  There are also other needs, like water and bio-breaks.

Just like productivity during the day generally gives people a 15 minute break every 3 hrs or so.  When we plan our projects, we need to include not just the time that it takes to perform the collective tasks, but other activities that will be happening during the course of the project.

Odd errors can occur when you get data that doesn’t fit the format, but matches close enough not to fail outright.  This type of error can be really hard to unravel. Preventing these messy errors, it is a good practice to implement a validation process before processing the data.  Some integration technologies come with validation, if yours doesn’t you should pre-process inbound data for validation.

In addition to this, pre validation can be used as a way to catch and handle other errors early.  Leveraging pre-processing with validation and error handling can give you a one stop place to catch, and report or correct systemic problems.

Regardless of the standards that you adopt for your integrations, you need to pick some.  EDI, XML, or any of the others.  You need to evaluate and decide on a standard that will do what you need.  And you need to be prepared to present it to your trading partners.

This also applies to other parts of your business.  Picking a standard platform for your development and standard tools for development.  And so forth.

Even small companies benefit from standardization.  Don’t be afraid to put your technology people in a room and ask them to provide standard recommendations.  But there are two cautions.

1.  The latest and greatest, may look cool to the tech people, but may also be a platform that is unproven and will change radically in a short time.

2.  Not all open source is good.  If you are going open source as part of your platform, look at the quality of the community that supports them.  (This is also true for non-open source technologies.)

The accuracy of the time-line will only be as accurate as the accuracy of the least accurate task estimate.

The time estimate part of project planning is a tricky thing.  Generally there is some guessing involved.  To keep us from falling short a good practice is to pad estimates to account for the unforeseeable.  Project planners can’t just pad and pad.  There is pressure from the other side to keep your estimates accurate and keep your time-line short.  Time is money after all.

So how do we put enough time in the time-line to not fall short, and keep the time-line from getting too long and expensive?  Well that is a dance that takes some practice to learn.

Just remember, when you pad your estimate to your project leader or get and estimate from someone on your project, that padding is necessary, and you should discuss it openly.

You may have noticed this before, but in case you haven’t, many many many projects are not completed on their due date.

I know this is shocking, and if you are a new project manager, you may want to leave the blog now.  Continuing to read this article may result in the following.

1. Better understanding of the factors that should go into a project plan time-line.
2. More effective communication with the key players in your project.
3. Increased ability to accurately predict the completion time of a project.

If this frightens you…

Well at least you have been warned. (more…)

Learn the basic tools.  Learn them first, and learn them well.  Sure, you can find some fancy XSLT tools or text editors.  And you can carry them around on a thumb drive and install them on all of you computers, workstations, laptops, and servers.  But sometime, some day, you will need to make a change, or fix a problem and you won’t have the fancy tool.  My experience says that this will also happen in the middle of the night when you are under pressure.  Knowing the basic tools like VI and Notepad, and any other tools that are a standard part of your systems will allow you to get your integration back up and running quickly and get you back to bed so you can go in and answer the questions about what happened with just that much more sleep.

This is like the Covey proverb, “Begin with the end in mind.”  In an integration project, the first step is always to define the target of where your data is going to end up.  This is true for a document conversion like XSLT or a EDI or Flat File transformation to XML or another version.  It is also true for ETL tasks where we are getting data in and out of a Database or repository.

Starting the work of transforming data or files without a well defined target for the data will cause you to do extra and unnecessary work, and to repeat some steps when the true destination is known.  It may seem obvious, but many many people fall into this trap, and then don’t understand why their projects keep exceeding their time budgets.

Keeping a copy of all of the documentation you create is a pretty general benefit.  It helps you in three major ways;

1. Having a personal copy means that if the systems that have the public copies become unavailable, you will still have access to them.
2. Some times projects that get shelved, lose their documentation.  If you have a personal copy, when the project comes back to life, you will not be starting over.
3. And you never know what future project you will be working on that will spark the memory, “Hey we solved a problem like this on this other project…”  And having the documentation for it will help you.

I have never regretted keeping a personal copy of documentation.  But I have always regretted knowing that I didn’t keep one when I could have used it.

While working for a certain company, I requested some integration information from a manager, and got the information in an attached ppt file.  I soon noticed that all of the managers used power point presentations to communicate everything.  Yes, we had interminable meetings where managers would show a slide show that they would read to us, but also received benefits, project updates and other data in ppt file attachments.

Well, I immediately realized that this was a key to communicating with my manager.  And after some observation determined that is was a part of my companies management culture.  (And yes, I started creating power point presentation to communicate with them more effectively.)

(more…)

At some point or another, you will create a set of best practices.  These are statements or procedures that your experience has taught you that you need to do to get the best, desired outcome. Many companies have these proudly included in a wiki page or document titled “Best Practices”.  Unfortunately this is not where they belong.

Best Practices should be the basis for your project plans.

Hopefully your project plans are started by using a project plan template.  If not, then you should.  On this template, your “Best Practices” should be represented.  Too many times I have been on projects where the project manager starts by gathing the information about what must be done.  Instead, start with how it should be done. Otherwise you will encounter the painful process of trying to apply best practices to a project that is already in motion.

Is it better to have a team, or the solo, maverick developer?  It depends.  The solo developer can work unimpeded by others, but doesn’t have anyone to work with to overcome obstacles.  Teams have meetings and have to coordinate their work.  Slowing them down.  As teams grow the problem gets worse.

Sometimes resource planning and project management pressure encourages us to allocate additional engineering resources to development projects.  And there are definitely times when having a team is the best way to work quickly.

However, on integration projects more than a couple people slows the progress of the project.

I was once involved in a merger of companies.  The acquiring company had a large team of developers working on replacing the functionality of my company, the acquired one.  I became a team of one and the plan was that I would maintain the systems until the migration.

Management soon became frustrated as new feature development stagnated.  So they called me and asked how long it would take to produce a specific new integration feature.  My estimate, 40hr over two weeks.  Then they called a meeting and asked the leader of the development team how long.  His estimate, 6 months or more.

This wasn’t solely a matter of competence.  Sure, I was still there because I was the best on my technology.  But the other team was not populated with slouches.  There were just too darn many of them.  They got in each others way.

I, on the other hand, had no one to coordinate with.  No one to slow me down.

Teams of one are not always the best, but teams of 50 are never the best.

Or as a friend of mine likes to say, “It takes one woman nine months to have one baby, but putting nine women on the job doesn’t get you a baby in only one months.”

Some times we overlook the basics. And from time to time it is good to ask and answer the basic questions. One of the most basic questions for us is to ask, “What is Data Integration?”  After asking, we need to provide an answer.

Data Integration is the process of transforming heterogeneous data into a useful homogeneous data set.

All of the techniques and disciplines that we use as integration engineer help us to transform (not must map) heterogeneous data (data in more than one form, source, type, etc)  into a useful (not common, and not necessarily the final) homogeneous (single form, schema, location, etc) data set.

The integration funnel view is where we have data in more than one format or more than one source or both and we integrate them with one destination.  (We also do the reverse, but that’s not very funnel like.)  To do this effectively we don’t do a unique integration for each format and source of data.

There are two versions of the funnel.  First is where we pick a format and map that data to our internal system.  Then other formats we modify to be in this preferred format, and then map then through the original process.

Second, we create a intermediary format of all of the required data, and them translate all external data to this format of convenience.  Using a convenient dataset as our intermediate format, we can apply mapping logic in the external translation, and business logic on the importation of the convenient data.

I prefer using the convenient dataset, but either way we are able to reuse the mapping logic and using that as leverage are able to do more rapid integrations.

Growing up on a farm, I learned that almost any problem could be solved, (at least temporarily) with the proper application of bailing wire and duct-tape.

However, my father was a mechanical engineer, and he had a different philosophy that he wanted me to learn.  Routinely after I had “fixed” something. He would show me how getting the right tool,  and part worked so much better and longer.

(more…)

I was at a business seminar where the instructor said, “Every task can be broken into three pieces; Inputs, Outputs, and Processes.”  I have thought about this and would like to echo that with more emphasis on integration projects and tasks.

Integrations can be broken into these three pieces as well.

Input:  Data that we get from outside whatever we are integrating.  We can be getting orders from a third party company, or we can be talking about getting data from an account DB on another server in our office.  In any case, and depending on the scope of our integration task we can define our inputs to be anything that is coming in externally to the product we are integrating.

Output:  Data that we transmit, broadcast or convey to others.  This can be our webpage, or orders we are sending to a vendor.  In any case and depending on the scope of the integration task we define our outputs to be anything that comes out of the products we are integrating.

Processes:  This is anything that we do to the data in-between input and output.  Translating data format, calculations, collation of data, or anything else that you can do with it count as a Process.

When we talk about getting the well defined target as the first step in the data mapping process it is not because starting with source data is harder.  And we don’t talk about sequence of project tasks because we are like little children who want to eat desert first.  There are two major benefits that doing an integration project (and other projects) in the right order bring; Cost, and Quality.

(more…)

To be quite honest, 80% of the work to pull off an integration is easy. It still has to be done right, but after you have done more that one supplier boarding there are many things that just make sense.

But then there is that 20% that is not so easy. 20% that you end up spending way too much time on. And its this 20% that produces the most exceptions, the most errors and the most cost.

We need the whole integration, we can’t just skip this part. But we can be aware of it and do the work to ensure it doesn’t grow or become a stubling block to our project.

Short list for me:

• new tech that is mandated or cool
• undefined support contacts
• one off integrations
• un assured delivery
• unmonitorable processes
• integrations without a well planned design

These are not in any particular order of occurrence or severity that I am aware of.

I am sure that there are other cases and causes for that 20%  that either don’t come to mind or that I haven’t encountered. (Lucky me)  Please feel free to add to this list.

And I expect that I will be blogging about some of these in more detail. 

A friend of mine once said in a meeting, “Let make sure we’re not trying to save the world when all we need to do is get lunch.”  This stuck with me and I often see teams of developers getting excited about what can be done to the point where they loose site of what needs to be done.

When this happens there is a tendency to try and “save the world” or at least far exceed the scope of the project or problem.  To avoid this tendency, it is good to have someone in the meeting and involved in the planning and scheduling of resources that will constantly ask, “How does doing this help us solve the problem?”  And this also means that we should have a well defined goal for our project.

(If you are your project, write down the purpose and focus and read it every day, or more often to stay on target.)

Projects that succumb to “save the world” tendencies have a propensity to get bogged down, experience massive scope creep, miss their deadlines and expend lots of energy that does not return in realized benefits to the developer, team or company.

It has been said that “the definition of insanity is to do the same thing over and over and expect different results“. While this may or may not be true, programs that can easily get caught in repetitive or infinite loops can certainly be the cause of crazy data and logs.  And so we need to recognize this potential and make sure we don’t write integrations susceptible to this ailment.

But there is another type of  “doing the same thing,” that can be just as maddening.  These are errors that continue to happen and no one bothers to fix them because the resolution is simple even if it is manual.  Like hitting the side of an old TV to clear up the picture.  It may work, but eventually all that time spent with the fuzzy picture, or making that quick manual change add up.  And that time is worth money.

What I recommend is that errors not only be tracked to ensure someone is alerted and they are resolved.  But errors instances are counted and reported so that support, development and management can see what the most frequent errors are.

Asking your support people to do the same thing over and over may not mean that you or they are insane, but it can result in something like it.  Find ways to automate or eliminate these repeating time wasters.  And with the data of how often and how long they took to fix, you can report how much time you are saving everyone.

and simple tools have simple problems.

In one of the early scenes in “The Patriot”, Mel Gibson’s character gives his young 11 and 12 year old sons some last minute instruction before ambushing a British patrol to rescue the older son.  He says, “Remember, aim small, miss small.”  The meaning behind this is simple.  If you are aiming small, say, at the button on your target’s shirt, and then you miss, you still actually hit the target.  This is “missing small”.

But if you aim big, say, at a whole person, or bear, and then you miss, you don’t hit your target, you may even hit something else that you didn’t intend.  This is “missing big”.

Selecting a tool to solve an integration problem is like aiming at a target.  Aiming small, is selecting a simple tool.  Aiming big is selecting a fancy tool.  Simple tools have simple problems, while fancy tools have fancy problems.