Starting the work of transforming data or files without a well defined target for the data will cause you to do extra and unnecessary work, and to repeat some steps when the true destination is known.  It may seem obvious, but many many people fall into this trap, and then don’t understand why their projects keep exceeding their time budgets.

1. Having a personal copy means that if the systems that have the public copies become unavailable, you will still have access to them.
2. Some times projects that get shelved, lose their documentation.  If you have a personal copy, when the project comes back to life, you will not be starting over.
3. And you never know what future project you will be working on that will spark the memory, “Hey we solved a problem like this on this other project…”  And having the documentation for it will help you.

I have never regretted keeping a personal copy of documentation.  But I have always regretted knowing that I didn’t keep one when I could have used it.

While working for a certain company, I requested some integration information from a manager, and got the information in an attached ppt file.  I soon noticed that all of the managers used power point presentations to communicate everything.  Yes, we had interminable meetings where managers would show a slide show that they would read to us, but also received benefits, project updates and other data in ppt file attachments.

Well, I immediately realized that this was a key to communicating with my manager.  And after some observation determined that is was a part of my companies management culture.  (And yes, I started creating power point presentation to communicate with them more effectively.)

But this power point preference is not pervasive.  I have worked at many companies since where almost no one used power point even when giving presentations.  So I have observed that with every company culture, there are communication norms that have developed.  And if people work there long enough they don’t even realize that they are creating a PDF file when it is not needed, because all important data comes in a PDF attachment.

For the outsider, or consultant, observing what methods of communication are used, and then responding with the same type or form of communication may open doors and establish you as part of the team faster than just having good information and advice.  Adopting the ‘language of the tribe’ so to speak, is a rite of passage that you should not try to avoid.

Best Practices should be the basis for your project plans.

Hopefully your project plans are started by using a project plan template.  If not, then you should.  On this template, your “Best Practices” should be represented.  Too many times I have been on projects where the project manager starts by gathing the information about what must be done.  Instead, start with how it should be done. Otherwise you will encounter the painful process of trying to apply best practices to a project that is already in motion.

Sometimes resource planning and project management pressure encourages us to allocate additional engineering resources to development projects.  And there are definitely times when having a team is the best way to work quickly.

However, on integration projects more than a couple people slows the progress of the project.

I was once involved in a merger of companies.  The acquiring company had a large team of developers working on replacing the functionality of my company, the acquired one.  I became a team of one and the plan was that I would maintain the systems until the migration.

Management soon became frustrated as new feature development stagnated.  So they called me and asked how long it would take to produce a specific new integration feature.  My estimate, 40hr over two weeks.  Then they called a meeting and asked the leader of the development team how long.  His estimate, 6 months or more.

This wasn’t solely a matter of competence.  Sure, I was still there because I was the best on my technology.  But the other team was not populated with slouches.  There were just too darn many of them.  They got in each others way.

I, on the other hand, had no one to coordinate with.  No one to slow me down.

Teams of one are not always the best, but teams of 50 are never the best.

Or as a friend of mine likes to say, “It takes one woman nine months to have one baby, but putting nine women on the job doesn’t get you a baby in only one months.”

Data Integration is the process of transforming heterogeneous data into a useful homogeneous data set.

All of the techniques and disciplines that we use as integration engineer help us to transform (not must map) heterogeneous data (data in more than one form, source, type, etc)  into a useful (not common, and not necessarily the final) homogeneous (single form, schema, location, etc) data set.

There are two versions of the funnel.  First is where we pick a format and map that data to our internal system.  Then other formats we modify to be in this preferred format, and then map then through the original process.

Second, we create a intermediary format of all of the required data, and them translate all external data to this format of convenience.  Using a convenient dataset as our intermediate format, we can apply mapping logic in the external translation, and business logic on the importation of the convenient data.

I prefer using the convenient dataset, but either way we are able to reuse the mapping logic and using that as leverage are able to do more rapid integrations.

However, my father was a mechanical engineer, and he had a different philosophy that he wanted me to learn.  Routinely after I had “fixed” something. He would show me how getting the right tool,  and part worked so much better and longer.

The same is true in software development and integration work.  However the bailing wire and duct-tape have been replaced with buzz-word products.

Many projects get bogged down as a new, cutting edge technology that sounded cool is added to the project when an existing tool or application will do the job.

The right tool for the right job is better than a cool tool that almost words for the job.

Integrations can be broken into these three pieces as well.

Input:  Data that we get from outside whatever we are integrating.  We can be getting orders from a third party company, or we can be talking about getting data from an account DB on another server in our office.  In any case, and depending on the scope of our integration task we can define our inputs to be anything that is coming in externally to the product we are integrating.

Output:  Data that we transmit, broadcast or convey to others.  This can be our webpage, or orders we are sending to a vendor.  In any case and depending on the scope of the integration task we define our outputs to be anything that comes out of the products we are integrating.

Processes:  This is anything that we do to the data in-between input and output.  Translating data format, calculations, collation of data, or anything else that you can do with it count as a Process.

Cost is the easiest to quantify.  And it resonates with businesses today more than ever.  Doing tasks out of order causes some tasks to be repeated.  Some work to get in the way of other work and delay it or just make it take longer.  Some solutions may be developed to problems that are solved making that work to go unused.  And more.

Quality is always harder to quantify.  And in a budget conscious business environment is often minimized.  But we can point out that integration solution that are good quality cost less to maintain.  And the accuracy, reliability and extendability of good quality integration solutions are investments in the health and longevity of the business.

But then there is that 20% that is not so easy. 20% that you end up spending way too much time on. And its this 20% that produces the most exceptions, the most errors and the most cost.

We need the whole integration, we can’t just skip this part. But we can be aware of it and do the work to ensure it doesn’t grow or become a stubling block to our project.

Short list for me:

• new tech that is mandated or cool
• undefined support contacts
• one off integrations
• un assured delivery
• unmonitorable processes
• integrations without a well planned design

These are not in any particular order of occurrence or severity that I am aware of.

I am sure that there are other cases and causes for that 20%  that either don’t come to mind or that I haven’t encountered. (Lucky me)  Please feel free to add to this list.

And I expect that I will be blogging about some of these in more detail. 

When this happens there is a tendency to try and “save the world” or at least far exceed the scope of the project or problem.  To avoid this tendency, it is good to have someone in the meeting and involved in the planning and scheduling of resources that will constantly ask, “How does doing this help us solve the problem?”  And this also means that we should have a well defined goal for our project.

(If you are your project, write down the purpose and focus and read it every day, or more often to stay on target.)

Projects that succumb to “save the world” tendencies have a propensity to get bogged down, experience massive scope creep, miss their deadlines and expend lots of energy that does not return in realized benefits to the developer, team or company.

But there is another type of  “doing the same thing,” that can be just as maddening.  These are errors that continue to happen and no one bothers to fix them because the resolution is simple even if it is manual.  Like hitting the side of an old TV to clear up the picture.  It may work, but eventually all that time spent with the fuzzy picture, or making that quick manual change add up.  And that time is worth money.

What I recommend is that errors not only be tracked to ensure someone is alerted and they are resolved.  But errors instances are counted and reported so that support, development and management can see what the most frequent errors are.

Asking your support people to do the same thing over and over may not mean that you or they are insane, but it can result in something like it.  Find ways to automate or eliminate these repeating time wasters.  And with the data of how often and how long they took to fix, you can report how much time you are saving everyone.

In one of the early scenes in “The Patriot”, Mel Gibson’s character gives his young 11 and 12 year old sons some last minute instruction before ambushing a British patrol to rescue the older son.  He says, “Remember, aim small, miss small.”  The meaning behind this is simple.  If you are aiming small, say, at the button on your target’s shirt, and then you miss, you still actually hit the target.  This is “missing small”.

But if you aim big, say, at a whole person, or bear, and then you miss, you don’t hit your target, you may even hit something else that you didn’t intend.  This is “missing big”.

Selecting a tool to solve an integration problem is like aiming at a target.  Aiming small, is selecting a simple tool.  Aiming big is selecting a fancy tool.  Simple tools have simple problems, while fancy tools have fancy problems.

This does not mean that we discard things like monitoring, or error handling. We need those things. It does mean that we remove developer level debug statements that are no longer needed. ( I think you know what I mean there.)

It also means that redundant functions and repeating instructions without any attempt at abstracting them is a poor practices.  And we can clearly see that.  If you have 20 places where you create and format a time-stamp the same way.  We know that it would be better to have a function or object that we call to do this work so that we don’t have 20 places to remember to update if we need to change the format.

Also, tasks that are necessary, but written in an obtuse and excessive way are not best.  Just because I can write a 150 line process to search and replace characters in a string does not mean that it is okay to do that.  It is best to do something small even if it is a bit austere.  (Regular Expressions are a good example.  it is a bit austere, but very small and effective.)

Finally we also need to limit the cool side features from our code.  An integration script is not the place to do work not related to the integration.  For example, we may need to move old logs to an archive.  But having our logging process check and move those logs files is really not what it is for.  It would be best if there was something else responsible for doing this work so that we are not burdened and slowed with this, and so that we eliminate this potential source of a point of failure.

That really is one of the core benefits realised by adhering to this rule.  Eliminations of unnecessary points of failure.

In that vein, sometimes bugs can be when the program does things that the programmer did intend.

All applications have a primary purpose.  After that they have secondary purposes.  And then many times there are aspects of applications that start to get in the way of the primary purpose.  A good example of that can be found in a popular word processing application from Microsoft.  Taking advantage of this application’s auto-correction feature, people can create a situation where typing in a common word will auto-correct it so a word that is not intended by the writer.

When this happens we may all laugh, but a secondary feature of the application, (auto-correction), is preventing the primary purpose, (composition), from being accomplished.

There are times in software development, both in user applications, and in back end scripting, when leveraging a secondary feature, subverts the primary purpose.  When that primary purpose is a mission critical function, we can get that call in the middle of the night.

All this does not mean that you test it once, and then leave it to age in the recesses of your file server.  This is a plan that needs to be reviewed periodically.  With every major release.  The parts of the disaster recovery plan like database restore activities need to be requested every few months to make sure that the tasks are repeatable and the backups are valid and ensure that a recovery is possible.

These periodic and onging tests and updates make sure that if you ever have to implement your plan, that you will be recovering to your current system, not to the system when the plan was created.

Because of this propensity to illustrate my explanations, people have jokingly said that I couldn’t talk if they took my dry erase markers away.  At the same time, being able to communicate with others that are not versed in programming, databases, chemestry, or any other disapline that you find yourself will be a useful skill that will help you with your immediate project, and can advance your career.

My general recommendation is to avoid using undefined or user defined data.  Relying on a user defined field that we use to solve a problem by working around some limitation (or perceived limitation) of the file data definition, may give us a feeling of power.  But over the long haul turns the implementation into a proprietary implementation that is not reusable with more than one trading partner.

And having each integration using a slightly unique implementation leads to chaos.  Both in suppport, and in the applicaiton of updates.

Undefined data exists.  People will use it.  You will have to deal with it as some time, but whenever possible. avaid making it a critical part of the implementation, and always provide it with a healty dose of error handling.