The Integration Engineer
When it just has to work.

Bugs are sometimes defined as when a program does anything that the user does not expect.  However sometimes users expect some unreasonable things.  So what is a bug?  Well I definitely think that it is more than when it does something that the programmer doesn’t expect.

In that vein, sometimes bugs can be when the program does things that the programmer did intend.

(more…)

Or as I like to call them, “Disaster Recovery Dreams.”  Sure, you need to have a plan before you can test it.  But before you really tell customers, management, team members or even yourself that you have a Disaster Recovery Plan, you need to have it tested.  Until them you are dreaming.

All this does not mean that you test it once, and then leave it to age in the recesses of your file server.  This is a plan that needs to be reviewed periodically.  With every major release.  The parts of the disaster recovery plan like database restore activities need to be requested every few months to make sure that the tasks are repeatable and the backups are valid and ensure that a recovery is possible.

(more…)

I know it may sound cleche to state that a picture is worth a thousand words.  But some things are just really difficult to communicate in text.  You may have noticed a few diagrams showing up in my posts here.  Well, I have found that there are times in a conversation with other programmers, and more often in conversations with non-technical collegues that I am saying, “I need a whiteboard.”

Because of this propensity to illustrate my explanations, people have jokingly said that I couldn’t talk if they took my dry erase markers away.  At the same time, being able to communicate with others that are not versed in programming, databases, chemestry, or any other disapline that you find yourself will be a useful skill that will help you with your immediate project, and can advance your career.

When the definition for a data field or element is undefined or otherwise open to the whim of the user, we are asking for trouble.  Not in the sense that this causes trouble.  But how it is then used that causes trouble.  And this is not to pick on the CDATA tag in XML or ZZ qualified elements in EDI.  It seems that every data format of any size of adoption commits this sin of categorisation.

My general recommendation is to avoid using undefined or user defined data.  Relying on a user defined field that we use to solve a problem by working around some limitation (or perceived limitation) of the file data definition, may give us a feeling of power.  But over the long haul turns the implementation into a proprietary implementation that is not reusable with more than one trading partner.

(more…)

Just like synergy, a projects time-line is generally greater than the sum of its tasks.  Taking the time to complete tasks in a project together, there will need to be time between tasks.  Some of this time will just be spent updating people, and coordination of project participants.  There are also other needs, like water and bio-breaks.

Just like productivity during the day generally gives people a 15 minute break every 3 hrs or so.  When we plan our projects, we need to include not just the time that it takes to perform the collective tasks, but other activities that will be happening during the course of the project.

Odd errors can occur when you get data that doesn’t fit the format, but matches close enough not to fail outright.  This type of error can be really hard to unravel. Preventing these messy errors, it is a good practice to implement a validation process before processing the data.  Some integration technologies come with validation, if yours doesn’t you should pre-process inbound data for validation.

In addition to this, pre validation can be used as a way to catch and handle other errors early.  Leveraging pre-processing with validation and error handling can give you a one stop place to catch, and report or correct systemic problems.

Regardless of the standards that you adopt for your integrations, you need to pick some.  EDI, XML, or any of the others.  You need to evaluate and decide on a standard that will do what you need.  And you need to be prepared to present it to your trading partners.

(more…)

The accuracy of the time-line will only be as accurate as the accuracy of the least accurate task estimate.

The time estimate part of project planning is a tricky thing.  Generally there is some guessing involved.  To keep us from falling short a good practice is to pad estimates to account for the unforeseeable.  Project planners can’t just pad and pad.  There is pressure from the other side to keep your estimates accurate and keep your time-line short.  Time is money after all.

(more…)

You may have noticed this before, but in case you haven’t, many many many projects are not completed on their due date.

I know this is shocking, and if you are a new project manager, you may want to leave the blog now.  Continuing to read this article may result in the following.

1. Better understanding of the factors that should go into a project plan time-line.
2. More effective communication with the key players in your project.
3. Increased ability to accurately predict the completion time of a project.

If this frightens you…

Well at least you have been warned. (more…)

Learn the basic tools.  Learn them first, and learn them well.  Sure, you can find some fancy XSLT tools or text editors.  And you can carry them around on a thumb drive and install them on all of your computers, workstations, laptops, and servers.  But sometime, some day, you will need to make a change, or fix a problem and you won’t have the fancy tool.

My experience says that this will also happen in the middle of the night when you are under pressure.  Knowing the basic tools like VI and Notepad, and any other tools that are a standard part of your systems will allow you to get your integration back up and running quickly and get you back to bed so you can go in and answer the questions about what happened with just that much more sleep.

(more…)

This is like the Covey proverb, “Begin with the end in mind.”  In an integration project, the first step is always to define the target of where your data is going to end up.  This is true for a document conversion like XSLT or a EDI or Flat File transformation to XML or another version.  It is also true for ETL tasks where we are getting data in and out of a Database or repository.

Starting the work of transforming data or files without a well defined target for the data will cause you to do extra and unnecessary work, and to repeat some steps when the true destination is known.  It may seem obvious, but many many people fall into this trap, and then don’t understand why their projects keep exceeding their time budgets.

(more…)

Keeping a copy of all of the documentation you create is a pretty general benefit.  It helps you in three major ways;

1. Having a personal copy means that if the systems that have the public copies become unavailable, you will still have access to them.
2. Some times projects that get shelved, lose their documentation.  If you have a personal copy, when the project comes back to life, you will not be starting over.
3. And you never know what future project you will be working on that will spark the memory, “Hey we solved a problem like this on this other project…”  And having the documentation for it will help you. (more…)

While working for a certain company, I requested some integration information from a manager, and got the information in an attached ppt file.  I soon noticed that all of the managers used power point presentations to communicate everything.  Yes, we had interminable meetings where managers would show a slide show that they would read to us, but also received benefits, project updates and other data in ppt file attachments.

Well, I immediately realized that this was a key to communicating with my manager.  And after some observation determined that is was a part of my companies management culture.  (And yes, I started creating power point presentation to communicate with them more effectively.)

(more…)

At some point or another, you will create a set of best practices.  These are statements or procedures that your experience has taught you that you need to do to get the best, desired outcome. Many companies have these proudly included in a wiki page or document titled “Best Practices”.  Unfortunately this is not where they belong.

Best Practices should be the basis for your project plans.

(more…)

Is it better to have a team, or the solo, maverick developer?  It depends.  The solo developer can work unimpeded by others, but doesn’t have anyone to work with to overcome obstacles.  Teams have meetings and have to coordinate their work.  Slowing them down.  As teams grow the problem gets worse.

Sometimes resource planning and project management pressure encourages us to allocate additional engineering resources to development projects.  And there are definitely times when having a team is the best way to work quickly.

(more…)

Some times we overlook the basics. And from time to time it is good to ask and answer the basic questions. One of the most basic questions for us is to ask, “What is Data Integration?”  After asking, we need to provide an answer.

Data Integration is the process of transforming heterogeneous data into a useful homogeneous data set.

(more…)

The integration funnel view is where we have data in more than one format or more than one source or both and we integrate them with one destination.  (We also do the reverse, but that’s not very funnel like.)  To do this effectively we don’t do a unique integration for each format and source of data.

There are two versions of the funnel.  First is where we pick a format and map that data to our internal system.  Then other formats we modify to be in this preferred format, and then map then through the original process.

(more…)

Growing up on a farm, I learned that almost any problem could be solved, (at least temporarily) with the proper application of bailing wire and duct-tape.

However, my father was a mechanical engineer, and he had a different philosophy that he wanted me to learn.  Routinely after I had “fixed” something. He would show me how getting the right tool,  and part worked so much better and longer.

(more…)

In order to use social media to engage your customers and / or potential customers in your products and services you actually have to use Social Media. This video starts a discussion about why to use Twitter for your business. The reasons for using Twitter can be applied to other forms of social media.

This video will be followed by 4 other videos that comprise the discussion about using Twitter as it relates to creating engagement with your customer base.  Both Existing and Future.

If you use or don’t use Twitter in your business, please share why or why not.

“640K ought to be enough for anybody.”  Attributed to Bill Gates (long long ago)

One thing that we should always try to do when building, buying or choosing a solution for e-commerce, data process, integration or any system, is that there will be a time when we have to move, to expand or extend it. We don’t have to solve all of the problems of the future, but we should be aware that we will have problems. And we will need to solve them.

(more…)