In that vein, sometimes bugs can be when the program does things that the programmer did intend.

All applications have a primary purpose.  After that they have secondary purposes.  And then many times there are aspects of applications that start to get in the way of the primary purpose.  A good example of that can be found in a popular word processing application from Microsoft.  Taking advantage of this application’s auto-correction feature, people can create a situation where typing in a common word will auto-correct it so a word that is not intended by the writer.

When this happens we may all laugh, but a secondary feature of the application, (auto-correction), is preventing the primary purpose, (composition), from being accomplished.

There are times in software development, both in user applications, and in back end scripting, when leveraging a secondary feature, subverts the primary purpose.  When that primary purpose is a mission critical function, we can get that call in the middle of the night.

All this does not mean that you test it once, and then leave it to age in the recesses of your file server.  This is a plan that needs to be reviewed periodically.  With every major release.  The parts of the disaster recovery plan like database restore activities need to be requested every few months to make sure that the tasks are repeatable and the backups are valid and ensure that a recovery is possible.

These periodic and onging tests and updates make sure that if you ever have to implement your plan, that you will be recovering to your current system, not to the system when the plan was created.

Because of this propensity to illustrate my explanations, people have jokingly said that I couldn’t talk if they took my dry erase markers away.  At the same time, being able to communicate with others that are not versed in programming, databases, chemestry, or any other disapline that you find yourself will be a useful skill that will help you with your immediate project, and can advance your career.

My general recommendation is to avoid using undefined or user defined data.  Relying on a user defined field that we use to solve a problem by working around some limitation (or perceived limitation) of the file data definition, may give us a feeling of power.  But over the long haul turns the implementation into a proprietary implementation that is not reusable with more than one trading partner.

And having each integration using a slightly unique implementation leads to chaos.  Both in suppport, and in the applicaiton of updates.

Undefined data exists.  People will use it.  You will have to deal with it as some time, but whenever possible. avaid making it a critical part of the implementation, and always provide it with a healty dose of error handling.

Just like productivity during the day generally gives people a 15 minute break every 3 hrs or so.  When we plan our projects, we need to include not just the time that it takes to perform the collective tasks, but other activities that will be happening during the course of the project.

In addition to this, pre validation can be used as a way to catch and handle other errors early.  Leveraging pre-processing with validation and error handling can give you a one stop place to catch, and report or correct systemic problems.

Regardless of the standards that you adopt for your integrations, you need to pick some.  EDI, XML, or any of the others.  You need to evaluate and decide on a standard that will do what you need.  And you need to be prepared to present it to your trading partners.

This also applies to other parts of your business.  Picking a standard platform for your development and standard tools for development.  And so forth.

Even small companies benefit from standardization.  Don’t be afraid to put your technology people in a room and ask them to provide standard recommendations.  But there are two cautions.

1.  The latest and greatest, may look cool to the tech people, but may also be a platform that is unproven and will change radically in a short time.

2.  Not all open source is good.  If you are going open source as part of your platform, look at the quality of the community that supports them.  (This is also true for non-open source technologies.)

The time estimate part of project planning is a tricky thing.  Generally there is some guessing involved.  To keep us from falling short a good practice is to pad estimates to account for the unforeseeable.  Project planners can’t just pad and pad.  There is pressure from the other side to keep your estimates accurate and keep your time-line short.  Time is money after all.

So how do we put enough time in the time-line to not fall short, and keep the time-line from getting too long and expensive?  Well that is a dance that takes some practice to learn.

Just remember, when you pad your estimate to your project leader or get and estimate from someone on your project, that padding is necessary, and you should discuss it openly.

I know this is shocking, and if you are a new project manager, you may want to leave the blog now.  Continuing to read this article may result in the following.

1. Better understanding of the factors that should go into a project plan time-line.
2. More effective communication with the key players in your project.
3. Increased ability to accurately predict the completion time of a project.

If this frightens you…

Well at least you have been warned.

Rule One:  The slowest horse determines the speed of the wagon.

Some people will be able to get their jobs done faster than others.  Some times this is predictable, and a consistent level of performance.  Some times people are just lucky.  But just because you have one race horse does not mean that the wagon will speed up.

First of all, that workaholic may begin to burn out if they are constantly pulling other peoples weight.  Don’t let them take on too much of the tasks.

Secondly, you need to give everyone else a chance to get that experience.

So pace the project with a base-line of performance.  Then whey you get done a little early, you can all look heroic.

Rule Two:  Determining the time of the project by analyzing the time of the tasks.

Yes this sounds simple.  And you may have been given a deadline from above.  But that does not mean that you should not start with the incremental tasks.

• Call a meeting, and have everyone report on how long their part will take.
• plug it all into a project plan, and see how much past your deadline it puts you.
• After that, open the discussion on how to get all the work done by the project deadline.
• If there is no way, then push back on those giving the deadline.
• But if it is possible, you are better off if the team figures out where to put the extra muscle to push the task along.

Rule Three:  Pad for the unforeseeable.

If you know that you need to pad a time estimate because experience tells you that it takes longer than the sum of the discrete tasks, then by all means pad the estimate.  But don’t hide that from your team, or if you are not the project leader, then don’t hide that from you project leader.

The need to pad times is just a reality.  People get interrupted both by life, and work.  There is nothing wrong with saying, “Well it takes 10 minutes to setup one customer, so lets say that we can get 4 done an hour.”  Then if you can beat that time you are ahead.

Rule Four:  Make sure you have regular checkpoints

If you are on a project that takes a week, you need to be checking in with the players, and the team as a whole more than just at the kick-off and wrap-up meetings.  Sometimes you need to have a daily meeting.  (sometimes more than that.)  But leaving even the best professionals off by themselves will always mean that you are blindsided when the deadline is missed.

Checking in won’t mean that things won’t take longer than they were planned, but they will allow you to know about them, and possibly compensate for one or more unforeseen delays.

Rule Five:  Participate in open discussions about the time-line.

From the very beginning, the project time-line should be published and discussed in every meeting about the project.  Sometimes having a conference room or some location where the time-line and project status can be posted will serve as a banner to let everyone know where the team stands.

When problems come up, have a quick brains storming session about how to correct the course and get the project back on the time-line.

My experience says that this will also happen in the middle of the night when you are under pressure.  Knowing the basic tools like VI and Notepad, and any other tools that are a standard part of your systems will allow you to get your integration back up and running quickly and get you back to bed so you can go in and answer the questions about what happened with just that much more sleep.

So at this point you need to get a piece of paper or a blank document and make a list of all of the tools that you use and need.  Pay specific attention to any tools that are not available universally.  Is there a more basic tool that can do the job?  Do you know how to use that tool?  If not, now is the time to learn it.

Starting the work of transforming data or files without a well defined target for the data will cause you to do extra and unnecessary work, and to repeat some steps when the true destination is known.  It may seem obvious, but many many people fall into this trap, and then don’t understand why their projects keep exceeding their time budgets.

Now it is time to take a look at the processes that you use in your integration projects and see if you are starting by defining your target.  To help you with this, you can take a look at the target based mapping posts on this site as well.

1. Having a personal copy means that if the systems that have the public copies become unavailable, you will still have access to them.
2. Some times projects that get shelved, lose their documentation.  If you have a personal copy, when the project comes back to life, you will not be starting over.
3. And you never know what future project you will be working on that will spark the memory, “Hey we solved a problem like this on this other project…”  And having the documentation for it will help you.

I have never regretted keeping a personal copy of documentation.  But I have always regretted knowing that I didn’t keep one when I could have used it.

So now it is time to examine what is the best way for you to keep your personal copies.  In years past, I have kept folders of documents on a flash drive or the hard drive of my laptop.  Well over the last year I have changed my strategy, I now use a service called Evernote.  It may not be for you, but if you haven’t checked them out and you don’t have a method of keeping your personal copies of documentation organized.

If Evernote is not for you, what is?  Try some different methods and post what you like to do in the comments to help other readers of this blog.

While working for a certain company, I requested some integration information from a manager, and got the information in an attached ppt file.  I soon noticed that all of the managers used power point presentations to communicate everything.  Yes, we had interminable meetings where managers would show a slide show that they would read to us, but also received benefits, project updates and other data in ppt file attachments.

Well, I immediately realized that this was a key to communicating with my manager.  And after some observation determined that is was a part of my companies management culture.  (And yes, I started creating power point presentation to communicate with them more effectively.)

But this power point preference is not pervasive.  I have worked at many companies since where almost no one used power point even when giving presentations.  So I have observed that with every company culture, there are communication norms that have developed.  And if people work there long enough they don’t even realize that they are creating a PDF file when it is not needed, because all important data comes in a PDF attachment.

For the outsider, or consultant, observing what methods of communication are used, and then responding with the same type or form of communication may open doors and establish you as part of the team faster than just having good information and advice.  Adopting the ‘language of the tribe’ so to speak, is a rite of passage that you should not try to avoid.

Best Practices should be the basis for your project plans.

Hopefully your project plans are started by using a project plan template.  If not, then you should.  On this template, your “Best Practices” should be represented.  Too many times I have been on projects where the project manager starts by gathing the information about what must be done.  Instead, start with how it should be done. Otherwise you will encounter the painful process of trying to apply best practices to a project that is already in motion.

Sometimes resource planning and project management pressure encourages us to allocate additional engineering resources to development projects.  And there are definitely times when having a team is the best way to work quickly.

However, on integration projects more than a couple people slows the progress of the project.

I was once involved in a merger of companies.  The acquiring company had a large team of developers working on replacing the functionality of my company, the acquired one.  I became a team of one and the plan was that I would maintain the systems until the migration.

Management soon became frustrated as new feature development stagnated.  So they called me and asked how long it would take to produce a specific new integration feature.  My estimate, 40hr over two weeks.  Then they called a meeting and asked the leader of the development team how long.  His estimate, 6 months or more.

This wasn’t solely a matter of competence.  Sure, I was still there because I was the best on my technology.  But the other team was not populated with slouches.  There were just too darn many of them.  They got in each others way.

I, on the other hand, had no one to coordinate with.  No one to slow me down.

Teams of one are not always the best, but teams of 50 are never the best.

Or as a friend of mine likes to say, “It takes one woman nine months to have one baby, but putting nine women on the job doesn’t get you a baby in only one months.”

Data Integration is the process of transforming heterogeneous data into a useful homogeneous data set.

All of the techniques and disciplines that we use as integration engineer help us to transform (not must map) heterogeneous data (data in more than one form, source, type, etc)  into a useful (not common, and not necessarily the final) homogeneous (single form, schema, location, etc) data set.

There are two versions of the funnel.  First is where we pick a format and map that data to our internal system.  Then other formats we modify to be in this preferred format, and then map then through the original process.

Second, we create a intermediary format of all of the required data, and them translate all external data to this format of convenience.  Using a convenient dataset as our intermediate format, we can apply mapping logic in the external translation, and business logic on the importation of the convenient data.

I prefer using the convenient dataset, but either way we are able to reuse the mapping logic and using that as leverage are able to do more rapid integrations.

However, my father was a mechanical engineer, and he had a different philosophy that he wanted me to learn.  Routinely after I had “fixed” something. He would show me how getting the right tool,  and part worked so much better and longer.

The same is true in software development and integration work.  However the bailing wire and duct-tape have been replaced with buzz-word products.

Many projects get bogged down as a new, cutting edge technology that sounded cool is added to the project when an existing tool or application will do the job.

The right tool for the right job is better than a cool tool that almost works for the job.

If you use or don’t use Twitter in your business, please share why or why not.

One thing that we should always try to do when building, buying or choosing a solution for e-commerce, data process, integration or any system, is that there will be a time when we have to move, to expand or extend it. We don’t have to solve all of the problems of the future, but we should be aware that we will have problems. And we will need to solve them.

Some of the biggest obstacles that I encounter are the failures to plan for growth or need to migrate to larger platforms that e-commerce systems are built with.  Sometimes the lack of forethought is unavoidable, like the original EDI format not having a place for an email address or anything like it.  They didn’t have email, so the best they could do was allow extended phone numbers.  But other times they are easily avoidable.  Like a 65536  row limit in an Excel spreadsheet.  This may sound like a lot of rows.  But I hit this limit all the time when working with data, and it make me use other tools.

Whenever possible, try to make sure you can grow the scope and size of your integration or system.  And also try to make sure that when you push data and features into a system, that you can get all of your data back out in a usable format for when it is time to migrate