The Integration Engineer
When it just has to work.

Even though 30% of searches are for local things.  Local businesses have a hard time getting found by keyword searches.  Sometimes this is because they don’t have a website.  But more often it is because Google, Bing, Yahoo or whatever, can’t tell if they are relevant for those search words.  So those local customers (customers that are geographically near a business with their product or service) are buying from someone in China or Toronto or some other place while the business in Peoria is wondering why know one ever comes in to buy.

As we customers do more and more of our initial shopping on the internet, businesses that don’t show up on a search are never visited.

My example: Read the rest of this entry »

“If you don’t have a website, you don’t have a business.”

Now let me explain this statement just a bit.  A business is a business when it starts to provide its goods and services to others.  In the past (distant past) people could and would “put out their shingle” to start serving their customers.  Times were simpler then.  And almost all markets were local.  At that time, the act of “putting out your shingle” was all you had to do to let your market know and you were in business.

In today’s market, a website is the modern day shingle.   Businesses aren’t really businesses if they don’t have a website.

But that is not all they need to do.

Some businesses are doing the equivalent to hanging their shingle out in the back yard or in back alleys where only people who know them will ever see it.  These are businesses that put up a website, but have used the wrong strategy, used the wrong keywords, and have no relationships with any other websites that can let search engines and people know where they are.

If you have a business, but aren’t getting visitors and customers from web traffic, you are probably doing something wrong.

Lets talk.  roy@theIntegrationEngineer.com

Or check out MarketMilestone.com to find out more.

If you are installing a Magento as a web store for yourself or for someone else, you will want to track how many visitors you have and how many of them convert into customers. Magento makes this really easy if you are using Google Analytic. Here is how it works.

First you will need to have a Google Analytics account.  If you don’t have one, you might want to check it out.  Its free and you can sign up at http://www.google.com/analytics

Read the rest of this entry »

Learn the basic tools.  Learn them first, and learn them well.  Sure, you can find some fancy XSLT tools or text editors.  And you can carry them around on a thumb drive and install them on all of you computers, workstations, laptops, and servers.  But sometime, some day, you will need to make a change, or fix a problem and you won’t have the fancy tool.  My experience says that this will also happen in the middle of the night when you are under pressure.  Knowing the basic tools like VI and Notepad, and any other tools that are a standard part of your systems will allow you to get your integration back up and running quickly and get you back to bed so you can go in and answer the questions about what happened with just that much more sleep.

Have a plan of what you want the reader to understand.

When you start taking notes, you won’t have this plan, and that is okay.  Notes are mostly for yourself.  But by the time you start to create a document that you want someone else to read and understand, you will need to have a plan on what the reader will take away from the document.  Ask yourself, “Is this a ‘how to’, or a ‘how works’ document?”

There is some overlap and subcategories for these two general types, but in basic if you are documenting how to start the system, or documenting how the parts of the system work together, you will use a different approach.  There will be some examples in the ‘how works’ document that will be like the ‘how to’  and any good ‘how to’ explains some of ‘how works’ that is inherent in the doing.  But these two approaches produce documents that look very differently. Read the rest of this entry »

This is like the Covey proverb, “Begin with the end in mind.”  In an integration project, the first step is always to define the target of where your data is going to end up.  This is true for a document conversion like XSLT or a EDI or Flat File transformation to XML or another version.  It is also true for ETL tasks where we are getting data in and out of a Database or repository.

Starting the work of transforming data or files without a well defined target for the data will cause you to do extra and unnecessary work, and to repeat some steps when the true destination is known.  It may seem obvious, but many many people fall into this trap, and then don’t understand why their projects keep exceeding their time budgets.

For those who may be unfamiliar with the EDI standard, and those who may have used EDI but have encountered a new, unfamiliar question.

This introduction aims at addressing the basic questions of convention and implementation of the EDI technology.  I have a list of topics, and hope that over time I can cover all of them.  I also hope that people reading here may have questions that I haven’t thought to write about.  Please let  me know what your questions are and I will see what I can do about providing answers. Read the rest of this entry »

Keeping a copy of all of the documentation you create is a pretty general benefit.  It helps you in three major ways;

1. Having a personal copy means that if the systems that have the public copies become unavailable, you will still have access to them.
2. Some times projects that get shelved, lose their documentation.  If you have a personal copy, when the project comes back to life, you will not be starting over.
3. And you never know what future project you will be working on that will spark the memory, “Hey we solved a problem like this on this other project…”  And having the documentation for it will help you.

I have never regretted keeping a personal copy of documentation.  But I have always regretted knowing that I didn’t keep one when I could have used it.

As with most strategies, we start out defining the goal, or the place we want to end up. With an EDI strategy roll out, our destination is a reliable bidirectional communication between our organization and one or more trading partners.

With this as the abstract version of our goal. We break the task into three sections. Data types that we will exchange both outbound and inbound. What transport or connectivity solutions we will used to ensure a robust and reliable transmission and reception of the data. And what data translation, storage and processing solutions we will employ.

Read the rest of this entry »

While working for a certain company, I requested some integration information from a manager, and got the information in an attached ppt file.  I soon noticed that all of the managers used power point presentations to communicate everything.  Yes, we had interminable meetings where managers would show a slide show that they would read to us, but also received benefits, project updates and other data in ppt file attachments.

Well, I immediately realized that this was a key to communicating with my manager.  And after some observation determined that is was a part of my companies management culture.  (And yes, I started creating power point presentation to communicate with them more effectively.)

Read the rest of this entry »

EDI and other data that is part of an e-commerce transaction needs to get to where it needs to go in a reliable way.  It needs to get there.  Get there once.  And be able to let the sender know if there was a problem with it, either not getting there, or being corrupted, in either case triggering a resend or some other remediation.

Read the rest of this entry »

At some point or another, you will create a set of best practices.  These are statements or procedures that your experience has taught you that you need to do to get the best, desired outcome. Many companies have these proudly included in a wiki page or document titled “Best Practices”.  Unfortunately this is not where they belong.

Best Practices should be the basis for your project plans.

Hopefully your project plans are started by using a project plan template.  If not, then you should.  On this template, your “Best Practices” should be represented.  Too many times I have been on projects where the project manager starts by gathing the information about what must be done.  Instead, start with how it should be done. Otherwise you will encounter the painful process of trying to apply best practices to a project that is already in motion.

Greetings. I gave a presentation yesterday on an introduction to market research. Not the normal topic for this blog. Well if you are looking for it, you can find it here.

Be sure to sign up to be notified when the video of the complete presentation becomes available.

Is it better to have a team, or the solo, maverick developer?  It depends.  The solo developer can work unimpeded by others, but doesn’t have anyone to work with to overcome obstacles.  Teams have meetings and have to coordinate their work.  Slowing them down.  As teams grow the problem gets worse.

Sometimes resource planning and project management pressure encourages us to allocate additional engineering resources to development projects.  And there are definitely times when having a team is the best way to work quickly.

However, on integration projects more than a couple people slows the progress of the project.

I was once involved in a merger of companies.  The acquiring company had a large team of developers working on replacing the functionality of my company, the acquired one.  I became a team of one and the plan was that I would maintain the systems until the migration.

Management soon became frustrated as new feature development stagnated.  So they called me and asked how long it would take to produce a specific new integration feature.  My estimate, 40hr over two weeks.  Then they called a meeting and asked the leader of the development team how long.  His estimate, 6 months or more.

This wasn’t solely a matter of competence.  Sure, I was still there because I was the best on my technology.  But the other team was not populated with slouches.  There were just too darn many of them.  They got in each others way.

I, on the other hand, had no one to coordinate with.  No one to slow me down.

Teams of one are not always the best, but teams of 50 are never the best.

Or as a friend of mine likes to say, “It takes one woman nine months to have one baby, but putting nine women on the job doesn’t get you a baby in only one months.”

Strangely enough this is a common question.  We all throw around PHP, and other sets of initials as if they were their own words.  There is some evidence in the annals of IT that these initials were used for “Personal Home Page,” however this really doesn’t help use the label and define the Hypertext Pre-Processing language used for development.

For me, Pre Hypertext Processing is the meaning of PHP.

PHP is a server side scripting language like ASP.

PHP allows dynamic activities to be embedded into web pages.  On the web-server php pages can access databases, directories, etc.  But to the web browser only a web page is presented. With care, a PHP page can create tools to do many clever things.  But more on that later.

Some times we overlook the basics. And from time to time it is good to ask and answer the basic questions. One of the most basic questions for us is to ask, “What is Data Integration?”  After asking, we need to provide an answer.

Data Integration is the process of transforming heterogeneous data into a useful homogeneous data set.

All of the techniques and disciplines that we use as integration engineer help us to transform (not must map) heterogeneous data (data in more than one form, source, type, etc)  into a useful (not common, and not necessarily the final) homogeneous (single form, schema, location, etc) data set.

During the course of setting up, developing on or making significant changes to a Magento installation, it is necessary to refresh Magento’s cache.  This is pretty straight forward operation, but sometimes simple things can be overlooked.  Here is a quick step by step guide to doing this.

There are two parts to this task.  First is the Administration Cache Refresh.  Second is the Cache File Removal. Read the rest of this entry »

The integration funnel view is where we have data in more than one format or more than one source or both and we integrate them with one destination.  (We also do the reverse, but that’s not very funnel like.)  To do this effectively we don’t do a unique integration for each format and source of data.

There are two versions of the funnel.  First is where we pick a format and map that data to our internal system.  Then other formats we modify to be in this preferred format, and then map then through the original process.

Second, we create a intermediary format of all of the required data, and them translate all external data to this format of convenience.  Using a convenient dataset as our intermediate format, we can apply mapping logic in the external translation, and business logic on the importation of the convenient data.

I prefer using the convenient dataset, but either way we are able to reuse the mapping logic and using that as leverage are able to do more rapid integrations.

Recently I needed to send our team an email.  Yeah okay, so I email a lot, but this was different.  I needed to send my team an email from a server that was not anywhere near a mail server, and that was pretty paired down.  I did a bunch of Googling for answers and came up with quite a few that didn’t work.

I hate it when that happens.  So as a form of perl tutorial, I am posting my working script here as an example of sending email from a limited server using perl and the MIME::Lite library.

Read the rest of this entry »

Growing up on a farm, I learned that almost any problem could be solved, (at least temporarily) with the proper application of bailing wire and duct-tape.

However, my father was a mechanical engineer, and he had a different philosophy that he wanted me to learn.  Routinely after I had “fixed” something. He would show me how getting the right tool,  and part worked so much better and longer.

Read the rest of this entry »