The Target

Open Office tables are divided into two tables. Invoice and Invoice Details. This can be mapped in two ways; The first way is to map the data into one common format and rely on whatever ETL tool is importing the data to catch and split the data. Second, a key can be acquired or constructed in the transformation and then the the data can be divided in to matching input formats. Then when these inputs are moved into the database, they will relate to each other on this key.

The choice of how you will do this will depend on your environment. Questions like, “Will I have enough data to provide a unique key?”, or “is there a way to get a key with an API call or database query?” The answers to these questions will determine what course you will take.

If the system ultimately receiving the data is asynchronous to the transformation, and you need to send the invoice and invoice details data separately, some care needs to be taken to ensure that the data can be related after it is separated.

So what data in the invoice can be used to tie the invoice to the invoice details? The first answer might be, “The Invoice Number.” But this number is not guaranteed to be unique across multiple vendors.  In EDI and cXML there are document unique identifiers.  Since we are using EDI, we can use a combination of the ISA Sender, Receiver, and Control number.  We will also want to use the GS control number, and ST control number in the event that more than one invoice is sent in a single envelope.  And we might as well tack on the actual invoice number from the BIG_02.

Database Tables

Invoice Table:	Invoice Details Table:

The Paper Map

Now that we know what the target looks like, we fill out the target side of the paper map.  Since we will create two “files” in our output.  We are creating the Invoice and the InvoiceDetails file, but we can use one paper map for both, and will distinguish this with a bar between the two “files”.  (I am saying files, but this could be a queue, or a post, or an insert over odbc, etc)

What’s Next

Today we went through the process of identifying the target, and creating a paper map with the target format identified.  We talked about some of the strategy that we use in deciding what to map and how to map it.  Next time we will identify the source, and begin mapping data from the source.

1.  A big Whiteboard

This is probably my number one requirement.  When I am thinking, I like to draw it out.  I haven’t found an application that gives me the same creative release and adaptability as my whiteboard.  After starting to do some of my work from home, I went out and acquired a 10 X 4 whiteboard to put in my home office.  Having the extra room is essential.  I can get a call, and walk over to the whiteboard and start drawing out what needs to be built to solve the problems while I am still on the call.

You may not need a white board as big as mine.  And you may have an electronic solution that you like better.  If so, please let me know, I like to try out gadgets.

2.  Spreadsheet

Now this may not sound earth shattering, but we are not just talking about the basics.  You will need to learn to write filters, import data, link cells and perform calculations.  If you think that a spreadsheet is like a ledger, then you are missing the power of a spreadsheet.  If you don’t think you have the skills you need, here are some links to tutorials for the two most popular spreadsheet software.

• Very Basic Open Office Tutorial http://www.tutorialsforopenoffice.org/
• More specific/advanced tutorials http://openoffice.blogs.com/openoffice/
• Some basic and advanced help for MS Excel http://www.internet4classrooms.com

3.  Text editor.

Familiarity with more than one is needed as you will find yourself on Windows and Unix servers and they will have different sets of tools.  One of the things that is the most frustrating is to not know how to use the native editor of the system you are on.  So get familiar with Notepad, and then get familiar with VI.  You can add in other tools like UltraEdit, TextPad, and more, but you should know the native ones first, if not best.

• Ultra Edit is a popular tool in some circles.  http://www.ultraedit.com/
• Text Pad is also popular.  http://www.textpad.com/
• Believe it or not, notepad has some tutorials.  http://bink.nu/news
• And here is a cheat sheet for VI commands.  http://downloads.techrepublic.com

4.  File compare.

One task that will need to be done is to compare a file with another file to detect changes or differences.  This happens in both new integrations, and in trouble shooting or investigating existing ones.  Some systems have native comparison tools, others don’t.  And there is much variety in how they work.  Here are some of the tools I have seen and used.

• Wiki has a great list.  I don’t know that I can improve on.

5.  DB query

Much of the time, integration involves a database somewhere sometime.  To work on an integration and not have to have a DBA sitting in your lap, you need to be competent to query the database.  And this involves using some tool.  Sometimes systems will have native tools, other times you will need to connect your own.  Here is a list of tutorials for DB Query tools that I have seen and used.

• Squirrel SQL http://squirrel-sql.sourceforge.net/
• SQL Worksheet
• TOAD
  

Summary

This is by no means attempting to be a comprehensive list of tools.  Such a list would be long, if it were possible.  And I don’t think that it is.  There are however some tools/skills that help us to be more effective as Integration Engineers.  Seeing what others use is helpful especially when we find that the tool we used to use is not longer around.

What tools, applications, or skills do you find you are falling back on often, on more than one project?  Or what new tools have you found that you think you will use often in the future?

Many moons ago, I was shown an database that was constructed back in the 50s.  It was hand held and consisted of a stack of cards that contained information about plant biology.  There was a series of holes that wrapped around the cards, some of the holes were notches, and others were not.

To query the database, one placed a pin through one of the wholes and let the cards that had a notch fall out of the stack.  You could continue this process by removing and adding pins to select different combination of data.

Once you had narrowed your search, you could read the cards, and retrieve the data.

This was a very manual process, and I doubt that anyone today would take the time to learn to use such a tool, let alone update or add records to it.

More Modern Databases

Replacing Card Catalogs in libraries, Rolodex on desktops, and filing rooms in offices, is our new electronic concept of databases.  At a rudimentary level, and fitting under the definition of the database that we started with fall many simple documents.

A post-it note is not a database, but a list of names and numbers on a contact list is.  It has regular structure, and the data can be retrieved using a routine.  And this is as far as most people need to go.  They see databases like spreadsheets.  They even use filters and link cells for calculations.

More than just a list

A database that just holds a list of records is a single table Database.  This is like a spreadsheet and the only real advantage is that Databases on computers are generally designed to be queried by other applications.  The contact list in your email client is not much more than a flat table.  But it is queried and updated from the email application.  This makes the application more useful.

Again, long ago, I had a watch that held phone numbers.  This worked pretty well, and had an alpha-numeric keyboard.  But the data was in a silo.  There was no way to get the data in or out other than the small screen and keypad.  Application bound, embedded databases are also silos of data.

Many applications now use, or have the option to use, an embedded database.  But databases that can be used by more than one application, and are treated like an object are even more useful.  When a database is shared between applications, we can gain exponential utility.  Every new application that can query the shared database becomes more efficient.

RDBMS More than a single table.

One of the things that we will discuss more later, is the concept of a relational database.  What this means is that we have more than one table, that have related fields.  A analogous example is the an email system with an integrated calender.  Now we have the schema or table that contains the contact and relate them to people invited to the events.  These systems have been around for quite a while and gives us a conceptual model of how related tables bring power to a Relational Database Management System (RDBMS).

Data in a Hierarchy

RDBMS is not the only way to organize data.  One way that some databases organize data is in a hierarchical schema.  In this fashion instead of tables, we have a conceptual model like a directory tree.  Data is located in a name space that is organized like files in directories.  Data in this type of structure gains some implicit structure and relationship as a parent child relationships exists in the data naturally.  For small sets of data this can make the database small and fast.  However the ability to build abstract structures, relationships and queries are more difficult with this structure, and the database really grows when extended to cover the features of multiple relational data reference.

All Shapes and Sizes

More basic than the most simple table or hierarchy based database is the file based system.  This is really what it sounds like.  A simple text file or an XML file can be used as a database.  These can support queries, inserts and joins.  Even though the database is in a standard file that can be opened and accessed with a common text editor, it has a structure and can be used to keep the application data organized.

Databases for the Integration Engineer

As integration Engineers, we will sometimes find ourselves working directly with and application’s database.  Sometimes this database will be a full blown enterprise level database implementation.  Many times, however, it will be something more simple like an embedded table or xml file that the application treats as its database.  Whatever the case, we need to learn and respect the structure of the data, and work to ensure that our integration preserves the integrity of the data structure.