And Magento doesn’t have an “out of the box” way to modify the order to customer profile relationship even though the solution is really simple.

Simple that is, if you are comfortable writing queries in the database.

Here are the steps to do this once you are in the mysql database:

Associate order with Customer Account

1.  Get customer account entity_id based on email:

SELECT entity_id FROM customer_entity WHERE email = ‘[email address]‘;

You can also pluck this id from the end of the url in the manage customers section of the admin panel if you are already there.

2.  Verify Order

SELECT entity_id, email, customer_id, created_at  FROM sales_order WHERE increment_id = [order number];

This step is really to verify that it is in fact the right order.  Associating the wrong order with the customer will make your store look less secure.

3.  Associate Order with Customer

UPDATE sales_order SET customer_id = [customer entity id] WHERE increment_id = [order number];

This sets the association up so that the customer can now see the status and order data from their customer profile.

And thats it.  The customer to order relationship has been corrected.

(Note, this will not change the listing of how the order was placed.  If placed as guest, the order view will still show this.  But I have never found this to be important.)

Out of the box, the Magento admin does not have the ability to change this.  But if you are comfortable in the database, then these steps will let you make this change in seconds.

1. Get order entity id

SELECT entity_id FROM sales_order WHERE increment_id = [order number];

2. Find order email address and get it’s value_id

SELECT value_id, value FROM sales_order_varchar WHERE entity_id = [order entity id];

This will return many values.  The email address should look distinctively like an email address.   In my current instance, the attribute ID is 124, but that may or may not be the same for you.  In any case, what we are looking for is the value_id for the email address that we will use in the next step.

3. Update email

UPDATE sales_order_varchar SET  value = ‘[new email address]‘ WHERE value_id = [value id for the email address];

And that is it, the email address has been corrected and the customer will get future emails relating to this order.

Many moons ago, I was shown an database that was constructed back in the 50s.  It was hand held and consisted of a stack of cards that contained information about plant biology.  There was a series of holes that wrapped around the cards, some of the holes were notches, and others were not.

To query the database, one placed a pin through one of the wholes and let the cards that had a notch fall out of the stack.  You could continue this process by removing and adding pins to select different combination of data.

Once you had narrowed your search, you could read the cards, and retrieve the data.

This was a very manual process, and I doubt that anyone today would take the time to learn to use such a tool, let alone update or add records to it.

More Modern Databases

Replacing Card Catalogs in libraries, Rolodex on desktops, and filing rooms in offices, is our new electronic concept of databases.  At a rudimentary level, and fitting under the definition of the database that we started with fall many simple documents.

A post-it note is not a database, but a list of names and numbers on a contact list is.  It has regular structure, and the data can be retrieved using a routine.  And this is as far as most people need to go.  They see databases like spreadsheets.  They even use filters and link cells for calculations.

More than just a list

A database that just holds a list of records is a single table Database.  This is like a spreadsheet and the only real advantage is that Databases on computers are generally designed to be queried by other applications.  The contact list in your email client is not much more than a flat table.  But it is queried and updated from the email application.  This makes the application more useful.

Again, long ago, I had a watch that held phone numbers.  This worked pretty well, and had an alpha-numeric keyboard.  But the data was in a silo.  There was no way to get the data in or out other than the small screen and keypad.  Application bound, embedded databases are also silos of data.

Many applications now use, or have the option to use, an embedded database.  But databases that can be used by more than one application, and are treated like an object are even more useful.  When a database is shared between applications, we can gain exponential utility.  Every new application that can query the shared database becomes more efficient.

RDBMS More than a single table.

One of the things that we will discuss more later, is the concept of a relational database.  What this means is that we have more than one table, that have related fields.  A analogous example is the an email system with an integrated calender.  Now we have the schema or table that contains the contact and relate them to people invited to the events.  These systems have been around for quite a while and gives us a conceptual model of how related tables bring power to a Relational Database Management System (RDBMS).

Data in a Hierarchy

RDBMS is not the only way to organize data.  One way that some databases organize data is in a hierarchical schema.  In this fashion instead of tables, we have a conceptual model like a directory tree.  Data is located in a name space that is organized like files in directories.  Data in this type of structure gains some implicit structure and relationship as a parent child relationships exists in the data naturally.  For small sets of data this can make the database small and fast.  However the ability to build abstract structures, relationships and queries are more difficult with this structure, and the database really grows when extended to cover the features of multiple relational data reference.

All Shapes and Sizes

More basic than the most simple table or hierarchy based database is the file based system.  This is really what it sounds like.  A simple text file or an XML file can be used as a database.  These can support queries, inserts and joins.  Even though the database is in a standard file that can be opened and accessed with a common text editor, it has a structure and can be used to keep the application data organized.

Databases for the Integration Engineer

As integration Engineers, we will sometimes find ourselves working directly with and application’s database.  Sometimes this database will be a full blown enterprise level database implementation.  Many times, however, it will be something more simple like an embedded table or xml file that the application treats as its database.  Whatever the case, we need to learn and respect the structure of the data, and work to ensure that our integration preserves the integrity of the data structure.

When businesses realize that the catalog data that they have may be worth as much as the products or services that they sell, they may be tempted to sell that instead.  Before they do, they need to think about who owns the data.

Lets say hypothetically that there is an integration tool that helps people that sell widgets, get widget pricing and data from people that make widgets. And this integration tool also helps people that sell widgets, place orders, and people who make widgets, invoice and collect payment from those that sell. The data that passes accross this integration tool is, in itself valuable to both the people that sell the widgets, and the people that buy widgets.

Other companies making widgets would be interested to know that there are people selling widgets that are getting shipping delays when they order from other widget makers. And people who sell widgets would like to know if there are people that make widgets cheaper than from those they are buying from.

But if I make widgets, I don’t want other widget makers to see what my prices are.   If I sell widgets, I don’t want other widget sellers to know what I am paying for my widgets. This information is something that makes me able to compete.  It is the type of information that the widget maker and seller want to own, and control when and if it is released.

But there is one person that knows all of this information about all of the widget makers and sellers.   That is the person that has access to the data of the integration tool.  It is also likely that the manufacturer and retailer have signed contracts about the disclosure of the information.  This information is confidential and can not be disclosed to other parties.

I was involved with one company that sold the information or information about it back to the manufacturers and retailers.  Since we had all of the data, we could easily report when there were discrepancies between orders, and shipments.  So we reported to the retailer patterns of  order anomalies they were having with all of their vendors.  We also reported catalog errors to the vendors so they could see what retailers continued to use bad or old part numbers.

And then the tricky part.  We convinced the vendors and retailers to let us have access to their data for “Industry Analytic”  products.  For this the information became abstracted so that no one could see information about a specific retailer or vendor.  But they could see industry trends or get a score card reports of a variety of metrics relating to their supply chain health.

This was a fun project to be a part of.  And the company I was with created whole new products and revenue streams.  But you don’t have to be a supply chain integration company to make use of data that you own.  If you have vendors here are a three questions that you can ask your database people to provide you with.

1. Order time to Fulfillment time.
2. Back Order or Out of Stock frequency.
3. Order Price to Invoice price accuracy.

And there are more questions to ask.  Asking them can help you to see places where your supply chain is leaky, and can help you start to leverage data that you own to improve your own bottom line.

Well, maybe it should be. The evil robot found this video, and thinks it is good enough to share. But don’t get your hopes up. This lays out some foundational information, but there are so many ways to use XML that this is well short of a complete overview.

But it was still fun to watch.

Once.

Video Description:

“Here you will learn what you NEED to know about XML. You will learn: XML syntax XML and CSS You will not learn XML and XSLT or JavaScript, because that is not necessary and it is very complicated. Have fun with YOUR commands.”

Continuing Mapping Exercise

Today we will identify our data source, and begin mapping the source data to the target data.  We identified our target format and placed that in the paper map last time.  If you didn’t read that post yet, you might want to review it quickly before continuing.  (read part One)

The Source

The source is an EDI Invoice.  We will pick an X12 4010 810 as our standard, and I am providing you a link to the Standards document here.  This is a sample standard and does not contain an exhaustive list of elements, just the ones that we are using.   TIE810

We then insert the source elements for the data that we have in this way, starting with the most straight forward data.  I am starting at the bottom and going up.  IT1_04 is the Unit Price.  And it will map directly.  So do ProductID and Quantity.  Quantity from IT1_02 and Product ID from IT1_07 where IT1_06 is “VC” (I will also look on down the line at other odd numbered elements greater than 7 that have a preceding qualifier of “VC”, this allows the execution of the map to be more flexible if the invoice line item is formatted freely.  You may not be able to do this depending on your mapping technology.)

After the easy ones, the direct mapping ones, we get to the concatenated ones.  InvoiceID is the concatenation of the ISA_06, ISA_08, ISA_12, GS_06,  ST_02 and BIG_02.  This is straightforward as well.

Conversion

The status will be pulled from the BIG_09 and we will want to convert this value.  Here is where the fun begins.  We may say, “Only this subset of values is accepted.”  List a set in our usage spec, and fail any that don’t comply.  Or we can try to map all of the invoices status that are in the standard to an internal status.  But this also leaves us with a possible gap if our Trading Partner doesn’t comply with the standard.

Or we can split the difference.  We map status codes that make sense to our system to their corresponding status.  And for all others, we map them to “OTHER” and note what they were into the Notes Field.  Doing this retains the data for someone to look at later.  It prevents failures on this point, and keeps the status from becoming abstracted from the original intent.

Homework time

Go ahead and map the other straight forward data, and document any of the transformation rules that you can.  When you are done you can check out the map that I completed up to this point.  They don’t have to be exactly the same, and only look at the example after you have given a go at doing it yourself.  Next week we will finish the mapping and talk about ways to solve some of the problems faced by the more complex and missing data.

.

The Target

Open Office tables are divided into two tables. Invoice and Invoice Details. This can be mapped in two ways; The first way is to map the data into one common format and rely on whatever ETL tool is importing the data to catch and split the data. Second, a key can be acquired or constructed in the transformation and then the the data can be divided in to matching input formats. Then when these inputs are moved into the database, they will relate to each other on this key.

The choice of how you will do this will depend on your environment. Questions like, “Will I have enough data to provide a unique key?”, or “is there a way to get a key with an API call or database query?” The answers to these questions will determine what course you will take.

If the system ultimately receiving the data is asynchronous to the transformation, and you need to send the invoice and invoice details data separately, some care needs to be taken to ensure that the data can be related after it is separated.

So what data in the invoice can be used to tie the invoice to the invoice details? The first answer might be, “The Invoice Number.” But this number is not guaranteed to be unique across multiple vendors.  In EDI and cXML there are document unique identifiers.  Since we are using EDI, we can use a combination of the ISA Sender, Receiver, and Control number.  We will also want to use the GS control number, and ST control number in the event that more than one invoice is sent in a single envelope.  And we might as well tack on the actual invoice number from the BIG_02.

Database Tables

Invoice Table:	Invoice Details Table:

The Paper Map

Now that we know what the target looks like, we fill out the target side of the paper map.  Since we will create two “files” in our output.  We are creating the Invoice and the InvoiceDetails file, but we can use one paper map for both, and will distinguish this with a bar between the two “files”.  (I am saying files, but this could be a queue, or a post, or an insert over odbc, etc)

What’s Next

Today we went through the process of identifying the target, and creating a paper map with the target format identified.  We talked about some of the strategy that we use in deciding what to map and how to map it.  Next time we will identify the source, and begin mapping data from the source.

1.  A big Whiteboard

This is probably my number one requirement.  When I am thinking, I like to draw it out.  I haven’t found an application that gives me the same creative release and adaptability as my whiteboard.  After starting to do some of my work from home, I went out and acquired a 10 X 4 whiteboard to put in my home office.  Having the extra room is essential.  I can get a call, and walk over to the whiteboard and start drawing out what needs to be built to solve the problems while I am still on the call.

You may not need a white board as big as mine.  And you may have an electronic solution that you like better.  If so, please let me know, I like to try out gadgets.

2.  Spreadsheet

Now this may not sound earth shattering, but we are not just talking about the basics.  You will need to learn to write filters, import data, link cells and perform calculations.  If you think that a spreadsheet is like a ledger, then you are missing the power of a spreadsheet.  If you don’t think you have the skills you need, here are some links to tutorials for the two most popular spreadsheet software.

• Very Basic Open Office Tutorial http://www.tutorialsforopenoffice.org/
• More specific/advanced tutorials http://openoffice.blogs.com/openoffice/
• Some basic and advanced help for MS Excel http://www.internet4classrooms.com

3.  Text editor.

Familiarity with more than one is needed as you will find yourself on Windows and Unix servers and they will have different sets of tools.  One of the things that is the most frustrating is to not know how to use the native editor of the system you are on.  So get familiar with Notepad, and then get familiar with VI.  You can add in other tools like UltraEdit, TextPad, and more, but you should know the native ones first, if not best.

• Ultra Edit is a popular tool in some circles.  http://www.ultraedit.com/
• Text Pad is also popular.  http://www.textpad.com/
• Believe it or not, notepad has some tutorials.  http://bink.nu/news
• And here is a cheat sheet for VI commands.  http://downloads.techrepublic.com

4.  File compare.

One task that will need to be done is to compare a file with another file to detect changes or differences.  This happens in both new integrations, and in trouble shooting or investigating existing ones.  Some systems have native comparison tools, others don’t.  And there is much variety in how they work.  Here are some of the tools I have seen and used.

• Wiki has a great list.  I don’t know that I can improve on.

5.  DB query

Much of the time, integration involves a database somewhere sometime.  To work on an integration and not have to have a DBA sitting in your lap, you need to be competent to query the database.  And this involves using some tool.  Sometimes systems will have native tools, other times you will need to connect your own.  Here is a list of tutorials for DB Query tools that I have seen and used.

• Squirrel SQL http://squirrel-sql.sourceforge.net/
• SQL Worksheet
• TOAD
  

Summary

This is by no means attempting to be a comprehensive list of tools.  Such a list would be long, if it were possible.  And I don’t think that it is.  There are however some tools/skills that help us to be more effective as Integration Engineers.  Seeing what others use is helpful especially when we find that the tool we used to use is not longer around.

What tools, applications, or skills do you find you are falling back on often, on more than one project?  Or what new tools have you found that you think you will use often in the future?

Mapping:

I practice and advocate the target oriented mapping approach.  So we will start with defining the target.  We will do this in our 6 column spreadsheet mapping template.  This is also referred to as a Paper Map.  To download a blank template, click here.  Once we have the target defined, we will add the source to the spreadsheet.  After both the source and the target have their fields on the Paper Map, we add the rules.  We will also need to insert some control structures to make everything flow correctly.  Once this is done we will have addressed the data format issues and constructing the map for our translator becomes just a technical exercise.

The Flat File Target:

What are we mapping?  This is the first question that the target defines.  In this example it is an invoice.  So we will want the following information in the following format.

RecordID|VendorID|DateOfInvoice|InvoiceNumber|DateOfPO|PONumber|POLineNumber|QTY|

~UnitCost|LineTotal|InvoiceTotal

Here are the parameters for these values:

RecordID:  A unique alpha-numeric.  It is derived by contacting the ISA_Sender, ISA_Control_Number, GS_Sender, GS_Control_Number, ST_Control_Number, and Line_Number where the line number exists.  Max size of 74 characters.

VendorID:  The ISA_Sender ID.  Alphanumeric with a max size of 15 characters.

DateOfInvoice:  A long date, max 10 character formatted (mmddCCyy)

InvoiceNumber:  Invoice number from the Vendor, Alpha numeric, max 22 characters.

DateOfPO:   A long date, max 10 character formatted (mmddCCyy)

PONumber:  PO number sent on the PO.  Alphanumeric, max 22 characters.

POLineNumber:  PO line number must be the same as the invoice line number.  AlphaNumeric, max 20 characters.

QTY:  Quantity of items invoiced.  Numeric, max 10 characters.

UnitCost: Cost per unit.  Currency, max 17 characters.

LineTotal:  Total of line cost.  Currency, max 17 characters.  sum(QTY*UnitCost)

InvoiceTotal:  Total of all line totals.  Currency, max 17 characters.  sum(all LineTotals for Invoice)

The EDI Source:

We are going to get our data directly from the EDI.  We could be doing API calls to resolve VendorID values or getting an incrementing RecordID, but we can get what we need from the EDI under the right circumstances and so that is what we will do here in this example.

For the record ID we are going to do a simple operation and concatenate several values to create a unique value.  This will almost always be unique.  And it will refer back to the original document as a bonus.  If we wanted to ensure that it was always unique we could include data and timestamps in the concatenation.

For the dates, we are converting the date from the EDI format into our preferred format.  Many systems understand date format conversions, so it might be as simple as a cast or convert command on the existing string of numbers.  If not, we will have to parse out the separate values and then string them back together concatenating in with dashes.

And there are two values that we will need to calculate. We will calculate the value for the invoice total before we create any lines.  And we will need to create the line totals.  Since these require processing the same data, we might create all of the totals and put them in an array to easy access.  Anyway, this is the time to be creative.

Constructing Rules:

Now that we have the source and the target we will makes the rules, and we already have some in mind.  We know that we are concatenating the record ID, reformatting the dates, and calculating the totals.  We also add in some other rules like the trimming of the space on the ISA_06 to Vendor ID operation.  We will also need to do that in the record ID or it will look odd.

Controls:

Controls are like rules, and they may be written in the same place if we need to.  But where rules describe how we completed our mapping, controls tell us when to and not to map something.  Sometimes mapping is conditional.  Controls tell the map what conditions allow for valid mapping when conditions exist.

There may be times that we don’t want the map to go on and complete the process.  Obviously if we have missing data.  If there is no invoice or PO number, you may want to stop with an exception.  Generally these types of controls are easy to build as you make the target field required.  To indicate this you may want to place an additional column to indicate what data is required.  On simple projects you can just put this in the notes.

There will also be other times.  For instance, it might be that some invoices have their own total.  If they do not match our calculated total, we want them to fail mapping and go to manual resolution.  A use case would be to have additional charges attached to the invoice. This extra line charges would need to be handled outside of our map.

Controls can be written to make the mapping very powerful and provide a gateway to your data that ensures it stays valid and accurate.  Some controls go beyond stopping the map.  Instead we use them to put conditional logic into our data mapping project.

Summary:

This example project provided a short exercise to familiarize you with using a paper map and the fundamentals of target mapping.  In this simple example we had a target specification, and a source specification.  We created rules to move and change data from source to target, and we discussed controls.  These are the fundamental tasks of mapping.

But this is not the end.  I know that as I worked on this project, I saw things that might improve the map.  I didn’t include them because I wanted to keep this mapping project simple.  But you don’t have to.  You can add your own improvements to this project and make it do what you want.  I would be happy to respond to these if you post them in the comments.

You can find the resoucres for this project in the tool box.

Theoretical discussion of integration of data and information.

Speaker is Alan Nash

Alan discusses two fundamental problems in information integration:

(1) How to answer a query over a public interface which combines data from several sources and

(2) How to create a single database conforming to the public interface which combines data from several sources.

Alan is using databases as both source and target and uses them in his examples of how data exchange and integration problems are addressed and solved.  This is a little bit heavy, but if you feel up to it, it is 53 minutes long.

This is a still a pretty high level discussion, even the practical examples are abstracted.  I have used a tool that helps do the tasks described here.  If you have this type of task, you might want to check out a company called Convertabase at http://www.convertabase.com

RFID is exciting technology for integration engineers.  All of this data coming in will have to be aggregated and integrated by someone.  Also, one of the focuses of integration is to provide more and better information to people and systems.  With more and better information people, businesses and systems can make better choices and decisions.

This is a good thing.

People talk about privacy issues and consumer protection.  I am sure that debate will continue to rage on long after the keyboard and mouse you are using have RFID tags embedded.  But the focus of this blog is how these technologies can be used for the collection and integration of information.  Implementing RFID technologies provides a hugely beneficial and accurate stream of data.

A character set is a collection or library of characters, (letters and symbols), and their identifying number.  Included with the printable characters, (letters and punctuation) are some unprintable yet important characters.  Characters are used to form messages.

Characters are not fonts.  Characters exist under the font that represent the definition of the character the  font is attempting to display.  When you change the font on a document the A is changed to an A, but the underlying character that identifies its meaning remains the same.  The font identifies how the character is displayed.  You can even convert to Wing Dings and the underlying character remains the same.

We can imagine that if I wrote this post using a character set that I created myself.  And then you came and tried you read it, without knowing what my character set was, you would see a bunch of garbage on the screen like if you go to a foreign language web page without the correct fonts loaded.  Even worse would be if you used the same characters, but had different Identifiers for them.  If an A is 001 (my set starts at A and moves on numerically) and you try to read it, (but in your character set you numbered the vowels after the consonants) and 001 to you is B.  Now all of the letters will be wrong.  And we get garbage.

Fortunately, some people got together early on and created a standard for characters.  The American Standard Code for Information Interchange created the character set we call ASCII.  The Extended Binary Coded Decimal Information Code was created by IBM, but they use ASCII now as well.

What is ASCII

ASCII is the acronym for American Standard Code for Information Interchange, and is a collection of characters defined from 0 to 127.  These definitions represent all of the standard English characters, numbers and symbols.  A number of other, unprintable, characters are also included.  You use one or two of these each time you hit the “Enter” key on your keyboard.  Depending on your operating systems, this sends the “carriage return” and or “line feed”

A “carriage return” comes from a printer where the head would move back and forth on the roller.  CR would tell the printer to move the head all the way to the left of its printing area.  A “line feed” is also from a printer perspective.  This tells the printer to roll the paper so that the head will be writing on the next line.  These are both examples of unprintable characters.  You can probably think of others.  For a complete list of ASCII characters, you can check out this table in my toolbox.

What is EBCDIC

EBCDIC is the acronym for Extended Binary Coded Decimal Information Code.  This was created by IBM back in the day.  IBM now uses ASCII just like everyone else, but there are legacies that are still with us.  Old terminals like VT100 and some legacy communications equipment still expect messages using the EBCDIC character set.

What is the big deal

As I said, some systems still want to use some of the characters in the EBCDIC system.  Even fancy new systems producing XML will sometimes fall into this trap and cause problems.  The one that I have run into is the use of | called ‘pipe’  ASCII and EBCDIC use different character IDs for this character.  And I have seen e-commerce systems, that are using ASCII for everything else, throw in an EBCDIC pipe as a control character.  When this happens, other systems will choke on it.

When you find yourself getting an invalid character message, but the characters look fine.  Remember that there are some twists that may exist in the underlying character set.  If you can, manually replace the character with the character that it looks like, (in my case the EBCDIC | with a ASCII | )  and see if the parser likes the file now.  If it does, you have encountered the character set problem as I have.  This can be a difficult problem to solve if you have never encountered it.

If the character that is causing problems is not a pipe, you may want to look at IBM’s ASCII to EBCDIC conversion table.  This can be difficult to communicate with others that have never encountered it, so using the ACSII and EBCDIC identifier designation can help explain what we are saying in email and documentation when we are trying to correct the issue.

Well as time went on, and I became more adept at using windows, I noticed that it tried to tell me what was wrong.  Windows was writing out a crash report.  And it had a message on the blue screen of death.  (a mostly incomprehensible message, but at least it was trying.)  And I discovered that if you looked for the error codes in Microsoft’s knowledge base, you could find the errors had corresponding articles for many of them.  (some of these articles were also incomprehensible.  )

Well when you write an application in perl or any other language you will need to do something other than just crash when you encounter an error.  There are two views on what to do with an error condition; write the message to a log, and write the error out in a separate error message or file.

I guess there could be a third view, and that is to do both.

Logging:

Writing errors to a log, can be just part of the normal logging that you do as your application processes.  If you aren’t logging, you might want to consider it.  There is value in having a log file of what your application is doing.  In order to keep your logging coherant, I create a logging object and then just pass my log messages to it.  This way they can all have the same format with timestamps and such without having to write it out every time.

Error Messages:

My application is processing messages, transforming CSV and other types of files.  So when errors in processing happen, I move the files to an error directory, and I also write a special error file with them so that I don’t have to parse through the log to find it.

Using alerts:

Writing messages to a log and moving files to an error directory are fairly passive ways to handle errors.  A more active way to handle and error is to send an email or trigger an alert of some kind.  You can also use a monitoring technology like nagios to watch for errors, and send alerts.

What are ways that you have handled errors and logging?

What is a Circular File?

This is not like sending the log data to /dev/null or some other black hole.  It is a file that can’t grow beyond a specified size, but never refuses to accept new data.  This is not a paradox.  Visualizing this file like a roll of paper instead of a sheet of paper.  On the roll, there is no end, it just goes around and around.

Say you have a small circular file, 10 MB in our example.  Then lets say, every minute, you add 1 MB of data to your file. For 9 minutes after you make a log entry, you can see this entry in the file.  Each time it moves down on minutes worth of space on the log.  At 10+ minutes, it is gone.  The circular file has rolled over and overwritten that location with more current entry.

Okay, so a 10 minute log may not be that useful, and we would probably want to used a bigger size.  But the point is, that if disaster strikes, the log won’t grow like crazy and create an even bigger problem by filling up the disk.

Why aren’t all logs like this?

Using a circular file for a log is very useful, but if you have an error, and then the log scrolls on for long enough for the error to be lost, all you have are the symptoms not the cause.  So you have saved yourself the problem of making, “The smtp server failed, and the Web tool started logging errors that it couldn’t send emails.  People kept clicking, resend mail until the log filled up the disk, and the app died.” explanations.  But they are replaced with the, “Yes we logged the error, and then the app started logging all of the following actions and the error fell out of the log.”  followed by an explanation of what a circular file is and why this bad news is really good news.

Its best to use more than one logging method.  Say we have a circular log that allows us to see a week of normal activity.  Lets pretend that is 10 GB.  And we have a log that is flat, and only logs critical errors.  Now we have a way to not get clobbered by all the actions that start happening when the rain comes, and we don’t lose that original error that started the mess.

Utopalog

Okay, I am making this up, but here is my dream strategy.  First, we use a circular log that can be size configured on the fly.  Second we use an error database.  We have two table structures; First Occurrence, and Last Occurrence.  Just as they sound, when we have a new error, debug, whatever, it gets logged in the First and Last Occurrence tables.  But the Next time it happens, the time on the Last Occurrence table is the only thing that gets updated.

To use this, we periodically truncate or archive the tables.  The log is self cleaning.  We can do this each night, or on demand, or pick your own schedule.  Now when a problem occurs, we can look for errors that happened before the beginning of the log, but we won’t have a mountain of data that will crush us.

Files are called “Flat Files” when they contain a single data structure.  Generally this structure is the column and row structure like a spreadsheet or table, but a file in binary or encrypted with a single encryption key could also be called a flat file.  Files that are not flat; marked up files like XML or HTML, EDI files, other formats like HL7 or SEF files and others.  Here I am going to briefly discuss two flat file types; Delimited Files, and Fixed Width Files.

What is a Delimited File?

Ok, to describe it briefly, a delimited file is a file where the data is organized in rows and columns.  Each row has a set of data, and each column has a type of data.  If it sounds like I am describing a spreadsheet, you are right on the money.  To make the column, each row has the columns separated with a character called a delimiter.  See the example below.

Tables of data and spreadsheets are both similar to a delimited file in the way they organize data.  In the delimited file all of the empty space, or white space is removed.  What we see here is a classic example of exporting a spreadsheet table as a comma delimited file.  In theory, this data can be imported by any other application that can read a delimited file.

Believe it of not, a space is a character, and takes up space in a file.  Back in the day people went out of their way to save space so that files could be send over slow modem connections.

What is a Fixed Width File?

There is another type of file, is is called a Fixed Width or Fixed Position file.  It is different from a delimited file in that the data fields are defined by the character position.  See the example below.

In a fixed width file, the delimiter characters are eliminated.  If the data is formulated such that the data fields are the same size, this format can be more compact than a delimited file. You can see here that we know the size of the Birthdate data, so we eliminate all the spaces between the Bdate and Department fields.  If all of the data was formatted for size like this, we could really make this file small, so that it only contains the data.

We also eliminate the pesky problem of delimiters found in data.  The issue of a comma delimited file containing a field that has a comma in the data.  How does the parser know that this comma is not really a delimiter, but is part of the data?  Anyway, that problem is eliminated in a fixed width file.

Comparison

This is not a contest of which format is superior.  Both file architectures are useful and both are used commonly enough that you need to be at ease working with both.  Delimited files are really easy to work with as long as your data is clean of the delimiter character.  Doing quick integration of data common in ETL tasks, delimited files are far more common that Fixed Width.  Continuous operations of data integration and importation many times find that Fixed Width or Position files are more reliable for the unattended operation, even ETL if it is unattended.

As with many things in integration work, we want to pick the best option.  Knowing and working with both fixed and delimited files will help you determine which is the right choice for the task you have before you.

– Data, Star Trek: Next Generation: “Where Silence Has Lease“

If the beginning of wisdom is to realize what it is that we do not know. NULL, by its definition is this not knowing.  We do not know what NULL is, this is why it is NULL.

What is this?

Null is not a number, or letter.  It may not even be a character.  Using Occam’s razor, things are either NULL or NOT NULL.  Things that are NULL are completely unknown at the time they are NULL.  And things that are NOT NULL are not completely unknown when they are NOT NULL.

I feel the hair splitting on my head, so let me explain one important point.  Things that are NULL are not destined to stay that way.  Where /dev/null/ is the black hole of output that we don’t need, NULL is not the data equivalent of a black hole.  Things that are currently NULL may become NOT NULL at any time, as soon as we know something about them.  Pretty much anything really.

Empty String Theory

Unlike string theory, which attempts to define the nature of the universe, the ‘empty string theory’ is that empty strings are the same thing as NULL.  They are not.  If they were NULL, we would not know that they were strings.  This seems quite clear.  NULL is NULL and an empty string is NOT NULL.

“But wait!”  I hear a Database Developer cry.  “I can create a field in my database that is a string, and allow it to be NULL.”  And this is correct, but this is not a contradiction.  When you allow a database field to be NULL you have allowed it to receive NULL as an input.  But it may or may not give NULL as an output.  Because we can put NULL into a variable, when we query for that variable, element, whatever, we may get NULL, or we may get an empty string or property defined value of that data type of the field, that just happens to be empty.

In final reply to this inquiry I must observe that NULL is not equal to anything.  Any conditional statement where NULL is compared to something will be false.  Where, “” represents an empty string.  If one writes the statement, ["" == ""], or in English, “if an empty string is equal to an empty string,” it evaluates as a true statement.  However if one writes the statement [NULL == NULL], or in English, “if NULL is equal to NULL,” it evaluates to a false statement.  Since an empty string is equal to another empty string, but NULL is not equal to NULL, then NULL cannot be the same thing as an empty string.

Why NULL is important

Just as early civilizations had no concept of zero, early discourses of NULL struggle with it.  Where we can argue if you have zero apples, is the same as having nothing.  Yet it can be the absence of something that is known.  Zero can be having none of something.  But NULL is not even knowing what the something is that you don’t have.

An apple basket can contain 0 apples as a way to say, “Hey, this basket is for apples!”  So is 0 greater than NULL.  Because with null we don’t know what the basket is for.  We may not even have a basket at all.

There are a great number of operations that we couldn’t do well or efficiently if we didn’t have NULL.  So understanding what we are asking when we look for NULL is important.  Whenever we have a search for things that don’t match something else, we are searching for NULL, even if the values themselves are not NULL.  We do this a lot, and we couldn’t do it without NULL.

I believe that Steven Covey says it this way, “Begin with the end in  mind.”  We do the same thing in data mapping, except that we begin with the end or target if you will.  The data target naturally leads us to the data sources that we need.

If one starts with the source, a lot of work goes into the project, like organizing the source data, before we know if we need any of it.  Sure, you may know that you will need the shipping addresses if you are mapping to a shipping transaction, but spending time gathering the information on the source of this data is still not effective until we know where we will be putting it.

A guide to mapping data from source to target

Spreadsheets are great for mapping.  We start with six columns.  From left to right we have source, source data type, rules, target, target data type, and notes.  We start wit the target and list the data elements that the target needs in the order that the target needs them to approximate the target form.  If this is a database table then this is really simple, and we are just listing the columns names in order.

If the target is something less list like, we will need to add some location information.  In XML we might want to include the xpath to the spreadsheet either in the cell above, or in a logical grouping.  Both approaches work and I will show you and example of each.

As we list the target we will also want to note the data type in the column to the right.  As we note the type of any data with specific formatting, like date or email addresses, we can place that definition here.

After we have the target well defined it is time to decide where this data will come from.  We probably already have some idea, so now it is time to work down the source column and fill in the data reference.  There may be more than one field that maps to a single field on the right.  And there may be one field on the left that maps to multiple fields on the right.  There may also be completely derived data that is calculated at runtime.  There can also be multiple data sources.  You will probably encounter all of these and more in your career as and integration engineer.

Upcomming Articles:

1. EDI mapping excersize.
2. XML mapping excersize.

My Answer:

First, let me just say that my answer was really, “40 hours or 2 calendar weeks.”  When asked if I could do it in 40 hours but in one week, I would say, “No.” and explain that I always had interruptions and other priorities to juggle.  There was never a time that I could only focus on just one issue for a whole week.  Thus it would not make sense to give them the expectation that I could be done in a week, even if the actual hours worked on project was 40 hours.

Second, let me say that these where all mapping change requests, or new document mapping requests.  I am extremely good at this, and have a system for keeping things simple and effective.  You can read more about that by reading some of my mapping posts here.  And following that system, I could pull of a changed or new map in 2 weeks with confidence.

When asked if I was sure that it wouldn’t take longer, I would shrug and say, “It never has before.”  Well not a detailed justification for my time spent, this was an expression of confidence to get my time estimate accepted and put on the plan.

Other Answers:

There are many ways to come up with a time estimate.  I took a class once, that discussed project planning and explained Gantt charts and time lines.  I had been using versions of these for years without knowing they had a name.  I called mine a “white board” but I did learn the lingo of projects that was useful when communicating with other how your time line is looking.

Project Planning is a great skill to have, but don’t get carried away.  Lots of time can be spent on making a project plan instead of actually working.  When you are working as a lone ranger, or small team, extensive planning can actually get in the way.

Team Work:

One of the things that let me get away with the “2 Week” time estimate was that I was the only one doing what I was doing.  Its not that I am anti social, it just that I was the only one left from what was once a team of 6.  I was helping support the product, building new features, and writing my own requirements based on tickets and user/manager responses.  And not having to coordinate with other people made me much much faster.  (but of course you have to know what you are doing to pull this off)

I remember sitting in a meeting with a group of developers.  They had been working on an integration process for 6 months and were still months away from completion.  I remember feeling really sorry for them when I realised that my boss had asked me to produce an integration solution for the same thing they had been working on.  Without knowing their situation, I responded with my 2 week estimate.

Well, they were floored, and didn’t believe I could do it.  I did, and they realised why.  I didn’t have team meetings, (it was just me).  I didn’t follow their coding standards and review processes, (not having to teach a team about what I had just done).  I had a test system that looked just like production that I built on, (they had a dev system, that all of their projects were being built on.  And this meant they were always in each others way.)

In the end, (after I had done it my way) they redeveloped it in their way and the project got completed.

This is not a knock on teams.  Teams are important to have.  But sometimes it is smarter and faster, just to give a guy a direction, and then get out of his way.  Teams help with stability and code longevity, but they can slow the process of creation down.  And the bigger the team, the slower it can go.

Mapping Steps

There are five steps to this mapping process.  Here is a list with a brief description of each one;

• Defining the target:  This is the first step to mapping.  We must know where we are going if we plan on getting there.  And using a Target based mapping process we decrease the time and efort of the other steps.
• Defining the Data Source:  This is the second step to mapping.  Once we know what the target looks like, we know what data we need to complete it.  So this naturally leads us to defining the source(s) of the data.  This can be a single input data file or record, or it can be multiple types of data from multiple sources.
• Defining the Processes:  Some of the data that we will need in our target will not be in the proper form or format in the source.  We will have to have a process defined in our map to convert the source data into the correct form for the target format.
• Handling customisation Points:  When we are mapping data, we will see points where we will want to make a decision on what to do.  Not all data is created or received in an equal form.  Thus we may have points in our mapping process where we will need to do a different process depending on the source or content of the data.
• Monitoring the Process:  After we have created a mapping process we will need to monitor it so that we can know when something unexpected happens.  To do this we must identify the right places in the mapping process for it to report its status to a monitoring process.  This is the last step in the mapping process.

Catalog Target

If you have a catalog, then you will have your own schema.  We are getting our schema from a standard Open Office DB template.  This way you can create your own version for this exercise.

The Product table in the Open Office Business table has these fields:

• ID, INT
• CategoryID, INT
• Discontinued, BOOLEAN
• LeadTime, VARCHAR
• ProductID, INT
• ProductDescription, VARCHAR
• ProductName, VARCHAR
• ReorderLevel, INT
• Serialnumber, VARCHAR
• SupplierID, INT
• UnitPrice, DECIMAL
• UnitsInStock, INT
• UnitsOnOrder, INT

These are the basic fields that we will be inserting data into.  We will probably have a process that will validate the supplier ID before inserting new records, but that will come in during the process step.

Next Steps

Following defining our target, we will define our source.  But we will do that in the next instalment of this exercise.   Download and create start your paper map now, we will show them in the next post as well.

One of the basic tasks Integration Engineers do is to compare files that we use or receive.  There are some interesting and useful tools that people can get out there to DIFF files.  But on Linux and Unix machines around the world there is a native tool that is almost always present.  Amazingly it is called DIFF.

Like some other command-line tools, its interface is not really intuitive.  Lets walk through the basics of how to get use out of this handy file comparing tool.  (If you are working with and comparing EDI files, you might want to look at the post on how to unwrap your EDI file so that our line by line comparison is more meaningful.)

How to used “diff”

You can get the real basics by executing “diff –help” to get the basic help and options for this application.  But in short, here is the thumb nail. “diff” is followed by some options.  Options are designated by a “-” and then a letter indicating the option.  Any options are then followed by the two file names that are being compared.  Lets look at an example.

Example:

We have two files, file1.txt and file2.txt

When we issue this command:  “diff file1.txt file2.txt” we get this result.

2c2
< And this is the first line of the first file.
—
> And this is the first line of the second file.
3a4
> Again.

• The first thing we see is “2c2″  This is line 2 of the first file, compared or changed to line 2 of second file.
• Next we have a < indicating the first file, and the line echoed.
• Following this we have a “—” as a separator between the line compared.
• Next we have > indicating the second file, and then that line is echoed.
• This is a comparison between to lines that were found to be different.
• For the next line that is shown, we have “3a4″ that indicates that there is a line added to the file.
• Finally, > indicates the second file followed by the line being echoed.

If we were to compare them in the other order, we end with these two lines:

4d3
< Again.

• Here, “4d3″ that the 4th line on the first file is deleted from the second file.
• Following this is < indicating the first file, and echoing the line.

Regular Options

Here are the list of options that “–help” gives you, with maybe some more explanation.

diff [-b] [-i] [-t] [-w] [-c] [-C] [-e] [-f] [-h] [-n] [-D string] [-l] [-r] [-s] [-S name] [fileone filetwo ] [directoryone directorytwo]

For comparing file, the first four options (-b, -i, -t, -w) are the most useful.  I don’t start with any options and add them as I need them to reduce the amount of change noise reported in the result set.

diff is your friend

Like many basic tools, “diff” is almost always there. And if you know how to use it effectively, it can really save time and frustration. Sure there are other cool file comparison tools. Some are even embedded into other products. But knowing how to use the basic tools that are always there will be a life saver in a crisis situation. And the only way to know how to use them is to actually use them sometimes.

Do you use “diff” with a set of options that does a specific task for you? If so, what are they, please share. And what other basic tools do you use?