Mapping Steps

There are five steps to this mapping process.  Here is a list with a brief description of each one;

• Defining the target:  This is the first step to mapping.  We must know where we are going if we plan on getting there.  And using a Target based mapping process we decrease the time and efort of the other steps.
• Defining the Data Source:  This is the second step to mapping.  Once we know what the target looks like, we know what data we need to complete it.  So this naturally leads us to defining the source(s) of the data.  This can be a single input data file or record, or it can be multiple types of data from multiple sources.
• Defining the Processes:  Some of the data that we will need in our target will not be in the proper form or format in the source.  We will have to have a process defined in our map to convert the source data into the correct form for the target format.
• Handling customisation Points:  When we are mapping data, we will see points where we will want to make a decision on what to do.  Not all data is created or received in an equal form.  Thus we may have points in our mapping process where we will need to do a different process depending on the source or content of the data.
• Monitoring the Process:  After we have created a mapping process we will need to monitor it so that we can know when something unexpected happens.  To do this we must identify the right places in the mapping process for it to report its status to a monitoring process.  This is the last step in the mapping process.

Catalog Target

If you have a catalog, then you will have your own schema.  We are getting our schema from a standard Open Office DB template.  This way you can create your own version for this exercise.

The Product table in the Open Office Business table has these fields:

• ID, INT
• CategoryID, INT
• Discontinued, BOOLEAN
• LeadTime, VARCHAR
• ProductID, INT
• ProductDescription, VARCHAR
• ProductName, VARCHAR
• ReorderLevel, INT
• Serialnumber, VARCHAR
• SupplierID, INT
• UnitPrice, DECIMAL
• UnitsInStock, INT
• UnitsOnOrder, INT

These are the basic fields that we will be inserting data into.  We will probably have a process that will validate the supplier ID before inserting new records, but that will come in during the process step.

Next Steps

Following defining our target, we will define our source.  But we will do that in the next instalment of this exercise.   Download and create start your paper map now, we will show them in the next post as well.

Industries have common ways to handle data.

This is something that we should understand.  Mandating that others use your standard usage implementation may work if you are Wal-Mart, but is not really necessary.  Most supply chain integrations are industry specific.  Probably most EDI integrations even the non-supply chain ones are also industry specific.  And you will find that most of the Trading Partners in a industry need to use the same data sets.  And you will find that most of them are using them in about the same way.

What this means is that when you set up a map for one Trading Partner, most of that mapping will also work for the next one.  This allows us to write maps that are flexible and can handle the variations in the data without having to have a map for each partner, or a crazy set of conditions that say, For Trading Partner 1 do xyz.”

This is of course for an inbound process of in other words, EDI data we receive from others.  And it is simple to accept a par number of other piece of data in more than one location and map that to our canonical.

What is more tricky to visualize is how this works on the outbound.  Well it is possible to map a part number or other piece of data to more than one location, this can be confusing to the trading partners that are receiving it.  Instead, we do a pre map manipulation or pre mapping for trading partners that step out of the normal map.  We could also have a post map process if your technology makes that the better choice.

Here is how that will look. First, we will have a map that is our standard preferred mapping.  This map takes our canonical data and maps it into our outbound EDI format.  Next, we create a process that will either modify the structure or content of our canonical to reflect the custom needs.  We also adjust the map to recognize and handle the new fields. Alternatively we could have a post map that maps our standard outbound EDI and alters data or structure.

Strategies:

1.  Broad mapping:

This is a mapping strategy that I have found works well on mapping data received well from Trading Partners.  One illustration can be found in the discussion of line item data.   Where a specific piece of data may be found in more than one location, you create your map to accept this data in all of the valid locations and out put that into your canonical.

It is also possible to do this on the outbound or map that formats data for consumption by your Trading Partners. This is not as straight forward, but there are times when you can see conditions in the data that cause the format or output to change. When these conditions exist, include them in the mapping and you will have a more agile and flexible map.

2.  Pre mapping:

This is a map before the map.  Well it doesn’t really have to be a whole map.  But the concept is this;  Data received or sent to a Trading Partner needs to be changed on a more Trading Partner specific basis.  It doesn’t actually need to be a TP condition, these pre maps can detect a data format, and alter it before sending it to the map.

For instance, I had a instance where we had a TP that was consistently sending us a DUNS number with the ID of “1″ and we needed “01″ for our system.   So we put together a pre-process that would look for these qualifiers of “1″ and altered it to “01″.  In this way our map didn’t have to have funky code, and we were able to integrate with these trading partners without having to ask them to change their vendor data in their DB that was trimming leading zeroes.  In any case, I have found that pre mapping is useful when there are consistent standards violations or data manipulation that can be corrected systematically.

3.  Post mapping:

This is similar to the pre mapping, but your process happens after the map.  This is a good use for mapping data out to your Trading Partner.  And again, they don’t have to be trading partner specific.  Data going out is altered slightly in form or format before it is delivered.

Again, I had a trading partner that transmitted catalog and part lookups in a format with a dash in the part number.  But they didn’t like getting the dash back on their orders.  So we instituted a simple mapping change that would strip dashes for them.  But later we realized that having a process to fix part number formats was something that more than one Vendor Trading Partner was requesting.  So we moved the mapping condition to a post mapping process that would look at the TP and find any part number rule and apply it.

Flexible and Agile is just better.

Okay, so that is a broad generalization.  But I can back it up with years of experience.  I never regret mapping broadly and implementing pre and post processes.  But I do regret when I have mapped narrowly, tried to have a single trading partner specific map, or include all types of TP specific logic into a map.

Continuing Mapping Exercise

Today we will identify our data source, and begin mapping the source data to the target data.  We identified our target format and placed that in the paper map last time.  If you didn’t read that post yet, you might want to review it quickly before continuing.  (read part One)

The Source

The source is an EDI Invoice.  We will pick an X12 4010 810 as our standard, and I am providing you a link to the Standards document here.  This is a sample standard and does not contain an exhaustive list of elements, just the ones that we are using.   TIE810

We then insert the source elements for the data that we have in this way, starting with the most straight forward data.  I am starting at the bottom and going up.  IT1_04 is the Unit Price.  And it will map directly.  So do ProductID and Quantity.  Quantity from IT1_02 and Product ID from IT1_07 where IT1_06 is “VC” (I will also look on down the line at other odd numbered elements greater than 7 that have a preceding qualifier of “VC”, this allows the execution of the map to be more flexible if the invoice line item is formatted freely.  You may not be able to do this depending on your mapping technology.)

After the easy ones, the direct mapping ones, we get to the concatenated ones.  InvoiceID is the concatenation of the ISA_06, ISA_08, ISA_12, GS_06,  ST_02 and BIG_02.  This is straightforward as well.

Conversion

The status will be pulled from the BIG_09 and we will want to convert this value.  Here is where the fun begins.  We may say, “Only this subset of values is accepted.”  List a set in our usage spec, and fail any that don’t comply.  Or we can try to map all of the invoices status that are in the standard to an internal status.  But this also leaves us with a possible gap if our Trading Partner doesn’t comply with the standard.

Or we can split the difference.  We map status codes that make sense to our system to their corresponding status.  And for all others, we map them to “OTHER” and note what they were into the Notes Field.  Doing this retains the data for someone to look at later.  It prevents failures on this point, and keeps the status from becoming abstracted from the original intent.

Homework time

Go ahead and map the other straight forward data, and document any of the transformation rules that you can.  When you are done you can check out the map that I completed up to this point.  They don’t have to be exactly the same, and only look at the example after you have given a go at doing it yourself.  Next week we will finish the mapping and talk about ways to solve some of the problems faced by the more complex and missing data.

.

The Target

Open Office tables are divided into two tables. Invoice and Invoice Details. This can be mapped in two ways; The first way is to map the data into one common format and rely on whatever ETL tool is importing the data to catch and split the data. Second, a key can be acquired or constructed in the transformation and then the the data can be divided in to matching input formats. Then when these inputs are moved into the database, they will relate to each other on this key.

The choice of how you will do this will depend on your environment. Questions like, “Will I have enough data to provide a unique key?”, or “is there a way to get a key with an API call or database query?” The answers to these questions will determine what course you will take.

If the system ultimately receiving the data is asynchronous to the transformation, and you need to send the invoice and invoice details data separately, some care needs to be taken to ensure that the data can be related after it is separated.

So what data in the invoice can be used to tie the invoice to the invoice details? The first answer might be, “The Invoice Number.” But this number is not guaranteed to be unique across multiple vendors.  In EDI and cXML there are document unique identifiers.  Since we are using EDI, we can use a combination of the ISA Sender, Receiver, and Control number.  We will also want to use the GS control number, and ST control number in the event that more than one invoice is sent in a single envelope.  And we might as well tack on the actual invoice number from the BIG_02.

Database Tables

Invoice Table:	Invoice Details Table:

The Paper Map

Now that we know what the target looks like, we fill out the target side of the paper map.  Since we will create two “files” in our output.  We are creating the Invoice and the InvoiceDetails file, but we can use one paper map for both, and will distinguish this with a bar between the two “files”.  (I am saying files, but this could be a queue, or a post, or an insert over odbc, etc)

What’s Next

Today we went through the process of identifying the target, and creating a paper map with the target format identified.  We talked about some of the strategy that we use in deciding what to map and how to map it.  Next time we will identify the source, and begin mapping data from the source.

Mapping:

I practice and advocate the target oriented mapping approach.  So we will start with defining the target.  We will do this in our 6 column spreadsheet mapping template.  This is also referred to as a Paper Map.  To download a blank template, click here.  Once we have the target defined, we will add the source to the spreadsheet.  After both the source and the target have their fields on the Paper Map, we add the rules.  We will also need to insert some control structures to make everything flow correctly.  Once this is done we will have addressed the data format issues and constructing the map for our translator becomes just a technical exercise.

The Flat File Target:

What are we mapping?  This is the first question that the target defines.  In this example it is an invoice.  So we will want the following information in the following format.

RecordID|VendorID|DateOfInvoice|InvoiceNumber|DateOfPO|PONumber|POLineNumber|QTY|

~UnitCost|LineTotal|InvoiceTotal

Here are the parameters for these values:

RecordID:  A unique alpha-numeric.  It is derived by contacting the ISA_Sender, ISA_Control_Number, GS_Sender, GS_Control_Number, ST_Control_Number, and Line_Number where the line number exists.  Max size of 74 characters.

VendorID:  The ISA_Sender ID.  Alphanumeric with a max size of 15 characters.

DateOfInvoice:  A long date, max 10 character formatted (mmddCCyy)

InvoiceNumber:  Invoice number from the Vendor, Alpha numeric, max 22 characters.

DateOfPO:   A long date, max 10 character formatted (mmddCCyy)

PONumber:  PO number sent on the PO.  Alphanumeric, max 22 characters.

POLineNumber:  PO line number must be the same as the invoice line number.  AlphaNumeric, max 20 characters.

QTY:  Quantity of items invoiced.  Numeric, max 10 characters.

UnitCost: Cost per unit.  Currency, max 17 characters.

LineTotal:  Total of line cost.  Currency, max 17 characters.  sum(QTY*UnitCost)

InvoiceTotal:  Total of all line totals.  Currency, max 17 characters.  sum(all LineTotals for Invoice)

The EDI Source:

We are going to get our data directly from the EDI.  We could be doing API calls to resolve VendorID values or getting an incrementing RecordID, but we can get what we need from the EDI under the right circumstances and so that is what we will do here in this example.

For the record ID we are going to do a simple operation and concatenate several values to create a unique value.  This will almost always be unique.  And it will refer back to the original document as a bonus.  If we wanted to ensure that it was always unique we could include data and timestamps in the concatenation.

For the dates, we are converting the date from the EDI format into our preferred format.  Many systems understand date format conversions, so it might be as simple as a cast or convert command on the existing string of numbers.  If not, we will have to parse out the separate values and then string them back together concatenating in with dashes.

And there are two values that we will need to calculate. We will calculate the value for the invoice total before we create any lines.  And we will need to create the line totals.  Since these require processing the same data, we might create all of the totals and put them in an array to easy access.  Anyway, this is the time to be creative.

Constructing Rules:

Now that we have the source and the target we will makes the rules, and we already have some in mind.  We know that we are concatenating the record ID, reformatting the dates, and calculating the totals.  We also add in some other rules like the trimming of the space on the ISA_06 to Vendor ID operation.  We will also need to do that in the record ID or it will look odd.

Controls:

Controls are like rules, and they may be written in the same place if we need to.  But where rules describe how we completed our mapping, controls tell us when to and not to map something.  Sometimes mapping is conditional.  Controls tell the map what conditions allow for valid mapping when conditions exist.

There may be times that we don’t want the map to go on and complete the process.  Obviously if we have missing data.  If there is no invoice or PO number, you may want to stop with an exception.  Generally these types of controls are easy to build as you make the target field required.  To indicate this you may want to place an additional column to indicate what data is required.  On simple projects you can just put this in the notes.

There will also be other times.  For instance, it might be that some invoices have their own total.  If they do not match our calculated total, we want them to fail mapping and go to manual resolution.  A use case would be to have additional charges attached to the invoice. This extra line charges would need to be handled outside of our map.

Controls can be written to make the mapping very powerful and provide a gateway to your data that ensures it stays valid and accurate.  Some controls go beyond stopping the map.  Instead we use them to put conditional logic into our data mapping project.

Summary:

This example project provided a short exercise to familiarize you with using a paper map and the fundamentals of target mapping.  In this simple example we had a target specification, and a source specification.  We created rules to move and change data from source to target, and we discussed controls.  These are the fundamental tasks of mapping.

But this is not the end.  I know that as I worked on this project, I saw things that might improve the map.  I didn’t include them because I wanted to keep this mapping project simple.  But you don’t have to.  You can add your own improvements to this project and make it do what you want.  I would be happy to respond to these if you post them in the comments.

You can find the resoucres for this project in the tool box.