Continuing Mapping Exercise

Today we will identify our data source, and begin mapping the source data to the target data.  We identified our target format and placed that in the paper map last time.  If you didn’t read that post yet, you might want to review it quickly before continuing.  (read part One)

The Source

The source is an EDI Invoice.  We will pick an X12 4010 810 as our standard, and I am providing you a link to the Standards document here.  This is a sample standard and does not contain an exhaustive list of elements, just the ones that we are using.   TIE810

We then insert the source elements for the data that we have in this way, starting with the most straight forward data.  I am starting at the bottom and going up.  IT1_04 is the Unit Price.  And it will map directly.  So do ProductID and Quantity.  Quantity from IT1_02 and Product ID from IT1_07 where IT1_06 is “VC” (I will also look on down the line at other odd numbered elements greater than 7 that have a preceding qualifier of “VC”, this allows the execution of the map to be more flexible if the invoice line item is formatted freely.  You may not be able to do this depending on your mapping technology.)

After the easy ones, the direct mapping ones, we get to the concatenated ones.  InvoiceID is the concatenation of the ISA_06, ISA_08, ISA_12, GS_06,  ST_02 and BIG_02.  This is straightforward as well.

Conversion

The status will be pulled from the BIG_09 and we will want to convert this value.  Here is where the fun begins.  We may say, “Only this subset of values is accepted.”  List a set in our usage spec, and fail any that don’t comply.  Or we can try to map all of the invoices status that are in the standard to an internal status.  But this also leaves us with a possible gap if our Trading Partner doesn’t comply with the standard.

Or we can split the difference.  We map status codes that make sense to our system to their corresponding status.  And for all others, we map them to “OTHER” and note what they were into the Notes Field.  Doing this retains the data for someone to look at later.  It prevents failures on this point, and keeps the status from becoming abstracted from the original intent.

Homework time

Go ahead and map the other straight forward data, and document any of the transformation rules that you can.  When you are done you can check out the map that I completed up to this point.  They don’t have to be exactly the same, and only look at the example after you have given a go at doing it yourself.  Next week we will finish the mapping and talk about ways to solve some of the problems faced by the more complex and missing data.

.

The Target

Open Office tables are divided into two tables. Invoice and Invoice Details. This can be mapped in two ways; The first way is to map the data into one common format and rely on whatever ETL tool is importing the data to catch and split the data. Second, a key can be acquired or constructed in the transformation and then the the data can be divided in to matching input formats. Then when these inputs are moved into the database, they will relate to each other on this key.

The choice of how you will do this will depend on your environment. Questions like, “Will I have enough data to provide a unique key?”, or “is there a way to get a key with an API call or database query?” The answers to these questions will determine what course you will take.

If the system ultimately receiving the data is asynchronous to the transformation, and you need to send the invoice and invoice details data separately, some care needs to be taken to ensure that the data can be related after it is separated.

So what data in the invoice can be used to tie the invoice to the invoice details? The first answer might be, “The Invoice Number.” But this number is not guaranteed to be unique across multiple vendors.  In EDI and cXML there are document unique identifiers.  Since we are using EDI, we can use a combination of the ISA Sender, Receiver, and Control number.  We will also want to use the GS control number, and ST control number in the event that more than one invoice is sent in a single envelope.  And we might as well tack on the actual invoice number from the BIG_02.

Database Tables

Invoice Table:	Invoice Details Table:

The Paper Map

Now that we know what the target looks like, we fill out the target side of the paper map.  Since we will create two “files” in our output.  We are creating the Invoice and the InvoiceDetails file, but we can use one paper map for both, and will distinguish this with a bar between the two “files”.  (I am saying files, but this could be a queue, or a post, or an insert over odbc, etc)

What’s Next

Today we went through the process of identifying the target, and creating a paper map with the target format identified.  We talked about some of the strategy that we use in deciding what to map and how to map it.  Next time we will identify the source, and begin mapping data from the source.

Mapping:

I practice and advocate the target oriented mapping approach.  So we will start with defining the target.  We will do this in our 6 column spreadsheet mapping template.  This is also referred to as a Paper Map.  To download a blank template, click here.  Once we have the target defined, we will add the source to the spreadsheet.  After both the source and the target have their fields on the Paper Map, we add the rules.  We will also need to insert some control structures to make everything flow correctly.  Once this is done we will have addressed the data format issues and constructing the map for our translator becomes just a technical exercise.

The Flat File Target:

What are we mapping?  This is the first question that the target defines.  In this example it is an invoice.  So we will want the following information in the following format.

RecordID|VendorID|DateOfInvoice|InvoiceNumber|DateOfPO|PONumber|POLineNumber|QTY|

~UnitCost|LineTotal|InvoiceTotal

Here are the parameters for these values:

RecordID:  A unique alpha-numeric.  It is derived by contacting the ISA_Sender, ISA_Control_Number, GS_Sender, GS_Control_Number, ST_Control_Number, and Line_Number where the line number exists.  Max size of 74 characters.

VendorID:  The ISA_Sender ID.  Alphanumeric with a max size of 15 characters.

DateOfInvoice:  A long date, max 10 character formatted (mmddCCyy)

InvoiceNumber:  Invoice number from the Vendor, Alpha numeric, max 22 characters.

DateOfPO:   A long date, max 10 character formatted (mmddCCyy)

PONumber:  PO number sent on the PO.  Alphanumeric, max 22 characters.

POLineNumber:  PO line number must be the same as the invoice line number.  AlphaNumeric, max 20 characters.

QTY:  Quantity of items invoiced.  Numeric, max 10 characters.

UnitCost: Cost per unit.  Currency, max 17 characters.

LineTotal:  Total of line cost.  Currency, max 17 characters.  sum(QTY*UnitCost)

InvoiceTotal:  Total of all line totals.  Currency, max 17 characters.  sum(all LineTotals for Invoice)

The EDI Source:

We are going to get our data directly from the EDI.  We could be doing API calls to resolve VendorID values or getting an incrementing RecordID, but we can get what we need from the EDI under the right circumstances and so that is what we will do here in this example.

For the record ID we are going to do a simple operation and concatenate several values to create a unique value.  This will almost always be unique.  And it will refer back to the original document as a bonus.  If we wanted to ensure that it was always unique we could include data and timestamps in the concatenation.

For the dates, we are converting the date from the EDI format into our preferred format.  Many systems understand date format conversions, so it might be as simple as a cast or convert command on the existing string of numbers.  If not, we will have to parse out the separate values and then string them back together concatenating in with dashes.

And there are two values that we will need to calculate. We will calculate the value for the invoice total before we create any lines.  And we will need to create the line totals.  Since these require processing the same data, we might create all of the totals and put them in an array to easy access.  Anyway, this is the time to be creative.

Constructing Rules:

Now that we have the source and the target we will makes the rules, and we already have some in mind.  We know that we are concatenating the record ID, reformatting the dates, and calculating the totals.  We also add in some other rules like the trimming of the space on the ISA_06 to Vendor ID operation.  We will also need to do that in the record ID or it will look odd.

Controls:

Controls are like rules, and they may be written in the same place if we need to.  But where rules describe how we completed our mapping, controls tell us when to and not to map something.  Sometimes mapping is conditional.  Controls tell the map what conditions allow for valid mapping when conditions exist.

There may be times that we don’t want the map to go on and complete the process.  Obviously if we have missing data.  If there is no invoice or PO number, you may want to stop with an exception.  Generally these types of controls are easy to build as you make the target field required.  To indicate this you may want to place an additional column to indicate what data is required.  On simple projects you can just put this in the notes.

There will also be other times.  For instance, it might be that some invoices have their own total.  If they do not match our calculated total, we want them to fail mapping and go to manual resolution.  A use case would be to have additional charges attached to the invoice. This extra line charges would need to be handled outside of our map.

Controls can be written to make the mapping very powerful and provide a gateway to your data that ensures it stays valid and accurate.  Some controls go beyond stopping the map.  Instead we use them to put conditional logic into our data mapping project.

Summary:

This example project provided a short exercise to familiarize you with using a paper map and the fundamentals of target mapping.  In this simple example we had a target specification, and a source specification.  We created rules to move and change data from source to target, and we discussed controls.  These are the fundamental tasks of mapping.

But this is not the end.  I know that as I worked on this project, I saw things that might improve the map.  I didn’t include them because I wanted to keep this mapping project simple.  But you don’t have to.  You can add your own improvements to this project and make it do what you want.  I would be happy to respond to these if you post them in the comments.

You can find the resoucres for this project in the tool box.

Theoretical discussion of integration of data and information.

Speaker is Alan Nash

Alan discusses two fundamental problems in information integration:

(1) How to answer a query over a public interface which combines data from several sources and

(2) How to create a single database conforming to the public interface which combines data from several sources.

Alan is using databases as both source and target and uses them in his examples of how data exchange and integration problems are addressed and solved.  This is a little bit heavy, but if you feel up to it, it is 53 minutes long.

This is a still a pretty high level discussion, even the practical examples are abstracted.  I have used a tool that helps do the tasks described here.  If you have this type of task, you might want to check out a company called Convertabase at http://www.convertabase.com

Files are called “Flat Files” when they contain a single data structure.  Generally this structure is the column and row structure like a spreadsheet or table, but a file in binary or encrypted with a single encryption key could also be called a flat file.  Files that are not flat; marked up files like XML or HTML, EDI files, other formats like HL7 or SEF files and others.  Here I am going to briefly discuss two flat file types; Delimited Files, and Fixed Width Files.

What is a Delimited File?

Ok, to describe it briefly, a delimited file is a file where the data is organized in rows and columns.  Each row has a set of data, and each column has a type of data.  If it sounds like I am describing a spreadsheet, you are right on the money.  To make the column, each row has the columns separated with a character called a delimiter.  See the example below.

Tables of data and spreadsheets are both similar to a delimited file in the way they organize data.  In the delimited file all of the empty space, or white space is removed.  What we see here is a classic example of exporting a spreadsheet table as a comma delimited file.  In theory, this data can be imported by any other application that can read a delimited file.

Believe it of not, a space is a character, and takes up space in a file.  Back in the day people went out of their way to save space so that files could be send over slow modem connections.

What is a Fixed Width File?

There is another type of file, is is called a Fixed Width or Fixed Position file.  It is different from a delimited file in that the data fields are defined by the character position.  See the example below.

In a fixed width file, the delimiter characters are eliminated.  If the data is formulated such that the data fields are the same size, this format can be more compact than a delimited file. You can see here that we know the size of the Birthdate data, so we eliminate all the spaces between the Bdate and Department fields.  If all of the data was formatted for size like this, we could really make this file small, so that it only contains the data.

We also eliminate the pesky problem of delimiters found in data.  The issue of a comma delimited file containing a field that has a comma in the data.  How does the parser know that this comma is not really a delimiter, but is part of the data?  Anyway, that problem is eliminated in a fixed width file.

Comparison

This is not a contest of which format is superior.  Both file architectures are useful and both are used commonly enough that you need to be at ease working with both.  Delimited files are really easy to work with as long as your data is clean of the delimiter character.  Doing quick integration of data common in ETL tasks, delimited files are far more common that Fixed Width.  Continuous operations of data integration and importation many times find that Fixed Width or Position files are more reliable for the unattended operation, even ETL if it is unattended.

As with many things in integration work, we want to pick the best option.  Knowing and working with both fixed and delimited files will help you determine which is the right choice for the task you have before you.

I believe that Steven Covey says it this way, “Begin with the end in  mind.”  We do the same thing in data mapping, except that we begin with the end or target if you will.  The data target naturally leads us to the data sources that we need.

If one starts with the source, a lot of work goes into the project, like organizing the source data, before we know if we need any of it.  Sure, you may know that you will need the shipping addresses if you are mapping to a shipping transaction, but spending time gathering the information on the source of this data is still not effective until we know where we will be putting it.

A guide to mapping data from source to target

Spreadsheets are great for mapping.  We start with six columns.  From left to right we have source, source data type, rules, target, target data type, and notes.  We start wit the target and list the data elements that the target needs in the order that the target needs them to approximate the target form.  If this is a database table then this is really simple, and we are just listing the columns names in order.

If the target is something less list like, we will need to add some location information.  In XML we might want to include the xpath to the spreadsheet either in the cell above, or in a logical grouping.  Both approaches work and I will show you and example of each.

As we list the target we will also want to note the data type in the column to the right.  As we note the type of any data with specific formatting, like date or email addresses, we can place that definition here.

After we have the target well defined it is time to decide where this data will come from.  We probably already have some idea, so now it is time to work down the source column and fill in the data reference.  There may be more than one field that maps to a single field on the right.  And there may be one field on the left that maps to multiple fields on the right.  There may also be completely derived data that is calculated at runtime.  There can also be multiple data sources.  You will probably encounter all of these and more in your career as and integration engineer.

Upcomming Articles:

1. EDI mapping excersize.
2. XML mapping excersize.

My Answer:

First, let me just say that my answer was really, “40 hours or 2 calendar weeks.”  When asked if I could do it in 40 hours but in one week, I would say, “No.” and explain that I always had interruptions and other priorities to juggle.  There was never a time that I could only focus on just one issue for a whole week.  Thus it would not make sense to give them the expectation that I could be done in a week, even if the actual hours worked on project was 40 hours.

Second, let me say that these where all mapping change requests, or new document mapping requests.  I am extremely good at this, and have a system for keeping things simple and effective.  You can read more about that by reading some of my mapping posts here.  And following that system, I could pull of a changed or new map in 2 weeks with confidence.

When asked if I was sure that it wouldn’t take longer, I would shrug and say, “It never has before.”  Well not a detailed justification for my time spent, this was an expression of confidence to get my time estimate accepted and put on the plan.

Other Answers:

There are many ways to come up with a time estimate.  I took a class once, that discussed project planning and explained Gantt charts and time lines.  I had been using versions of these for years without knowing they had a name.  I called mine a “white board” but I did learn the lingo of projects that was useful when communicating with other how your time line is looking.

Project Planning is a great skill to have, but don’t get carried away.  Lots of time can be spent on making a project plan instead of actually working.  When you are working as a lone ranger, or small team, extensive planning can actually get in the way.

Team Work:

One of the things that let me get away with the “2 Week” time estimate was that I was the only one doing what I was doing.  Its not that I am anti social, it just that I was the only one left from what was once a team of 6.  I was helping support the product, building new features, and writing my own requirements based on tickets and user/manager responses.  And not having to coordinate with other people made me much much faster.  (but of course you have to know what you are doing to pull this off)

I remember sitting in a meeting with a group of developers.  They had been working on an integration process for 6 months and were still months away from completion.  I remember feeling really sorry for them when I realised that my boss had asked me to produce an integration solution for the same thing they had been working on.  Without knowing their situation, I responded with my 2 week estimate.

Well, they were floored, and didn’t believe I could do it.  I did, and they realised why.  I didn’t have team meetings, (it was just me).  I didn’t follow their coding standards and review processes, (not having to teach a team about what I had just done).  I had a test system that looked just like production that I built on, (they had a dev system, that all of their projects were being built on.  And this meant they were always in each others way.)

In the end, (after I had done it my way) they redeveloped it in their way and the project got completed.

This is not a knock on teams.  Teams are important to have.  But sometimes it is smarter and faster, just to give a guy a direction, and then get out of his way.  Teams help with stability and code longevity, but they can slow the process of creation down.  And the bigger the team, the slower it can go.

Mapping Steps

There are five steps to this mapping process.  Here is a list with a brief description of each one;

• Defining the target:  This is the first step to mapping.  We must know where we are going if we plan on getting there.  And using a Target based mapping process we decrease the time and efort of the other steps.
• Defining the Data Source:  This is the second step to mapping.  Once we know what the target looks like, we know what data we need to complete it.  So this naturally leads us to defining the source(s) of the data.  This can be a single input data file or record, or it can be multiple types of data from multiple sources.
• Defining the Processes:  Some of the data that we will need in our target will not be in the proper form or format in the source.  We will have to have a process defined in our map to convert the source data into the correct form for the target format.
• Handling customisation Points:  When we are mapping data, we will see points where we will want to make a decision on what to do.  Not all data is created or received in an equal form.  Thus we may have points in our mapping process where we will need to do a different process depending on the source or content of the data.
• Monitoring the Process:  After we have created a mapping process we will need to monitor it so that we can know when something unexpected happens.  To do this we must identify the right places in the mapping process for it to report its status to a monitoring process.  This is the last step in the mapping process.

Catalog Target

If you have a catalog, then you will have your own schema.  We are getting our schema from a standard Open Office DB template.  This way you can create your own version for this exercise.

The Product table in the Open Office Business table has these fields:

• ID, INT
• CategoryID, INT
• Discontinued, BOOLEAN
• LeadTime, VARCHAR
• ProductID, INT
• ProductDescription, VARCHAR
• ProductName, VARCHAR
• ReorderLevel, INT
• Serialnumber, VARCHAR
• SupplierID, INT
• UnitPrice, DECIMAL
• UnitsInStock, INT
• UnitsOnOrder, INT

These are the basic fields that we will be inserting data into.  We will probably have a process that will validate the supplier ID before inserting new records, but that will come in during the process step.

Next Steps

Following defining our target, we will define our source.  But we will do that in the next instalment of this exercise.   Download and create start your paper map now, we will show them in the next post as well.

Industries have common ways to handle data.

This is something that we should understand.  Mandating that others use your standard usage implementation may work if you are Wal-Mart, but is not really necessary.  Most supply chain integrations are industry specific.  Probably most EDI integrations even the non-supply chain ones are also industry specific.  And you will find that most of the Trading Partners in a industry need to use the same data sets.  And you will find that most of them are using them in about the same way.

What this means is that when you set up a map for one Trading Partner, most of that mapping will also work for the next one.  This allows us to write maps that are flexible and can handle the variations in the data without having to have a map for each partner, or a crazy set of conditions that say, For Trading Partner 1 do xyz.”

This is of course for an inbound process of in other words, EDI data we receive from others.  And it is simple to accept a par number of other piece of data in more than one location and map that to our canonical.

What is more tricky to visualize is how this works on the outbound.  Well it is possible to map a part number or other piece of data to more than one location, this can be confusing to the trading partners that are receiving it.  Instead, we do a pre map manipulation or pre mapping for trading partners that step out of the normal map.  We could also have a post map process if your technology makes that the better choice.

Here is how that will look. First, we will have a map that is our standard preferred mapping.  This map takes our canonical data and maps it into our outbound EDI format.  Next, we create a process that will either modify the structure or content of our canonical to reflect the custom needs.  We also adjust the map to recognize and handle the new fields. Alternatively we could have a post map that maps our standard outbound EDI and alters data or structure.

Strategies:

1.  Broad mapping:

This is a mapping strategy that I have found works well on mapping data received well from Trading Partners.  One illustration can be found in the discussion of line item data.   Where a specific piece of data may be found in more than one location, you create your map to accept this data in all of the valid locations and out put that into your canonical.

It is also possible to do this on the outbound or map that formats data for consumption by your Trading Partners. This is not as straight forward, but there are times when you can see conditions in the data that cause the format or output to change. When these conditions exist, include them in the mapping and you will have a more agile and flexible map.

2.  Pre mapping:

This is a map before the map.  Well it doesn’t really have to be a whole map.  But the concept is this;  Data received or sent to a Trading Partner needs to be changed on a more Trading Partner specific basis.  It doesn’t actually need to be a TP condition, these pre maps can detect a data format, and alter it before sending it to the map.

For instance, I had a instance where we had a TP that was consistently sending us a DUNS number with the ID of “1″ and we needed “01″ for our system.   So we put together a pre-process that would look for these qualifiers of “1″ and altered it to “01″.  In this way our map didn’t have to have funky code, and we were able to integrate with these trading partners without having to ask them to change their vendor data in their DB that was trimming leading zeroes.  In any case, I have found that pre mapping is useful when there are consistent standards violations or data manipulation that can be corrected systematically.

3.  Post mapping:

This is similar to the pre mapping, but your process happens after the map.  This is a good use for mapping data out to your Trading Partner.  And again, they don’t have to be trading partner specific.  Data going out is altered slightly in form or format before it is delivered.

Again, I had a trading partner that transmitted catalog and part lookups in a format with a dash in the part number.  But they didn’t like getting the dash back on their orders.  So we instituted a simple mapping change that would strip dashes for them.  But later we realized that having a process to fix part number formats was something that more than one Vendor Trading Partner was requesting.  So we moved the mapping condition to a post mapping process that would look at the TP and find any part number rule and apply it.

Flexible and Agile is just better.

Okay, so that is a broad generalization.  But I can back it up with years of experience.  I never regret mapping broadly and implementing pre and post processes.  But I do regret when I have mapped narrowly, tried to have a single trading partner specific map, or include all types of TP specific logic into a map.

Finish The Paper Map

Today we will finish the three part series mapping exercise.  If you haven’t gotten a chance to read the first two posts in this exercise you may want to start with them, or to review them before moving on to the conclusion.  (read part One or read part Two)

In last post, we began mapping an EDI Invoice to invoice tables in an Open Office Database.  I chose these as Open Office is free and anyone can get a copy to repeat this exercise, and because Invoices are common documents that need to be handled in an e-commerce and supply chain scenario.

Compete Mapping the Invoice Header

At this point we have mapped all of the direct mapping, the data that just moves from one location in the EDI, to another in the Database.  We have also done some simple operations such as concatenation to form unique IDs for our invoice records using data in the invoice to preserve referential integrity to the source document.  And we have done some conversions for invoice Status.  If we though about it, we also indicated that dates were mapped and converted.

Sometimes we will have large conversions, or specific conditions that we want to key a conditional conversion on.  We can either code these into the maps, or we can use some API call that the technology we are using provides us.  In our example we are going to code them into the map.  But you may want to do this in an API or another process in your environment.

Status Conversion

We ask in our specification that all BIG_07 be the value “BS”  as “Bill Of Sale”  But we don’t want to error out on the invoice if there is any other value.  So we export and code a conversion condition into the map.  If it is BS, we map BOS to the Status Field, but if it is any other value, we attempt to look that value up in our list of codes, and format a note to be inserted into notes with this text;  “Invoice Status of __ as [converted value] please take appropriate action.”  And if the Status Code is not in our table we insert, “Invoice Status of __ was unknown.” into the notes.

This way our mapping of this converted value does not rely on the validity of the code, but handles any value presented.  And this keeps us from having the people who handle EDI translation exceptions from having to deal with overrides that belong to AP.

Location Codes or Text

Depending on the way you do business, you may only be able to handle account codes for shipping and billing.  Or you may want the full billing address for each Invoice in the database.  For us, we will do a little of both.  When we encounter the N1 loop we will look first at the N1_04.  We will match this with an API call to another application to validate that it is a known Billing Account.  If it is, we insert the account code into the billing field, and can skip the rest of the data in the N1.  If you are unfamiliar with EDI loops, you might want to take a look at this overview of repeating EDI segments.

But if it is new, or not in the system, we will need to have someone in AP verify that we want to remit payment to this location.  So we process the N1 lines, and either insert them into the Billing Field, or insert them as a note on the invoice.  We will probably need a Billing Account Code that indicates that user action is required if this happens.  We are not going to handle this in our mapping.  We will let the remittance application handle this process.  There are as many ways to handle these situation as there are companies.  But in your environment the capabilities of your systems will indicate how to make the most robust integration.

Multiple Mapping Location Options

Some times there will be more than one place where we can get a piece of data.  Sometimes this will like creating a total by multiplying all of the line totals.  But other times this might be that the Product ID could be in any of the IT1 odd numbered elements from 7-25  and we need to find the one that is qualified with “VC”  found on the respective IT1 even numbered elements 6-24.  And sometimes we will have a billing term, that might be found in the ITD that is for the whole invoice, but might also  be found on a line by line location.  (or even more complex, both, with one superseding the other location when found.)  Luckily, we have examples in our mapping project for you to become familiar with.

Double Checking Your Paper Map

Walk through this paper map again and see what details have changed along the way.  Try to complete the mapping on your own, and then you can take a look at the map that I completed.  Because we haven’t actually coded a map to convert the transaction, there will be a few things that we will have over looked.  There is  nothing wrong with that.  We will fill those in when we write the map that we will execute.  This is the final step in making a paper map, putting it to use.

So here is the  paper map that I completed.  I will be creating a conversion for this process and processing some test documents.  When that happens, I wil post the transformer, and the sample DB and files.  You will be able to look at how that works and tinker with the code to see if your mapping works the same.  Until then ask me any questions about the mapping process in the comments section of this post.