Validate to Standard, not to spec
After you have created your usage specification, it can be useful to use a validation tool to check to verify that new trading partners comply during your boarding process. I have used Foresight’s EDI Analyzer for this many times, and it lets be quickly see where the EDI file departs from the specification.
There is a temptation to use this same validation in the production integration. But this would be a mistake. I’m not saying, “Don’t validate.” I highly encourage validation on both standard compliance and required data validation in mapping and integration. But to use the usage specification has a side effect that I witnessed once. (Only once.)
Validating Transactional EDI to Usage Specification
We had an integration requirement once that required that we only allow EDI elements that were in our usage specification. This was outside of our normal process and we all soon learned why.
When we validated EDI files based on the standard, we would error out documents that were corrupted, or in violation of the full EDi specification. (having mangled segments, or invalid UOMs and such.) But this one time, we thought it would be a good idea to also fail documents that had any extra data that we weren’t going to use. So we pulled all of the segments and elements out so that we were just down to what was in our usage specification.
This worked just like we told it to. As when the next file came in that was in bad form with extra trailing delimiters on the line item, it failed. And then the next with an extra segment that we didn’t care about? Well it failed too. As so on.
We soon were buried under failed EDI files that were valid EDI, and did contain the data we needed, but simply had something extra. And we were calling them all exceptions.
At first we added in a few of the common segments that were being sent. This relieved the lion’s share of the errors. But it still cropped up from time to time. Each time we would have failures for this reason we had to explain why. This grew tedious and embarrassing.
Eventually we re-mapped the doc-type using the complete standard in our validation. Magically all of these errors went away, and so did the embarrassing explanations.
Here is the way the process should work, generalized.
1. EDI envelop is validated against EDI standard.
2. EDI document is validated against EDI standard
3. Mission critical values are validated against Usage Specification.
4. File is processed.
What this does is to establish an inherent hierarchy of exceptions. Transactions that fail on step 1 are all falling because the envelope is corrupt or in error. Also, Transactions that fail on step 2 are failing for a general EDI reason. Both of these steps will have exceptions that can be handled by anyone with EDI knowledge. Operational and Application rules don’t necessarily need to be known to understand how to handle errors at these levels.
Step 3 and 4 are exactly the opposite. Errors that happen because of mission critical data that is missing or in error must be handled by someone that knows what the mission critical form is supposed to look like. And files that pass validation but fail later in processing may need to be corrected and the validation process updated. Or they may point to an error in the application itself. In any case, much more knowledge of the whole process is required at these stages.
Take Away
First, by sharing this experience I hope that everyone can appreciate that sometimes requirements make us to funny things. But I also wanted to share the approach to validation that I have found most helpful. I have seen implementation that don’t do any EDI or Enveloping validation. These are just as dangerous and frustrating as my experience. But now all of the errors are pushed down to step 3 and 4.
I’d also like to invite you to subscribe to my newsletter. It comes out once a month, starting at the end of this month. I’ve got some great ideas for it, and think it will be terrific.
Subscribe to "The Integration Engineer" by Email
Find out about the tools and services available at The Integration Engineer's Consulting site.
