Passive Monitor
I remember a day when we had a problem with our order processing. When we finally figured out what was wrong, there were a few embarrassed people. We had introduced some new monitoring to the system, but had done so in such a way that we added a point of failure to the ordering process.
This was and is a bad thing. And it can be avoided easily by making sure that you use a passive monitoring system. By its very nature the state of a passive system has no impact on the system being monitored.
File Based System
In a file based system, the pieces that are being monitored create files to report status to the monitor. The files sit on the file system where the monitor picks them up and processes them. In this way, if the monitor develops problems, goes down, or gets behind in processing, the production process stays unaffected. (Unless you fill up the disk with files. But this too can be managed and monitored, passively.)
By designing the status files so that they contain the relevant timestamps and data, it doesn’t even matter in which order they are picked up and processed by the monitor. And the health of the monitor can be watched by watching the depth and age of files waiting for processing.
Queue Based System
In a queue based system, the process being monitored writes its status to a queue. This can be a queue that is part of the system, or residing on the database that the product relies on. It should not be a queue that is on a non-essential system. Doing so places the queue as a point of failure for monitoring, or worse, for the product.
Similar to the file system, the monitor goes out and reads from the queue. This allows the same flexibility for the monitor to get behind without impairing production processes. And the queue depth and age can be monitored in the same way as the file system to check on the health and activity of the monitor.
Missing points of data
This can be frustrating when trying to plug all of the time stamps for the life cycle of your process onto a map, graph, or other representation. And when there is only a small amount of traffic on your system, losing even one data point can seem important or even critical. But including some deductive logic into the monitor, these points can be filled in with the data that exists.
This is the same technique I talked about on my post about Sanjay Shaw and RFID in Supply Chain Managment. When we gather data points from more locations than needed, and one goes missing, we just gloss it over with an estimated point. And with a passive system we can insert the real point of data later if it shows up.
We had a problem that sometimes we would not get the final status. Orders were going out, but the monitor was not getting the last status of the system that delivered them. We were getting the Order responses. So we tied in the order response status to the order, and when an order that had no transmission time received a status, we filled in the transmitted time with the Order Response transmission time. This was not entirely accurate, but stopped hundreds of tickets from being created for investigation where the only problem was the status file.
Insurance
To ensure that your passive system is truly passive, the best test is to turn it off, shut it down, and block access to it and any peices of it that can be turned off and still have the production system running. This test will show you if you have missed or added something that introduces a new point of failure to your environment. Anything less will not ensure that the monitor is passive.
Subscribe to "The Integration Engineer" by Email
Find out about the tools and services available at The Integration Engineer's Consulting site.
April 19th, 2010 at 8:29 am
[...] I recommend is that errors not only be tracked to ensure someone is alerted and they are resolved. But errors instances are counted and reported [...]