This was and is a bad thing.  And it can be avoided easily by making sure that you use a passive monitoring system.  By its very nature the state of a passive system has no impact on the system being monitored.

File Based System

In a file based system, the pieces that are being monitored create files to report status to the monitor.  The files sit on the file system where the monitor picks them up and processes them.  In this way, if the monitor develops problems, goes down, or gets behind in processing, the production process stays unaffected. (Unless you fill up the disk with files.  But this too can be managed and monitored, passively.)

By designing the status files so that they contain the relevant timestamps and data, it doesn’t even matter in which order they are picked up and processed by the monitor.  And the health of the monitor can be watched by watching the depth and age of files waiting for processing.

Queue Based System

In a queue based system, the process being monitored writes its status to a queue.  This can be a queue that is part of the system, or residing on the database that the product relies on.  It should not be a queue that is on a non-essential system.  Doing so places the queue as a point of failure for monitoring, or worse, for the product.

Similar to the file system, the monitor goes out and reads from the queue.  This allows the same flexibility for the monitor to get behind without impairing production processes.  And the queue depth and age can be monitored in the same way as the file system to check on the health and activity of the monitor.

Missing points of data

This can be frustrating when trying to plug all of the time stamps for the life cycle of your process onto a map, graph, or other representation.  And when there is only a small amount of traffic on your system, losing even one data point can seem important or even critical.  But including some deductive logic into the monitor, these points can be filled in with the data that exists.

This is the same technique I talked about on my post about Sanjay Shaw and RFID in Supply Chain Managment.  When we gather data points from more locations than needed, and one goes missing, we just gloss it over with an estimated point.  And with a passive system we can insert the real point of data later if it shows up.

We had a problem that sometimes we would not get the final status.  Orders were going out, but the monitor was not getting the last status of the system that delivered them.  We were getting the Order responses.  So we tied in the order response status to the order, and when an order that had no transmission time received a status, we filled in the transmitted time with the Order Response transmission time.  This was not entirely accurate, but stopped hundreds of tickets from being created for investigation where the only problem was the status file.

Insurance

To ensure that your passive system is truly passive, the best test is to turn it off, shut it down, and block access to it and any peices of it that can be turned off and still have the production system running.  This test will show you if you have missed or added something that introduces a new point of failure to your environment.  Anything less will not ensure that the monitor is passive.

Convenient Points of Monitoring

Some systems and technology lend themselves to easy monitoring. These points should be monitored and their data included in the aggregation of monitoring data.  These will very from technology to technology, but these point where data is easy to gather, is well documented, and in a friendly format should be collected and aggregated.

For example, it may not be easy or even possible to monitor when an item in a queue becomes stuck.  But it is possible and may even be easy to monitor how many items are in a queue.  By monitoring the easy point of data, we can find evidence that the queue “may” have stuck items because the number begins to grow beyond a threshold that we have determined.

Helpful Points of Monitoring

Some points are helpful in supporting a system. Having a supportable system should be a requirement, so any point where monitoring would be helpful should be monitored. Only the most extreme obstacles should prevent this monitoring from being in place.

If you have chosen good technologies to be working with, or are lucky enough to have inherited good technologies that have convenient points of monitoring in locations that are helpful to your process, then there should be quite a bit of overlap with the previous group of monitoring points.  Where there is no overlap, implementing a passive monitoring option is a good and recommended strategy.

In the example from the convenient points, the number of items in the queue can become helpful in that it alerts us to a condition that may exist.  When this is combined with other factors, like duration, it can become helpful.  Extending the example so that we are not only checking on the size of the queue, but how long it has been at a certain size, we can know even more information and be in an even better position to alert that a “stuck” condition exists.

Necessary Points of Monitoring

Some points are more than helpful, they are absolutely necessary. These are points where there is a mission criticality to the data. It is beyond mere supportability, but the achievement of the data transfer. Any point that does not have any other points that back it up can fall into this category.

Some times these critical points may only be seen when we expand upon the data that we can see.  If we need to know that our invoice files have been delivered to the accounting application.  But the method of transport is so passive that we can’t tell during transport that it was successful or not.  We may have to build an audit monitor into the system.  We may have to have something on the accounting application side, and the delivery side report as to how many invoice documents were sent and how many received.

Then we can compare these numbers to know if something was lost.  We may even be able to build in a document identification into the  audit so that we will even know which files were lost and only resend those.  Being creative and building a monitors that cover the critical areas of data flow are essential, so even if you have to use primitive methods like file counts, when you put these things together you can achieve a robust and stable environment.

Monitoring the Monitors

One can get carried away in this thinking so that we are monitoring the data. And then we monitor the monitoring system, and monitoring the system that monitors the monitors, and so forth. This strategy creates complexity where simplicity is needed. It is also false positive prone, and thus is hard on the support personnel that need to respond to these alerts.

Instead of having this strategy of cascading points of failure. I recommend redundant points of monitoring. Thus monitoring points support each other. If the first point fails, but the next point in the path completes, then there is no need to alert on the monitoring point that failed. Using this strategy we can utilize monitoring methods that have inherent flaws that cause them to fail regularly, but build them into a reliable, robust, fault tolerant system. An example of this is RFID tags for monitoring the movement of materials.

Applying this strategy makes it possible to cover the critical monitoring needs without having to go to extreems of auditing, or expensive systems changes.

Initial Idea for B2T

First, I need to explain the background of the idea.  I know that not everyone on twitter is, er, human.  Yep, we have “people” that are not real flesh and blood on twitter, tweeting away about what they are doing and why you should visit their site, or buy their product.  Shocking I know, but some of them have quite a large following.  (other twitter bots I think)

So then I thought, (again, in context to the airline example that Jason was blogging about), “Hey, what if I could follow a flight on twitter?  Then I could know when it was due to arrive! When it had landed! And where to meet my buddy in baggage claim!”  (Yes, I think with lots of exclamation marks.)  But I still think this is a cool idea.  Each tweet could have a short statement about the flight status and a link to the airline’s page for people that needed to look up more about it.

Okay, then I had my second thought about how to use this.  You see so far I was still thinking, “customer service by managing the twiterverse”  which is fine, I am a customer expecting good service as well.  But then I put this together with another connection.  “What about Order notifications?”

B2T Expanded to Supply Chain

This is where Jason mentions a supplier monitoring their reputation and customer attitudes by following Southwest’s example.  But I thought, “What if Dell, let me follow my order on twitter? I could know when it was shipped!  When it would arrive!  And even know updates like delays, or arriving early!”  (Again, I think with lots of exclamations.)

Have worked on supply chain systems that sent out many email notifications.  Some users liked these, and some didn’t.  Using a twitter model, a user/customer could follow the orders or processes that were important to them on a case by case basis.  BPM could be followed in the same fashion.  And so forth, just by clicking on the follow like on the checkout confirmation page or equivalent.

What do you think?

What are other ways that you can integrate twitter with e-commerce, B2B, or Supply Chains?  I want to know your ideas.  And also, what do we call this?  (“B2T”, “B2Tweet”, “BtweeB”, etc)