Well as time went on, and I became more adept at using windows, I noticed that it tried to tell me what was wrong.  Windows was writing out a crash report.  And it had a message on the blue screen of death.  (a mostly incomprehensible message, but at least it was trying.)  And I discovered that if you looked for the error codes in Microsoft’s knowledge base, you could find the errors had corresponding articles for many of them.  (some of these articles were also incomprehensible.  )

Well when you write an application in perl or any other language you will need to do something other than just crash when you encounter an error.  There are two views on what to do with an error condition; write the message to a log, and write the error out in a separate error message or file.

I guess there could be a third view, and that is to do both.

Logging:

Writing errors to a log, can be just part of the normal logging that you do as your application processes.  If you aren’t logging, you might want to consider it.  There is value in having a log file of what your application is doing.  In order to keep your logging coherant, I create a logging object and then just pass my log messages to it.  This way they can all have the same format with timestamps and such without having to write it out every time.

Error Messages:

My application is processing messages, transforming CSV and other types of files.  So when errors in processing happen, I move the files to an error directory, and I also write a special error file with them so that I don’t have to parse through the log to find it.

Using alerts:

Writing messages to a log and moving files to an error directory are fairly passive ways to handle errors.  A more active way to handle and error is to send an email or trigger an alert of some kind.  You can also use a monitoring technology like nagios to watch for errors, and send alerts.

What are ways that you have handled errors and logging?

What is a Circular File?

This is not like sending the log data to /dev/null or some other black hole.  It is a file that can’t grow beyond a specified size, but never refuses to accept new data.  This is not a paradox.  Visualizing this file like a roll of paper instead of a sheet of paper.  On the roll, there is no end, it just goes around and around.

Say you have a small circular file, 10 MB in our example.  Then lets say, every minute, you add 1 MB of data to your file. For 9 minutes after you make a log entry, you can see this entry in the file.  Each time it moves down on minutes worth of space on the log.  At 10+ minutes, it is gone.  The circular file has rolled over and overwritten that location with more current entry.

Okay, so a 10 minute log may not be that useful, and we would probably want to used a bigger size.  But the point is, that if disaster strikes, the log won’t grow like crazy and create an even bigger problem by filling up the disk.

Why aren’t all logs like this?

Using a circular file for a log is very useful, but if you have an error, and then the log scrolls on for long enough for the error to be lost, all you have are the symptoms not the cause.  So you have saved yourself the problem of making, “The smtp server failed, and the Web tool started logging errors that it couldn’t send emails.  People kept clicking, resend mail until the log filled up the disk, and the app died.” explanations.  But they are replaced with the, “Yes we logged the error, and then the app started logging all of the following actions and the error fell out of the log.”  followed by an explanation of what a circular file is and why this bad news is really good news.

Its best to use more than one logging method.  Say we have a circular log that allows us to see a week of normal activity.  Lets pretend that is 10 GB.  And we have a log that is flat, and only logs critical errors.  Now we have a way to not get clobbered by all the actions that start happening when the rain comes, and we don’t lose that original error that started the mess.

Utopalog

Okay, I am making this up, but here is my dream strategy.  First, we use a circular log that can be size configured on the fly.  Second we use an error database.  We have two table structures; First Occurrence, and Last Occurrence.  Just as they sound, when we have a new error, debug, whatever, it gets logged in the First and Last Occurrence tables.  But the Next time it happens, the time on the Last Occurrence table is the only thing that gets updated.

To use this, we periodically truncate or archive the tables.  The log is self cleaning.  We can do this each night, or on demand, or pick your own schedule.  Now when a problem occurs, we can look for errors that happened before the beginning of the log, but we won’t have a mountain of data that will crush us.