Circular Files
A circular file is not a nickname for the waste can. Circular files, sometimes called round files, are useful in some applications and support tasks. With a normal log file or repository, the log grows as logged events are added to the log. The obvious danger is that if the space where the log is located becomes saturated when the log grows to fill this space. Many applications will shut down and refuse to restart if this happens. For some applications, having the log write to a circular file is the answer.
What is a Circular File?
This is not like sending the log data to /dev/null or some other black hole. It is a file that can’t grow beyond a specified size, but never refuses to accept new data. This is not a paradox. Visualizing this file like a roll of paper instead of a sheet of paper. On the roll, there is no end, it just goes around and around.
Say you have a small circular file, 10 MB in our example. Then lets say, every minute, you add 1 MB of data to your file. For 9 minutes after you make a log entry, you can see this entry in the file. Each time it moves down on minutes worth of space on the log. At 10+ minutes, it is gone. The circular file has rolled over and overwritten that location with more current entry.
Okay, so a 10 minute log may not be that useful, and we would probably want to used a bigger size. But the point is, that if disaster strikes, the log won’t grow like crazy and create an even bigger problem by filling up the disk.
Why aren’t all logs like this?
Using a circular file for a log is very useful, but if you have an error, and then the log scrolls on for long enough for the error to be lost, all you have are the symptoms not the cause. So you have saved yourself the problem of making, “The smtp server failed, and the Web tool started logging errors that it couldn’t send emails. People kept clicking, resend mail until the log filled up the disk, and the app died.” explanations. But they are replaced with the, “Yes we logged the error, and then the app started logging all of the following actions and the error fell out of the log.” followed by an explanation of what a circular file is and why this bad news is really good news.
Its best to use more than one logging method. Say we have a circular log that allows us to see a week of normal activity. Lets pretend that is 10 GB. And we have a log that is flat, and only logs critical errors. Now we have a way to not get clobbered by all the actions that start happening when the rain comes, and we don’t lose that original error that started the mess.
Utopalog
Okay, I am making this up, but here is my dream strategy. First, we use a circular log that can be size configured on the fly. Second we use an error database. We have two table structures; First Occurrence, and Last Occurrence. Just as they sound, when we have a new error, debug, whatever, it gets logged in the First and Last Occurrence tables. But the Next time it happens, the time on the Last Occurrence table is the only thing that gets updated.
To use this, we periodically truncate or archive the tables. The log is self cleaning. We can do this each night, or on demand, or pick your own schedule. Now when a problem occurs, we can look for errors that happened before the beginning of the log, but we won’t have a mountain of data that will crush us.
Subscribe to "The Integration Engineer" by Email
Find out about the tools and services available at The Integration Engineer's Consulting site.
September 21st, 2010 at 8:30 am
[...] This post was Twitted by RoyHayward [...]
September 23rd, 2010 at 6:20 am
Roy, great explanation of what a circular file is and how it works. Do you have some links or info on how to create them and quirks for different OS’es?
September 23rd, 2010 at 8:41 am
Paul,
Nice to hear from you. Circular files or circular logs are less of an OS specific thing are are more an application specific thing.
For instance, MS SQL Server has their transaction log, which can be a circular log, that only holds a specific size and can then delete old transactions past a certain size.
If I recall correctly, exchange server does it a bit differently. It uses a round-robin approach where the log is actually two files. The first file, when it reaches its size limit, moves. The new file then begins to grow until it reaches the size limit. Then new file replace the first file and a new file is creates. This process repeats indefinitely keeping the log from growing beyond the collective size limit. When the log is viewed or exported, a concatenation of the two is what the user sees, looking like one log.
This is a great question. And this reply is getting really long. I think I will have to create a couple of new articles to explore how to create and include circular files and logs in applications.
If you have any examples or ideas, please let me know.
Thanks
Roy