Leadership By Numbers

Here's how I got compacting under control, where it had been running 16 hours a day and trying to compact over 3,800 files. Now, compacting is rarely run more than on a hundred databases a night--and everything works great.

Several weeks after I started my new job, I was looking over the server consoles and surprised to see the compactor task running. Compacting was scheduled to begin at midnight, and only for a set of sub-directories. I realized that compacting was taking nearly 16 hours to complete, even for the small set of directories listed in the program document. After some quick analysis, the culprit was obvious--too many files were being compacted during the system backup. It was not a good setup, because each night only a subset of directories were benefiting from compacting. While one set of directories were getting compacting, all the others were expanding. The net result being that compacting was not effective, until during the weekend when all the databases could be compacted over a 24 hour period.

The backups are taking 12 hours because my company has not yet implemented transaction log-enabled backups. As soon as I can, I will work on getting the transaction log backups to function correctly. In the meantime, I decided to look at the compacting.

What I found, was that most of the time the compactor was churning a lot of disk I/O and eating up a lot of CPU cycles and producing “0” benefits. The program files were configured to look for databases with 15% or more free space. The problem is that with smaller email files (e.g., <50 M) there is always going to be a certain amount of free space that cannot be squeezed out. Out of 100 files with 15% free space, only a handful of them recover any significant space. If you check your own logs, you'll see that the space recovered on many smaller mail files is “0.”

Here's the dilemma: the servers are short on space and have to be compacted as much as they can, but there are too many files to check in the time allotted. There is no way that I can compact 3,800 databases in an evening, while backups are running.

After considering some choices, I decided that compacting needs an additional flag: the database size. A 500 M mail file that is 80% in use is worth the time to compact it. Conversely, a 50 M file that is 85% in use just isn't going to give back any space and I'd like to tell the compactor to ignore it. What was needed was a list, generated nightly, which could tell the compactor which databases to compact.

Here's what I did:

1. Scheduled the catalog task to run every night. It reads in the size and freespace of each database.

2. Created an agent to read the catalog and write an text file of all databases that meet a criteria of (1) being large enough and (2) having enough freespace. I store the criteria defaults in a profile document so they are easy to change and access.

3. When the agent runs, it reads the catalog, writes out an indirect text file on the Domino server and records the list of databases it selected in a document.

4. Finally, a program document is run that is scheduled to load up the compact task and read in the indirect text file. In this manner, the compact task will only work on the databases that most need it. I use another Administration Secret to configure my program document.

Special thanks to Julian Robichaux for his LotusScript to HTML converter, and while I didn't mention his OpenLog application in my code, you should know it's in there in my use.