Console Log Magic
Category Administration IBM/Lotus Domino Linux Open Source
So, if an administrator decides to create a console.log file with "start consolelog" then after the server is restarted, there won't be anymore console recording. To make the console logging stick, there is a Notes.ini configuration: CONSOLE_LOG_ENABLED= (1 is yes and 0 is no).
Accessing the Console Log: Now, why exactly is it that I'm working with a console log? After all, the data could also be found in the miscellaneous view of the log.nsf. That log file is more of a general purpose resource. The log.nsf creates Notes documents, and those are harder to be read and processed outside of a Notes client. It's just easier to have a big ol' text file like a console.log that can be quickly filtered.At my work site the connection protocol for administrating a system varies somewhat from team to team. The Unix admins prefer ftp and telnet, the Linux crowd only installs ssh. I don't have a problem with the Windows servers, as I can always (easily) share a directory and map a drive to them. But, it's a pain to always have to work with the limitations of a telnet session or ssh.
I'd like to be able to map (or mount) a Linux/Unix directory from my local workstation. And, while I'm filling out my wish list, I'd like to be able to process these directories with my own tools. For instance, my own bash scripts on my own schedule, or Python (which has not been loaded onto the production systems).To get the tools I need, I've added Ubuntu 8.04 to my Windows system through Virtualbox: http://www.leadershipbynumbers.com/ms.nsf/d6plinks/BMAA-7EHS9K. In the "seamless" mode it's a very unobtrusive addition.
Now, I can some powerful Linux/Unix utilities which are not available on Windows. Let me start with sshfs:sshfs allows directory mapping from Linux to a ssh server. It's sort of like a secure FTP.Here are the steps:
FTP :I rely on CurlFtpFS which allows me to mount an FTP connection as a directory. Both CurlFtpFS and sshfs rely on fuse so, again, your workstation login identity should be a member of the fuse group, and so should the mount point. It's not necessary to be SU to invoke curlftpfs.Here's what it looks like for an anonymous FTP connection:
$ mkdir sunet
$ curlftpfs ftp://ftp.sunet.se/ sunet/
$ cd sunet/
If you need to authenticate, just add the name a password in the url: as in ftp://user:pass@site.com/. Some people like to use CurlFtpFS to log in the root directory, rather than the home directory of the login identity. The simplest technique for relocating the login directory is to add ../ to the login url (e.g., curlftpfs ftp://ftp.myserver.com/../../.. /mnt/ftp). Kumar Appaiah has a good article on "Fun with Fuse" (http://linuxgazette.net/134/appaiah.html) which you might find useful for learning more about Fuse.
As an aside, I'd like point out that even if I could load up and run CurlFtpFS, fuse, sshfs, Python, etc., from a Windows client, without a line of Linux code involved, I'd still want to use Linux. Not because I'm a Microsoft-hater, but because of the Linux framework that supports package updates. It's really, really nice to get automated updates for the code that I use.
Pausing and Restarting the Console Log and Some Caveats. Great ! Now I can access my console.log and do things like the following: cat console.log | grep "2008/05/20?" | grep Error. I can create a scheduled task to sort the console.log and then write it out in HTML(ish) format into the /data/domino/html directory for public review. I can run Python against the file and do all sorts of reports. It's all good. Except for one little, tiny problem.Domino is still writing to the console.log when I'm trying to access it. I'm going to get caught in a tug-o-war over permission rights.
If it's a short request (e.g., "tail -100 console.log"), I'm probably not going to have a problem, but otherwise I'll find that I've been denied access. If I'm persistent, eventually I'll find that the console.log is suddenly (and, inexplicably) available. No fuss at all. Wonder what I did?Turns out, nothing much, because if a connection is made to the console.log, while Domino is trying to write to it, then Domino will let go. Within a few minutes, Domino will actually revert the CONSOLE_LOG_ENABLED to 0 !! No more active console.log, which seems to defeat the purpose of what I'm trying to do in the first place.
Of course, it's a good thing that Domino is sensitive to my brute force attempts at reading active files. I know better than to work with the notes.ini while the server is running (I've missed that bullet, but many admins have brought their server to a halt by making text edits to notes.ini on a live server). There is a workaround, though.In the beginning of this article, I pointed out the "start consolelog" and "stop consolelog" console commands. Before working on the console.log, run "stop consolelog." You'll need to give it a minute or two, because it's designed to gracefully disconnect, giving a few moments to cache out anything still in RAM, in case you're running it for debugging and the server faults.
With the console.log inactive, you can now fully access it with your Linux client and start processing.
Enabling the console log is a quick tool to capture Domino debugging data that is easily sortable. I'd like to take a couple of minutes to explain some of the more unique aspects of the console.log, and how I've become very fond of my hybrid Win-Linux workstation, which can easily process these logs.
Because my Domino servers run on AIX, Linux and Windows, I need a workstation that supports them all. The simplest solution has been to run two OS on a single workstation through virtualization. After a couple of years of using just about everything that can be run on an x86 system, I've settled into a Windows XP as host and Ubuntu 8.04 as guest with Sun's VirtualBox.
I'm going to explain how I do the following:Because my Domino servers run on AIX, Linux and Windows, I need a workstation that supports them all. The simplest solution has been to run two OS on a single workstation through virtualization. After a couple of years of using just about everything that can be run on an x86 system, I've settled into a Windows XP as host and Ubuntu 8.04 as guest with Sun's VirtualBox.
- Creating the console.log
- Accessing the console.log
- Pausing and restarting the console.log and some caveats
So, if an administrator decides to create a console.log file with "start consolelog" then after the server is restarted, there won't be anymore console recording. To make the console logging stick, there is a Notes.ini configuration: CONSOLE_LOG_ENABLED=
Accessing the Console Log: Now, why exactly is it that I'm working with a console log? After all, the data could also be found in the miscellaneous view of the log.nsf. That log file is more of a general purpose resource. The log.nsf creates Notes documents, and those are harder to be read and processed outside of a Notes client. It's just easier to have a big ol' text file like a console.log that can be quickly filtered.At my work site the connection protocol for administrating a system varies somewhat from team to team. The Unix admins prefer ftp and telnet, the Linux crowd only installs ssh. I don't have a problem with the Windows servers, as I can always (easily) share a directory and map a drive to them. But, it's a pain to always have to work with the limitations of a telnet session or ssh.
I'd like to be able to map (or mount) a Linux/Unix directory from my local workstation. And, while I'm filling out my wish list, I'd like to be able to process these directories with my own tools. For instance, my own bash scripts on my own schedule, or Python (which has not been loaded onto the production systems).To get the tools I need, I've added Ubuntu 8.04 to my Windows system through Virtualbox: http://www.leadershipbynumbers.com/ms.nsf/d6plinks/BMAA-7EHS9K. In the "seamless" mode it's a very unobtrusive addition.
Now, I can some powerful Linux/Unix utilities which are not available on Windows. Let me start with sshfs:sshfs allows directory mapping from Linux to a ssh server. It's sort of like a secure FTP.Here are the steps:
- sshfs is in the Ubuntu universe repository, if it hasn't been installed.
- mkdir /media/*xyz* (where it will be mounted as the remote server directory via ssh)
- sudo chown *your identity* /media/*xyz* (so you can access it)
- sudo adduser *your identity* fuse (this puts your identity with the fuse group, if you are not already in)
- logout and login (to bind fuse to your identity, a logout is required)
- sshfs *serveraccount*@*server.org.com*:/local/notesdata /media/*xyz* (this will mount the server directory as /media/ using the scp protocol)
- fusermount -u /media/*xyz* (this will unmount the directory setting, so that you can run sshfs on another server for mapping a second server's directory to the same /media/*xyz*)
FTP :I rely on CurlFtpFS which allows me to mount an FTP connection as a directory. Both CurlFtpFS and sshfs rely on fuse so, again, your workstation login identity should be a member of the fuse group, and so should the mount point. It's not necessary to be SU to invoke curlftpfs.Here's what it looks like for an anonymous FTP connection:
$ mkdir sunet
$ curlftpfs ftp://ftp.sunet.se/ sunet/
$ cd sunet/
If you need to authenticate, just add the name a password in the url: as in ftp://user:pass@site.com/. Some people like to use CurlFtpFS to log in the root directory, rather than the home directory of the login identity. The simplest technique for relocating the login directory is to add ../ to the login url (e.g., curlftpfs ftp://ftp.myserver.com/../../.. /mnt/ftp). Kumar Appaiah has a good article on "Fun with Fuse" (http://linuxgazette.net/134/appaiah.html) which you might find useful for learning more about Fuse.
As an aside, I'd like point out that even if I could load up and run CurlFtpFS, fuse, sshfs, Python, etc., from a Windows client, without a line of Linux code involved, I'd still want to use Linux. Not because I'm a Microsoft-hater, but because of the Linux framework that supports package updates. It's really, really nice to get automated updates for the code that I use.
Pausing and Restarting the Console Log and Some Caveats. Great ! Now I can access my console.log and do things like the following: cat console.log | grep "2008/05/20?" | grep Error. I can create a scheduled task to sort the console.log and then write it out in HTML(ish) format into the /data/domino/html directory for public review. I can run Python against the file and do all sorts of reports. It's all good. Except for one little, tiny problem.Domino is still writing to the console.log when I'm trying to access it. I'm going to get caught in a tug-o-war over permission rights.
If it's a short request (e.g., "tail -100 console.log"), I'm probably not going to have a problem, but otherwise I'll find that I've been denied access. If I'm persistent, eventually I'll find that the console.log is suddenly (and, inexplicably) available. No fuss at all. Wonder what I did?Turns out, nothing much, because if a connection is made to the console.log, while Domino is trying to write to it, then Domino will let go. Within a few minutes, Domino will actually revert the CONSOLE_LOG_ENABLED to 0 !! No more active console.log, which seems to defeat the purpose of what I'm trying to do in the first place.
Of course, it's a good thing that Domino is sensitive to my brute force attempts at reading active files. I know better than to work with the notes.ini while the server is running (I've missed that bullet, but many admins have brought their server to a halt by making text edits to notes.ini on a live server). There is a workaround, though.In the beginning of this article, I pointed out the "start consolelog" and "stop consolelog" console commands. Before working on the console.log, run "stop consolelog." You'll need to give it a minute or two, because it's designed to gracefully disconnect, giving a few moments to cache out anything still in RAM, in case you're running it for debugging and the server faults.
With the console.log inactive, you can now fully access it with your Linux client and start processing.
- 
