sysklogd/man/klogd.8

.\"                                                              -*- nroff -*-
.\" Copyright 1994-1996  Dr. Greg Wettstein, Enjellic Systems Development.
.\" Copyright 1997-2007  Martin Schulze <joey@infodrom.org>
.\" Copyright 2018-2019  Joachim Nilsson <troglobit@gmail.com>
.\"
.\" May be distributed under the GNU General Public License
.\"
.Dd Oct 30, 2019
.Dt klogd 8
.Os "sysklogd (2.0)"
.Sh NAME
.Nm klogd
.Nd Kernel Log Daemon
.Sh SYNOPSIS
.Nm
.Op Fl 2ndiInopsvx
.Op Fl c Ar NUM
.Op Fl f Ar FILE
.Op Fl k Ar FILE
.Sh DESCRIPTION
.Nm
is a system daemon which intercepts and logs Linux kernel messages.
.Sh OPTIONS
.Bl -tag -width Ds
.It Fl c Ar NUM
Sets the default log level of console messages to
.Ar NUM
(1-8).
.It Fl d
Enable debugging mode.  This will generate potentially LOTS of output to
stderr.
.It Fl f Ar FILE
Log messages to the specified filename rather than to the syslog facility.
.It Fl i | Fl I
Signal the currently executing klogd daemon.  Both of these switches control
the loading/reloading of symbol information.  The
.Fl i
switch signals the daemon to reload the kernel module symbols.  The
.Fl I
switch signals for a reload of both the static kernel symbols and the
kernel module symbols.
.It Fl n
Avoid auto-backgrounding.  This is needed especially if
.Nm
is started and controlled by 
.Xr init 8 .
.It Fl o
Execute in one-shot mode.  This causes
.Nm
to read and log all the messages that are found in the kernel message
buffers.  After a single read and log cycle the daemon exits.
.It Fl p
Enable paranoia mode.  This option controls when klogd loads kernel
module symbol information.  This mode causes klogd to load the kernel
module symbol information whenever a kernel "Oops" string is detected in
the kernel message stream.
.It Fl s
Force
.Nm
to use the system call interface to the kernel message buffers.
.It Fl k Ar FILE
Use the specified file as the source of kernel symbol information.
.It Fl v
Print version and exit.
.It Fl x
Omits EIP translation and therefore doesn't read the System.map file.
.It Fl 2
When symbols are expanded, print the line twice.  Once with addresses
converted to symbols, once with the raw text.  This allows external
programs such as ksymoops do their own processing on the original data.
.Sh OVERVIEW
The functionality of klogd has been typically incorporated into other
versions of syslogd but this seems to be a poor place for it.  In the
modern Linux kernel a number of kernel messaging issues such as
sourcing, prioritization and resolution of kernel addresses must be
addressed.  Incorporating kernel logging into a separate process offers
a cleaner separation of services.
.Pp
In Linux there are two potential sources of kernel log information: the 
.Pa /proc
file system and the syscall,
.Fn sys_syslog
interface, although ultimately
they are one and the same.
.Nm
is designed to choose whichever source of information is the most
appropriate.  It does this by first checking for the presence of a
mounted
.Pa /proc
file system.  If this is found the 
.Pa /proc/kmsg
file is used as the source of kernel log
information.  If the proc file system is not mounted 
.Nm
uses a system call to obtain kernel messages.  The command line switch
.Fl s
can be used to force
.Nm
to use the system call interface as its messaging source.
.Pp
If kernel messages are directed through the 
.Nm syslogd
daemon,
.Nm ,
as of version 1.1, has the ability to properly prioritize kernel
messages.  Prioritization of kernel log messages was added around Linux
version 0.99pl13.  The raw kernel messages are of the form:
.Bd -literal -offset indent
<[0-7]>Something said by the kernel.
.Ed
.Pp
The priority of the kernel message is encoded as a single numeric
digit enclosed inside the
.Ql <>
pair.  The definitions of these values is given in the kernel include
file kernel.h.  When a message is received from the kernel the klogd
daemon reads this priority level and assigns the appropriate priority
level to the syslog message.  If file output,
.Fl f Ar FILE ,
is used the prioritization sequence is left pre-pended to the kernel
message.
.Pp
The klogd daemon can also be used in a 'one-shot' mode for reading the
kernel message buffers.  One shot mode is selected by specifying the
.Fl o
switch on the command line.  Output will be directed to either the
syslog daemon or to an alternate file specified by the
.Fl -f Ar FILE
switch.
.Pp
For example, to read all the kernel messages after a system
boot and record them in a file called krnl.msg the following
command would be given:
.Bd -literal -offset indent
klogd -o -f ./krnl.msg
.Ed
.Sh KERNEL ADDRESS RESOLUTION
If the kernel detects an internal error condition a general protection
fault will be triggered.  As part of the GPF handling procedure the
kernel prints out a status report indicating the state of the
processor at the time of the fault.  Included in this display are the
contents of the microprocessor's registers, the contents of the kernel
stack and a tracing of what functions were being executed at the time
of the fault.
.Pp
This information is
.Sy EXTREMELY IMPORTANT
in determining what caused the internal error condition.  The
difficulty comes when a kernel developer attempts to analyze this
information.  The raw numeric information present in the protection
fault printout is of very little use to the developers.  This is due
to the fact that kernels are not identical and the addresses of
variable locations or functions will not be the same in all kernels.
In order to correctly diagnose the cause of failure a kernel developer
needs to know what specific kernel functions or variable locations
were involved in the error.
.Pp
As part of the kernel compilation process a listing is created which
specified the address locations of important variables and function in
the kernel being compiled.  This listing is saved in a file called
System.map in the top of the kernel directory source tree.  Using this
listing a kernel developer can determine exactly what the kernel was
doing when the error condition occurred.
.Pp
The process of resolving the numeric addresses from the protection
fault printout can be done manually or by using the
.Xr ksymoops 8
program which is included in Linux kernel sources before 1999, or
from this location:
.Lk https://www.kernel.org/pub/linux/utils/kernel/ksymoops
.Pp
As a convenience
.Nm
will attempt to resolve kernel numeric addresses to their symbolic forms
if a kernel symbol table is available at execution time.  If you require
the original address of the symbol, use the
.Fl 2
switch to preserve the numeric address.  A symbol table may be specified
by using the
.Fl k Ar FILE
switch on the command line.  If a symbol file is not explicitly
specified the following filenames will be tried:
.Pp
.Bl -tag -width /usr/src/linux/System.map -compact -offset indent
.It Pa /boot/System.map
.It Pa /System.map
.It Pa /usr/src/linux/System.map
.El
.Pp
Version information is supplied in the system maps as of Linux 1.3.43.
This version information is used to direct an intelligent search of the
list of symbol tables.  This feature is useful since it provides support
for both production and experimental kernels.
.Pp
For example a production kernel may have its map file stored in
.Pa /boot/System.map .
If an experimental or test kernel is compiled with
the sources in the 'standard' location of
.Pa /usr/src/linux
the system
map will be found in
.Pa /usr/src/linux/System.map .
When
.Nm
starts under the experimental kernel the map in
.Pa /boot/System.map
will be bypassed in favor of the map in
.Pa /usr/src/linux/System.map .
.Pp
Modern kernels as of 1.3.43 properly format important kernel addresses
so that they will be recognized and translated by klogd.  Earlier
kernels require a source code patch be applied to the kernel sources.
This patch is supplied with the sysklogd sources.
.Pp
The process of analyzing kernel protections faults works very well with
a static kernel.  Additional difficulties are encountered when
attempting to diagnose errors which occur in loadable kernel modules.
Loadable kernel modules are used to implement kernel functionality in a
form which can be loaded or unloaded at will.  The use of loadable
modules is useful from a debugging standpoint and can also be useful in
decreasing the amount of memory required by a kernel.
.Pp
The difficulty with diagnosing errors in loadable modules is due to the
dynamic nature of the kernel modules.  When a module is loaded the
kernel will allocate memory to hold the module, when the module is
unloaded this memory will be returned back to the kernel.  This dynamic
memory allocation makes it impossible to produce a map file which
details the addresses of the variable and functions in a kernel loadable
module.  Without this location map it is not possible for a kernel
developer to determine what went wrong if a protection fault involves a
kernel module.
.Pp
.Nm
has support for dealing with the problem of diagnosing protection faults
in kernel loadable modules.  At program start time or in response to a
signal the daemon will interrogate the kernel for a listing of all
modules loaded and the addresses in memory they are loaded at.
Individual modules can also register the locations of important
functions when the module is loaded.  The addresses of these exported
symbols are also determined during this interrogation process.
.Pp
When a protection fault occurs an attempt will be made to resolve kernel
addresses from the static symbol table.  If this fails the symbols from
the currently loaded modules are examined in an attempt to resolve the
addresses.  At the very minimum this allows klogd to indicate which
loadable module was responsible for generating the protection fault.
Additional information may be available if the module developer chose to
export symbol information from the module.
.Pp
Proper and accurate resolution of addresses in kernel modules requires
that
.Nm
be informed whenever the kernel module status changes.  The
.Fl i
and
.Fl I
switches can be used to signal the currently executing daemon that
symbol information be reloaded.  Of most importance to proper resolution
of module symbols is the
.Fl i
switch.  Each time a kernel module is loaded or removed from the kernel
the following command should be executed:
.Bd -literal -offset indent
klogd -i
.Ed
.Pp
The
.Fl p
switch can also be used to insure that module symbol information is up
to date.  This switch instructs
.Nm
to reload the module symbol information whenever a protection fault
is detected.  Caution should be used before invoking the program in
'paranoid' mode.  The stability of the kernel and the operating
environment is always under question when a protection fault occurs.
Since the klogd daemon must execute system calls in order to read the
module symbol information there is the possibility that the system may
be too unstable to capture useful information.  A much better policy is
to insure that klogd is updated whenever a module is loaded or unloaded.
Having uptodate symbol information loaded increases the probability of
properly resolving a protection fault if it should occur.
.Pp
Included in the sysklogd source distribution is a patch to the
modules-2.0.0 package which allows the
.Xr insmod 8 ,
.Xr rmmod 8 ,
and
.Xr modprobe 8
utilities to automatically signal
.Nm
whenever a module is inserted or removed from the kernel.  Using this
patch will insure that the symbol information maintained in klogd is
always consistent with the current kernel state.
.Sh CONSOLE LOG LEVEL
The
.Nm
daemon allows the ability to alter the presentation of kernel messages
to the system console.  Consequent with the prioritization of kernel
messages was the inclusion of default messaging levels for the kernel.
In a stock kernel the the default console log level is set to 7.  Any
messages with a priority level numerically lower than 7 (higher
priority) appear on the console.
.Pp
Messages of priority level 7 are considered to be 'debug' messages and
will thus not appear on the console.  Many administrators, particularly
in a multi-user environment, prefer that all kernel messages be handled
by klogd and either directed to a file or to the syslogd daemon.  This
prevents 'nuisance' messages such as line printer out of paper or disk
change detected from cluttering the console.
.Pp
When
.Fl c
is given on the commandline,
.Nm
will execute a system call to inhibit all kernel messages from being
displayed on the console.  Former versions always issued this system
call and defaulted to all kernel messages except for panics.  This is
handled differently currently so
.Nm
doesn't need to set this value anymore.  The
.Ar NUM
argument given to the
.Fl c
switch specifies the priority level of messages which will be directed
to the console.  Note that messages of a priority value LOWER than the
indicated number will be directed to the console.
.Pp
For example, to have the kernel display all messages with a priority
level of 3,
.Ql (KERN_ERR)
or more severe the following command would be executed:
.Bd -literal -offset indent
klogd \-c 4
.Ed
.Pp
The definitions of the numeric values for kernel messages are given in
the file
.Pa kernel.h
which can be found in the
.Pa /usr/include/linux
directory if the kernel sources are installed.  These values parallel
the syslog priority values which are defined in the file
.Pa syslog.h
found in the
.PA /usr/include/sys
sub-directory.
.Pp
The console log level is usually configured with the
.Xr sysctl 8
program, directly or via its configuration file
.Pa /etc/sysctl.conf .
In this file the following line 
.Bd -lilteral -offset indent
kernel.printk = 4 4 1 7
.Ed
.Pp
corresponds to the sampe setting above.
.Sh SIGNALS
.Nm
responds to eight signals:
.Pp
.Bl -tag -width TERM -compact
.It HUP
.It INT
.It KILL
.It TERM
The SIGINT, SIGKILL, SIGTERM and SIGHUP signals cause the daemon to
close its kernel log sources and terminate gracefully.
.It TSTP
.It CONT
The SIGTSTP and SIGCONT signals are used to start and stop kernel
logging.  Upon receipt of SIGTSTP the daemon will close its log sources
and spin in an idle loop.  Subsequent receipt of SIGCONT cause the
daemon to go through its initialization sequence and re-choose an input
source.  Using SIGSTOP and SIGCONT in combination the kernel log input
can be re-chosen without stopping and restarting the daemon.  For
example if the
.PA /proc
file system is to be un-mounted the following command sequence should be
used:
.Bd -literal -offset indent
kill -TSTP pid
umount /proc
kill -CONT pid
.Ed
.Pp
Notations will be made in the system logs with
.Ql LOG_INFO
priority documenting the start/stop of logging.
.It USR1
.It USR2
The SIGUSR1 and SIGUSR2 signals are used to initiate loading/reloading
of kernel symbol information.  Receipt of SIGUSR1 will cause the kernel
module symbols to be reloaded.  Signaling the daemon with SIGUSR2 will
cause both the static kernel symbols and the kernel module symbols to be
reloaded.
.Pp
Provided that the
.Pa System.map
file is placed in an appropriate location the signal of generally
greatest usefulness is SIGUSR1.  It is designed to be used to signal the
daemon when kernel modules are loaded/unloaded.  Sending this signal to
the daemon after a kernel module state change will insure that proper
resolution of symbols will occur if a protection fault occurs in the
address space occupied by a kernel module.
.El
.Pp
.Sh FILES
.Bl -tag -width TERM -compact
.It Pa /proc/kmsg
One source for kernel messages for
.Nm klogd
.It Pa /var/run/klogd.pid
The file containing the process id of 
.Nm
.It Pa /boot/System.map , Pa /System.map , Pa /usr/src/linux/System.map
Default locations for kernel system maps
.Ed
.Sh BUGS
Probably numerous.  Well formed unidiffs and/or GitHub pull
requests appreciated.
.Sh SEE ALSO
.Xr syslogd 8
.Xr syslog 2
.Xr klogctl 2
.Sh AUTHORS
The kernel log daemon
.Nm
was originally written by Steve Lord <lord@cray.com>, Greg Wettstein
made major improvements.  Martin Schulze <joey@infodrom.org> fixed some
bugs and took over maintenance.  Later Joachim Nilsson, @troglobit
picked up maintenance at GitHub.