Note: this is by far the most important and complex patch of the whole
series, please review it carefully; thank you very much!
For this patch, we decided to keep the original function's design and
skeleton, to avoid regressions and behavior changes, while fixing the
various bugs and overflows. And like the "Harden file2str()" patch, this
patch does not fail when about to overflow, but truncates instead: there
is information available about this process, so return it to the caller;
also, we used INT_MAX as a limit, but a lower limit could be used.
The easy changes:
- Replace sprintf() with snprintf() (and check for truncation).
- Replace "if (n == 0 && rbuf == 0)" with "if (n <= 0 && tot <= 0)" and
do break instead of return: it simplifies the code (only one place to
handle errors), and also guarantees that in the while loop either n or
tot is > 0 (or both), even if n is reset to 0 when about to overflow.
- Remove the "if (n < 0)" block in the while loop: it is (and was) dead
code, since we enter the while loop only if n >= 0.
- Rewrite the missing-null-terminator detection: in the original
function, if the size of the file is a multiple of 2047, a null-
terminator is appended even if the file is already null-terminated.
- Replace "if (n <= 0 && !end_of_file)" with "if (n < 0 || tot <= 0)":
originally, it was equivalent to "if (n < 0)", but we added "tot <= 0"
to handle the first break of the while loop, and to guarantee that in
the rest of the function tot is > 0.
- Double-force ("belt and suspenders") the null-termination of rbuf:
this is (and was) essential to the correctness of the function.
- Replace the final "while" loop with a "for" loop that behaves just
like the preceding "for" loop: in the original function, this would
lead to unexpected results (for example, if rbuf is |\0|A|\0|, this
would return the array {"",NULL} but should return {"","A",NULL}; and
if rbuf is |A|\0|B| (should never happen because rbuf should be null-
terminated), this would make room for two pointers in ret, but would
write three pointers to ret).
The hard changes:
- Prevent the integer overflow of tot in the while loop, but unlike
file2str(), file2strvec() cannot let tot grow until it almost reaches
INT_MAX, because it needs more space for the pointers: this is why we
introduced ARG_LEN, which also guarantees that we can add "align" and
a few sizeof(char*)s to tot without overflowing.
- Prevent the integer overflow of "tot + c + align": when INT_MAX is
(almost) reached, we write the maximal safe amount of pointers to ret
(ARG_LEN guarantees that there is always space for *ret = rbuf and the
NULL terminator).
1/ Replace sprintf() with snprintf() (and check for truncation).
2/ Prevent an integer overflow of ub->siz. The "tot_read--" is needed to
avoid an off-by-one overflow in "ub->buf[tot_read] = '\0'". It is safe
to decrement tot_read here, because we know that tot_read is equal to
ub->siz (and ub->siz is very large).
We believe that truncation is a better option than failure (implementing
failure instead should be as easy as replacing the "tot_read--" with
"tot_read = 0").
1/ Use a "size_t num" instead of an "unsigned num" (also, do not store
the return value of sscanf() into num, it was unused anyway).
2/ Check the return value of strchr() and strrchr().
3/ Never jump over the terminating null byte with "S = tmp + 2".
1/ Prevent an integer overflow of t.
2/ Avoid an infinite loop if s contains characters other than comma,
spaces, +, -, and digits.
3/ Handle all possible return values of snprintf().
1/ Do not read past the terminating null byte when hashing the name.
2/ S[x] is used as an index, but S is "char *S" (signed) and hence may
index the array out-of-bounds. Bit-mask S[x] with 127 (the array has 128
entries).
3/ Use a size_t for j, not an int (strlen() returns a size_t).
Notes:
- These are (mostly) theoretical problems, because the contents of
/proc/PID/status are (mostly) trusted.
- The "name" member of the status_table_struct has 8 bytes, and
"RssShmem" occupies exactly 8 bytes, which means that "name" is not
null-terminated. This is fine right now, because status2proc() uses
memcmp(), not strcmp(), but it is worth mentioning.
This function is unused (SIGNAL_STRING is defined by default, and if it
is not, procps does not compile -- for example, there is no "outbuf" in
help_pr_sig()) but fix it anyway. There are two bugs:
- it accepts non-hexadecimal characters (anything >= 0x30);
- "(c - (c>0x57) ? 0x57 : 0x30)" is always equal to 0x57.
Reference(s):
proc/readproc.c: In function 'statm2proc'
proc/readproc.c:627:9: warning: variable 'num' set but not used [-Wunused-but-set-variable]
ps/output.c: In function 'pr_context':
ps/output.c:1273:14: warning: unused variable 'tried_load' [-Wunused-variable]
ps/output.c:1272:16: warning: unused variable 'ps_is_selinux_enabled' [-Wunused-variable]
ps/output.c:1272:16: warning: 'ps_is_selinux_enabled' defined but not used [-Wunused-variable]
ps/output.c:1273:14: warning: 'tried_load' defined but not used [-Wunused-variable]
ps/output.c:1837:18: warning: 'shortsort_array_count' defined but not used [-Wunused-const-variable=]
ps/output.c:1803:18: warning: 'aix_array_count' defined but not used [-Wunused-const-variable=]
ps/parser.c: In function 'arg_type':
ps/parser.c:1098:3: warning: this 'if' clause does not guard... [-Wmisleading-indentation]
ps/parser.c:1099:34: note: ...this statement, but the latter is misleadingly indented as if it is guarded by the 'if'
ps/sortformat.c: In function 'format_parse':
ps/sortformat.c:241:1: warning: label 'out' defined but not used [-Wunused-label]
ps/stacktrace.c:176:13: warning: 'stack_trace_sigsegv' defined but not used [-Wunused-function]
watch.c: In function 'process_ansi':
watch.c:234:5: warning: this 'if' clause does not guard... [-Wmisleading-indentation]
watch.c:237:2: note: ...this statement, but the latter is misleadingly indented as if it is guarded by the 'if'
Signed-off-by: Jim Warner <james.warner@comcast.net>
On MacOS the system already has user_from_uid and group_from_gid.
These are renamed pwcache_get_user and pwcache_get_group.
For the old library, pwcache_get_user needs to be exported
for skill.
References:
https://gitlab.com/procps-ng/procps/issues/34
Signed-off-by: Craig Small <csmall@enc.com.au>
Since we're about to break the ABI/API anyway, why not
go ahead and add yet another field to our proc_t which
the newlib branch has had for awhile. This then allows
the top program to offer 'control group name' and will
also permit a few reductions in that ps program logic.
And let's also clean up some unrelated warnings below.
Clang warnings:
proc/readproc.c:1178:50: warning: address of array 'ent->d_name' will always evaluate to 'true' [-Wpointer-bool-conversion]
if(unlikely(unlikely(!ent) || unlikely(!ent->d_name))) return 0;
~~~~~~^~~~~~
proc/readproc.c:1205:50: warning: address of array 'ent->d_name' will always evaluate to 'true' [-Wpointer-bool-conversion]
if(unlikely(unlikely(!ent) || unlikely(!ent->d_name))) return 0;
~~~~~~^~~~~~
Signed-off-by: Jim Warner <james.warner@comcast.net>
The former variable length structure created potential
problems for library users like that referenced below.
We will now parallel the same approach newlib uses for
the configure options --enable-oomem & --with-systemd.
Thus, the --enable-oomem and OOMEM_ENABLE #define have
been eliminated and the --with-systemd option (#define
WITH_SYSTEMD) will hereafter impact one function only.
The proc_t struct itself will now *never* be impacted.
Reference(s):
https://gitlab.com/procps-ng/procps/issues/31
Signed-off-by: Jim Warner <james.warner@comcast.net>
Since support already exists in the newlib branch this
represents an equivalent master branch implementation,
and this commit message is shared with 2 more patches.
Beginning with linux-4.5, the following new fields are
being added under that /proc/<pid>/status pseudo file:
. RssAnon - size of resident anonymous memory
. RssFile - size of resident file mappings
. RssShmem - size of resident shared memory
p.s. Locked resident memory support was also added but
isn't directly related to the kernel 4.5 enhancements.
p.p.s. Archlinux, Debian-stretch and Fedora-23 already
are currently using a 4.5 linux kernel (as of 6/2/16).
Signed-off-by: Jim Warner <james.warner@comcast.net>
Multiple scanf()s use the GNU-permitted %Lu. This is not supported in
other libraries and isn't to the POSIX specification. The L modifier
is only used for floats in POSIX.
Replacing %Lu with %llu is the same for GNU libc (scanf(3) says as much)
but means other libraries will work fine.
Closes: #19
References:
http://pubs.opengroup.org/onlinepubs/009695399/functions/fscanf.html
This commit adds a lxc container name to every proc_t.
If a process is not running in a container, then a '-'
will be provided, making such a field always sortable.
Unlike other proc_t character pointers, lxc containers
will find many duplicate shared values. So rather than
strdup 'em (with a later free required upon reuse), we
try to keep track of those already seen and share that
address among all tasks running within each container.
We rely on the lines in the task's cgroup subdirectory
which may initially seem somewhat unsophisticated. But
the lxc library itself uses a similar approach when it
is called to list active containers. In that case, the
/proc/net/unix directory is parsed for the '/lxc' eye-
catcher, with potential complications from hashed path
and names that are too long (something we don't face).
[ too bad docker abandoned lxc - our commit won't do ]
[ anything for the users of those kind of containers ]
Reference(s):
https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1424253https://bugs.launchpad.net/ubuntu/+source/procps/+bug/1424253
Signed-off-by: Jim Warner <james.warner@comcast.net>
Under some circumstances the ksh shell doesn't fork new processes
when executing scripts and the script is interpreted by the
parent process. That makes the execution faster, but it means
ksh needs to reuse the /proc/PID/cmdline for the new script name
and arguments while the file length needs to stay untouched.
The fork is skipped only when the new cmdline is shorter than
the parent's cmdline and the rest of the file is filled
with '\0'. This is perfectly ok until we try to read the cmdline
of such process. As the read_unvectored() function replaces
all zeros with chosen separator, these trailing zeros are replaced
with spaces in case of the ps tool. Consequently it appends
multiple spaces at the end of the arguments string even when these
zeros do not represent any separators and therefore shouldn't
be replaced.
With this commit the read_unvectored() function skips the
replacement of trailing zeros and separates valid content only.
Reference: https://bugzilla.redhat.com/show_bug.cgi?id=1057600
While 'invisible' thread subdirectories are accessible
under /proc/ with stat/opendir calls, they have always
been treated as non-existent, as is true with readdir.
This patch trades the /proc/#/ns access convention for
the more proper /proc/#/task/#/ns approach when thread
access is desired. In addition some namespace code has
been simplified and made slightly more efficient given
the calloc nature of proc_t acquisition and its reuse.
Reference(s):
commit a01ee3c0b3
Signed-off-by: Jim Warner <james.warner@comcast.net>
One recent patch to dynamic buffer management involved
over-allocating the buffer increase to lessen calls to
xrealloc. That was successful, but the actual increase
amount did not attempt to optimize size or alignments.
With this commit, we'll copy an approach recently used
by the top program and round up buffer sizes to 1 KiB.
More importantly, while buffers are quickly reaching a
KiB optimum multiple, no memcpy will ever be employed!
To illustrate just how effective top's algorithm would
be, just change the initial and subsequent allocations
from the current 1024 bytes to just a single byte then
add an fprintf. Those one byte reallocations while on
the way to optimum buffer size will be a one-time cost
and won't represent any recurring performance penalty.
( gosh, that top program *must be* one fart smeller, )
( or was that a smart feller, i can't remember which )
Reference)s):
commit 6d605f521c
commit a45dace4b8
Signed-off-by: Jim Warner <james.warner@comcast.net>
Each process in Linux has a /proc/<pid>/ns directory which contains
symbolic links to pipes that identify which namespaces that process
belongs to. This patch adds support for ps to display that information
optionally.
Signed-off-by: Aristeu Rozanski <arozansk@redhat.com>
When utility buffers were introduced for file2str read
requests, a subtle change was inadvertently introduced
such that a read of zero no longer returns a -1 value.
This commit ensures that zero bytes read returns a -1.
And although the solution differs from a merge request
submitted by sergey.senozhatsky@gmail.com, a thank you
is offered for revealing this potential abend problem.
References(s):
commit a45dace4b8http://gitorious.org/procps/procps/merge_requests/11
Signed-off-by: Jim Warner <james.warner@comcast.net>
Signed-off-by: Craig Small <csmall@enc.com.au>
When dynamic buffers were recently introduced for read
of the status, stat and statm subdirectories one extra
call to read() was required for end-of-file detection.
This patch avoids most all such extra calls to read().
Additionally, the frequency of memory reallocations is
reduced by overallocating each increase more than 25%.
Reference)s):
commit a45dace4b8
Signed-off-by: Jim Warner <james.warner@comcast.net>
Signed-off-by: Craig Small <csmall@enc.com.au>
readproc.c: In function 'stat2proc' :
readproc.c:516: warning: use of assignment suppression and length modifier together in gnu_scanf format
readproc.c:516: warning: use of assignment suppression and length modifier together in gnu_scanf format
Signed-off-by: Gilles Espinasse <g.esp@free.fr>
A recent Debian bug report, dealing with release 3.2.8
and its even more restrictive buffer sizes (1024) used
in stat, statm and status reads via file2str calls, is
a reminder of what could yet happen to procps-ng. Size
needs are determined by kernel evolution and/or config
options so that bug could resurface even though buffer
size is currently 4 times the old procps-3.2.8 limits.
Those sizes were raised from 1024 to 4096 bytes in the
patch submitted by Eric Dumazet, and referenced below.
This patch makes libprocps immune to future changes in
the amount of stuff that is ultimately found in a proc
'stat', 'statm' or 'status' subdirectory. We now trade
the former static buffer of 4096 bytes for dynamically
allocated buffers whose size can be increased by need.
Even though this change is solely an internal one, and
in no way directly affects the API or the ABI, libtool
suggests that the LIBprocps_REVISION be raised. I hope
Craig remembers to do that just before a next release.
We don't want a repeat of the procps-ng-3.3.4 boo-boo,
but with no API/ABI impact that probably can't happen.
p.s. A big thanks to Jaromir Capik <jcapik@redhat.com>
who reviewed my original version and, of course, found
some of my trademark illogic + unnecessary code. After
his coaxing, he helped make this a much better commit.
Reference(s):
. procps-3.2.8
http://bugs.debian.org/702965
. allow large list of groups
commit 7933435584
Signed-off-by: Jim Warner <james.warner@comcast.net>
Reviewed by: Jaromir Capik <jcapik@redhat.com>
Current linux kernels output no more than 32 groups
in /proc/{pid}/status.
Plan is to increase this limit.
This patch allows ps to not core dump if the buffer used to read status
file was too small.
# ps aux
Signal 11 (SEGV) caught by ps (procps-ng version 3.3.3).
ps:display.c:59: please report this bug
Also increases the size of the buffer from 1024 to 4096, since even with
32 groups we are close to the limit.
cat /proc/12731/status | wc
39 128 961
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
In preparation for top scrollable environment display,
the new flag PROC_EDITENVRCVT was added to mirror the
existing single vector string handling for cgroup and
cmdline.
Signed-off-by: Jim Warner <james.warner@comcast.net>
The control group hierarchies for any particular task
could conceivably grow quite large. However, the
library might impose an arbitrary limit of 1024 bytes
via fill_cgroup_cvt.
Two utility buffers of 128 KiB each were already
available for command line use. This commit simply
trades the smaller 1024 byte stack based buffers for
those much larger existing ones. Thus, truncation
can be avoided with no additional run-time costs.
Signed-off-by: Jim Warner <james.warner@comcast.net>
Some inconsistencies have emerged during development
of support for these relatively new proc_t fields.
For example, a PROC_FILLCGROUP flag (via file2strvec)
could return NULL in cgroup whereas PROC_EDITCGRPCVT
(via fill_cgroup_cvt) *almost* guaranteed a return
address (as is true for PROC_EDITCMDLCVT and cmdline).
But even PROC_EDITCGRPCVT could return NULL if the
kernel version was less than 2.6.24. Then with NULL
ps would display a "-" while top would show "n/a".
And while unlikely, with the PROC_FILLSTATUS flag (via
status2proc) a NULL supgid address was theoretically
possible and both ps and top would then show "n/a".
This commit standardizes the following usage:
. PROC_FILLSTATUS (via status2proc)
guarantees a valid supgid address
representing either a true comma
delimited list or "-"
. PROC_FILLCGROUP plus
PROC_EDITCGRPCVT (via fill_cgroup_cvt)
guarantees a cgroup single vector
representing either a true control
group hierarchy or "-"
And as was true before, the following remains true:
PROC_FILLCOM or
PROC_FILLARG (via file2strvec)
may return a NULL cmdline pointer
. PROC_FILLCGROUP (via file2strvec)
may return a NULL cgroup pointer
. PROC_FILLCOM or
PROC_FILLARG plus
PROC_EDITCMDLCVT (via fill_cmdline_cvt)
guarantees a cmdline single vector
representing either a true command
line or a bracketed program name
. PROC_FILLSTATUS plus
PROC_FILLSUPGRP (via supgrps_from_supgids)
guarantees a valid supgrp address
representing either a true comma
delimited list or "-"
Signed-off-by: Jim Warner <james.warner@comcast.net>
Commit a5881b5a4e, addressing
command lines with a trailing space, produced an undesirable
side effect in the fill_cgroup_cvt() function.
This patch restores correct cgroup behavior while still
producing command lines with no trailing space.
When PROC_FILLARG was used (invoking file2strvec)
command lines contained no trailing space.
When PROC_EDITCMDLCVT was used (invoking read_unvectored)
command lines contained a trailing space.
Now both routes to a cmdline act the same -- no trailing space.
The CodingStyle document suggests programs should allow for
cmdlines of at least 128k. Only the ps program can display
such a cmdline, and only with multiple -w switches.
The library function file2strvec can essentially return
a cmdline of unlimited length. However, the library function
fill_cmdline_cvt used an arbitrary upper limit of 2048 for
buffers automatically allocated on the stack.
This patch raises the fill_cmdline_cvt upper limit to 128k via
dymaically acquired utility buffers ensured by the openproc
function.
It also makes indentation consistent in the openproc function.
There were numerous library memory allocation inconsistencies.
Some were checked for failure and others were not.
All library source modules were modified to utilize the alloc.h
memory rouines which are consistent in dealing with errors.
ps
. added tgid support
. made process/task naming consistent
top
. added tgid support
. standardized pid related width logic
documents
. added tgid to ps & top man documents
library
. fixed OOMEM_ENABLE readtask bug
. corrected header typo for tgid
Library changes
readproc
. added readeither to more efficiently
fill both process and thread proc_t
. added readproctab3, uses readeither
. included task path support in:
fill_cgroup_cvt, fill_cmdline_cvt,
read_unvectored
. QUICK_THREADS #define allows copying
process info vs. repeatedly reading
. simple_nexttid no longer values ppid
. path var made non-static in readtask
. documented 'proc_data_t' in .h file
. tweaked some c comments & formatting
library.map
. added new readeither, readproctab3
Program changes
ps
. exploits readproctab3 where possible
. improved args/comm compliance
top
. exploits readeither
Library changes
readproc
. added support for supplementary groups
. eliminated 2 potential mem leak sources
. shortcut used for multi-threaded str
vectors & ptrs was obsoleted
. freeing of proc_t related dynamic
memory now rests with the library
. standardized/normalized many c comments
sysinfo
. corrected note regarding glibc & cpuinfo
library.map
. made the visible freeproc accessable
Program changes
pmap
. initialized buffer for new readproc i/f
. eliminated now obsolete free() call
ps
. added width aware supgrp support
. initialized buffers for new readproc i/f
. eliminated now obsolete free() calls
top
. added supgrp support as variable width
. eliminated now obsolete free() calls
. expoilted library freeproc function
. corrected -h|v args text & spacing
. updated some c comments
Documentation changes
ps.1
. added supgid and supgrp
top.1
. added supgid and supgrp
. addition of above required renumbering
many fields in section 3a. DESCRIPTIONS
This will revert change at Oct 2002 when autotools support was
removed.
Unlike before the package developers are expected to use
./autogen.sh to generate ./configure script, and run make after
that. The build system is also able to create, with make dist, a
tar ball release which compiles correctly, and has files which
seemed to be important to have.
The patch removes few unnecessary files, but no everything. Files
procps.lsm and procps.spec in git repository are useless as is,
but I left them lying around for someone who can make more
justified call about removal of them.
Last, but not least package version number is set 3.3.0 to
distinct this procps from the sourceforge's upstream procps.
Please notice that libproc relese is kept as 3.2.8.
Signed-off-by: Sami Kerola <kerolasa@iki.fi>
Library Changes
. added PROC_EDITCMDLCVT flag
. added an internal (static) fill_cmdline_cvt function:
- reads and "escapes" /proc/#/cmdline
- returns result as a single string in a single vector
- callers are guaranteed a cmdline (no more NULL)
. added vectorize_this_str function, exploited by
fill_cgroup_cvt, fill_cmdline_cvt
. generalized read_cmdline function as read_unvectored, now
exploited by fill_cgroup_cvt, fill_cmdline_cvt, read_cmdline
( cgroup and cmdline no longer need be converted to string )
( vectors before being transformed to final representation )
. fixed bug regarding skipped group numbers (when enabled)
. escape_str made responsible for all single byte translation
with distinction between control chars + other unprintable
. added escaped_copy function for already escaped strings
. reorganized parts of proc_t to restore formatting standards
( displacement changes shouldn't matter with new version # )
. former ZAP_SUSEONLY #define now OOMEM_ENABLE
. added to library.map: escaped_copy; read_cmdline
Top Program Changes
. exploited the new PROC_EDITCMDLCVT provision
. eliminated now obsolete #include "proc/escape.h"
. changed the P_WCH display format if no kernel symbol table
. fixed very old bug in lflgs for out-of-view sort fields
. former ZAP_SUSEONLY #define now OOMEM_ENABLE
Ps Program Changes
. exploited the new PROC_EDITCMDLCVT provision
. exploited the new escaped_copy function
. consolidated pr_args and pr_comm into pr_argcom
Signed-off-by: Jan Görig <jgorig@redhat.com>
This source patchset addresses the following:
Library Extension (readproc)
. added PROC_EDITCGRPCVT flag
. added an internal (static) fill_cgroup_cvt function:
. reads AND parses /proc/#/cgroup
. returns result as a single string in a single vector
. thus no changes to proc_t structure or free memory logic
Program Enhancements (top)
. removed parse_cgroup logic in favor of libproc
. eliminated cgroup sort recurring overhead
. converted WCHAN field to variable width
. generalized variable width field logic in task_show
. real-time notation under P col more distinctive as 'rt'
Program Cosmetic (top)
. CGROUP now known as CGROUPS (plural)
. moved jan's attribution from top.c to 'Notes' in top.h
. numerous comments tweaked
Document Enhancements (top)
. documented CGROUPS field (required renumbering all fields)
. adapted narratives for the 3 current variable width fields
. expanded real-time scheduling notes, 'RT' now shown as 'rt'
Signed-off-by: Jan Görig <jgorig@redhat.com>
