We use a simple file lock that is created with O_CREAT|O_EXCL.
This should fix the concurrency issues with multiple processes
running xbps-rindex -a/-c on the same repository/arch combo.
If xbps_repo_open() was called with the lock arg set, xbps_repo_close()
will now unlock the repo file lock, without the need to set it.
This avoids the need to always unlock the file lock even if it wasn't
locked previously. This also introduceds an ABI/API break, but this
way it's cleaner.
This is magnituds faster than before; some results:
$ for f in 1 2 3; do time xbps-rindex -r /var/cache/xbps/ &>/dev/null; done
real 0m0.624s
user 0m2.163s
sys 0m0.032s
real 0m0.590s
user 0m2.159s
sys 0m0.023s
real 0m0.584s
user 0m2.144s
sys 0m0.039s
$ for f in 1 2 3; do time LD_PRELOAD=$PWD/lib/libxbps.so.2.0.0 ./bin/xbps-rindex/xbps-rindex -r /var/cache/xbps &>/dev/null; done
real 0m0.037s
user 0m0.030s
sys 0m0.010s
real 0m0.036s
user 0m0.032s
sys 0m0.007s
real 0m0.037s
user 0m0.035s
sys 0m0.006s
$
- xbps_repo_open() accepts a third argument (bool) to acquire a POSIX file
lock on the repository archive.
- xbps_repo_close() accepts a second argument (bool) to release a POSIX file
lock on the repository archive.
This avoids the issue of multiple xbps-rindex(8) processes being blocked
even for different repositories on the same architecture, resulting in
unnecessary contention.
- Removed -c --clean mode. It's not too useful and adds a considerable
amount of gratuitous code. It takes almost the same time than
adding all pkgs from scratch.
- When creating the repository data always add the meta plist at the
2nd position in the archive, to optimize its access.
- Misc improvements.
In some tasks the single threaded implementation outperms the multithreaded
one. Use it where it really makes a difference. The _multi() routines do not
spawn any thread if _SC_NPROCESSORS_ONLN == 1.
Bump XBPS_API_VERSION.
This routine will spawn a thread per core to process N items stored
in the specified array, the last thread gets the remainder of items left.
Results have shown that xbps benefits if there is a considerable amount
of items and number of threads being spawned.
Use it in xbps_pkgdb_foreach_cb(), xbps-pkgdb(8), xbps-query(8)
and xbps-rindex(8).
On UP systems there's no overhead because pthread(3) is not used at all.
WIP! investigate if it can be used in libxbps (xbps_rpool_foreach()),
and finish conversion of xbps-rindex(8) -c.
The list of required external deps is now confuse, libarchive and openssl.
libxbps now includes a wrapper for proplib prefixed with xbps_ rather than prop_.
These are the core interfaces in the new API:
rpool - Interface to interact with the repository pool.
rindex - Interface to interact with repository indexes.
pkgdb - Interface to interact with local packages.
transaction - Interface to interact with a transaction.
This also brings new repository index format, making the index file
per architecture and being incompatible with previous versions.
The transaction frequency flush option has been removed, and due to
the nature of package states it was causing more harm than good.
More changes coming soon, but the API shall remain stable from now on.
