CVS log for src/sys/sys/atomic.h

Up to [cvs.NetBSD.org] / src / sys / sys

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

Move attributes to the front of the __do_atomic_store declarations
(to make it compatible with c++ and other more nitpicky compiker invocations).
Should fix the build for sparc.

sys/atomic.h: Fix atomic_store_* on sparcv7, sparcv8.

These did not cooperate with the hash-locked scheme of the other
atomic operations, with the effect that, for instance, a typical
naive spin lock based on atomic_*,

volatile unsigned locked = 0;
lock()
{
	while (atomic_swap_uint(&locked, 1))
		continue;
	membar_acquire();
}
unlock()
{
	membar_release();
	atomic_store_relaxed(&locked, 0);
}

would fail to achieve mutual exclusion.

For this case, we need to use atomic_swap_* (or, for 8- or 16-bit
objects, atomic_cas_32 loops, since there is no atomic_swap_8 or
atomic_swap_16).

The new machine/types.h macro __HAVE_HASHLOCKED_ATOMICS says whether
these contortions are necessary.

Note that this _requires_ the use of atomic_store_*(p, v), not
regular stores *p = v, to work with the r/m/w atomic operations.

atomic_loadstore(9): Use membar_acquire/release.

Introduce membar_acquire/release.  Deprecate membar_enter/exit.

The names membar_enter/exit were unclear, and the documentation of
membar_enter has disagreed with the implementations on sparc,
powerpc, and even x86(!) for the entire time it has been in NetBSD.

The terms `acquire' and `release' are ubiquitous in the literature
today, and have been adopted in the C and C++ standards to mean
load-before-load/store and load/store-before-store, respectively,
which are exactly the orderings required by acquiring and releasing a
mutex, as well as other useful applications like decrementing a
reference count and then freeing the underlying object if it went to
zero.

Originally I proposed changing one word in the documentation for
membar_enter to make it load-before-load/store instead of
store-before-load/store, i.e., to make it an acquire barrier.  I
proposed this on the grounds that

(a) all implementations guarantee load-before-load/store,
(b) some implementations fail to guarantee store-before-load/store,
and
(c) all uses in-tree assume load-before-load/store.

I verified parts (a) and (b) (except, for (a), powerpc didn't even
guarantee load-before-load/store -- isync isn't necessarily enough;
need lwsync in general -- but it _almost_ did, and it certainly didn't
guarantee store-before-load/store).

Part (c) might not be correct, however: under the mistaken assumption
that atomic-r/m/w then membar-w/rw is equivalent to atomic-r/m/w then
membar-r/rw, I only audited the cases of membar_enter that _aren't_
immediately after an atomic-r/m/w.  All of those cases assume
load-before-load/store.  But my assumption was wrong -- there are
cases of atomic-r/m/w then membar-w/rw that would be broken by
changing to atomic-r/m/w then membar-r/rw:

https://mail-index.netbsd.org/tech-kern/2022/03/29/msg028044.html

Furthermore, the name membar_enter has been adopted in other places
like OpenBSD where it actually does follow the documentation and
guarantee store-before-load/store, even if that order is not useful.
So the name membar_enter currently lives in a bad place where it
means either of two things -- r/rw or w/rw.

With this change, we deprecate membar_enter/exit, introduce
membar_acquire/release as better names for the useful pair (r/rw and
rw/w), and make sure the implementation of membar_enter guarantees
both what was documented _and_ what was implemented, making it an
alias for membar_sync.

While here, rework all of the membar_* definitions and aliases.  The
new logic follows a rule to make it easier to audit:

	membar_X is defined as an alias for membar_Y iff membar_X is
	guaranteed by membar_Y.

The `no stronger than' relation is (the transitive closure of):

- membar_consumer (r/r) is guaranteed by membar_acquire (r/rw)
- membar_producer (w/w) is guaranteed by membar_release (rw/w)
- membar_acquire (r/rw) is guaranteed by membar_sync (rw/rw)
- membar_release (rw/w) is guaranteed by membar_sync (rw/rw)

And, for the deprecated membars:

- membar_enter (whether r/rw, w/rw, or rw/rw) is guaranteed by
  membar_sync (rw/rw)
- membar_exit (rw/w) is guaranteed by membar_release (rw/w)

(membar_exit is identical to membar_release, but the name is
deprecated.)

Finally, while here, annotate some of the instructions with their
semantics.  For powerpc, leave an essay with citations on the
unfortunate but -- as far as I can tell -- necessary decision to use
lwsync, not isync, for membar_acquire and membar_consumer.

Also add membar(3) and atomic(3) man page links.

CTASSERT requires <lib/libkern/libkern.h>.

Mostly merge changes from HEAD upto 20200411

Pull up following revision(s) (requested by riastradh in ticket #508):

	distrib/sets/lists/comp/mi: revision 1.2294
	share/man/man9/Makefile: revision 1.443
	sys/sys/atomic.h: revision 1.18
	share/man/man9/atomic_loadstore.9: revision 1.1
	share/man/man9/atomic_loadstore.9: revision 1.2

New atomic load/store operations for the kernel.

Guarantee no fusing and no tearing, and can optionally impose
ordering relative to other memory operations.

Unordered:
- atomic_load_relaxed
- atomic_store_relaxed

Ordered:
- atomic_load_acquire
- atomic_load_consume
- atomic_store_release

These are intended to match C11 semantics, and can be defined in
terms of the C11 atomic API when ready.

Document relation to atomic_ops(3) and membar_ops(3).

Fix brain fart in alignment criterion.

Rework modified atomic_load/store_* to work on const pointers.

Add KCSAN instrumentation for atomic_{load,store}_*.

New atomic load/store operations for the kernel.

Guarantee no fusing and no tearing, and can optionally impose
ordering relative to other memory operations.

Unordered:
- atomic_load_relaxed
- atomic_store_relaxed

Ordered:
- atomic_load_acquire
- atomic_load_consume
- atomic_store_release

These are intended to match C11 semantics, and can be defined in
terms of the C11 atomic API when ready.

Add support for Kernel Memory Sanitizer (kMSan). It detects uninitialized
memory used by the kernel at run time, and just like kASan and kCSan, it
is an excellent feature. It has already detected 38 uninitialized variables
in the kernel during my testing, which I have since discreetly fixed.

We use two shadows:
 - "shad", to track uninitialized memory with a bit granularity (1:1).
   Each bit set to 1 in the shad corresponds to one uninitialized bit of
   real kernel memory.
 - "orig", to track the origin of the memory with a 4-byte granularity
   (1:1). Each uint32_t cell in the orig indicates the origin of the
   associated uint32_t of real kernel memory.

The memory consumption of these shadows is consequent, so at least 4GB of
RAM is recommended to run kMSan.

The compiler inserts calls to specific __msan_* functions on each memory
access, to manage both the shad and the orig and detect uninitialized
memory accesses that change the execution flow (like an "if" on an
uninitialized variable).

We mark as uninit several types of memory buffers (stack, pools, kmem,
malloc, uvm_km), and check each buffer passed to copyout, copyoutstr,
bwrite, if_transmit_lock and DMA operations, to detect uninitialized memory
that leaves the system. This allows us to detect kernel info leaks in a way
that is more efficient and also more user-friendly than KLEAK.

Contrary to kASan, kMSan requires comprehensive coverage, ie we cannot
tolerate having one non-instrumented function, because this could cause
false positives. kMSan cannot instrument ASM functions, so I converted
most of them to __asm__ inlines, which kMSan is able to instrument. Those
that remain receive special treatment.

Contrary to kASan again, kMSan uses a TLS, so we must context-switch this
TLS during interrupts. We use different contexts depending on the interrupt
level.

The orig tracks precisely the origin of a buffer. We use a special encoding
for the orig values, and pack together in each uint32_t cell of the orig:
 - a code designating the type of memory (Stack, Pool, etc), and
 - a compressed pointer, which points either (1) to a string containing
   the name of the variable associated with the cell, or (2) to an area
   in the kernel .text section which we resolve to a symbol name + offset.

This encoding allows us not to consume extra memory for associating
information with each cell, and produces a precise output, that can tell
for example the name of an uninitialized variable on the stack, the
function in which it was pushed on the stack, and the function where we
accessed this uninitialized variable.

kMSan is available with LLVM, but not with GCC.

The code is organized in a way that is similar to kASan and kCSan, so it
means that other architectures than amd64 can be supported.

Add Kernel Concurrency Sanitizer (kCSan) support. This sanitizer allows us
to detect race conditions at runtime. It is a variation of TSan that is
easy to implement and more suited to kernel internals, albeit theoretically
less precise than TSan's happens-before.

We do basically two things:

 - On every KCSAN_NACCESSES (=2000) memory accesses, we create a cell
   describing the access, and delay the calling CPU (10ms).

 - On all memory accesses, we verify if the memory we're reading/writing
   is referenced in a cell already.

The combination of the two means that, if for example cpu0 does a read that
is selected and cpu1 does a write at the same address, kCSan will fire,
because cpu1's write collides with cpu0's read cell.

The coverage of the instrumentation is the same as that of kASan. Also, the
code is organized in a way similar to kASan, so it is easy to add support
for more architectures than amd64. kCSan is compatible with KCOV.

Reviewed by Kamil.

Be consistent about semicolons in macros: always put them in the macro
definitions and in the macro invocations so that the invocations look
like statements, and we don't and up with extra semis. Fixes lint build.

Add KASAN instrumentation on the atomic functions. Use macros to simplify.
These macros are prerequisites for future changes.

update from HEAD

Sync with HEAD

Pull up following revision(s) (requested by riastradh in ticket #600):
	common/lib/libc/arch/alpha/atomic/membar_ops.S: revision 1.7
	lib/libc/atomic/membar_ops.3: revision 1.4
	sys/arch/alpha/include/types.h: revision 1.50
	sys/sys/atomic.h: revision 1.13
Introduce membar_datadep_consumer.
Discussed briefly on tech-kern without objection:
https://mail-index.netbsd.org/tech-kern/2014/11/20/msg018054.html
https://mail-index.netbsd.org/tech-kern/2015/01/07/msg018326.html

Introduce membar_datadep_consumer.

Discussed briefly on tech-kern without objection:

https://mail-index.netbsd.org/tech-kern/2014/11/20/msg018054.html
https://mail-index.netbsd.org/tech-kern/2015/01/07/msg018326.html

Rebase to HEAD as of a few days ago.

sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs.  ("Protocol error: too many arguments")

sync with head

Declare 16 and 8 bit atomic CAS - while it is not usually sane to expect
these functions to work in MI code, we need them to provide some of the
__sync_* functions gcc and other compilers expect nowadays on some
architectures.

sync to netbsd-5

sync with head

Pull up following revision(s) (requested by dholland in ticket #1314):
	sys/sys/atomic.h: revision 1.11
make atomics usable from c++!

make atomics usable from c++!

Sync with HEAD.

sync with head.

sync with head.

Remove clause 3 and 4 from TNF licenses

sync with HEAD

sync with head.

Sync with HEAD.

Correct a comment.

sync with head.

Add prototypes for atomic_cas_foo_ni().

sync with head

Sync with HEAD

Standalone programs (via libkern) end up pulling this header in, so
check for !_STANDALONE as well before trying to include <stdint.h>

sync with HEAD

Sync with HEAD.

sync with head

Sync with HEAD.

file atomic.h was added on branch matt-armv6 on 2008-01-09 01:58:05 +0000

file atomic.h was added on branch yamt-lazymbuf on 2007-12-07 17:34:52 +0000

Fix another proto.

Fix atomic_inc protos again.

atomic_add_* works on signed integers.

Remove the sub-word atomic ops because they can not be implemented correctly
on some architectures and so are misleading. The same effect can be had by
building on top of the 32-bit CAS. As a side effect this makes it somewhat
easier to complete and document the operations across all platforms.

Sync with HEAD.

Sync with HEAD.

Sync with HEAD

Pull in atomic.h from the thorpej-atomic branch.

Fix a typo.

Add <sys/atomic.h>, defining the API for atomic operations and memory
barriers.

file atomic.h was initially added on branch thorpej-atomic.