The NetBSD Project

CVS log for src/common/lib/libc/arch/aarch64/atomic/membar_ops.S

[BACK] Up to [cvs.NetBSD.org] / src / common / lib / libc / arch / aarch64 / atomic

Request diff between arbitrary revisions


Default branch: MAIN


Revision 1.4 / (download) - annotate - [select for diffs], Sat Apr 9 23:32:51 2022 UTC (9 months, 4 weeks ago) by riastradh
Branch: MAIN
CVS Tags: netbsd-10-base, netbsd-10, HEAD
Changes since 1.3: +10 -6 lines
Diff to previous 1.3 (colored)

Introduce membar_acquire/release.  Deprecate membar_enter/exit.

The names membar_enter/exit were unclear, and the documentation of
membar_enter has disagreed with the implementations on sparc,
powerpc, and even x86(!) for the entire time it has been in NetBSD.

The terms `acquire' and `release' are ubiquitous in the literature
today, and have been adopted in the C and C++ standards to mean
load-before-load/store and load/store-before-store, respectively,
which are exactly the orderings required by acquiring and releasing a
mutex, as well as other useful applications like decrementing a
reference count and then freeing the underlying object if it went to
zero.

Originally I proposed changing one word in the documentation for
membar_enter to make it load-before-load/store instead of
store-before-load/store, i.e., to make it an acquire barrier.  I
proposed this on the grounds that

(a) all implementations guarantee load-before-load/store,
(b) some implementations fail to guarantee store-before-load/store,
and
(c) all uses in-tree assume load-before-load/store.

I verified parts (a) and (b) (except, for (a), powerpc didn't even
guarantee load-before-load/store -- isync isn't necessarily enough;
need lwsync in general -- but it _almost_ did, and it certainly didn't
guarantee store-before-load/store).

Part (c) might not be correct, however: under the mistaken assumption
that atomic-r/m/w then membar-w/rw is equivalent to atomic-r/m/w then
membar-r/rw, I only audited the cases of membar_enter that _aren't_
immediately after an atomic-r/m/w.  All of those cases assume
load-before-load/store.  But my assumption was wrong -- there are
cases of atomic-r/m/w then membar-w/rw that would be broken by
changing to atomic-r/m/w then membar-r/rw:

https://mail-index.netbsd.org/tech-kern/2022/03/29/msg028044.html

Furthermore, the name membar_enter has been adopted in other places
like OpenBSD where it actually does follow the documentation and
guarantee store-before-load/store, even if that order is not useful.
So the name membar_enter currently lives in a bad place where it
means either of two things -- r/rw or w/rw.

With this change, we deprecate membar_enter/exit, introduce
membar_acquire/release as better names for the useful pair (r/rw and
rw/w), and make sure the implementation of membar_enter guarantees
both what was documented _and_ what was implemented, making it an
alias for membar_sync.

While here, rework all of the membar_* definitions and aliases.  The
new logic follows a rule to make it easier to audit:

	membar_X is defined as an alias for membar_Y iff membar_X is
	guaranteed by membar_Y.

The `no stronger than' relation is (the transitive closure of):

- membar_consumer (r/r) is guaranteed by membar_acquire (r/rw)
- membar_producer (w/w) is guaranteed by membar_release (rw/w)
- membar_acquire (r/rw) is guaranteed by membar_sync (rw/rw)
- membar_release (rw/w) is guaranteed by membar_sync (rw/rw)

And, for the deprecated membars:

- membar_enter (whether r/rw, w/rw, or rw/rw) is guaranteed by
  membar_sync (rw/rw)
- membar_exit (rw/w) is guaranteed by membar_release (rw/w)

(membar_exit is identical to membar_release, but the name is
deprecated.)

Finally, while here, annotate some of the instructions with their
semantics.  For powerpc, leave an essay with citations on the
unfortunate but -- as far as I can tell -- necessary decision to use
lwsync, not isync, for membar_acquire and membar_consumer.

Also add membar(3) and atomic(3) man page links.

Revision 1.3 / (download) - annotate - [select for diffs], Sat Apr 9 12:07:37 2022 UTC (9 months, 4 weeks ago) by riastradh
Branch: MAIN
Changes since 1.2: +2 -2 lines
Diff to previous 1.2 (colored)

aarch64/membar_ops: Fix wrong symbol end.

Revision 1.1.28.1 / (download) - annotate - [select for diffs], Wed Aug 11 17:19:01 2021 UTC (17 months, 3 weeks ago) by martin
Branch: netbsd-9
CVS Tags: netbsd-9-3-RELEASE
Changes since 1.1: +11 -7 lines
Diff to previous 1.1 (colored) next main 1.2 (colored)

Pull up following revision(s) (requested by skrll in ticket #1331):

	common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
	common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
	common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
	common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
	common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
	common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
	common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
	common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
	common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
	common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
	common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
	common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
	common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
	common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
	common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
	common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
	common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
	common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops.  I don't understand why thos=
e
   are there.  Is it some architectural thing, or for a CPU bug, or just
   over-caution maybe?  They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
   predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

 From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
    This built-in function is not a full barrier, but rather an acquire
    barrier. This means that references after the operation cannot move to
    (or be speculated to) before the operation, but previous memory stores
    may not be globally visible yet, and previous memory loads may not yet
    be satisfied.

void __sync_lock_release (type *ptr, ...)
    This built-in function is not a full barrier, but rather a release
    barrier. This means that all previous memory stores are globally
    visible, and all previous memory loads have been satisfied, but
    following memory reads are not prevented from being speculated to
    before the barrier.

Revision 1.2 / (download) - annotate - [select for diffs], Tue Oct 13 21:22:12 2020 UTC (2 years, 3 months ago) by skrll
Branch: MAIN
CVS Tags: cjep_sun2x-base1, cjep_sun2x-base, cjep_sun2x, cjep_staticlib_x-base1, cjep_staticlib_x-base, cjep_staticlib_x
Changes since 1.1: +11 -7 lines
Diff to previous 1.1 (colored)

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

Revision 1.1.26.2 / (download) - annotate - [select for diffs], Tue Apr 21 19:37:41 2020 UTC (2 years, 9 months ago) by martin
Branch: phil-wifi
Changes since 1.1.26.1: +0 -0 lines
Diff to previous 1.1.26.1 (colored) to branchpoint 1.1 (colored) next main 1.2 (colored)

Ooops, restore accidently removed files from merge mishap

Revision 1.1.26.1, Tue Apr 21 18:41:14 2020 UTC (2 years, 9 months ago) by martin
Branch: phil-wifi
Changes since 1.1: +1 -1 lines
FILE REMOVED

Sync with HEAD

Revision 1.1.4.2 / (download) - annotate - [select for diffs], Tue Aug 19 23:45:12 2014 UTC (8 years, 5 months ago) by tls
Branch: tls-maxphys
Changes since 1.1.4.1: +55 -0 lines
Diff to previous 1.1.4.1 (colored) to branchpoint 1.1 (colored) next main 1.2 (colored)

Rebase to HEAD as of a few days ago.

Revision 1.1.4.1, Sun Aug 10 05:47:35 2014 UTC (8 years, 5 months ago) by tls
Branch: tls-maxphys
Changes since 1.1: +0 -55 lines
FILE REMOVED

file membar_ops.S was added on branch tls-maxphys on 2014-08-19 23:45:12 +0000

Revision 1.1 / (download) - annotate - [select for diffs], Sun Aug 10 05:47:35 2014 UTC (8 years, 5 months ago) by matt
Branch: MAIN
CVS Tags: tls-maxphys-base, tls-maxphys-20171202, prg-localcount2-base3, prg-localcount2-base2, prg-localcount2-base1, prg-localcount2-base, prg-localcount2, phil-wifi-base, phil-wifi-20200421, phil-wifi-20200411, phil-wifi-20200406, phil-wifi-20191119, phil-wifi-20190609, pgoyette-localcount-base, pgoyette-localcount-20170426, pgoyette-localcount-20170320, pgoyette-localcount-20170107, pgoyette-localcount-20161104, pgoyette-localcount-20160806, pgoyette-localcount-20160726, pgoyette-localcount, pgoyette-compat-merge-20190127, pgoyette-compat-base, pgoyette-compat-20190127, pgoyette-compat-20190118, pgoyette-compat-1226, pgoyette-compat-1126, pgoyette-compat-1020, pgoyette-compat-0930, pgoyette-compat-0906, pgoyette-compat-0728, pgoyette-compat-0625, pgoyette-compat-0521, pgoyette-compat-0502, pgoyette-compat-0422, pgoyette-compat-0415, pgoyette-compat-0407, pgoyette-compat-0330, pgoyette-compat-0322, pgoyette-compat-0315, pgoyette-compat, perseant-stdc-iso10646-base, perseant-stdc-iso10646, netbsd-9-base, netbsd-9-2-RELEASE, netbsd-9-1-RELEASE, netbsd-9-0-RELEASE, netbsd-9-0-RC2, netbsd-9-0-RC1, netbsd-8-base, netbsd-8-2-RELEASE, netbsd-8-1-RELEASE, netbsd-8-1-RC1, netbsd-8-0-RELEASE, netbsd-8-0-RC2, netbsd-8-0-RC1, netbsd-8, netbsd-7-nhusb-base-20170116, netbsd-7-nhusb-base, netbsd-7-nhusb, netbsd-7-base, netbsd-7-2-RELEASE, netbsd-7-1-RELEASE, netbsd-7-1-RC2, netbsd-7-1-RC1, netbsd-7-1-2-RELEASE, netbsd-7-1-1-RELEASE, netbsd-7-1, netbsd-7-0-RELEASE, netbsd-7-0-RC3, netbsd-7-0-RC2, netbsd-7-0-RC1, netbsd-7-0-2-RELEASE, netbsd-7-0-1-RELEASE, netbsd-7-0, netbsd-7, matt-nb8-mediatek-base, matt-nb8-mediatek, localcount-20160914, is-mlppp-base, is-mlppp, bouyer-xenpvh-base2, bouyer-xenpvh-base1, bouyer-xenpvh-base, bouyer-xenpvh, bouyer-socketcan-base1, bouyer-socketcan-base, bouyer-socketcan, ad-namecache-base3, ad-namecache-base, ad-namecache
Branch point for: tls-maxphys, phil-wifi, netbsd-9

Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

This form allows you to request diff's between any two revisions of a file. You may select a symbolic revision name using the selection box or you may type in a numeric name using the type-in text box.




CVSweb <webmaster@jp.NetBSD.org>