- Separate the failure cases into different assertions: end/default constructed vs rehashed or erased.
- Update the assertion error for AssertIsValid to not mention the end iterator case because end iterators are considered valid by AssertIsValid.
- Also fix an out-of-date comment for skip_empty_or_deleted.
PiperOrigin-RevId: 485402559
Change-Id: I593056abdc6c3565d0396fb885923fef643bf4e4
- Check for invalid generation before checking for other types of invalid iterators.
- Check specifically for dereferencing end() iterators.
PiperOrigin-RevId: 483725646
Change-Id: Ibca19c48b1b242384683580145be8fb9ae707bc8
Note: btree_iterator::operator- is still O(N) because in the worst case (end()-begin()), we will have at least one operation per node in the tree, and there are at least N/M nodes, where M (a constant) is the maximum number of values per node.
PiperOrigin-RevId: 481716874
Change-Id: Ic0225b7509208ed96b75a2dc626d2aa4a24f4946
Also remove the transfer implementations from btree_set.h and flat_hash_set.h, which are equivalent to the default implementations.
Motivation: this will simplify upcoming changes related to trivial relocation.
PiperOrigin-RevId: 477493403
Change-Id: I75babef4c93dec3a8105f86c58af54199bb1ec9c
Certain core libraries in Chrome build with these warnings [1];
btree_map and btree_set cannot be used in those libraries until these
warnings are fixed.
[1] https://crbug.com/1292951
PiperOrigin-RevId: 476908396
Change-Id: I32e9ea1eec911e329d6ff00f04fa2e9cfde8660a
The btree data structure poisons regions of memory it's not using. It leaves
these regions poisoned when it frees memory. This means that a custom memory
allocator that tries to reuse freed memory will trigger an ASAN
use-after-poison error.
The fix is to unpoison each memory region right before freeing it.
PiperOrigin-RevId: 475309671
Change-Id: I29d55c298d3d89a83e1f960deb6e93118891ff83
Addresses failures with the following, in some files:
-Wshorten-64-to-32
-Wimplicit-int-conversion
-Wsign-compare
-Wsign-conversion
-Wtautological-unsigned-zero-compare
(This specific CL focuses on .cc files in */internal/.)
Bug: chromium:1292951
PiperOrigin-RevId: 473868797
Change-Id: Ibe0b76e33f9e001d59862beaac54fb47bacd39b2
Addresses failures with the following, in some files:
-Wshorten-64-to-32
-Wimplicit-int-conversion
-Wsign-compare
-Wsign-conversion
-Wtautological-unsigned-zero-compare
(This specific CL focuses on tests.)
Bug: chromium:1292951
PiperOrigin-RevId: 473055916
Change-Id: I40cdd6c87ba9d0a5fb2db5746cff04f14ee829c2
Addresses failures with the following, in some files:
-Wshorten-64-to-32
-Wimplicit-int-conversion
-Wsign-compare
-Wsign-conversion
-Wtautological-unsigned-zero-compare
(This specific CL focuses on miscellaneous non-test source files.)
Bug: chromium:1292951
PiperOrigin-RevId: 473054605
Change-Id: Ifd7b24966613ca915511a3a607095508068200b8
In some cases we can do a bit better by using
std::min(std::numeric_limits<size_type>::max() / 2, allocator<T>::max_size())
They may help in some cases, particularly on 32-bit platforms.
PiperOrigin-RevId: 471846886
Change-Id: I5bd63de5dd8aec3de6530a33d8904dd6e9bd015e
The problem is that the underlying storage adds redzone after the actual data so from compiler's perspective it looks like a valid memory. In the outlined version the memory is returned in call.i.i with unknown size so the access check can't be removed. The workaround is to always outline the call to InitializeData for ASAN builds.
Outlined version:
%call.i.i = call noundef i32* @absl::FixedArray<int, 4ul, std::__u::allocator<int> >::Storage::InitializeData()(...), !dbg !28
store i32* %call.i.i, i32** %data_.i.i, align 8, !dbg !27
%arrayidx = getelementptr inbounds i32, i32* %call.i.i, i64 5, !dbg !29
%24 = bitcast i32* %arrayidx to i8*, !dbg !29
call void @llvm.asan.check.memaccess(i8* %24, i32 36), !dbg !29
store i32 0, i32* %arrayidx, align 4, !dbg !29
Inlined version:
%arrayidx = getelementptr inbounds %"class.absl::FixedArray", %"class.absl::FixedArray"* %7, i64 0, i32 0, i32 0, i32 1, i64 20, !dbg !40
%27 = bitcast i8* %arrayidx to i32*, !dbg !40
>>>>>>>>>>>>>>> call to @llvm.asan.check.memaccess removed <<<<<<<<<<<<<<
store i32 0, i32* %27, align 4, !dbg !40
Workaround for ASAN stack safety analysis problem with FixedArray container annotations.
PiperOrigin-RevId: 471583635
Change-Id: I0d74eed5782a1cbd340ca4aca1bce71b63b06d43
Addresses failures with the following, in some files:
-Wshorten-64-to-32
-Wimplicit-int-conversion
-Wsign-compare
-Wsign-conversion
-Wtautological-unsigned-zero-compare
(This specific CL focuses on .cc files in */internal/.)
Bug: chromium:1292951
PiperOrigin-RevId: 471561809
Change-Id: I7abd6d83706f5ca135f1ce3458192a498a6280b9
Addresses failures with the following, in some files:
-Wshorten-64-to-32
-Wimplicit-int-conversion
-Wsign-compare
-Wsign-conversion
-Wtautological-unsigned-zero-compare
(This specific CL focuses on .cc files in */internal/.)
Bug: chromium:1292951
PiperOrigin-RevId: 471549854
Change-Id: Id685d0e4666212926f4e001b8ef4930b6a33a4cc
Corrects the computation of max_size(), so that it accounts for the
size of the objects.
PiperOrigin-RevId: 471343778
Change-Id: I68e222cefaa0295b8d8c38d00308a29df4165e81
Addresses failures with the following, in some files:
-Wshorten-64-to-32
-Wimplicit-int-conversion
-Wsign-compare
-Wsign-conversion
-Wtautological-unsigned-zero-compare
(This specific CL focuses on .h and win32 .inc files.)
Bug: chromium:1292951
PiperOrigin-RevId: 463835431
Change-Id: If8e5f7f651d5cd96035e23e4623bdb08a7fedabe
This ensures that emmintrin.h is included with clang-cl. Otherwise, errors like
this occur:
.../absl/container/internal/raw_hash_set.h(536,8): error: unknown type name '__m128i'
inline __m128i _mm_cmpgt_epi8_fixed(__m128i a, __m128i b) {
This aligns the include ifdef guards and the guards that use the provided APIs.
The includes are also reordered, so they appear after the config header which
provides the internal macro values.
PiperOrigin-RevId: 456530271
Change-Id: I86dfd0022fd06fe7aa132138ec4d1bd14a86ba84
This change introduces the symbol
ABSL_INTERNAL_NEED_REDUNDANT_CONSTEXPR_DECL
to guard redundant declarations of static constexpr data
members that are needed prior to C++17.
This change also introduces the symbol
ABSL_INTERNAL_CPLUSPLUS_LANG, which is supposed to be set
to the same value as __cplusplus, except it uses _MSVC_LANG
on MSVC so that the value is correct on MSVC.
Neither of these new symbols should be used outside of Abseil.
Fixes#1191
PiperOrigin-RevId: 453923908
Change-Id: I1316c52c19fa0c168b93cced0c817e4cb7c9c862
Avoid using value move/swap and delete those functions from slot_policy types. There was only one use of params_type::move in `erase`.
PiperOrigin-RevId: 453237739
Change-Id: Ie81c6dba6c4db34e97a067d2c0defcded8044a5a
https://pastebin.com/CmnzwUFN
The key idea is to avoid using 16 byte NEON and use 8 byte NEON which has lower latency for BitMask::Match. Even though 16 byte NEON achieves higher throughput, in SwissMap it's very important to catch these Matches with low latency as probing on average happens at most once.
I also introduced NonIterableMask as ARM has really great cbnz instructions and additional AND on scalar mask had 1 extra latency cycle
PiperOrigin-RevId: 453216147
Change-Id: I842c50d323954f8383ae156491232ced55aacb78
We were using `init_type`s for temp values that we would move into slots, but in this case, we need to have actual slots. We use node handles for managing slots outside of nodes.
Also, in btree::copy_or_move_values_in_order, pass the slots from the iterators rather than references to values. This allows for moving from map keys instead of copying for standard layout types.
In the test, fix a couple of ClangTidy warnings from missing includes and calling `new` instead of `make_unique`.
PiperOrigin-RevId: 452062967
Change-Id: I870e89ae1aa5b3cfa62ae6e75b73ffc3d52e731c
Benchmarks: https://pastebin.com/tZ7dr67W. Works well especially on smaller ranges.
After a week on spending optimizing NEON SIMD where I almost managed to make hash tables work with NEON SIMD without performance hits (still 1 cycle to optimize and I gave up a little), I found an interesting optimization for aarch64 to use cls instruction (count leading sign bits).
The loop has a property that ctrl_ group is not matched against count when the first slot is empty or deleted.
```
void skip_empty_or_deleted() {
while (IsEmptyOrDeleted(*ctrl_)) {
uint32_t shift = Group{ctrl_}.CountLeadingEmptyOrDeleted();
ctrl_ += shift;
slot_ += shift;
}
...
}
```
However, `kEmpty` and `kDeleted` have format of `1xxxxxx0` and `~ctrl & (ctrl >> 7)` always sets the lowest bit to 1.
In naive implementation, it does +1 to start counting zero bits, however, in aarch64 we may start counting one bits immediately. This saves 1 cycle and 5% of iteration performance.
Then it becomes hard to find a supported and sustainable C++ version of it.
`__clsll` is not supported by GCC and was supported only since clang 8, `__builtin_clrsb` is not producing optimal codegen for clang. `__rbit` is not supported by GCC and there is no intrinsic to do that, however, in clang we have `__builtin_bitreverse{32,64}`. For now I decided to enable this only for clang, only if they have appropriate builtins.
PiperOrigin-RevId: 451168570
Change-Id: I7e9256a60aecdc88ced4e6eb15ebc257281b6664
Add a new (internal) feature test macro to detect whether the wrappers are no-ops on a given platform.
Note that one-arg __builtin_prefetch(x) is equivalent to __builtin_prefetch(x, 0, 3), per `man BUILTIN_PREFETCH(3)` and gcc docs.
PiperOrigin-RevId: 449508660
Change-Id: I144e750205eec0c956d8dd62bc72e10bdb87c4f7
Evan Brown
Abseil Team
Gennadiy Rozental
Derek Mauro
Tom Rybka
Jose Renau
Greg Falcon