AVX2 popcount

[TheHippo]

Description

Add an AVX2 SIMD implementation of the five slice population-count routines (popcntSlice and the And/Or/Xor/Mask variants) on amd64, selected at runtime via a CPUID/XGETBV feature check, with the scalar Go path kept as a fallback. It also removes the old scalar assembly, which was more or less dead code: that code was gated behind a !go1.9 build tag, meaning it stopped being compiled as of Go 1.9 - released in August 2017, roughly nine years ago - and has not been built by any supported toolchain since. The build tags are reworked accordingly.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Performance improvement
• Code refactoring
• Documentation update
• Test improvements
• Build/CI changes

Changes Made

What was changed?

• New popcnt_avx2_amd64.s / popcnt_avx2_amd64.go: AVX2 kernels plus the _hasAVX2 detection and the popcnt*Slice dispatch wrappers.
• popcnt_generic.go build tag retargeted to !amd64 || appengine.
• Removed popcnt_asm.go and popcnt_amd64.s.

Why was it changed?

The old amd64 assembly was guarded by !go1.9, so on every modern toolchain it was never compiled and all population counting fell through to a scalar loop calling math/bits.OnesCount64 one 64-bit word at a time. That intrinsic already lowers to the hardware POPCNT instruction, but POPCNT is scalar - nothing counted bits using the wide vector (SIMD) registers, which is the speedup this change adds (the removed assembly was itself just a scalar POPCNTQ loop, so it would not have helped either). These routines are hot: a bitmap container is a fixed 1024-word (8 KB) array, and a popcnt*Slice runs after essentially every bitmap/bitmap AND/OR/XOR/ANDNOT, plus in rank(), getCardinalityInRange() and computeCardinality().

How was it changed?

• The kernel counts bits four 64-bit words (256 bits) at a time. AVX2 has no single "count the set bits in a vector" instruction, so it uses the well-known Mula/Lemire trick: split each byte into its two 4-bit halves, look up each half's bit count in a tiny 16-entry table (one VPSHUFB does all 32 lookups at once), then add the per-byte counts together (VPSADBW). When a slice length isn't a multiple of four, the leftover 1-3 words are finished with the CPU's ordinary one-word popcount instruction (POPCNTQ), so the result is correct for any length.
• AVX2 is only used when it is actually safe. At startup the code asks the CPU whether it supports AVX2 and whether the OS has enabled the wide registers AVX2 needs (using the CPUID and XGETBV instructions). If either check fails, the SIMD path is never taken.
• The original scalar code stays as the fallback. amd64 CPUs without AVX2 use it at runtime, and builds for App Engine or non-amd64 architectures (arm, etc.) compile it directly. Behavior is identical on every path; only speed differs.

Testing

go test .

Passes (incl. go test -tags appengine . for the fallback). A differential test checks the AVX2 output against the Go reference across edge lengths {0,1,2,3,4,5,7,8,15,16,17,31,63,64,65,1023,1024,1025}, and a dispatch test forces the scalar path to exercise it on amd64 too.

Fuzzing

No serialization/parsing surface is touched; the existing fuzz harness continues to build and run.

Performance Impact

Significant speedup on the popcount functions.

Running Benchmarks

go test -bench=BenchmarkPopcnt -run=^$

Performance Analysis

Full 1024-word container (AMD Ryzen AI 7 PRO 350):

popcntSlice      599 ns/op -> 331 ns/op   (~1.8x)
popcntAndSlice  1108 ns/op -> 408 ns/op   (~2.7x)

• Trade-off: adds amd64 assembly to maintain. The scalar Go reference remains the source of truth and is what all other architectures use.
• benchstat reports a 13.0% speed improvement over the entire benchmark suite (geomean).

Breaking Changes

None. Public API and behavior are unchanged.

AI Disclosure

This change was developed with AI assistance (Claude). I reviewed code carefully, understand how it works, and take responsibility for it. Correctness was verified independently of the model: a differential test checks the AVX2 output against the scalar Go reference across many slice lengths (including the non-vector tail), go test and go test -tags appengine pass, go vet's asm checker is clean, and the speedups were confirmed with benchmarks.

[lemire]

Merging.