Fix filtered CAGRA search returning incorrect recall due to kHostMmap#2303
Open
rupakroynv wants to merge 1 commit into
Open
Fix filtered CAGRA search returning incorrect recall due to kHostMmap#2303rupakroynv wants to merge 1 commit into
rupakroynv wants to merge 1 commit into
Conversation
… filter bitset cuvs_cagra::get_preference() returned MemoryType::kHostMmap, which the benchmark framework used to provide the filter bitset as a file-backed mmap pointer. GPU kernels cannot read host mmap memory on non-ATS GPUs (A100, H100, V100), causing the filter to be silently non-functional and recall to collapse to approximately the filter pass rate regardless of itopk. On ATS-capable GPUs (GH200), the file-backed mmap pages are not pre-faulted, triggering a cascade of concurrent ATS page faults during graph traversal and producing incorrect results. Fix 1: Change get_preference() to return MemoryType::kDevice so the framework explicitly copies the filter bitset and dataset to device memory before CAGRA's kernels access them. This is correct on all GPUs. Fix 2: With kDevice, the framework copies the full dataset to device before calling set_search_dataset(). The original set_search_param() then called copy_with_padding() unconditionally, doubling device memory usage (e.g. 2x 38.4 GB for deep-100M) and causing OOM. Skip copy_with_padding() when the dataset is already on device and no 16-byte alignment padding is required. Fix 3: sub_indices_ accumulates corrupted state when the kDevice path triggers the dataset update code path. For single-index CAGRA, sub_indices_ is always logically empty, so reset it via placement new in copy() to prevent bad_array_new_length when the copy constructor runs. Tested on H100 80GB (no ATS) and GH200 (ATS) with deep-1M/deep-100M, 5% filter pass rate, inner product distance. Recall with kHostMmap (unpatched): 0.0495 across all itopk values. Recall with kDevice (fix): 0.45-0.9999 scaling correctly with itopk, reaching >0.99 at itopk=1024. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Filtered CAGRA search returns near-zero recall when the benchmark harness passes the filter bitset using
MemoryType::kHostMmap. This affects the ANN benchmark (CUVS_CAGRA_ANN_BENCH) when running filtered search workloads.Root Cause
cuvs_cagra::get_preference()returnedMemoryType::kHostMmapfor both the dataset and the filter bitset.kHostMmapallocates a file-backedMAP_PRIVATEmmap where pages are not pre-faulted — the CPU page table has no mappings until a page fault occurs on first access.Inside CAGRA's search dispatch (
cagra.cuh),bitset_view::count()is called to estimate the filtering rate. That function wraps the host mmap pointer inraft::make_device_vector_viewand passes it toraft::popc, which launches a CUDA kernel treating the host pointer as device memory. On non-ATS GPUs this silently reads invalid memory, producing wrong counts and corrupting graph traversal. On ATS-capable GPUs the kernel can access host memory via hardware page migration, but the un-faulted file-backed pages trigger a cascade of serialized ATS page faults across thousands of concurrent graph-traversal threads, causing severe performance degradation.Fix
Three changes in
cpp/bench/ann/src/cuvs/cuvs_cagra_wrapper.h:get_preference(): Changedataset_memory_typefromkHostMmaptokDevice. This causes the benchmark harness to place both the dataset and the filter bitset in device memory before passing them to CAGRA, socount()and all GPU accesses operate on valid device pointers.set_search_param(): Skip the redundantcopy_with_padding()call when the dataset is already in device memory and no padding is needed. Previously, changing tokDevicewould causecopy_with_padding()to allocate a second device copy of the full dataset, causing OOM on GPUs where the dataset already fills most of device memory.copy(): Resetsub_indices_with placement new before invoking the copy constructor. WhenkDeviceis used,sub_indices_can be left in a corrupted state from a prior code path, causing UB in the copy constructor.Notes