Skip to content

Remove diskann-platform; tidy diskann-disk instrumentation & reader modules#1205

Open
suri-kumkaran wants to merge 1 commit into
mainfrom
users/suryangupta/remove_diskann_platform
Open

Remove diskann-platform; tidy diskann-disk instrumentation & reader modules#1205
suri-kumkaran wants to merge 1 commit into
mainfrom
users/suryangupta/remove_diskann_platform

Conversation

@suri-kumkaran

Copy link
Copy Markdown
Contributor

Folds the diskann-platform crate into diskann-disk, decouples diskann-providers from
perf instrumentation, and reorganizes the affected diskann-disk modules. No behavior change;
pure refactor.

Changes

  • Decouple providers: move Timer out of diskann-providers (its one timing use now uses
    std::time::Instant); drop providers' diskann-platform, opentelemetry deps and perf_test
    feature.
  • Delete diskann-platform: perf counters now live as a private detail of Timer; the io
    primitives (io_uring / Windows IOCP / file handles) move next to their only consumer,
    aligned_file_reader. Workspace, CI path filters, and docs updated.
  • Consolidate instrumentation: Timer + platform perf counters now sit in
    utils/instrumentation/ beside PerfLogger (perf → Timer → PerfLogger). Canonical path is
    utils::instrumentation::Timer.
  • Relocate aligned_file_reader: utils/search/provider/ (its sole consumer area) and
    reorganized internally into reader/, factory/, traits/, platform/.
  • Simplify disk-search setup: add DiskVertexProviderFactory::from_disk_index_path, removing
    the benchmark's coupling to the low-level reader factory.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors the disk-index stack by removing the diskann-platform crate, moving platform I/O and perf/timing utilities into diskann-disk, and updating dependent crates (diskann-tools, diskann-benchmark, diskann-providers) to use the new module boundaries.

Changes:

  • Removed diskann-platform from the workspace and relocated its platform-specific I/O primitives into diskann-disk::search::provider::aligned_file_reader::platform.
  • Consolidated instrumentation by moving Timer into diskann-disk::utils::instrumentation and decoupling diskann-providers from perf instrumentation dependencies/features.
  • Reorganized aligned_file_reader under diskann-disk::search::provider, and added DiskVertexProviderFactory::from_disk_index_path to simplify higher-level setup.

Reviewed changes

Copilot reviewed 55 out of 61 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
diskann-tools/src/utils/search_disk_index.rs Updates imports to the relocated aligned reader traits and instrumentation paths.
diskann-tools/src/utils/build_pq.rs Switches Timer usage to diskann-disk instrumentation and tidies provider imports.
diskann-tools/src/utils/build_disk_index.rs Moves Timer dependency from providers to diskann-disk instrumentation.
diskann-tools/src/bin/compute_multivec_groundtruth.rs Updates Timer import location.
diskann-tools/src/bin/compute_groundtruth.rs Updates Timer import location.
diskann-tools/Cargo.toml Drops diskann-providers/perf_test from the tools perf_test feature wiring.
diskann-providers/src/utils/mod.rs Removes the providers-local Timer module export.
diskann-providers/src/model/pq/pq_construction.rs Replaces Timer usage with std::time::Instant for simple timing.
diskann-providers/Cargo.toml Removes opentelemetry/diskann-platform deps and drops the perf_test feature.
diskann-platform/src/win/thread_safe_handle.rs Deleted as part of removing diskann-platform.
diskann-platform/src/win/ssd_io_context.rs Deleted as part of removing diskann-platform.
diskann-platform/src/macos/ssd_io_context.rs Deleted as part of removing diskann-platform.
diskann-platform/src/macos/mod.rs Deleted as part of removing diskann-platform.
diskann-platform/src/linux/mod.rs Deleted as part of removing diskann-platform.
diskann-platform/src/lib.rs Deleted as part of removing diskann-platform.
diskann-platform/Cargo.toml Deleted as part of removing diskann-platform.
diskann-disk/src/utils/mod.rs Removes old utils::aligned_file_reader exports now that it moved under search/provider.
diskann-disk/src/utils/instrumentation/timer/windows.rs New Windows perf/timing helpers moved under diskann-disk instrumentation.
diskann-disk/src/utils/instrumentation/timer/macos.rs New macOS perf/timing helpers moved under diskann-disk instrumentation.
diskann-disk/src/utils/instrumentation/timer/linux.rs New Linux perf/timing helpers moved under diskann-disk instrumentation.
diskann-disk/src/utils/instrumentation/timer.rs Replaces diskann_platform::* with per-OS timer backends inside diskann-disk.
diskann-disk/src/utils/instrumentation/perf_logger.rs Switches PerfLogger to use diskann-disk’s Timer.
diskann-disk/src/utils/instrumentation/mod.rs Re-exports Timer from the new instrumentation module layout.
diskann-disk/src/utils/aligned_file_reader/mod.rs Deleted; aligned file reader moved under search/provider.
diskann-disk/src/storage/quant/pq/pq_generation.rs Moves Timer import to diskann-disk instrumentation.
diskann-disk/src/storage/quant/generator.rs Moves Timer import to diskann-disk instrumentation.
diskann-disk/src/search/provider/mod.rs Exposes the relocated aligned_file_reader module.
diskann-disk/src/search/provider/disk_vertex_provider.rs Updates aligned reader imports to the new module location.
diskann-disk/src/search/provider/disk_vertex_provider_factory.rs Adds from_disk_index_path and updates aligned reader factory imports.
diskann-disk/src/search/provider/disk_sector_graph.rs Updates aligned reader imports to the new module location.
diskann-disk/src/search/provider/disk_provider.rs Switches setup to DiskVertexProviderFactory::from_disk_index_path and updates imports.
diskann-disk/src/search/provider/cached_disk_vertex_provider.rs Updates aligned reader trait import path.
diskann-disk/src/search/provider/aligned_file_reader/traits/mod.rs Adjusts trait module structure and visibility.
diskann-disk/src/search/provider/aligned_file_reader/traits/aligned_reader_factory.rs New aligned reader factory trait definition.
diskann-disk/src/search/provider/aligned_file_reader/traits/aligned_file_reader.rs Updates imports to the new aligned read location.
diskann-disk/src/search/provider/aligned_file_reader/reader/windows.rs Updates platform primitive imports to the new internal platform module.
diskann-disk/src/search/provider/aligned_file_reader/reader/storage_provider.rs Updates imports to the new aligned reader module paths.
diskann-disk/src/search/provider/aligned_file_reader/reader/mod.rs New module that selects concrete reader implementations by platform.
diskann-disk/src/search/provider/aligned_file_reader/reader/linux.rs Updates IOContext import to the new platform module.
diskann-disk/src/search/provider/aligned_file_reader/platform/windows/ssd_io_context.rs New Windows IO context type colocated with aligned reader platform primitives.
diskann-disk/src/search/provider/aligned_file_reader/platform/windows/mod.rs Trims exports and rehomes IOContext and Win32 types for aligned readers.
diskann-disk/src/search/provider/aligned_file_reader/platform/windows/io_completion_port.rs Fixes internal imports and modernizes logging formatting.
diskann-disk/src/search/provider/aligned_file_reader/platform/windows/file_io.rs Fixes internal imports to the new platform module structure.
diskann-disk/src/search/provider/aligned_file_reader/platform/windows/file_handle.rs Adds dead_code allowances and improves formatting in errors/logs.
diskann-disk/src/search/provider/aligned_file_reader/platform/mod.rs New platform dispatch module for aligned reader primitives.
diskann-disk/src/search/provider/aligned_file_reader/platform/linux.rs Documents/ensures Linux IOContext keeps the registered FD alive.
diskann-disk/src/search/provider/aligned_file_reader/mod.rs New aligned file reader module root under search/provider.
diskann-disk/src/search/provider/aligned_file_reader/factory/virtual_storage.rs Updates internal module paths after refactor.
diskann-disk/src/search/provider/aligned_file_reader/factory/mod.rs New factory module root for aligned reader factories.
diskann-disk/src/search/provider/aligned_file_reader/factory/file.rs Updates internal imports after module reorganization.
diskann-disk/src/search/provider/aligned_file_reader/aligned_read.rs New typed alignment-checked IO read request type.
diskann-disk/src/build/builder/core.rs Updates tests/imports to use relocated Timer and aligned reader factories.
diskann-disk/Cargo.toml Removes diskann-platform, updates platform deps (io-uring/windows-sys), keeps perf feature wiring.
diskann-disk/benches/benchmarks/aligned_file_reader_bench.rs Updates aligned reader import path for benchmarks.
diskann-disk/benches/benchmarks_iai/aligned_file_reader_bench_iai.rs Updates aligned reader import path for IAI benchmarks.
diskann-benchmark/src/disk_index/search.rs Uses DiskVertexProviderFactory::from_disk_index_path to reduce low-level coupling.
diskann-benchmark/Cargo.toml Drops diskann-providers/perf_test from disk-index feature wiring.
Cargo.toml Removes diskann-platform from workspace members and dependencies.
Cargo.lock Removes diskann-platform package entries and reflects updated platform deps.
agents.md Updates tier listing to remove diskann-platform.
.github/workflows/disk-benchmarks.yml Updates path filters to stop triggering on the removed diskann-platform crate.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +23 to +24
#[cfg(test)]
pub(crate) use factory::VirtualAlignedReaderFactory;
Comment on lines +11 to +14
#[cfg(test)]
mod virtual_storage;
#[cfg(test)]
pub(crate) use virtual_storage::VirtualAlignedReaderFactory;
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 7.14286% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.82%. Comparing base (53257e9) to head (3184897).

Files with missing lines Patch % Lines
...rc/search/provider/disk_vertex_provider_factory.rs 0.00% 8 Missing ⚠️
diskann-disk/src/search/provider/disk_provider.rs 0.00% 5 Missing ⚠️

❌ Your patch status has failed because the patch coverage (7.14%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1205      +/-   ##
==========================================
+ Coverage   89.79%   90.82%   +1.02%     
==========================================
  Files         488      488              
  Lines       93312    93319       +7     
==========================================
+ Hits        83791    84757     +966     
+ Misses       9521     8562     -959     
Flag Coverage Δ
miri 90.82% <7.14%> (+1.02%) ⬆️
unittests 90.78% <7.14%> (+1.33%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
diskann-disk/src/build/builder/core.rs 95.35% <ø> (ø)
...earch/provider/aligned_file_reader/aligned_read.rs 100.00% <ø> (ø)
...earch/provider/aligned_file_reader/factory/file.rs 100.00% <ø> (ø)
...der/aligned_file_reader/factory/virtual_storage.rs 100.00% <ø> (ø)
...rch/provider/aligned_file_reader/platform/linux.rs 100.00% <ø> (ø)
...earch/provider/aligned_file_reader/reader/linux.rs 82.65% <ø> (ø)
...der/aligned_file_reader/reader/storage_provider.rs 98.36% <ø> (ø)
...src/search/provider/cached_disk_vertex_provider.rs 94.11% <ø> (ø)
...kann-disk/src/search/provider/disk_sector_graph.rs 96.95% <ø> (ø)
...n-disk/src/search/provider/disk_vertex_provider.rs 85.27% <ø> (ø)
... and 13 more

... and 43 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants