Skip to content

Read and parse a local file once on the chunked GPU read path (#3373)#3374

Open
brendancol wants to merge 2 commits into
mainfrom
issue-3373
Open

Read and parse a local file once on the chunked GPU read path (#3373)#3374
brendancol wants to merge 2 commits into
mainfrom
issue-3373

Conversation

@brendancol

Copy link
Copy Markdown
Contributor

Closes #3373.

_read_geotiff_gpu_chunked read a local source file off disk twice and parsed its IFDs twice at dask-graph-build time. The per-tile byte-count cap block did read_all + parse_header + parse_all_ifds and discarded the result; the GDS qualification block right after repeated the same read and parse. IFD parsing is O(tile_count), so a COG with 10k+ tiles paid that cost twice (plus a second full-file read) on every open_geotiff(gpu=True, chunks=...) call.

Change

  • Read the bytes once and parse the header/IFDs once before the cap loop, run the cap check on that single parse, then reuse the same raw / header / ifds for the GDS qualification probe.

Contract preserved:

  • The per-tile cap still raises ValueError (the denial-of-service guard) before any GDS work runs.
  • A metadata-parse failure, an empty IFD list, or a bad overview level leaves raw/header/ifds as None and falls through to the CPU path (_read_geotiff_dask) instead of raising, exactly as before.
  • The local-file-only guard and the _FileSource context-manager scoping (hardened for the free-threaded race) are unchanged.

This is graph-construction overhead only, not a decode-throughput change.

Backends

GPU read path (gpu=True, chunks=...). The dask+cupy chunked read still returns a lazy cupy-backed array; numpy, cupy-eager, and dask+numpy paths are untouched.

Tests

Added three regression tests in xrspatial/geotiff/tests/gpu/test_reader.py:

  • A read_all / parse_header / parse_all_ifds invocation counter proving the local chunked GPU read parses once (it counted 2 each before the fix).
  • The per-tile DoS cap still raises at graph build.
  • A synthetic metadata-parse failure routes to the CPU path without raising, and the result matches the source.
  • Full tests/gpu/ (487 passed, 3 skipped) and golden_corpus/test_dask_gpu.py pass.

No ASV: geotiff is not in the performance-labeler benchmarked set, so the performance label does not trigger benchmarks here.

_read_geotiff_gpu_chunked read the source file off disk twice and parsed
its IFDs twice at dask-graph-build time: the per-tile byte-count cap
block did read_all + parse_header + parse_all_ifds and discarded the
result, then the GDS qualification block repeated the same read and
parse. IFD parsing is O(tile_count), so a 10k+ tile COG paid that cost
twice (plus a second full-file read) per open_geotiff(gpu=True,
chunks=...) call.

Merge the two passes: read the bytes once and parse header/IFDs once
before the cap loop, run the cap check on that single parse, then reuse
the same raw/header/ifds for the GDS qualification probe.

Preserves the existing contract: the per-tile cap still raises
ValueError before any GDS work; a metadata-parse failure (or empty IFD
list, or bad overview level) leaves raw/header/ifds as None / falls
through to the CPU path instead of raising; the local-file-only guard
and the _FileSource context-manager scoping (hardened for the
free-threaded race) are unchanged.

Graph-build path only, not a decode-throughput change.

Adds 3 regression tests: a read_all/parse invocation counter proving the
file is read and parsed once (it counted 2 each before the fix), the DoS
per-tile cap still raising, and a synthetic parse failure routing to the
CPU path without raising.

Closes #3373

@brendancol brendancol left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review: Read and parse a local file once on the chunked GPU read path (#3373)

Blockers (must fix before merge)

None.

Suggestions (should fix, not blocking)

None.

Nits (optional improvements)

  • xrspatial/geotiff/_backends/gpu.py:1416: the if not ifds: raise ValueError("No IFDs found in TIFF file") inside the qualification try is now only reachable when parse_all_ifds returns an empty list (a non-None, falsy value). That is the same path the old code took, and the broad except below routes it to the CPU fallback, so behavior is unchanged. A one-line comment noting that an empty list is intentionally routed to CPU would save the next reader the trace. Optional.

What looks good

  • The over-cap ValueError is raised in the cap loop (line ~1400), which sits before and outside the qualification try/except, so the denial-of-service guard still propagates. Verified.
  • Parse failure, an empty IFD list, and a bad overview level all leave raw/header/ifds as None or fall through the broad except, routing to _read_geotiff_dask without raising, which matches the prior two-parse semantics.
  • header.byte_order and raw are only dereferenced under if ifds is not None:, where a successful parse guarantees both are set.
  • The _FileSource context-manager scoping (hardened for the free-threaded race) is preserved: the read still happens inside the with block.
  • The invocation-counter test has teeth. It counts 2 reads/parses on the pre-fix code and 1 after, so the regression cannot silently return.

Checklist

  • Behavior matches the prior implementation (cap raises, failures fall through)
  • GPU dask path still returns a lazy cupy-backed array
  • Cap / DoS guard preserved
  • Edge cases (empty IFDs, bad overview, parse failure) covered
  • No premature materialization or unnecessary copies (one fewer read, one fewer parse)
  • Benchmark not needed (graph-build path, geotiff not in the benchmarked set)
  • README feature matrix not applicable (no API change)
  • Tests present and meaningful (487 passed, 3 skipped in tests/gpu/)

@brendancol brendancol left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review (follow-up): #3373

The single optional nit from the first pass is addressed in d5c96b0: the empty-IFD-list path now carries a comment explaining it falls through to the CPU reader.

No remaining findings. The fix preserves the cap/DoS raise, the parse-failure and empty-IFD fall-through to the CPU path, and the _FileSource scoping. The 3373 regression tests still pass (3 passed), and the wider tests/gpu/ suite was green (487 passed, 3 skipped).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

open_geotiff(gpu=True, chunks=...) reads and parses a local file twice at graph build

1 participant