refactor(bigquery-jdbc): optimize and unify concurrent metadata methods

[keshavdandeva]

b/535543457

This PR significantly refactors and optimizes the concurrent execution model for JDBC metadata operations (e.g., getTables, getColumns, getProcedures, getExportedKeys) in BigQueryDatabaseMetaData.java.

Previously, these methods relied on bespoke, scattered threading logic that was susceptible to thread starvation deadlocks (where bounded pools wait on tasks submitted to the same pool) and suffered from sequential dataset listing bottlenecks.

This PR introduces a unified, robust Two-Tier Scatter-Gather Architecture, eliminating deadlocks, restoring parallel execution for broad dataset scans, and standardizing error handling across all metadata fetchers.

Key Changes

1. Unified Scatter-Gather Concurrency Framework:

   • Replaced custom Runnable implementations in each metadata method with shared helper functions (fetchAndPopulateQueueAsync, processTargetTablesConcurrently, processTargetRoutinesConcurrently).
   • Introduced a two-tier thread pool strategy to prevent thread pool starvation (deadlocks):
     • Tier 1 (Orchestrator): The overarching fetchAndPopulateQueueAsync task now runs on the unbounded queryExecutor (via connection.getExecutorService()). This thread safely blocks while coordinating API calls.
     • Tier 2 (API Workers): The bounded metadataExecutor is now exclusively used for concurrent BigQuery HTTP API calls (listTables, getTable, etc.). Workers never wait on other workers, completely removing the deadlock risk.

2. Parallelized Dataset Discovery (Phase 1):

   • Operations that require wildcard scanning across datasets now submit parallel dataset listing tasks (listTables / listRoutines) to the API pool.
   • The orchestrator gathers these listing results to construct a targeted list of parallel detail fetch tasks (Phase 2). This drastically speeds up broad schema discovery compared to the previous sequential listing behavior.

3. Robust Resource Management & Cancellation:

   • Centralized activeFutures tracking inside the new concurrent helpers.
   • If an InterruptedException or SQLException occurs, the finally blocks aggressively cancel all active futures, preventing runaway background API calls and memory leaks.

4. Specific Bug Fixes:

   • getExportedKeys Catalog Fallback: Fixed an issue where an exact schema match string without wildcards was incorrectly evaluated as a regex pattern during the fallback flow, by explicitly passing isPattern=false to getTargetDatasets().

5. Performance Benchmarks:

   • Rigorous benchmarking proves the new architecture eliminates thread starvation and maximizes throughput. While shallow targeted scans maintain parity (~1s), deep, full-catalog scans (e.g., getColumns(null, null, null, null)) are up to 2.6x faster (reducing latency from 23.5s to 8.8s on a 12,000-column project) because the API worker pool is no longer bottlenecked by blocked orchestrator threads.

[gemini-code-assist]

Code Review

This pull request refactors BigQueryDatabaseMetaData.java to streamline metadata retrieval methods (such as getProcedures, getTables, and getColumns) by introducing concurrent helper methods like processTargetRoutinesConcurrently and processTargetTablesConcurrently. The review feedback highlights several critical issues with this refactoring: an architectural regression where heavy metadata retrieval now blocks synchronously on the caller thread; a performance bottleneck due to sequential routine listing for wildcard patterns; a potential NullPointerException when handling execution exceptions; silent failures on thread interruption in waitForTasksCompletion; and a missing empty schema check that leads to redundant API calls.

[keshavdandeva]

/gemini review

[gemini-code-assist]

Code Review

This pull request refactors BigQueryDatabaseMetaData.java to streamline metadata fetching by introducing unified concurrent processing helpers and an asynchronous queue population mechanism, significantly reducing boilerplate code across various metadata methods. The review feedback highlights several critical issues where valid null parameters (such as columnNamePattern, schemaPattern, routineNamePattern, and tableNamePattern) are passed directly to compileSqlLikePattern, which would trigger NullPointerExceptions. Additionally, a bug was identified in getColumns where literal column name filtering is ignored due to conditional regex compilation. Lastly, an improvement was suggested to prevent redundant nesting of SQLExceptions during asynchronous error handling.

[logachev]

This is for both methods processTargetTablesConcurrently and processTargetRoutinesConcurrently.

• You can simplify the code, there is no need in this micro-optimization of special-casing single table. Calling it in executor thread has nearly 0 overhead compared to the cost of network call
• Similar, no need to handle Literal/non-Literal in different ways. It is less error-prone to:
  • If not literal: build list of tables first
  • Process list of tables regardless of where it came from, from Literal list or was just resolved.

Also for (2), it might not be a good idea to schedule a task from within a task. E.g. what if there are MAX_THREADS top-level tasks? They'll deadlock since their child tasks can't be scheduled.

[keshavdandeva]

Okay so, regarding removing micro-optimization: I guess you are right, makes sense to remove it and make code uniform and easier to read and flow through one pipeline.

regarding deadlock risk: I want to clarify that there is no nested task scheduling (no deadlock risk). The listing tasks submitted in Stage 1 do not schedule child tasks onto the executor. They simply return an in-memory List<Callable<Void>> At L4788, the calling thread calls future.get() and waits for 100% of the Stage 1 listing tasks to finish completely. Only after all worker threads from Stage 1 have returned to the pool, we submit the Stage 2 detail tasks (processSingleTable) to metadataExecutor. I added comments so it is easier to read/understand