From bbadc6b9c8f24343ab2142257b381b7be1441b7e Mon Sep 17 00:00:00 2001 From: Joachim Meyer Date: Fri, 10 Jul 2026 12:21:13 -0600 Subject: [PATCH] Agents - Start from anew; clear out folder Removing the historic agents file to create a new tutorial for the NASA Gen-AI hackweek. --- .agents/POSTGIS_OPPORTUNITIES.md | 228 ------- .agents/implement-mcp-server-improvements.md | 235 ------- .agents/initial-setup-snowexsqlbot.md | 20 - .agents/mcp-research-prompt.md | 147 ---- .agents/plan-mcp-server-improvements.md | 621 ----------------- ...arch-mcp-server-and-agent-documentation.md | 402 ----------- .agents/research-snowexsql-tutorial.md | 637 ------------------ 7 files changed, 2290 deletions(-) delete mode 100644 .agents/POSTGIS_OPPORTUNITIES.md delete mode 100644 .agents/implement-mcp-server-improvements.md delete mode 100644 .agents/initial-setup-snowexsqlbot.md delete mode 100644 .agents/mcp-research-prompt.md delete mode 100644 .agents/plan-mcp-server-improvements.md delete mode 100644 .agents/research-mcp-server-and-agent-documentation.md delete mode 100644 .agents/research-snowexsql-tutorial.md diff --git a/.agents/POSTGIS_OPPORTUNITIES.md b/.agents/POSTGIS_OPPORTUNITIES.md deleted file mode 100644 index 2e611339..00000000 --- a/.agents/POSTGIS_OPPORTUNITIES.md +++ /dev/null @@ -1,228 +0,0 @@ -# PostGIS Optimization Opportunities - -## Overview -By leveraging PostGIS on the server side and sending WKT strings from the client, we can avoid heavy geospatial dependencies in Lambda while maintaining full spatial functionality. - -## Current Implementation Status - -### βœ… Implemented -- **from_area() for Point/Layer Measurements**: Uses PostGIS `ST_Intersects`, `ST_Buffer`, `ST_Transform` for spatial filtering - -## 🎯 High Priority Opportunities - -### 1. **Raster Operations (RasterMeasurements.from_area)** -**Current Issue**: Requires shapely's `from_shape()` in Lambda -**Solution**: Accept WKT from client, use PostGIS functions - -```python -# Lambda Handler -def _handle_raster_from_area_postgis(event, session): - wkt = event.get('shp_wkt') or event.get('pt_wkt') - crs = event.get('crs', 26912) - buffer_dist = event.get('buffer') - - if pt_wkt and buffer_dist: - geom_sql = f"ST_Buffer(ST_Transform(ST_GeomFromText('{wkt}', {crs}), 4326)::geography, {buffer_dist})::geometry" - else: - geom_sql = f"ST_Transform(ST_GeomFromText('{wkt}', {crs}), 4326)" - - query = text(f""" - SELECT ST_AsTiff( - ST_Clip( - ST_Union(raster), - ({geom_sql}), - TRUE - ) - ) - FROM image_data - WHERE ST_Intersects(raster, ({geom_sql})) - """) - # ... execute and return -``` - -**Benefits**: -- Raster clipping works in Lambda -- No shapely/rasterio needed in Lambda -- Fast server-side processing - -### 2. **Distance-Based Queries** -**New Feature**: Find measurements within X meters of a point - -```python -# Client API -df = PointMeasurements.find_within_distance( - pt=Point(x, y), - distance=1000, # meters - crs=26912, - type='depth' -) - -# Lambda uses PostGIS ST_DWithin -query = text(f""" - SELECT * - FROM point_data - WHERE ST_DWithin( - geom::geography, - ST_Transform(ST_GeomFromText('{pt_wkt}', {crs}), 4326)::geography, - {distance} - ) -""") -``` - -**Benefits**: -- Uses PostGIS spatial index (extremely fast) -- No need to buffer geometries -- Geography type handles meters correctly - -### 3. **Bounding Box Queries** -**New Feature**: Query by bounding box (xmin, ymin, xmax, ymax) - -```python -# Client API -df = PointMeasurements.from_bbox( - bbox=(xmin, ymin, xmax, ymax), - crs=26912, - type='depth' -) - -# Lambda uses PostGIS ST_MakeEnvelope -query = text(f""" - SELECT * - FROM point_data - WHERE ST_Intersects( - geom, - ST_Transform(ST_MakeEnvelope({xmin}, {ymin}, {xmax}, {ymax}, {crs}), 4326) - ) -""") -``` - -**Benefits**: -- Common use case for map viewers -- Very efficient with spatial indexes -- No client-side geometry construction needed - -## πŸ”„ Medium Priority - -### 4. **Nearest Neighbor Queries** -**New Feature**: Find N nearest measurements to a point - -```python -df = PointMeasurements.find_nearest( - pt=Point(x, y), - n=10, - crs=26912, - type='depth' -) - -# Uses PostGIS <-> operator and ORDER BY distance -query = text(f""" - SELECT *, ST_Distance(geom::geography, point::geography) as distance - FROM point_data - CROSS JOIN ( - SELECT ST_Transform(ST_GeomFromText('{pt_wkt}', {crs}), 4326)::geography as point - ) pt - WHERE type_id = (SELECT id FROM measurement_type WHERE name = :type) - ORDER BY geom::geography <-> pt.point - LIMIT :n -""") -``` - -### 5. **Spatial Aggregations** -**New Feature**: Group measurements by proximity - -```python -# Find average depth within grid cells -df = PointMeasurements.aggregate_by_grid( - bbox=(xmin, ymin, xmax, ymax), - cell_size=100, # meters - type='depth', - agg='mean' -) - -# Uses PostGIS ST_SnapToGrid -``` - -## πŸš€ Advanced Opportunities - -### 6. **Line-of-Sight / Path Queries** -Query measurements along a path (e.g., flight line, transect) - -```python -df = PointMeasurements.along_path( - path=LineString([...]), - buffer=50, - type='depth' -) -``` - -### 7. **Temporal-Spatial Queries** -Combine spatial and temporal proximity - -```python -# Find measurements near location X within 1 day of date Y -df = PointMeasurements.find_nearby_in_time( - pt=Point(x, y), - date=datetime(2020, 2, 1), - spatial_buffer=500, - temporal_window=timedelta(days=1) -) -``` - -### 8. **Spatial Joins** -Join different measurement types by proximity - -```python -# Find all SMP profiles within 10m of pits -df = LayerMeasurements.join_nearby( - reference_type='density', # pits - join_type='smp', - max_distance=10 -) -``` - -## Implementation Pattern - -For all PostGIS operations, follow this pattern: - -1. **Client Side** (lambda_client.py): - - Convert Shapely geometries to WKT - - Send WKT + parameters to Lambda - -2. **Lambda Handler** (lambda_handler.py): - - Construct PostGIS SQL query - - Use WKT with `ST_GeomFromText` - - Let database do spatial operations - -3. **Benefits**: - - βœ… No heavy dependencies in Lambda - - βœ… Fast database-side processing - - βœ… Scales to millions of geometries - - βœ… Uses PostGIS spatial indexes - -## Performance Notes - -PostGIS spatial indexes (`GIST`) make these operations extremely fast: -- `ST_Intersects`: Uses index, very fast -- `ST_DWithin`: Uses index, very fast -- `ST_Distance` with ORDER BY: Uses index with KNN operator `<->` -- Without spatial index: Linear scan, slow - -Ensure all geometry columns have spatial indexes: -```sql -CREATE INDEX idx_point_data_geom ON point_data USING GIST(geom); -CREATE INDEX idx_layer_data_site_geom ON site USING GIST(geom); -``` - -## Summary - -By systematically moving spatial operations to PostGIS: -1. Lambda stays lightweight and fast -2. Database does what it's optimized for -3. Spatial queries scale efficiently -4. No dependency hell in serverless environment - -**Next Steps**: -1. Implement raster WKT support -2. Add distance-based queries -3. Add bounding box queries -4. Consider advanced features based on user needs diff --git a/.agents/implement-mcp-server-improvements.md b/.agents/implement-mcp-server-improvements.md deleted file mode 100644 index a3dbd881..00000000 --- a/.agents/implement-mcp-server-improvements.md +++ /dev/null @@ -1,235 +0,0 @@ -# Implementation Summary: MCP Server Improvements - ---- -**Date:** 2026-03-16 -**Author:** AI Assistant (Claude Sonnet 4.6) -**Status:** Complete -**Plan Reference:** [plan-mcp-server-improvements.md](plan-mcp-server-improvements.md) - ---- - -## Overview - -Completed all four planned improvement phases for the `snowexsql` MCP server -(`snowexsql/mcp_server.py`). The server is now more robust, safer by default, -and significantly more LLM-friendly. - -**Implementation Duration:** 2026-03-16 (single session) - -**Final Status:** βœ… Complete - -## Plan Adherence - -**Plan Followed:** [plan-mcp-server-improvements.md](plan-mcp-server-improvements.md) - -**Deviations from Plan:** - -- **Deviation 1:** Test file created at `tests/unit/test_mcp_server.py` instead - of `tests/test_mcp_server.py`. - - **Reason:** The root `tests/conftest.py` has `autouse=True` on a - `db_session` fixture that tries to connect to a live PostgreSQL instance. - This would fail for all MCP unit tests since they need no DB. A - `tests/unit/` subdirectory with its own `conftest.py` overriding the DB - fixtures cleanly isolates unit tests from the DB test infrastructure. - - **Impact:** None on functionality; adds a `tests/unit/` directory and - `tests/unit/conftest.py` as two additional created files. - -- **Deviation 2:** `snowex_get_layer_sites` `filters: dict | None` parameter - also removed (not explicitly in Phase 2 scope, but caught by the plan's - success criterion "no opaque dict params remain"). - - **Reason:** The success criterion was absolute; `get_layer_sites` was the - only remaining tool with a `filters: dict` parameter after Phase 2. - - **Impact:** `get_layer_sites` now only accepts `site_names`; callers can - no longer pass arbitrary filter kwargs. This is acceptable because the - `get_sites` Lambda endpoint has limited filtering support anyway. - -## Phases Completed - -### Phase 1: Fix `verbose` Wiring and Default Limit -- βœ… **Status:** Complete -- **Completion Date:** 2026-03-16 -- **Summary:** Added `filters['verbose'] = verbose` and - `filters.setdefault('limit', 100)` before the `from_filter()` call in - `snowex_query_measurements`. Updated docstring. Note: Phase 1 changes were - subsequently superseded by the Phase 2 rewrite (which folds them in - directly), but Phase 1 was valid as an intermediate state. - -### Phase 2: Replace `filters: dict` with Explicit Parameters -- βœ… **Status:** Complete -- **Completion Date:** 2026-03-16 -- **Summary:** Rewrote `snowex_query_measurements`, `snowex_spatial_query`, - and `snowex_get_unique_values` to use explicit named parameters. Each tool - now builds its internal filters dict from non-`None` kwargs. `measurement_type` - maps to the `'type'` filter key. Also removed the `filters` dict from - `snowex_get_layer_sites` (deviation above). - -### Phase 3: Add `snowex_discover` Tool and Block Dates -- βœ… **Status:** Complete -- **Completion Date:** 2026-03-16 -- **Summary:** Removed `"dates"` from `METADATA_PROPERTIES`. Added an explicit - error guard in `snowex_get_metadata` for `property_name='dates'` that directs - the agent to `snowex_get_unique_values` with a scoping filter. Added the new - `snowex_discover` tool (returns types, instruments, campaigns, observers, DOIs, - units β€” and sites for layer class β€” in one call, never dates). Updated `AGENTS.md` - with approximate campaign date ranges and guidance on scoped date queries. - -### Phase 4: Write Test Suite -- βœ… **Status:** Complete -- **Completion Date:** 2026-03-16 -- **Summary:** Created `tests/unit/test_mcp_server.py` with 39 unit tests - covering all 8 tools. All tests mock `snowexsql.mcp_server.client` via - `unittest.mock.patch`. Created `tests/unit/conftest.py` to override the - DB connection fixtures and `tests/unit/__init__.py`. - -## Files Modified - -**Created:** -- `tests/unit/__init__.py` β€” Empty init for the new unit test package -- `tests/unit/conftest.py` β€” DB fixture overrides so unit tests don't need Postgres -- `tests/unit/test_mcp_server.py` β€” 39 unit tests covering all MCP tools - -**Modified:** -- `snowexsql/mcp_server.py` β€” All four phases of improvements -- `AGENTS.md` β€” Added campaign date ranges, scoped date query guidance, - `all_dates` warning - -**Deleted:** -No files deleted. - -## Key Changes Summary - -1. **`snowex_query_measurements` (mcp_server.py)** - - Signature: replaced opaque `filters: dict` with 13 explicit named params - - `measurement_type` parameter maps to `'type'` filter key internally - - `verbose` is always passed; `limit` defaults to 100 - - Files: `snowexsql/mcp_server.py:81-150` - -2. **`snowex_spatial_query` (mcp_server.py)** - - Signature: replaced `filters: dict | None` with 10 explicit named params - - `measurement_type` β†’ `'type'` mapping; `limit` defaults to 100 - - Files: `snowexsql/mcp_server.py:205-310` - -3. **`snowex_get_unique_values` (mcp_server.py)** - - Signature: replaced `filters: dict | None` with 8 explicit named params - - `limit` defaults to 1000 (higher default for unique-value discovery) - - Files: `snowexsql/mcp_server.py:315-415` - -4. **`snowex_discover` (mcp_server.py, new tool)** - - Combined metadata discovery in one call; no `all_dates` - - Per-section error handling; layer class includes Sites section - - Files: `snowexsql/mcp_server.py:205` (inserted before spatial_query) - -5. **Dates blocked in `snowex_get_metadata` (mcp_server.py)** - - `"dates"` removed from `METADATA_PROPERTIES` - - Explicit error guard returns helpful redirect message - - Files: `snowexsql/mcp_server.py:20-29, 155-170` - -6. **`AGENTS.md` update** - - Campaigns section now has approximate date ranges per campaign - - `all_dates` annotated with warning about full-table scan - - Scoped date query examples added - -## Verification Results - -### Automated Verification - -- βœ… `python -m pytest tests/unit/test_mcp_server.py -v` β€” 39 passed, 0 failed -- βœ… `python -c "... assert 'measurement_type' in sig.parameters ..."` β€” prints `OK` -- βœ… `python -c "from snowexsql.mcp_server import snowex_discover; print('OK')"` β€” prints `OK` -- βœ… No `dict` type annotation on any tool parameter remains (only internal variable annotations) - -**Command Output:** -``` -============================= test session starts ============================== -platform linux -- Python 3.12.6, pytest-8.4.1 -collected 39 items -... 39 passed in 1.56s ============================== -``` - -### Manual Verification - -- ⏸️ Start MCP server and inspect tool schema β€” pending (requires MCP client) -- ⏸️ `snowex_query_measurements(measurement_class='point')` with no filters β€” pending (requires live Lambda) -- ⏸️ `verbose=True` vs `verbose=False` column difference β€” pending (requires live Lambda) -- ⏸️ `snowex_discover(measurement_class='point')` with real data β€” pending -- ⏸️ `snowex_spatial_query` with UTM point + buffer β€” pending - -## Issues Encountered - -### Issue 1: Test DB Fixtures Block Unit Tests -- **Impact:** All 39 unit tests failed at setup because `conftest.py` has an - `autouse=True` `db_session` fixture that tries to connect to Postgres. -- **Resolution:** Moved test file to `tests/unit/` subdirectory with a local - `conftest.py` that overrides `sqlalchemy_engine`, `connection`, and - `db_session` to no-ops. -- **Files Affected:** `tests/unit/conftest.py` (created) - -### Issue 2: `grep 'filters: dict'` Matches Internal Variables -- **Impact:** The plan's grep success check matched local variable annotations - (`filters: dict = {...}`) in addition to the one remaining opaque parameter - (`snowex_get_layer_sites`). -- **Resolution:** Removed the `filters` parameter from `snowex_get_layer_sites` - (the only real opaque-dict parameter remaining). Internal variable annotations - are benign false positives for the grep check. -- **Files Affected:** `snowexsql/mcp_server.py` - -## Testing Summary - -**Tests Added:** -- `tests/unit/test_mcp_server.py:TestSnowExTestConnection` β€” 3 tests for connection tool -- `tests/unit/test_mcp_server.py:TestListMeasurementTypes` β€” 2 tests -- `tests/unit/test_mcp_server.py:TestSnowExQueryMeasurements` β€” 9 tests including verbose, limit, type mapping -- `tests/unit/test_mcp_server.py:TestSnowExGetMetadata` β€” 5 tests including dates guard -- `tests/unit/test_mcp_server.py:TestSnowExSpatialQuery` β€” 5 tests including WKT modes -- `tests/unit/test_mcp_server.py:TestSnowExGetUniqueValues` β€” 5 tests -- `tests/unit/test_mcp_server.py:TestSnowExGetLayerSites` β€” 4 tests -- `tests/unit/test_mcp_server.py:TestSnowExDiscover` β€” 6 tests including partial failure - -**Test Coverage:** -- Unit tests: 39 tests across all 8 MCP tools -- Integration tests: 0 new (existing Lambda integration tests in `tests/deployment/` cover the client layer) -- Edge cases tested: invalid measurement class, Lambda exceptions, `dates` blocked, POINT without buffer, partial `all_*` failure, `measurement_type` β†’ `type` mapping - -**All Tests Passing:** βœ… Yes (39/39) - -## Performance Observations - -Performance was not a primary concern for this implementation. The `all_dates` -removal (Phase 3) is a significant performance protection β€” it prevents agents -from accidentally triggering a full-table scan on the 29 GB+ points table. - -## Documentation Updated - -- βœ… `AGENTS.md` β€” Added campaign date ranges table, `all_dates` warning, - scoped date query examples in both MCP and direct client forms -- βœ… `snowexsql/mcp_server.py` β€” All tool docstrings updated to reflect new - parameter signatures; `snowex_get_metadata` docstring updated to note dates - exclusion; `snowex_discover` docstring explains orientation use case - -## Remaining Work - -All planned work has been completed. Manual verification steps remain pending -(require live Lambda access and an MCP client). - -## Next Steps - -1. Complete manual verification (see Manual Verification section above) -2. Run `/validate .agents/plan-mcp-server-improvements.md` for systematic validation -3. Create git commit: `/commit` -4. Create pull request: `/pr` - -**Recommended Actions:** -- Manually test the MCP server with a live Lambda connection -- Verify LLM tool schema visibility in Claude Desktop or another MCP client - -## References - -**Plan Document:** -- [Plan: MCP Server Improvements](plan-mcp-server-improvements.md) - -**Research Documents:** -- [Research: MCP Server and Agent Documentation](research-mcp-server-and-agent-documentation.md) - ---- - -**Implementation completed by AI Assistant (Claude Sonnet 4.6) on 2026-03-16** diff --git a/.agents/initial-setup-snowexsqlbot.md b/.agents/initial-setup-snowexsqlbot.md deleted file mode 100644 index 5ef8eaed..00000000 --- a/.agents/initial-setup-snowexsqlbot.md +++ /dev/null @@ -1,20 +0,0 @@ -I would like to build a prototype of https://github.com/uw-ssec/llmaven. - -I will be using these RSE plugins with Claude Code command line: https://github.com/uw-ssec/rse-plugins and following the "research", "plan", "implement" protocol. - -My prototype example is going to be the NASA SnowEx mission. I am a core developer for the API and SQL database currently posted here: https://github.com/SnowEx/snowexsql. My goal is to showcase a "snowexsql-bot" that can answer questions about the mission, and that also enables plain language querying of the database. - -Here is my general plan: - -* gather all relevant scientific and technical literature on snowex and the associated open source software and datasets surrounding it -* build an MCP server to establish protocols for database querying (this is something I already initiated in a local branch) -* build agents and skills to set guardrails on the capabilities and components of the LLM -* build a custom RAG LLM to be deployed locally, using open source weightings on hugging face -* implement the custom LLM via the SSEC recommended agentic RAG approach -* test, iterate and improve -* deploy to AWS - -My resources include free AWS credits, a Windows 11 laptop with 16 GB RAM and using WSL2 in VSCode. - -Help me get set up for my first prompt to Claude within the RSE "research" plugin and let me know if I've missed anything important. - diff --git a/.agents/mcp-research-prompt.md b/.agents/mcp-research-prompt.md deleted file mode 100644 index 67f061c1..00000000 --- a/.agents/mcp-research-prompt.md +++ /dev/null @@ -1,147 +0,0 @@ -# SnowExSQL Bot β€” Research Prompt: MCP Server + Agent Documentation - -## Instructions - -Run this in Claude Code after installing the RSE plugins and checking out -your `minimal-mcp` branch: - -```bash -/plugin marketplace add uw-ssec/rse-plugins -``` - -Make sure your `minimal-mcp` branch is checked out so Claude Code can -inspect the existing work. - ---- - -## The Prompt - -``` -/research - -## Topic: SnowExSQL MCP Server and Agent Documentation - -### Context -I am a core developer of snowexsql (https://github.com/SnowEx/snowexsql), -the Python client library for accessing NASA SnowEx campaign data stored -in a PostgreSQL/PostGIS database on AWS. - -I have a `minimal-mcp` branch where I've started building an MCP server -that wraps the snowexsql Lambda Client. I need two things from this -research session: - -1. A thorough understanding of the snowexsql database schema, Lambda - Client interface, and valid query parameters β€” documented as a - durable agent context file that can live in the snowexsql repo -2. An assessment of my existing MCP server work on the `minimal-mcp` - branch, with recommendations for completing it - -### CRITICAL ARCHITECTURE CONSTRAINT -All database access goes through an AWS Lambda Client. There is no -direct database access permitted. The chain is: - - MCP Server β†’ Lambda Client β†’ AWS Lambda β†’ PostgreSQL/PostGIS DB - -No raw SQL is generated. No direct SQLAlchemy sessions are established -by the caller. - -### Research Area 1: Document the Database and Lambda Client for Agents - -The goal is to produce a comprehensive reference document (suitable for -committing as AGENTS.md or similar in the snowexsql repo) that any AI -agent or coding assistant can read once instead of re-discovering the -schema and API every session. This document should cover: - -#### Database Schema -- All database tables: points, layers, etc. -- Every column in each table with its type, meaning, and constraints -- Which columns are filterable via the API -- The geometry/spatial columns and their SRID/coordinate system -- Relationships between tables (e.g., how site_id links sites to - measurements, how spatial joins work across tables) - -#### Lambda Client Interface -- Inspect my local branch code and document the complete API surface -- Every method/function the Lambda Client exposes -- Parameters for each method: name, type, required vs optional, - valid values -- Return types and serialization format (JSON, GeoDataFrame, etc.) -- Authentication: how the client authenticates with AWS (IAM roles, - credentials, environment variables) -- Error handling: what errors can be returned, timeout behavior -- Any rate limits, payload size constraints, or cold start implications - -#### Valid Parameter Catalog -Generate (or document how to generate) a catalog of valid enum-like -values for key filter parameters. These are the values a user or agent -needs to know to construct valid queries: -- All valid `type` values per table (e.g., "depth", "swe", "density") -- All valid `instrument` values (e.g., "pit ruler", "magnaprobe", "mesa") -- All valid `observers`/`surveyors` values (e.g., "ASO Inc.", - "UAVSAR team, JPL", "USGS") -- All valid `site_name` values (e.g., "Grand Mesa") -- Sample `site_id` values and their naming convention -- Available date ranges per campaign/dataset -- Any other parameters with constrained valid values - -#### Example Query Patterns -Document 10-15 representative queries that researchers actually perform, -showing the mapping from research intent to Lambda Client call. Draw from: -- Snow observations cookbook (https://projectpythia.org/snow-observations-cookbook/) -- Common patterns visible in the codebase - -Categories to cover: -- Simple filtered queries (single table, one or two filters) -- Date range queries -- Spatial queries (point + buffer, polygon area) -- Discovery queries (what instruments/types/dates are available?) -- Cross-table queries (e.g., point measurements near a raster footprint) -- Raster-specific queries (with their special constraints) - -### Research Area 2: Assess and Plan the MCP Server - -Examine my `minimal-mcp` branch and evaluate what exists: - -#### Current State Assessment -- What MCP tools are already defined? -- What works, what's stubbed out, what's missing? -- How does it currently invoke the Lambda Client? -- What's the current project structure? - -#### MCP Best Practices -- Review the MCP specification for tool design patterns -- How should tools be granular vs. composite? (one tool per table? - one tool per query pattern? a single flexible query tool?) -- Naming conventions for tools and parameters -- How to write good tool descriptions so an LLM knows when/how - to use each tool -- Input validation: what should the MCP server validate before - invoking Lambda? -- Error responses: how to surface Lambda errors through MCP - -#### Gap Analysis and Recommendations -- What tools need to be added to cover the query patterns from - Research Area 1? -- Is a "discovery" tool needed for parameter exploration? -- How should spatial queries be handled (coordinate input format, - buffer specification)? -- How should results be formatted for display in a chat interface? -- Should raster queries return metadata only, or attempt to return - data? What are the payload size implications? -- Testing strategy: how to test MCP tools against the Lambda Client - -### Output Requirements - -Produce TWO documents: - -1. **Agent Context Document** β€” A self-contained reference file suitable - for committing to the snowexsql repo (as AGENTS.md or - docs/agent_context.md). It should be complete enough that any AI - agent reading it can construct valid queries without further - exploration. Include the schema, Lambda Client API, parameter - catalog, and example patterns. - -2. **MCP Server Assessment and Plan** β€” An evaluation of the - minimal-mcp branch with a concrete list of what to build next, - ordered by priority. This feeds into the /plan phase. -``` \ No newline at end of file diff --git a/.agents/plan-mcp-server-improvements.md b/.agents/plan-mcp-server-improvements.md deleted file mode 100644 index 71969361..00000000 --- a/.agents/plan-mcp-server-improvements.md +++ /dev/null @@ -1,621 +0,0 @@ -# Implementation Plan: MCP Server Improvements - ---- -**Date:** 2026-03-16 -**Author:** AI Assistant (Claude Sonnet 4.6) -**Status:** Draft -**Related Documents:** -- [Research: MCP Server and Agent Documentation](research-mcp-server-and-agent-documentation.md) - ---- - -## Overview - -The `snowexsql` MCP server (`snowexsql/mcp_server.py`) is already functional -with 7 tools covering the core point and layer query workflows. This plan -addresses four quality gaps identified in the research phase, then adds a test -suite. - -The primary motivation is LLM usability: the current `filters: dict` parameter -on `snowex_query_measurements` is opaque β€” an agent cannot discover valid keys -from the tool schema alone and must either guess or call a discovery tool first. -Converting to explicit keyword parameters makes the tool self-documenting. -Secondary fixes address a dangling `verbose` parameter and missing limit guard. -A new combined discovery tool reduces the round-trips an agent needs before -querying. - -**Goal:** A fully tested MCP server where every tool parameter is -schema-discoverable, queries are safe by default (bounded limit), and the -`verbose` flag works end-to-end. - -**Motivation:** Agents interacting with the MCP server should be able to -construct valid queries from the tool schema alone without consulting external -documentation. The current opaque `filters: dict` pattern defeats this. - ---- - -## Current State Analysis - -**Existing Implementation:** -- `snowexsql/mcp_server.py:82-118` β€” `snowex_query_measurements`: accepts - `filters: dict` (opaque), `verbose: bool` (accepted but not passed through) -- `snowexsql/mcp_server.py:160-212` β€” `snowex_spatial_query`: accepts - `filters: dict | None` for supplementary filters -- `snowexsql/mcp_server.py:216-245` β€” `snowex_get_unique_values`: accepts - `filters: dict | None` -- `snowexsql/mcp_server.py:18` β€” `MEASUREMENT_CLASSES = ["point", "layer"]` -- `snowexsql/mcp_server.py:20-29` β€” `METADATA_PROPERTIES` list - -**Current Behavior:** -- `snowex_query_measurements` accepts a `verbose` parameter but the body - calls `dataset.from_filter(**filters)` without including `verbose`, so the - flag has no effect (`mcp_server.py:114`). -- No default `limit` is applied. An agent that omits `limit` from the - `filters` dict on a large table will receive a `LargeQueryCheckException` - from the Lambda. -- Filter keys (`type`, `instrument`, `campaign`, etc.) are only documented in - the docstring, not in the tool schema. MCP clients that surface the schema - as JSON Schema (including Claude.ai) do not expose docstring content as - parameter-level hints. -- No tool returns all metadata categories in a single call; an agent needs - multiple `snowex_get_metadata` calls to orient itself. -- No test file exists for `mcp_server.py`. - -**Current Limitations:** -- `verbose=True` silently does nothing -- Queries without `limit` can raise exceptions rather than returning results -- `filters: dict` is undiscoverable from the tool schema -- No combined discovery path; multiple round-trips required for orientation -- Zero test coverage on the MCP layer - ---- - -## Desired End State - -**New Behavior:** -- `snowex_query_measurements` has named parameters for every valid filter - (`measurement_type`, `instrument`, `campaign`, `date`, - `date_greater_equal`, `date_less_equal`, `observer`, `doi`, - `value_greater_equal`, `value_less_equal`, `site`, `limit`, `verbose`). - Each has a type annotation and default. The tool schema is fully - self-documenting. -- `verbose=True` correctly triggers denormalized output (instrument name, - campaign name, observer, etc.) via the Lambda handler's verbose path. -- Queries default to `limit=100` if not specified; agents can raise or lower - this explicitly. -- `snowex_spatial_query` and `snowex_get_unique_values` similarly replace - their `filters: dict | None` with named optional parameters. -- A new `snowex_discover(measurement_class)` tool returns all metadata - categories (types, instruments, campaigns, observers, DOIs, units) - in a single formatted string. Dates are deliberately excluded (see below). -- `snowex_get_metadata(..., property_name='dates')` is blocked with an error - message directing agents to use `snowex_get_unique_values` with a scoping - filter (campaign, site, or instrument) instead. -- `AGENTS.md` includes approximate campaign date ranges as static facts so - agents can orient temporally without any query. -- `tests/test_mcp_server.py` provides unit test coverage for all tools using - mocked Lambda client calls. - -**Success Looks Like:** -- An LLM with only the tool schema (no docstring) can construct a valid - `snowex_query_measurements` call -- `snowex_query_measurements(..., verbose=True)` returns more columns than - `verbose=False` -- `snowex_query_measurements('point')` with no other arguments returns up to - 100 results rather than raising an exception -- `snowex_discover('point')` returns a formatted string covering types, - instruments, campaigns, observers, DOIs, and units β€” but not dates -- `pytest tests/test_mcp_server.py -v` passes with no failures - ---- - -## What We're NOT Doing - -- **Raster/image data tools** β€” `RasterMeasurements` is being downgraded; no - raster tools will be added to the MCP server -- **Changing the Lambda Client or api.py** β€” All changes are confined to - `mcp_server.py` and the new test file -- **Changing `snowex_get_layer_sites`** β€” No `filters: dict` issue; signature stays the same -- **Exposing `all_dates` through any MCP tool** β€” Unscoped date queries - on the points table are a full-table distinct scan on 29 GB of data. No - agent use case requires every date ever recorded; agents should always - scope date discovery to a campaign, site, or instrument via - `snowex_get_unique_values`. `BaseDataset.all_dates` is left intact for - direct API users who know what they are doing. -- **Materialized views or other DB infrastructure changes** β€” Out of scope - for this plan; the date problem is solved by simply not offering the - unscoped query through MCP -- **MCP protocol-level tests** β€” Unit tests mock the client; no full - MCP protocol round-trip tests -- **Map/visualization tools** β€” Out of scope (discussed but deferred) - -**Rationale:** Keeping changes confined to `mcp_server.py` minimises risk. -The Lambda Client API is the correct abstraction boundary; the MCP server -should adapt to it, not the other way around. - ---- - -## Implementation Approach - -**Technical Strategy:** -All changes are in `snowexsql/mcp_server.py`. The Lambda Client API is -unchanged. Each fix is independently testable. - -For the signature expansion (Phase 2), `snowex_query_measurements` will -build the `filters` dict internally from the named parameters and pass it to -`dataset.from_filter(**filters)`. The key mapping is: - -| MCP parameter | filters dict key | Notes | -|---------------------|----------------------|------------------------------------------------| -| `measurement_type` | `type` | Renamed to avoid shadowing Python builtin | -| `instrument` | `instrument` | Direct mapping | -| `campaign` | `campaign` | Direct mapping | -| `date` | `date` | Direct mapping | -| `date_greater_equal`| `date_greater_equal` | Direct mapping | -| `date_less_equal` | `date_less_equal` | Direct mapping | -| `observer` | `observer` | Direct mapping | -| `doi` | `doi` | Direct mapping | -| `value_greater_equal`| `value_greater_equal`| Direct mapping | -| `value_less_equal` | `value_less_equal` | Direct mapping | -| `site` | `site` | Layer-only; ignored by point queries | -| `limit` | `limit` | Direct mapping; default 100 | -| `verbose` | `verbose` | Extracted by lambda_handler before forwarding | - -The `verbose` key is passed inside the filters dict because -`lambda_handler._handle_class_action` extracts it via -`filters.pop('verbose', False)` at `lambda_handler.py:220` before forwarding -the remaining filters to the API class. - -**Key Architectural Decisions:** - -1. **Decision:** Rename `type` β†’ `measurement_type` in the MCP parameter name - - **Rationale:** `type` is a Python builtin; using it as a parameter name - would shadow it and cause linting warnings - - **Trade-offs:** Agents see `measurement_type` in the schema but the - underlying API filter key is `type`; the mapping is handled internally - - **Alternatives considered:** Keeping `type` as the parameter name β€” works - at runtime but is bad practice and confuses linters - -2. **Decision:** Apply `verbose` by including it in the filters dict passed - to `from_filter`, not as a separate argument - - **Rationale:** `_LambdaDatasetClient.from_filter()` does not accept - `verbose` directly; the Lambda handler extracts it from `filters` - (`lambda_handler.py:220`) - - **Trade-offs:** Slightly unintuitive that `verbose` goes through `filters`; - but requires no changes outside `mcp_server.py` - - **Alternatives considered:** Modifying `_LambdaDatasetClient` β€” rejected - to keep changes confined to the MCP layer - -3. **Decision:** Expand `filters: dict | None` in `snowex_spatial_query` and - `snowex_get_unique_values` as well as `snowex_query_measurements` - - **Rationale:** Consistency β€” an agent should not need to use dict syntax - in some tools and named params in others - - **Trade-offs:** Slightly more code; but the filter set for spatial/unique - queries is a subset of the main query filters - - **Alternatives considered:** Leaving spatial/unique tools unchanged β€” - rejected for consistency - -**Patterns to Follow:** -- Existing `@mcp.tool()` decorator pattern β€” see `mcp_server.py:69-78` -- Error return pattern (`return f"Error: {e}"`) β€” see `mcp_server.py:117` -- `_df_to_json(df)` for serialisation β€” see `mcp_server.py:32-42` -- Mocking pattern for Lambda client in tests β€” see - `tests/deployment/test_lambda_client.py:36-39` - ---- - -## Implementation Phases - -### Phase 1: Fix Verbose Wiring and Add Default Limit - -**Objective:** Two targeted one-line fixes to `snowex_query_measurements` that -correct immediately observable bugs without changing the tool signature. - -**Tasks:** - -- [x] Wire `verbose` into the filters dict before calling `from_filter` - - File: `snowexsql/mcp_server.py:113-115` - - Changes: Before `df = dataset.from_filter(**filters)`, add - `filters['verbose'] = verbose` - -- [x] Add default limit guard - - File: `snowexsql/mcp_server.py:113-115` - - Changes: Before the `from_filter` call, add - `filters.setdefault('limit', 100)` - -- [x] Update the `snowex_query_measurements` docstring to reflect the new - default behaviour - - File: `snowexsql/mcp_server.py:87-110` - - Changes: Update the `limit` description from "ALWAYS set this" to - "Max number of records (default 100)" - -**Dependencies:** None. - -**Verification:** -- [ ] Call `snowex_query_measurements('point', {}, verbose=True)` in a Python - REPL with the live Lambda β€” result should have more columns than - `verbose=False` -- [ ] Call `snowex_query_measurements('point', {})` with no limit β€” should - return up to 100 records, not raise `LargeQueryCheckException` - ---- - -### Phase 2: Replace `filters: dict` with Explicit Parameters - -**Objective:** Rewrite the three tools that currently accept opaque `dict` -parameters so that every valid filter key is a named, typed parameter visible -in the MCP tool schema. - -**Tasks:** - -- [x] Rewrite `snowex_query_measurements` signature and body - - File: `snowexsql/mcp_server.py:81-118` - - New signature (all filter params optional with `None` default except - `limit=100` and `verbose=False`): - ```python - def snowex_query_measurements( - measurement_class: str, - measurement_type: str | None = None, - instrument: str | None = None, - campaign: str | None = None, - date: str | None = None, - date_greater_equal: str | None = None, - date_less_equal: str | None = None, - observer: str | None = None, - doi: str | None = None, - value_greater_equal: float | None = None, - value_less_equal: float | None = None, - site: str | None = None, - limit: int = 100, - verbose: bool = False, - ) -> str: - ``` - - Body: build `filters` dict from non-`None` params, mapping - `measurement_type` β†’ `'type'`; always include `limit` and `verbose` - - Note: Phase 1 fixes (`verbose` wiring, default limit) are naturally - superseded by this rewrite; the Phase 1 intermediate state is still - valid and can be left or replaced cleanly - -- [x] Rewrite `snowex_spatial_query` to expand its supplementary `filters` - - File: `snowexsql/mcp_server.py:160-212` - - Replace `filters: dict | None = None` with the same named optional params - (excluding `site` which is layer-only and less relevant for spatial - queries, but can be included for completeness): - `measurement_type`, `instrument`, `campaign`, `date`, - `date_greater_equal`, `date_less_equal`, `observer`, `doi`, - `value_greater_equal`, `value_less_equal`, `limit` - - Body: build `query_filters` dict from non-`None` params; pass as - `**query_filters` to `dataset.from_area(...)` - -- [x] Rewrite `snowex_get_unique_values` to expand its supplementary `filters` - - File: `snowexsql/mcp_server.py:216-245` - - Replace `filters: dict | None = None` with named optional params: - `measurement_type`, `instrument`, `campaign`, `date`, - `date_greater_equal`, `date_less_equal`, `observer`, `doi`, `limit` - - Body: build `query_filters` dict from non-`None` params; pass as - `**query_filters` to `dataset.from_unique_entries(columns, ...)` - -- [x] Update all three docstrings to reflect the new parameter list and drop - any reference to passing a `filters` dict - -**Dependencies:** Phase 1 (or Phase 1 changes are folded in directly here). - -**Verification:** -- [ ] Run `python -c "from snowexsql.mcp_server import snowex_query_measurements; import inspect; print(inspect.signature(snowex_query_measurements))"` β€” should show all named parameters -- [ ] No `dict` type annotation remains on any of the three rewritten tools - ---- - -### Phase 3: Add `snowex_discover` Tool - -**Objective:** Add a single combined-discovery tool that returns all metadata -categories for a measurement class in one call, reducing agent round-trips. - -**Tasks:** - -- [x] Block `dates` in `snowex_get_metadata` - - File: `snowexsql/mcp_server.py` β€” inside `snowex_get_metadata`, add a - guard before the existing `property_name not in METADATA_PROPERTIES` check: - ```python - if property_name == "dates": - return ( - "Error: unscoped date queries are disabled (full-table scan on " - "29 GB data). Use snowex_get_unique_values with a campaign, site, " - "or instrument filter instead. Example: " - "snowex_get_unique_values('point', ['date'], campaign='SnowEx20')" - ) - ``` - - Remove `"dates"` from `METADATA_PROPERTIES` at `mcp_server.py:20-29` - so it does not appear as a valid option in the tool description - -- [x] Add `snowex_discover` function and `@mcp.tool()` decorator - - File: `snowexsql/mcp_server.py` β€” insert after `snowex_get_metadata` - - Tool description should clearly state this is for initial orientation, - distinct from `snowex_get_metadata` (which is per-property), and that - date ranges are intentionally omitted (use `snowex_get_unique_values` - scoped by campaign/site for dates) - - Calls `all_types`, `all_instruments`, `all_campaigns`, `all_observers`, - `all_dois`, `all_units` β€” **not** `all_dates` - - Assembles a formatted string with section headers: - ``` - ## Types - depth - swe - ... - - ## Instruments - magnaprobe - ... - - ## Campaigns - SnowEx20 - ... - ``` - - For `layer` class, also include `## Sites` from `all_sites` - - Handle errors per-section (if one `all_*` call fails, report the error - for that section and continue) - -- [x] Add campaign date ranges to `AGENTS.md` - - File: `AGENTS.md` β€” in the "Valid Parameter Catalog β†’ Campaigns" section - - Add approximate date ranges for each known campaign (e.g. - "SnowEx20: Jan–Feb 2020, Grand Mesa CO") so agents can orient - temporally without any query - - Note in the section that fine-grained date discovery should use - `snowex_get_unique_values` scoped by campaign or site - -**Dependencies:** Phase 2 (schema is stable before adding new tools). - -**Verification:** -- [ ] `snowex_discover('point')` returns a multi-section string with - Types, Instruments, Campaigns, Observers, DOIs, Units β€” no Dates section -- [ ] `snowex_discover('layer')` additionally includes a Sites section -- [ ] `snowex_discover('invalid')` returns a clear error string - ---- - -### Phase 4: Write Test Suite - -**Objective:** Create `tests/test_mcp_server.py` with unit tests for all -tools. Tests mock `snowexsql.mcp_server.client` so no network calls are made. - -**Tasks:** - -- [x] Create `tests/test_mcp_server.py` - - File: `tests/test_mcp_server.py` (new file) - - Import all tool functions directly from `snowexsql.mcp_server` - - Use `unittest.mock.patch('snowexsql.mcp_server.client', ...)` as a - fixture or context manager - -- [x] Write tests for `snowex_test_connection` - - Test: connected=True returns success string with version - - Test: connected=False returns failure string - - Test: exception returns error string - -- [x] Write tests for `list_measurement_types` - - Test: merges point and layer types, returns sorted deduplicated list - - Test: returns newline-separated string - -- [x] Write tests for `snowex_query_measurements` - - Test: valid `measurement_class='point'` calls `from_filter` with correct - kwargs (verify `filters['type']` is set when `measurement_type` is passed) - - Test: `measurement_type` is mapped to `type` key in the filters dict - - Test: `verbose=True` passes `verbose=True` in filters - - Test: default `limit=100` is applied when not specified - - Test: explicit `limit` overrides the default - - Test: invalid `measurement_class` returns error string - - Test: Lambda exception returns error string - - Test: DataFrame result is returned as JSON string - -- [x] Write tests for `snowex_get_metadata` - - Test: valid property calls correct `all_*` attribute - - Test: invalid property name returns error string - - Test: `property_name='dates'` returns error string directing agent to - `snowex_get_unique_values` with a filter - - Test: `sites` on `point` class returns error string - - Test: `sites` on `layer` class returns newline-separated list - -- [x] Write tests for `snowex_spatial_query` - - Test: POINT WKT without buffer returns error string - - Test: POINT WKT with buffer calls `from_area(pt=..., buffer=..., crs=...)` - - Test: POLYGON WKT calls `from_area(shp=..., crs=...)` - - Test: supplementary filter params are passed through correctly - - Test: missing shapely import returns helpful error string - -- [x] Write tests for `snowex_get_unique_values` - - Test: calls `from_unique_entries(columns, ...)` with correct args - - Test: result is returned as JSON string - - Test: filter params passed through correctly - -- [x] Write tests for `snowex_get_layer_sites` - - Test: `site_names=None` calls `get_sites()` with no name filter - - Test: list of names calls `get_sites(site_names=[...])` - - Test: exception returns error string - -- [x] Write tests for `snowex_discover` - - Test: `'point'` returns string containing "## Types" and "## Instruments" - - Test: `'point'` result does NOT contain "## Dates" - - Test: `'layer'` returns string additionally containing "## Sites" - - Test: invalid class returns error string - - Test: partial failure (one `all_*` raises) still returns other sections - -**Dependencies:** Phases 1–3 (tests cover the final tool signatures). - -**Verification:** -- [ ] `pytest tests/test_mcp_server.py -v` passes with no failures -- [ ] No test makes a network call (verify with `pytest --co -q` and inspect - that no `requests` calls are made) - ---- - -## Success Criteria - -### Automated Verification - -- [ ] `pytest tests/test_mcp_server.py -v` passes with no failures -- [ ] `pytest tests/ -v -m "not integration and not handler"` passes - (existing tests unbroken) -- [ ] `python -c "from snowexsql.mcp_server import snowex_query_measurements; import inspect; sig = inspect.signature(snowex_query_measurements); assert 'measurement_type' in sig.parameters; assert 'instrument' in sig.parameters; assert 'limit' in sig.parameters; print('OK')"` prints `OK` -- [ ] `python -c "from snowexsql.mcp_server import snowex_discover; print('OK')"` prints `OK` -- [ ] `grep -n 'filters: dict' snowexsql/mcp_server.py` returns no matches - (all opaque dict params replaced) - -### Manual Verification - -- [ ] Start the MCP server (`snowexsql-mcp`) and inspect it with an MCP - client β€” `snowex_query_measurements` tool schema shows individual filter - parameters, not a `filters` object -- [ ] Call `snowex_query_measurements(measurement_class='point')` with no - other parameters β€” returns up to 100 records as JSON, no exception -- [ ] Call `snowex_query_measurements(measurement_class='point', verbose=True, limit=3)` β€” result JSON has more keys than the same call with `verbose=False` -- [ ] Call `snowex_discover(measurement_class='point')` β€” returns multi-section - text with real data from the live database -- [ ] Call `snowex_spatial_query(measurement_class='point', geometry_wkt='POINT (743683 4321095)', buffer=500.0)` β€” returns JSON records or empty array, no exception - ---- - -## Testing Strategy - -**Unit Tests** (`tests/test_mcp_server.py`): -- Mock `snowexsql.mcp_server.client` entirely -- Each tool function is tested by calling it directly -- Verify correct delegation to the Lambda client mock -- Verify correct error handling (exceptions become error strings) -- Verify output format (JSON strings, newline-separated strings) - -**Integration Tests** (existing, no new tests added here): -- Existing `tests/deployment/test_lambda_client.py` marked - `@pytest.mark.integration` covers the Lambda round-trip -- The MCP tools delegate to the same client, so Lambda integration is - already covered - -**Test Data Requirements:** -- No live database connection needed for unit tests -- Mock return values: `pd.DataFrame({'value': [1.0], 'geom': ['POINT(0 0)']})` for DataFrame-returning methods; `['depth', 'swe']` for list-returning properties - ---- - -## Migration Strategy - -**Backward Compatibility:** -The tool signature change in Phase 2 is a **breaking change to the MCP tool -interface** β€” any agent or client that passes `filters` as a positional or -keyword argument will break. However: -- The `snowexsql-mcp` server has no versioning; breaking changes are - acceptable at this stage -- Agents that used `filters={'type': 'depth'}` will need to use - `measurement_type='depth'` instead -- The `AGENTS.md` and research docs will reflect the new signatures - -**Rollback Plan:** The branch is `minimal-mcp`. If issues arise, revert -`mcp_server.py` to the pre-Phase-2 state. The Lambda Client is unchanged -throughout. - ---- - -## Risk Assessment - -**Potential Risks:** - -1. **Risk:** Phase 2 signature change breaks an existing user's integration - - **Likelihood:** Low (server is new and not yet publicly documented with - the old signature) - - **Impact:** Medium - - **Mitigation:** The change is on a feature branch; document the new - signatures clearly in commit message and `AGENTS.md` - -2. **Risk:** `verbose` key in the filters dict causes unexpected behaviour - in the Lambda handler if it reaches a code path that doesn't pop it - - **Likelihood:** Low (`lambda_handler.py:220` always pops `verbose` before - forwarding filters) - - **Impact:** Low (at worst, `verbose` appears as an unrecognised filter - key and raises `ValueError` on the server side) - - **Mitigation:** Verified in `lambda_handler.py:220` that `verbose` is - always popped from filters before `_get_measurements_by_class` is called - -3. **Risk:** `snowex_discover` is slow because each `all_*` property is a - separate Lambda request - - **Likelihood:** Medium (6 HTTP round-trips; each fast in steady state, - but cold starts add latency) - - **Impact:** Low–Medium (annoying but not broken; `all_dates` was the - worst offender and is now excluded) - - **Mitigation:** The six remaining properties query small lookup tables - (campaigns, observers, instruments, types, DOIs, units) or use EXISTS - subqueries. None touch the full 29 GB points table. Document in the - tool description that multiple backend requests are made. A future - improvement could add a `get_all_metadata` batch action to - `lambda_handler.py` to collapse these to a single invocation. - ---- - -## Edge Cases and Error Handling - -**Edge Cases:** - -1. **Case:** `snowex_query_measurements` called with only `measurement_class`, - no other params - - **Expected Behavior:** Returns up to 100 records (default limit) - - **Implementation:** `filters.setdefault('limit', 100)` in Phase 1; always - included via `limit=100` default in Phase 2 - -2. **Case:** `site` parameter passed to `snowex_query_measurements` with - `measurement_class='point'` - - **Expected Behavior:** Lambda returns a `ValueError` ("site is not an - allowed filter") which surfaces as an error string to the agent - - **Implementation:** No special handling needed; the error propagates from - the Lambda and is caught by the existing `except Exception as e:` block - -3. **Case:** `snowex_discover` β€” one `all_*` property call times out - - **Expected Behavior:** That section shows an error message; other - sections still appear - - **Implementation:** Wrap each `all_*` call in its own try/except in - `snowex_discover` - -**Error Scenarios:** - -1. **Error:** Lambda timeout during `snowex_discover` - - **Handling:** Per-section try/except; failing section shows - `"(error: Request timed out...)"` inline; other sections complete - -2. **Error:** Invalid `measurement_class` string - - **Handling:** `_get_measurement_dataset()` raises `ValueError`; caught - in each tool and returned as `f"Error: {e}"` - ---- - -## Documentation Updates - -- [ ] Update `AGENTS.md` MCP server section (currently it doesn't document - MCP tools β€” no update strictly needed, but the tool signatures in the - research doc should be noted as superseded) -- [ ] Docstrings on all modified/new tool functions must be complete and - accurate after Phase 2 rewrites β€” particularly the parameter descriptions - for named filter params - ---- - -## Open Questions - -*(None β€” all decisions resolved before plan was written.)* - ---- - -## References - -**Research Documents:** -- [Research: MCP Server and Agent Documentation](research-mcp-server-and-agent-documentation.md) - -**Files Analyzed:** -- `snowexsql/mcp_server.py` -- `snowexsql/lambda_client.py` -- `snowexsql/lambda_handler.py` -- `snowexsql/api.py` -- `tests/api/test_point_measurements.py` -- `tests/deployment/test_lambda_client.py` -- `pyproject.toml` - ---- - -## Review History - -### Version 1.0 β€” 2026-03-16 -- Initial plan created diff --git a/.agents/research-mcp-server-and-agent-documentation.md b/.agents/research-mcp-server-and-agent-documentation.md deleted file mode 100644 index 3c629e68..00000000 --- a/.agents/research-mcp-server-and-agent-documentation.md +++ /dev/null @@ -1,402 +0,0 @@ -# Research: SnowExSQL MCP Server and Agent Documentation - ---- -**Date:** 2026-03-16 -**Author:** AI Assistant (Claude Sonnet 4.6) -**Status:** Active -**Branch:** `minimal-mcp` -**Related Documents:** `AGENTS.md` (repo root β€” agent context document produced alongside this) - ---- - -## Research Question - -Document the SnowExSQL database schema and Lambda Client API for agents, and -assess the existing MCP server on the `minimal-mcp` branch to identify what -works, what is missing, and what to build next. - -## Executive Summary - -The `minimal-mcp` branch contains a fully operational MCP server -(`snowexsql/mcp_server.py`) with 7 tools covering the core query patterns for -point and layer measurements. The server correctly uses `SnowExLambdaClient` -as its sole database access mechanism and is already registered as a -`pyproject.toml` entry point (`snowexsql-mcp`). - -The server is further along than a stub β€” all tools have real implementations -that reach the Lambda backend. The main gaps are: (1) the `verbose` parameter -is accepted but not passed through to `from_filter`; (2) raster/image data is -not exposed at all; (3) no default `limit` enforcement means agents can -accidentally trigger `LargeQueryCheckException`; and (4) the `filters: dict` -parameter type on the primary query tool is opaque to LLMs β€” they cannot -discover valid keys without calling a discovery tool first. - -The Agent Context Document (`AGENTS.md`) has been written to the repo root and -is the primary output of Research Area 1. - -## Scope - -**What This Research Covers:** -- The complete database schema (all tables, columns, types, relationships) -- The Lambda Client API surface: every method, parameter, and return type -- Valid filter parameter catalog with known enum-like values -- Fifteen representative example query patterns -- Full assessment of the existing MCP server (all 7 tools) -- Gap analysis and prioritized build plan for the MCP server - -**What This Research Does NOT Cover:** -- Live database content (no network requests were made) -- Deployment infrastructure details (AWS CDK, Terraform, etc.) -- Authentication for writing/uploading data (read-only client) - ---- - -## Key Findings - -### Finding 1 β€” Database Schema - -The SnowEx database uses a normalized relational schema with three data tables -and five lookup tables. See `AGENTS.md` for the full schema reference. - -**Critical relationships:** -- Layer geometry comes from the parent `sites` row (join on `site_id`), not - from `layers` itself. Any spatial query on layers must go through `sites`. -- Point geometry lives directly on the `points` row (`geom` column). -- The `campaign_observations` table uses single-table inheritance (STI) to - serve both `PointObservation` and `ImageObservation` rows via a `type` - discriminator column. - -**Relevant Files:** -- `snowexsql/tables/site.py:30-87` β€” Site model with all field condition columns -- `snowexsql/tables/layer_data.py:9-27` β€” LayerData; note `value` is `Text` -- `snowexsql/tables/point_data.py:10-34` β€” PointData; `value` is `Float` -- `snowexsql/tables/single_location.py:1-13` β€” Mixin providing `datetime`, `elevation`, `geom` -- `snowexsql/tables/campaign_observation.py:11-36` β€” STI parent -- `docs/database_structure.rst` β€” Narrative documentation with Mermaid ER diagram - -**Key Patterns:** -- All tables are in the `public` schema (`ForeignKey('public.sites.id')`) -- Sessions always run in UTC (`"-c timezone=UTC"` in `db.py`) -- `geom` columns use geoalchemy2 `Geometry("POINT")` with no SRID declared in - the model β€” the database SRID is detected at query time in `api.py:from_area()` - -### Finding 2 β€” Lambda Client Architecture - -**Relevant Files:** -- `snowexsql/lambda_client.py:21-748` β€” Complete client implementation -- `snowexsql/lambda_handler.py:197-301` β€” Server-side routing (for understanding what the Lambda actually does) - -**How It Works:** -1. `SnowExLambdaClient.__init__()` creates a `requests.Session` with retry - logic (3 attempts on 5xx) and dynamically creates dataset accessor - attributes by importing `snowexsql.api` and discovering classes ending in - `Measurements`. -2. Every accessor method call becomes a JSON POST to the Lambda Function URL: - `{"action": "PointMeasurements.from_filter", "filters": {...}}` -3. `_LambdaDatasetClient.__getattr__()` intercepts any attribute access and - routes it to either `_get_property()` (for `all_*`) or - `_create_method_proxy()` (for known methods). -4. `from_area()` is handled specially: geometries are converted to WKT strings - before transmission; PostGIS spatial filtering happens server-side. -5. Responses are deserialized to DataFrame; if a `geom` or `geometry` column - is present and geopandas is available, converted to GeoDataFrame. - -**Authentication:** None required from the caller. The Lambda Function URL is -public HTTPS. The Lambda itself authenticates to the database via -`DB_SECRET_NAME` (AWS Secrets Manager). - -**Known Timeout Risk:** The 30-second timeout can be hit on cold starts or -large property queries (`all_instruments` on the 29 GB+ points table). The -Lambda uses `EXISTS` subqueries for instrument lists to mitigate this -(`api.py:594-606`). - -### Finding 3 β€” MCP Server Current State - -**File:** `snowexsql/mcp_server.py` (301 lines, all production code) - -**Entry Point:** Registered in `pyproject.toml:61` as: -```toml -snowexsql-mcp = "snowexsql.mcp_server:main" -``` -and the `mcp` optional dependency group is defined: -```toml -mcp = ["mcp[cli]>=1.1"] -``` - -Install with: `pip install 'snowexsql[mcp]'` -Run with: `snowexsql-mcp` (calls `FastMCP.run()`) - -#### Tools Inventory - -| Tool name | Status | Description | -|------------------------------|-------------|----------------------------------------------------------| -| `list_measurement_types` | βœ… Working | Merges point + layer `all_types`, returns sorted list | -| `snowex_query_measurements` | ⚠️ Partial | Primary query; `verbose` param not wired through | -| `snowex_get_metadata` | βœ… Working | Discovery tool; routes `all_*` properties | -| `snowex_spatial_query` | βœ… Working | WKT-in / JSON-out spatial queries | -| `snowex_get_unique_values` | βœ… Working | `from_unique_entries` wrapper | -| `snowex_get_layer_sites` | βœ… Working | `get_sites()` wrapper | -| `snowex_test_connection` | βœ… Working | Health check | - -#### Tool Analysis - -**`list_measurement_types()`** β€” No parameters. Good as a quick orientation -tool. Merges both tables which is user-friendly. - -**`snowex_query_measurements(measurement_class, filters, verbose)`** β€” The -primary workhorse. Issues: -- `verbose` is accepted but not passed to `from_filter()` (`mcp_server.py:114`) -- `filters: dict` is opaque; an LLM can't know valid keys without calling - `snowex_get_metadata` first. The docstring lists them but structured - parameter definitions would be better. -- No default `limit` enforcement β€” an agent that omits `limit` in `filters` - on a large table will get a `LargeQueryCheckException`. - -**`snowex_get_metadata(measurement_class, property_name)`** β€” Well designed. -The `METADATA_PROPERTIES` list at the module level makes the valid values -clear. The guard for `sites` being layer-only is correct. - -**`snowex_spatial_query(...)`** β€” Good implementation. Accepts WKT strings -(easy for LLMs to generate). Correctly dispatches on `geometry.geom_type`. -Default CRS of `26912` matches data storage. - -**`snowex_get_unique_values(...)`** β€” Correctly documents that it only works -with direct model columns, not relationship attributes. The docstring lists -known columns. - -**`snowex_get_layer_sites(...)`** β€” Simple wrapper. Works correctly. - -**`snowex_test_connection()`** β€” Good health-check tool. - -#### Helper Functions - -**`_df_to_json(df)`** β€” Converts GeoDataFrame or DataFrame to JSON records -string. Correctly handles geometry dtype by converting to string. Uses -`orient='records'` with `indent=2`. This is the right approach for MCP. - -**`_get_measurement_dataset(measurement_class)`** β€” Maps `"point"` β†’ -`client.point_measurements`, `"layer"` β†’ `client.layer_measurements`. Clean -and simple. - -#### Missing Capabilities - -1. **`verbose` parameter** β€” Accepted by `snowex_query_measurements` but not - used; the verbose/non-verbose behavior difference (column richness) is - invisible to agents. -2. **Default limit** β€” No automatic limit applied; agents must always include - `limit` in the filters dict or risk exceptions. -3. **Combined discovery** β€” No single tool returns all metadata (types + - instruments + campaigns + dates) in one call. -4. **`verbose=True` for `from_filter`** β€” The verbose flag in the handler - (`lambda_handler.py:220`) is extracted from `filters.pop('verbose', False)`, - so putting `verbose=True` inside the `filters` dict *would* work, but the - MCP tool doesn't surface this cleanly. - ---- - -## Architecture Overview - -``` -Claude / LLM Agent - β”‚ - β”‚ (MCP protocol) - β–Ό -FastMCP Server (mcp_server.py) - β”‚ - β”‚ Python method calls - β–Ό -SnowExLambdaClient (lambda_client.py) - β”‚ - β”‚ HTTP POST JSON - β–Ό -AWS Lambda Function URL (public HTTPS) - β”‚ - β”‚ Boto3 / Secrets Manager - β–Ό -PostgreSQL 17 / PostGIS (AWS RDS) -``` - -The MCP server is a thin adapter layer. It does: -1. Input validation (valid measurement class, valid property name) -2. Geometry parsing (WKT β†’ shapely object for `from_area()`) -3. DataFrame β†’ JSON serialization -4. Error string formatting - -It does not do: SQL generation, direct DB connections, or credential handling. - ---- - -## Component Interactions - -### Request Flow for `snowex_query_measurements` - -``` -Agent calls snowex_query_measurements( - measurement_class='point', - filters={'type': 'depth', 'limit': 100} -) - ↓ -_get_measurement_dataset('point') - β†’ returns client.point_measurements (_LambdaDatasetClient) - ↓ -dataset.from_filter(type='depth', limit=100) - β†’ _create_method_proxy('from_filter')(type='depth', limit=100) - β†’ shapes payload: {'filters': {'type': 'depth', 'limit': 100}} - β†’ _invoke_lambda('PointMeasurements.from_filter', filters={...}) - β†’ HTTP POST to Lambda URL with JSON body - ↓ -Lambda parses action = 'PointMeasurements.from_filter' - β†’ _handle_class_action('PointMeasurements', 'from_filter', event, tmp_creds) - β†’ _get_measurements_by_class(PointMeasurements, {'type': 'depth'}, limit=100) - β†’ PointMeasurements.from_filter(type='depth', limit=100) - β†’ SQLAlchemy query with joins and filters - β†’ returns DataFrame β†’ serialized to JSON - ↓ -Lambda returns {'action': '...', 'data': [...], 'count': N} - ↓ -_LambdaDatasetClient converts to GeoDataFrame - ↓ -mcp_server._df_to_json(df) β†’ JSON string - ↓ -Agent receives JSON records string -``` - ---- - -## MCP Server Gap Analysis and Prioritized Build Plan - -### Priority 1 β€” Fix the `verbose` Wiring (1 hour) - -**Issue:** `snowex_query_measurements` accepts `verbose: bool = False` but -never passes it to `from_filter`. - -**Fix:** Add `verbose` to the filters dict before calling `from_filter`: -```python -filters['verbose'] = verbose -df = dataset.from_filter(**filters) -``` - -This works because `lambda_handler._handle_class_action` extracts verbose -from `filters.pop('verbose', False)` before forwarding to the API class. - -### Priority 2 β€” Add Default Limit Guard (30 min) - -**Issue:** An agent that doesn't set `limit` in filters will get a -`LargeQueryCheckException` on large tables. - -**Fix:** Apply a safe default limit in `snowex_query_measurements` if the -filters dict doesn't already contain `limit`: -```python -if 'limit' not in filters: - filters['limit'] = 100 # or make it configurable -``` -Also update the docstring to drop the "ALWAYS set this" advisory once the -default is in place. - -### Priority 3 β€” Structured Filter Parameters (4 hours) - -**Issue:** `filters: dict` is opaque. An LLM must either call -`snowex_get_metadata` first or guess valid keys. - -**Option A (preferred for LLMs):** Replace the `filters: dict` parameter with -explicit keyword parameters in `snowex_query_measurements`: -```python -@mcp.tool() -def snowex_query_measurements( - measurement_class: str, - type: str | None = None, - instrument: str | None = None, - campaign: str | None = None, - date: str | None = None, - date_greater_equal: str | None = None, - date_less_equal: str | None = None, - observer: str | None = None, - doi: str | None = None, - value_greater_equal: float | None = None, - value_less_equal: float | None = None, - site: str | None = None, # layer only - limit: int = 100, - verbose: bool = False, -) -> str: -``` -Build the filters dict inside the function from the non-None params. This -makes every valid key discoverable from the tool schema. - -**Option B (minimal change):** Keep `filters: dict` but add a JSON Schema -annotation to the docstring describing valid keys. Some MCP clients expose -this in the tool UI. - -Option A is better for LLM usability. Option B is faster to implement. - -### Priority 4 β€” Combined Discovery Tool (1 hour) - -**Issue:** An agent needs to call `snowex_get_metadata` several times to get -a full picture of available data. - -**New tool:** `snowex_discover(measurement_class)` β€” returns all metadata -for a class in a single call: -```python -@mcp.tool() -def snowex_discover(measurement_class: str) -> str: - """Return a summary of all available metadata for a measurement class. - - Returns types, instruments, campaigns, and approximate date range - in a single call. Use this for initial orientation. - """ -``` -This reduces the number of round-trips an agent must make before constructing -a valid query. - -### Priority 5 β€” Testing Strategy - -**Current test coverage of MCP server:** None (no test file for -`mcp_server.py` exists in `tests/`). - -**Recommended approach:** - -1. **Unit tests** β€” Test each tool function directly (no MCP protocol needed): -```python -# tests/test_mcp_server.py -from unittest.mock import MagicMock, patch - -def test_snowex_query_measurements_calls_from_filter(): - with patch('snowexsql.mcp_server.client') as mock_client: - mock_client.point_measurements.from_filter.return_value = pd.DataFrame(...) - result = snowex_query_measurements('point', {'limit': 5}) - assert isinstance(result, str) - assert '[' in result # JSON array -``` - -2. **Integration tests** β€” Add `@pytest.mark.integration` to tests that call - the live Lambda. Follow the pattern in `tests/deployment/test_lambda_client.py`. - -3. **MCP protocol tests** β€” Use `mcp.test_client` (from the `mcp` package) to - test the full MCP protocol round-trip if needed. - ---- - -## Summary of Files to Change - -| File | Change | -|------------------------------|-----------------------------------------------------------------| -| `snowexsql/mcp_server.py` | Fix verbose wiring (P1), add limit default (P2), structured params (P3), discovery tool (P4) | -| `tests/test_mcp_server.py` | Create new; unit tests for all tools | - -No changes needed to `lambda_client.py`, `api.py`, or `lambda_handler.py` -for MCP completion β€” the server correctly uses the existing client API. - ---- - -## References - -- `snowexsql/mcp_server.py` β€” MCP server (301 lines) -- `snowexsql/lambda_client.py` β€” Lambda client (748 lines) -- `snowexsql/api.py` β€” API classes (1098 lines) -- `snowexsql/lambda_handler.py` β€” Lambda handler (503 lines) -- `snowexsql/tables/` β€” SQLAlchemy models (16 files) -- `docs/database_structure.rst` β€” Schema documentation -- `docs/data_notes.rst` β€” Per-dataset notes -- `pyproject.toml` β€” Package configuration -- `tests/api/test_point_measurements.py` β€” Unit test patterns -- `tests/deployment/test_lambda_client.py` β€” Integration test patterns -- `AGENTS.md` β€” Agent context document produced by this research session diff --git a/.agents/research-snowexsql-tutorial.md b/.agents/research-snowexsql-tutorial.md deleted file mode 100644 index dc675ead..00000000 --- a/.agents/research-snowexsql-tutorial.md +++ /dev/null @@ -1,637 +0,0 @@ -# Research: SnowEx Database Tutorial β€” Lambda API & Data Access Patterns - ---- -**Date:** 2026-03-09 -**Author:** AI Assistant -**Status:** Active -**Related Documents:** None yet - ---- - -## Research Question - -What code patterns, data structures, API classes, and existing examples exist to support building a new tutorial that orients new users to the SnowEx database via the Lambda client β€” covering spatial area-of-interest queries, campaign/year-based filtering, and the two main data types (PointMeasurements and LayerMeasurements)? - -## Executive Summary - -The SnowEx database is a PostgreSQL/PostGIS database hosted on AWS EC2. A new, credential-free access pattern was recently implemented using an AWS Lambda Function URL backed by AWS Secrets Manager. The `SnowExLambdaClient` in `snowexsql/lambda_client.py` proxies calls to the two main API classes β€” `PointMeasurements` and `LayerMeasurements` β€” both defined in `snowexsql/api.py`. These classes expose `from_filter()` and `from_area()` methods that return GeoDataFrames without any direct database credentials required from the user. - -A working prototype tutorial exists at `snowexsql/docs/gallery/lambda_example.ipynb`. It demonstrates connection testing, bounding box spatial queries for both data types, and basic visualization with `contextily` basemaps. The old gallery examples (`getting_started_example.ipynb`, `api_intro_example.ipynb`) use the legacy `get_db()` + credentials pattern and are now out of date. The new cookbook tutorial should be a MyST markdown file (`.md`) to be placed in the cookbook's `notebooks/` directory and registered in `myst.yml`. - -The tutorial should cover: (1) initializing the Lambda client, (2) discovering what data exist in the database, (3) querying by spatial bounding box, (4) querying by campaign/year, and (5) illustrating the structural difference between point and layer data. The `RasterMeasurements` class should be excluded per project instructions. - -## Scope - -**What This Research Covers:** -- The new Lambda client architecture and usage pattern -- `PointMeasurements` and `LayerMeasurements` API classes, methods, and columns -- Database schema for points, layers, and sites tables -- The `lambda_example.ipynb` prototype as a starting point -- The cookbook structure (MyST, `myst.yml`, `notebooks/` placement) -- Campaign discovery, spatial bounding box queries, verbose mode -- Available measurement types for each data class - -**What This Research Does NOT Cover:** -- `RasterMeasurements` / `ImageData` (being downgraded; excluded per instructions) -- Direct database connection via `get_db()` (legacy, being replaced) -- Any non-Python access methods -- The full history of how data were collected in the field - ---- - -## Key Findings - -### 1. Lambda Client Initialization and Architecture - -The core user-facing entry point is `SnowExLambdaClient` in `snowexsql/lambda_client.py`. - -**Relevant Files:** -- `snowexsql/snowexsql/lambda_client.py:21-120` β€” Full `SnowExLambdaClient` class definition -- `snowexsql/snowexsql/lambda_client.py:354-734` β€” `_LambdaDatasetClient` internal proxy class -- `snowexsql/snowexsql/api.py:104-293` β€” `BaseDataset` class with `from_filter`, `from_area`, `from_unique_entries` - -**How It Works:** -1. User instantiates `SnowExLambdaClient()` β€” no arguments needed. -2. The client uses a hardcoded public Lambda Function URL (`DEFAULT_FUNCTION_URL` at `lambda_client.py:51-54`). -3. `_create_measurement_clients()` (line 134) auto-discovers all classes in `snowexsql.api` whose names end in `'Measurements'` and creates snake_case attributes: `client.point_measurements`, `client.layer_measurements`, `client.raster_measurements`. -4. `get_measurement_classes()` (line 182) returns these as a dict keyed by CamelCase names, making them drop-in replacements for direct imports. -5. All HTTP calls go through `_invoke_lambda()` (line 261) which POST's JSON to the Function URL with `{'action': ..., ...kwargs}` payload. - -**Key Pattern β€” Initialization:** -```python -from snowexsql.lambda_client import SnowExLambdaClient - -client = SnowExLambdaClient() - -# Get measurement classes as drop-in replacements for direct API imports -classes = client.get_measurement_classes() -PointMeasurements = classes['PointMeasurements'] -LayerMeasurements = classes['LayerMeasurements'] - -# Test connection -result = client.test_connection() -# Returns: {'connected': True, 'version': 'PostgreSQL 16.10 ...'} -``` - -**URL Resolution Precedence (lambda_client.py:78-81):** -1. Constructor argument `function_url` -2. Environment variable `SNOWEX_LAMBDA_URL` -3. `DEFAULT_FUNCTION_URL` class constant (`'https://izwsawyfkxss5vawq5v64mruqy0ahxek.lambda-url.us-west-2.on.aws'`) - -**HTTP Transport (lambda_client.py:107-115):** Uses `requests.Session` with retry strategy (3 retries, exponential backoff, on 429/500/502/503/504). Timeout is 30 seconds. - ---- - -### 2. PointMeasurements β€” Structure and Access Patterns - -**Relevant Files:** -- `snowexsql/snowexsql/api.py:605-731` β€” `PointMeasurements` class -- `snowexsql/snowexsql/tables/point_data.py` β€” `PointData` ORM model (table: `points`) -- `snowexsql/snowexsql/tables/single_location.py` β€” `SingleLocationData` mixin (adds `geom`, `datetime`, `elevation`) - -**Database Table:** `points` - -**Core Columns returned (non-verbose):** -| Column | Type | Description | -|--------|------|-------------| -| `id` | Integer | Primary key | -| `value` | Float | The measurement value | -| `datetime` | DateTime | Timestamp of measurement | -| `elevation` | Float | Elevation in meters | -| `geom` | Geometry | Point geometry (direct on points table) | - -**Verbose Mode Additional Columns (api.py:622-648):** -- `date` (from datetime), `observation_name`, `obs_description` -- `type` (measurement type name), `units`, `derived` -- `instrument_name`, `instrument_model`, `instrument_specifications` -- `campaign_name`, `observer_name` - -**Available Types (confirmed from lambda_example.ipynb output):** -``` -['two_way_travel', 'depth', 'swe', 'density'] -``` -- `two_way_travel` β€” GPR two-way travel time -- `depth` β€” Snow depth (from magnaprobe, mesa, camera, pit rule) -- `swe` β€” Snow water equivalent -- `density` β€” Snow density point measurements - -**Instruments (confirmed from api_intro_example.ipynb):** -`magnaprobe`, `mesa`, `camera`, `pit rule` (and others discoverable via `all_instruments`) - ---- - -### 3. LayerMeasurements β€” Structure and Access Patterns - -**Relevant Files:** -- `snowexsql/snowexsql/api.py:741-928` β€” `LayerMeasurements` class -- `snowexsql/snowexsql/tables/layer_data.py` β€” `LayerData` ORM model (table: `layers`) -- `snowexsql/snowexsql/tables/site.py` β€” `Site` ORM model (table: `sites`) - -**Database Tables:** `layers` joined to `sites` - -**Key difference from PointMeasurements:** Layer data does NOT have a geometry column directly. Instead, each layer record links to a `Site` via `site_id`, and the `Site` holds the geometry (`Site.geom`). This is why all spatial queries on `LayerMeasurements` require a join to the `sites` table. - -**Core Columns returned (non-verbose):** -| Column | Type | Description | -|--------|------|-------------| -| `id` | Integer | Primary key | -| `depth` | Float | Depth from surface (cm) | -| `bottom_depth` | Float | Bottom of layer (nullable) | -| `value` | Text | Measurement value (stored as text, requires numeric conversion) | -| `geom` | Geometry | From `Site.geom` (join required) | - -**Verbose Mode returns all site metadata (api.py:796-824):** -- `depth`, `bottom_depth`, `value` -- `site_name`, `site_description`, `slope_angle`, `aspect`, `air_temp`, `total_depth` -- `weather_description`, `precip`, `sky_cover`, `wind` -- `ground_condition`, `ground_roughness`, `ground_vegetation`, `vegetation_height`, `tree_canopy` -- `date` (Site.datetime), `geom`, `geom_wkt` -- `type`, `units`, `type_derived` -- `instrument_name`, `instrument_model`, `instrument_specifications` - -**Available Types (confirmed from lambda_example.ipynb output):** -``` -['density', 'grain_size', 'grain_type', 'hand_hardness', 'manual_wetness', - 'comments', 'permittivity', 'liquid_water_content', 'snow_temperature', - 'force', 'sample_signal', 'reflectance', 'specific_surface_area', - 'equivalent_diameter'] -``` - -**Important Note:** `value` is a `Text` column in LayerData (api.py line 21 in layer_data.py), so numeric operations require conversion: `pd.to_numeric(df['value'], errors='coerce')`. This pattern is demonstrated in `lambda_example.ipynb` (cell `655eeecd`). - -**Additional ALLOWED_QRY_KWARGS for LayerMeasurements (api.py:746-750):** -- Includes `site` (filter by site name or list of site names) in addition to base class kwargs - ---- - -### 4. The `from_filter()` Method - -**Defined in:** `snowexsql/snowexsql/api.py:325-369` - -**Allowed filter kwargs (BaseDataset.ALLOWED_QRY_KWARGS, api.py:107-113):** -```python -["campaign", "date", "instrument", "type", "utm_zone", - "date_greater_equal", "date_less_equal", - "value_greater_equal", "value_less_equal", "doi", "observer"] -``` -Plus special kwarg: `limit` - -**Size guard:** Default `MAX_RECORD_COUNT = 1000`. If query would return more without explicit `limit`, raises `LargeQueryCheckException` (api.py:150-156). - -**Examples from api_intro_example.ipynb:** -```python -# Simple date + instrument filter -df = PointMeasurements.from_filter( - date=date(2020, 5, 28), instrument='camera' -) - -# With explicit limit override -df = PointMeasurements.from_filter( - date=date(2020, 1, 28), - instrument="magnaprobe", - limit=3000 -) - -# Filter by campaign -df = PointMeasurements.from_filter( - campaign='SnowEx 2020', - type='depth', - limit=5000 -) -``` - -**Verbose mode:** -```python -df = PointMeasurements.from_filter( - type='depth', - limit=100, - verbose=True -) -# Returns extra columns: campaign_name, observer_name, instrument_name, etc. -``` - ---- - -### 5. The `from_area()` Method β€” Spatial Bounding Box Queries - -**Defined in:** `snowexsql/snowexsql/api.py:372-526` - -**Signature:** -```python -def from_area(cls, verbose=False, shp=None, pt=None, buffer=None, crs=26912, **kwargs) -``` - -**Two spatial input modes:** -1. `shp` β€” A shapely geometry (Polygon, MultiPolygon, etc.) -2. `pt` + `buffer` β€” A shapely Point with a buffer distance (in CRS units) - -**CRS handling (api.py:432-486):** The method auto-detects the database SRID and transforms the input geometry to match, using `ST_Transform`. Default `crs=26912` (UTM Zone 12N). For WGS84 lat/lon input, pass `crs=4326`. - -**Lambda client-side `from_area` (lambda_client.py:520-611):** The Lambda proxy handles `from_area` specially in `_handle_from_area_server_side()`. It converts the shapely geometry to WKT and sends it as `shp_wkt` or `pt_wkt` + `buffer`, delegating PostGIS spatial filtering to the server. - -**Bounding box examples from lambda_example.ipynb:** -```python -from shapely.geometry import box - -# Boise Basin area (Idaho) -bbox_polygon = box( - minx=-116.14, # min longitude (west) - miny=43.73, # min latitude (south) - maxx=-116.04, # max longitude (east) - maxy=43.8 # max latitude (north) -) - -# Query layer data with date range and type filters -df = LayerMeasurements.from_area( - shp=bbox_polygon, - date_greater_equal=date(2020, 1, 1), - date_less_equal=date(2022, 12, 30), - crs=4326, - type='snow_temperature', - limit=600, - verbose=True -) - -# Grand Mesa, Colorado β€” point data -bbox_polygon = box( - minx=-108.195487, miny=39.031819, - maxx=-108.189329, maxy=39.036568 -) -df = PointMeasurements.from_area( - shp=bbox_polygon, - crs=4326, - type='depth', - limit=30000, - verbose=False -) -``` - ---- - -### 6. Discovery Properties (Catalog Exploration) - -Both classes expose catalog-exploration properties via the Lambda proxy. In the `_LambdaDatasetClient`, any attribute starting with `all_` (e.g., `all_types`, `all_instruments`) is routed to `_get_property()` which invokes Lambda with action `ClassName.property_name`. - -**Available on both PointMeasurements and LayerMeasurements:** -| Property | Returns | -|----------|---------| -| `all_types` | List of measurement type names | -| `all_instruments` | List of instrument names | -| `all_campaigns` | List of campaign names | -| `all_dates` | List of distinct dates | -| `all_observers` | List of observer names | -| `all_dois` | List of DOI strings | -| `all_units` | List of unit strings | - -**Additional on LayerMeasurements only:** -| Property | Returns | -|----------|---------| -| `all_sites` | List of site names | - -**Usage:** -```python -# Discover what's available -all_layer_types = LayerMeasurements.all_types -all_point_types = PointMeasurements.all_types -all_campaigns = LayerMeasurements.all_campaigns -``` - ---- - -### 7. Campaign Filtering - -**Campaigns correspond to SnowEx field campaigns (from snowex_data_overview.ipynb):** -| Campaign | Year(s) | Location | Notes | -|----------|---------|----------|-------| -| SnowEx 2017 | 2017 | Grand Mesa & Senator Beck Basin, CO | IOP | -| SnowEx 2020 | 2019-2020 | Grand Mesa + Western U.S. | IOP + TS | -| SnowEx 2021 | 2020-2021 | Western U.S. | TS | -| SnowEx 2023 | 2022-2023 | Alaska Tundra & Boreal Forest | IOP | - -**Filtering by campaign:** -```python -df = PointMeasurements.from_filter( - campaign='SnowEx 2020', - type='depth', - limit=5000 -) -``` - -**Filtering by date range:** -```python -from datetime import date - -df = LayerMeasurements.from_filter( - date_greater_equal=date(2020, 1, 1), - date_less_equal=date(2020, 12, 31), - type='density', - limit=1000 -) -``` - -**Combined area + campaign filter:** -```python -df = LayerMeasurements.from_area( - shp=bbox_polygon, - crs=4326, - campaign='SnowEx 2020', - type='snow_temperature', - limit=500, - verbose=True -) -``` - ---- - -### 8. Database Schema Overview - -``` -points (PointData) layers (LayerData) -β”œβ”€β”€ id (PK) β”œβ”€β”€ id (PK) -β”œβ”€β”€ value (Float) β”œβ”€β”€ depth (Float) -β”œβ”€β”€ datetime β”œβ”€β”€ bottom_depth (Float) -β”œβ”€β”€ elevation (Float) β”œβ”€β”€ value (Text) ← numeric conversion needed -β”œβ”€β”€ geom (geometry) ← direct └── site_id (FK β†’ sites.id) -β”œβ”€β”€ measurement_type_id (FK) -└── observation_id (FK) sites (Site) - β”‚ β”œβ”€β”€ id (PK) - ↓ β”œβ”€β”€ name (String) ← pit_id -point_observations β”œβ”€β”€ datetime -β”œβ”€β”€ instrument_id (FK) β”œβ”€β”€ geom (geometry) ← geometry for layers -β”œβ”€β”€ campaign_id (FK) β”œβ”€β”€ campaign_id (FK) -└── observer_id (FK) β”œβ”€β”€ slope_angle, aspect, air_temp - β”œβ”€β”€ total_depth, weather_description -campaigns └── [many site metadata fields] -β”œβ”€β”€ id (PK) -└── name (e.g. 'SnowEx 2020') -``` - ---- - -### 9. Return Value β€” GeoDataFrame - -Both `from_filter()` and `from_area()` return a `geopandas.GeoDataFrame`. The Lambda client handles conversion from JSON response to GeoDataFrame client-side in `_to_geodataframe()` (lambda_client.py:630-720), parsing PostGIS WKB hex strings, WKT strings, or GeoJSON dicts into shapely geometry objects. - -**Default CRS after Lambda conversion:** `EPSG:4326` (WGS84) β€” set in `_to_geodataframe()` at line 657. - -**Reprojection for basemap plotting:** -```python -import contextily as ctx -df_web = df.to_crs(epsg=3857) # Web Mercator for contextily -ctx.add_basemap(ax, source=ctx.providers.OpenStreetMap.Mapnik) -``` - ---- - -### 10. Existing Lambda Example Notebook Contents - -**File:** `snowexsql/docs/gallery/lambda_example.ipynb` - -The notebook demonstrates 5 distinct patterns useful for the new tutorial: - -1. **Connection test** (cell `15159007`): Initialize client, call `test_connection()` -2. **Layer types discovery** (cell `cd2cb76d`): `LayerMeasurements.all_types` -3. **Layer data by bounding box** (cell `4a53cbe1`): `from_area()` with Boise Basin bbox, `snow_temperature`, date range, `verbose=True` -4. **Layer data visualization** (cells `588cf2f5`, `655eeecd`): Map plot + boxplot by depth band (requires numeric conversion of `value` column) -5. **Point types discovery** (cell `779cdc44`): `PointMeasurements.all_types` -6. **Point data by bounding box (Grand Mesa)** (cells `7b24ceaa`, `c6d895ff`): `from_area()` with `depth` type, `verbose=False` -7. **Point data map** (cell `6cfbb523`): Spatial plot of snow depths -8. **Point data by filter** (cell `6b47e731`): `PointMeasurements.from_filter(type='swe', limit=10000)` - ---- - -### 11. Cookbook Structure for New Tutorial - -**Existing notebook placement:** `notebooks/` directory in `snow-observations-cookbook/` - -**TOC registration:** `myst.yml` (lines 12-28). The new tutorial should be added under the "Data Access" section: -```yaml -- title: Data Access - children: - - file: notebooks/snowex_data_overview.ipynb - - file: notebooks/snowexsql_database.ipynb - - file: notebooks/snowexsql-lambda-tutorial.md # ← new file -``` - -**MyST markdown format:** Files use MyST markdown with code fences for Python (`\`\`\`python`). The `environment.yml` already includes `snowexsql` (from git master), `geopandas`, `shapely`, `contextily`, and `matplotlib`. - -**Existing markdown example:** `notebooks/how-to-cite.md` (simple markdown, no executable code). MyST markdown with code cells uses `{code-cell} ipython3` directive syntax for executable cells. - ---- - -## Architecture Overview - -``` -User (Tutorial) - β”‚ - β–Ό -SnowExLambdaClient() (snowexsql/lambda_client.py:21) - β”‚ - β”‚ get_measurement_classes() β†’ dict - β”‚ - β”œβ”€β”€ PointMeasurements (_LambdaDatasetClient proxy) - β”‚ β”‚ - β”‚ β”œβ”€β”€ .from_filter(**kwargs) - β”‚ β”œβ”€β”€ .from_area(shp=..., crs=4326, ...) - β”‚ β”œβ”€β”€ .all_types - β”‚ └── .all_instruments - β”‚ - └── LayerMeasurements (_LambdaDatasetClient proxy) - β”‚ - β”œβ”€β”€ .from_filter(**kwargs) - β”œβ”€β”€ .from_area(shp=..., crs=4326, ...) - β”œβ”€β”€ .all_types - └── .all_sites - - β”‚ (HTTP POST to Lambda Function URL) - β–Ό -AWS Lambda (public Function URL) - β”‚ - β”‚ (credentials via AWS Secrets Manager) - β–Ό -PostgreSQL/PostGIS (AWS EC2) - β”œβ”€β”€ points table ← PointMeasurements - β”œβ”€β”€ layers table ← LayerMeasurements - └── sites table ← Site metadata + geometry for layers -``` - ---- - -## Component Interactions - -**Flow for `from_area()` call with Lambda client:** -1. User calls `LayerMeasurements.from_area(shp=bbox_polygon, crs=4326, type='snow_temperature', limit=600)` (`lambda_client.py:483`) -2. `_handle_from_area_server_side()` converts shapely geometry to WKT string (`lambda_client.py:580`) -3. HTTP POST sent to Lambda Function URL with payload: `{'action': 'LayerMeasurements.from_area', 'shp_wkt': '...', 'crs': 4326, 'filters': {'type': 'snow_temperature', 'limit': 600}}` -4. Lambda handler receives, reconstructs SQLAlchemy query using PostGIS `ST_Transform` + `ST_Intersects` -5. Database executes spatial + attribute filters, returns records as JSON -6. Lambda client receives JSON, `_to_geodataframe()` parses WKB hex geometry to shapely objects (`lambda_client.py:650`) -7. Returns `geopandas.GeoDataFrame` with `EPSG:4326` CRS to user - -**Flow for `from_filter()` call:** -1. User calls `PointMeasurements.from_filter(type='swe', limit=10000)` -2. Lambda proxy wraps kwargs into `{'action': 'PointMeasurements.from_filter', 'filters': {'type': 'swe', 'limit': 10000}}` -3. HTTP POST to Lambda β†’ database query β†’ JSON response -4. `pd.DataFrame(result['data'])` β†’ `_to_geodataframe()` β†’ GeoDataFrame returned - ---- - -## Code Examples - -### Complete Tutorial Setup Pattern -```python -# lambda_example.ipynb (cell aca5a2c4 + 15159007) -from datetime import date -import geopandas as gpd -import matplotlib.pyplot as plt -import contextily as ctx -import pandas as pd -from shapely.geometry import box - -from snowexsql.lambda_client import SnowExLambdaClient - -# Initialize client β€” no credentials needed! -client = SnowExLambdaClient() -classes = client.get_measurement_classes() -PointMeasurements = classes['PointMeasurements'] -LayerMeasurements = classes['LayerMeasurements'] - -# Verify connection -result = client.test_connection() -print(f"Connected: {result.get('connected', False)}") -``` - -### Spatial Bounding Box Query β€” Layer Data -```python -# lambda_example.ipynb (cells b5b2a3b9 + 4a53cbe1) -bbox_polygon = box( - minx=-116.14, miny=43.73, - maxx=-116.04, maxy=43.8 -) -df = LayerMeasurements.from_area( - shp=bbox_polygon, - date_greater_equal=date(2020, 1, 1), - date_less_equal=date(2022, 12, 30), - crs=4326, - type='snow_temperature', - limit=600, - verbose=True -) -# df has 29 columns including site metadata when verbose=True -# IMPORTANT: df['value'] is Text β†’ must convert: pd.to_numeric(df['value']) -``` - -### Spatial Bounding Box Query β€” Point Data -```python -# lambda_example.ipynb (cells 7b24ceaa + c6d895ff) -bbox_polygon = box( - minx=-108.195487, miny=39.031819, - maxx=-108.189329, maxy=39.036568 -) -df = PointMeasurements.from_area( - shp=bbox_polygon, - crs=4326, - type='depth', - limit=30000 -) -# df has id, value, datetime, elevation, geom, geometry columns -# df['value'] is Float (no conversion needed for PointMeasurements) -``` - -### Filter by Campaign -```python -# api.py supports campaign kwarg in ALLOWED_QRY_KWARGS (line 107) -df = PointMeasurements.from_filter( - campaign='SnowEx 2020', - type='depth', - limit=5000 -) -``` - -### Type Discovery -```python -# lambda_example.ipynb (cells cd2cb76d + 779cdc44) -layer_types = LayerMeasurements.all_types -# ['density', 'grain_size', 'grain_type', 'hand_hardness', ...] - -point_types = PointMeasurements.all_types -# ['two_way_travel', 'depth', 'swe', 'density'] -``` - ---- - -## Technical Decisions - -- **Lambda as public gateway:** Database only accepts connections from Lambda (not public internet). Lambda Function URL is public HTTPS. This eliminates the need for VPN, credentials, or AWS account from end users. - - **Rationale:** Simplifies onboarding for the research community; secures the database. - -- **`_LambdaDatasetClient` dynamic proxy:** Uses `__getattr__` to intercept any `all_*` property or known method call and route it to Lambda. This means the Lambda client automatically picks up new properties/methods without manual updates. - - **Rationale:** `api.py` can evolve without requiring parallel changes to `lambda_client.py`. - -- **`from_area` server-side PostGIS:** The Lambda client sends WKT geometry to the server and PostGIS performs the spatial filter. This is more efficient than fetching all data and filtering client-side. - - **Location:** `lambda_client.py:520-611` - -- **LayerData `value` as Text:** `LayerData.value` is `Column(Text)` in the ORM (`layer_data.py:20`). This is because layer data includes various types including grain type strings. Numeric conversion with `pd.to_numeric(..., errors='coerce')` is needed for analysis. - -- **GeoDataFrame CRS defaults to EPSG:4326:** After Lambda response, geometry is always set to `EPSG:4326` (`lambda_client.py:657`). Users need to reproject to EPSG:3857 for contextily basemaps. - ---- - -## Dependencies and Integrations - -- **`snowexsql`** (from `git+https://github.com/SnowEx/snowexsql.git@master`): Core library; install via `environment.yml` -- **`geopandas`**: Returned by all data methods; needed for spatial operations and `.to_crs()` -- **`shapely`**: Required for `from_area()` input geometry creation (`box()`, `Point`, etc.) -- **`contextily`**: Used in lambda_example.ipynb for basemap tiles; available in `environment.yml` -- **`matplotlib`**: Standard plotting; available in `environment.yml` -- **`pandas`**: Used for `pd.to_numeric()` conversion of layer values; available in `environment.yml` -- **`requests`**: Internal to `SnowExLambdaClient` for HTTP calls; bundled with snowexsql - ---- - -## Edge Cases and Constraints - -- **Query size limit:** Default `MAX_RECORD_COUNT = 1000`. Must pass `limit=N` explicitly for larger queries. `LargeQueryCheckException` is raised if exceeded without `limit`. (`api.py:115, 150-156`) - -- **LayerData value is Text:** `pd.to_numeric(df['value'], errors='coerce')` is required before mathematical operations. Demonstrated at `lambda_example.ipynb:cell 655eeecd`. - -- **CRS mismatch:** `from_area()` defaults to `crs=26912` (UTM Zone 12N) in direct API, but Lambda examples use `crs=4326` for lat/lon bounding boxes. Always specify `crs` explicitly to avoid coordinate mismatch. - -- **Lambda timeout:** 30-second timeout (`lambda_client.py:57`). Large queries (>30,000 records) may timeout. Use `limit` parameter and verify queries work before removing limits. - -- **Geometry column naming:** Lambda returns data with both `geom` (WKB hex) and `geometry` (parsed shapely) columns. The `geometry` column is the active GeoDataFrame geometry. - -- **`verbose=False` for LayerMeasurements still joins Site:** Even in non-verbose mode, `_add_base_joins()` joins to the `sites` table to return `Site.geom` (`api.py:836-841`). This is required for the GeoDataFrame to have geometry. - -- **`all_types` scope:** `PointMeasurements.all_types` uses `EXISTS` subquery filtering to only show types that have actual records in the `points` table (`api.py:708-715`). Same for `LayerMeasurements.all_types` (`api.py:856-866`). - ---- - -## Open Questions - -1. What are the exact campaign name strings to use in `from_filter(campaign=...)`? The `all_campaigns` property should be queried to get the exact names. -2. Are there recommended bounding boxes for each SnowEx campaign site (Grand Mesa, Boise Basin, Alaska sites) that would work as good tutorial examples? -3. Does `from_filter(verbose=True)` work on the Lambda path? The lambda_example.ipynb uses `verbose=True` only with `from_area`. Should be tested. -4. What is the expected behavior when `from_area` returns 0 records? The lambda client returns empty DataFrame (`lambda_client.py:606`). - ---- - -## References - -- Files analyzed: 14 files - - `snowexsql/docs/gallery/lambda_example.ipynb` - - `snowexsql/snowexsql/lambda_client.py` - - `snowexsql/snowexsql/api.py` - - `snowexsql/snowexsql/tables/point_data.py` - - `snowexsql/snowexsql/tables/layer_data.py` - - `snowexsql/snowexsql/tables/site.py` - - `snowexsql/docs/gallery/api_intro_example.ipynb` - - `snowexsql/docs/gallery/getting_started_example.ipynb` - - `snowexsql/docs/gallery/what_is_in_the_db_example.ipynb` - - `snowexsql/docs/gallery/overview_example.ipynb` - - `snowexsql/docs/gallery/index.md` - - `snow-observations-cookbook/myst.yml` - - `snow-observations-cookbook/environment.yml` - - `snow-observations-cookbook/notebooks/how-to-cite.md` - ----