diff --git a/.env.example b/.env.example index d507cbe..ccaf010 100644 --- a/.env.example +++ b/.env.example @@ -26,8 +26,9 @@ GUARDRAILS_HUB_API_KEY="" HF_TOKEN="" # SHA-256 hex digest of your bearer token (64 lowercase hex chars) AUTH_TOKEN="" -KAAPI_AUTH_URL="" -KAAPI_AUTH_TIMEOUT=5 +# Comma-separated source IPs allowed to call this service (kaapi-backend). +# Empty disables the check; required when ENVIRONMENT=production. +ALLOWED_IPS="127.0.0.1" # URL for the guardrails API — required for the multiple_validators evaluation script GUARDRAILS_API_URL="http://localhost:8001/api/v1/guardrails/" diff --git a/.env.test.example b/.env.test.example index 5169239..5efd1e7 100644 --- a/.env.test.example +++ b/.env.test.example @@ -25,5 +25,6 @@ OPENAI_API_KEY="" GUARDRAILS_HUB_API_KEY="" # SHA-256 hex digest of your bearer token (64 lowercase hex chars) AUTH_TOKEN="" -KAAPI_AUTH_URL="" -KAAPI_AUTH_TIMEOUT=5 +# Comma-separated source IPs allowed to call this service (kaapi-backend). +# Empty disables the check; required when ENVIRONMENT=production. +ALLOWED_IPS="127.0.0.1" diff --git a/backend/.guardrails/hub_registry.json b/backend/.guardrails/hub_registry.json index cd71baa..e7b757b 100644 --- a/backend/.guardrails/hub_registry.json +++ b/backend/.guardrails/hub_registry.json @@ -6,7 +6,7 @@ "exports": [ "BanList" ], - "installed_at": "2026-05-12T05:28:55.036165+00:00", + "installed_at": "2026-07-23T14:06:37.542360+00:00", "package_name": "guardrails-grhub-ban-list" }, "guardrails/llm_critic": { @@ -14,7 +14,7 @@ "exports": [ "LLMCritic" ], - "installed_at": "2026-05-12T05:29:05.596003+00:00", + "installed_at": "2026-07-23T14:06:45.369592+00:00", "package_name": "guardrails-grhub-llm-critic" }, "guardrails/llamaguard_7b": { @@ -22,7 +22,7 @@ "exports": [ "LlamaGuard7B" ], - "installed_at": "2026-05-12T05:29:16.995441+00:00", + "installed_at": "2026-07-23T14:06:51.618169+00:00", "package_name": "guardrails-grhub-llamaguard-7b" }, "guardrails/profanity_free": { @@ -30,7 +30,7 @@ "exports": [ "ProfanityFree" ], - "installed_at": "2026-05-12T05:29:28.079410+00:00", + "installed_at": "2026-07-23T14:07:02.770910+00:00", "package_name": "guardrails-grhub-profanity-free" }, "guardrails/nsfw_text": { diff --git a/backend/README.md b/backend/README.md index ce07174..42151bf 100644 --- a/backend/README.md +++ b/backend/README.md @@ -191,19 +191,32 @@ echo -n "your-plain-text-token" | shasum -a 256 Set the resulting digest as `AUTH_TOKEN` in your `.env` / `.env.test`. -## Multi-tenant API Key Configuration +## Caller Authentication -Ban List and LLM Prompt Config APIs use `X-API-KEY` auth instead of bearer token auth. +kaapi-guardrails is an internal service. Its only caller is kaapi-backend, which authenticates the +end user and resolves the tenant before calling. -Required environment variables: -- `KAAPI_AUTH_URL`: Base URL of the Kaapi auth service used to verify API keys. -- `KAAPI_AUTH_TIMEOUT`: Timeout in seconds for auth verification calls. +Every request (except the health check) must carry: -At runtime, the backend calls: -- `GET {KAAPI_AUTH_URL}/apikeys/verify` -- Header: `X-API-KEY: ` +- `Authorization: Bearer ` +- `X-ORGANIZATION-ID: ` and `X-PROJECT-ID: ` — the tenant, resolved by kaapi-backend +- and originate from an IP listed in `ALLOWED_IPS` -If verification succeeds, tenant's scope (`organization_id`, `project_id`) is resolved from the auth response and applied to tenant-scoped CRUD operations (for example Ban Lists and LLM Prompt Configs). +`ALLOWED_IPS` is a comma-separated list of source IPs allowed to reach this service: + +```bash +ALLOWED_IPS="10.0.3.14" +``` + +Leave it empty to disable the check for local development. It is required when +`ENVIRONMENT=production` — the service will not start without it. + +Rejections: +- Source IP not whitelisted → `403` (checked before the token) +- Missing or invalid token → `401` +- Missing or non-integer tenant headers → `422` + +Tenant scope is never read from the query string or request body. ## Guardrails AI Setup diff --git a/backend/app/api/API_USAGE.md b/backend/app/api/API_USAGE.md index af2b82a..2c6f263 100644 --- a/backend/app/api/API_USAGE.md +++ b/backend/app/api/API_USAGE.md @@ -18,14 +18,16 @@ Example local base URL: ## Authentication -This API currently uses two auth modes: +This service is internal. Its only caller is kaapi-backend, which authenticates the end user and +resolves the tenant before calling. -1. Bearer token auth (`Authorization: Bearer `) - - Used by validator config and guardrails endpoints. - - The server validates your plaintext bearer token against a SHA-256 digest stored in `AUTH_TOKEN`. -2. multi-tenant API key auth (`X-API-KEY: `) - - Used by ban list and LLM prompt config endpoints. - - The API key is verified against `KAAPI_AUTH_URL` and resolves tenant's scope (`organization_id`, `project_id`). +Every request must carry: + +- `Authorization: Bearer ` — validated against the SHA-256 digest in `AUTH_TOKEN` +- `X-ORGANIZATION-ID: ` and `X-PROJECT-ID: ` — the tenant, resolved by kaapi-backend +- and must arrive from an IP listed in `ALLOWED_IPS` + +Tenant scope is never read from the query string or request body. Notes: - `GET /utils/health-check/` is public. @@ -74,13 +76,15 @@ Base path: ## 2.1 Create validator config Endpoint: -- `POST /api/v1/guardrails/validators/configs/?organization_id=1&project_id=101` +- `POST /api/v1/guardrails/validators/configs/` Example (PII input validator): ```bash -curl -X POST "http://localhost:8001/api/v1/guardrails/validators/configs/?organization_id=1&project_id=101" \ +curl -X POST "http://localhost:8001/api/v1/guardrails/validators/configs/" \ -H "Authorization: Bearer " \ + -H "X-ORGANIZATION-ID: 1" \ + -H "X-PROJECT-ID: 101" \ -H "Content-Type: application/json" \ -d '{ "type": "pii_remover", @@ -95,7 +99,7 @@ curl -X POST "http://localhost:8001/api/v1/guardrails/validators/configs/?organi ## 2.2 List validator configs Endpoint: -- `GET /api/v1/guardrails/validators/configs/?organization_id=1&project_id=101` +- `GET /api/v1/guardrails/validators/configs/` Optional filters: - `ids=&ids=` @@ -106,32 +110,38 @@ Optional filters: Example: ```bash -curl -X GET "http://localhost:8001/api/v1/guardrails/validators/configs/?organization_id=1&project_id=101&stage=input" \ - -H "Authorization: Bearer " +curl -X GET "http://localhost:8001/api/v1/guardrails/validators/configs/?stage=input" \ + -H "Authorization: Bearer " \ + -H "X-ORGANIZATION-ID: 1" \ + -H "X-PROJECT-ID: 101" ``` ## 2.3 Get validator config by id Endpoint: -- `GET /api/v1/guardrails/validators/configs/{id}?organization_id=1&project_id=101` +- `GET /api/v1/guardrails/validators/configs/{id}` Example: ```bash -curl -X GET "http://localhost:8001/api/v1/guardrails/validators/configs/?organization_id=1&project_id=101" \ - -H "Authorization: Bearer " +curl -X GET "http://localhost:8001/api/v1/guardrails/validators/configs/" \ + -H "Authorization: Bearer " \ + -H "X-ORGANIZATION-ID: 1" \ + -H "X-PROJECT-ID: 101" ``` ## 2.4 Update validator config Endpoint: -- `PATCH /api/v1/guardrails/validators/configs/{id}?organization_id=1&project_id=101` +- `PATCH /api/v1/guardrails/validators/configs/{id}` Example: ```bash -curl -X PATCH "http://localhost:8001/api/v1/guardrails/validators/configs/?organization_id=1&project_id=101" \ +curl -X PATCH "http://localhost:8001/api/v1/guardrails/validators/configs/" \ -H "Authorization: Bearer " \ + -H "X-ORGANIZATION-ID: 1" \ + -H "X-PROJECT-ID: 101" \ -H "Content-Type: application/json" \ -d '{ "is_enabled": false, @@ -142,13 +152,15 @@ curl -X PATCH "http://localhost:8001/api/v1/guardrails/validators/configs/?organization_id=1&project_id=101" \ - -H "Authorization: Bearer " +curl -X DELETE "http://localhost:8001/api/v1/guardrails/validators/configs/" \ + -H "Authorization: Bearer " \ + -H "X-ORGANIZATION-ID: 1" \ + -H "X-PROJECT-ID: 101" ``` ## 3) Runtime Validator Discovery @@ -163,7 +175,9 @@ Example: ```bash curl -X GET "http://localhost:8001/api/v1/guardrails/" \ - -H "Authorization: Bearer " + -H "Authorization: Bearer " \ + -H "X-ORGANIZATION-ID: 1" \ + -H "X-PROJECT-ID: 101" ``` ## 4) Guardrail Execution @@ -192,11 +206,11 @@ Example: ```bash curl -X POST "http://localhost:8001/api/v1/guardrails/?suppress_pass_logs=true" \ -H "Authorization: Bearer " \ + -H "X-ORGANIZATION-ID: 1" \ + -H "X-PROJECT-ID: 101" \ -H "Content-Type: application/json" \ -d '{ "request_id": "2a6f6d5c-5b9f-4f6b-92e4-cf7d67f87932", - "organization_id": 1, - "project_id": 101, "input": "Amit Gupta phone number is 919611188278", "validators": [ { @@ -264,7 +278,7 @@ Possible failure response: ## 5) Ban List APIs (multi-tenant) -These endpoints manage tenant-scoped ban lists and use `X-API-KEY` auth. +These endpoints manage tenant-scoped ban lists. The tenant comes from the `X-ORGANIZATION-ID` / `X-PROJECT-ID` headers. Base path: - `/api/v1/guardrails/ban_lists` @@ -278,7 +292,9 @@ Example: ```bash curl -X POST "http://localhost:8001/api/v1/guardrails/ban_lists/" \ - -H "X-API-KEY: " \ + -H "Authorization: Bearer " \ + -H "X-ORGANIZATION-ID: 1" \ + -H "X-PROJECT-ID: 101" \ -H "Content-Type: application/json" \ -d '{ "name": "Safety Banned Terms", @@ -298,7 +314,9 @@ Example: ```bash curl -X GET "http://localhost:8001/api/v1/guardrails/ban_lists/?offset=0&limit=20" \ - -H "X-API-KEY: " + -H "Authorization: Bearer " \ + -H "X-ORGANIZATION-ID: 1" \ + -H "X-PROJECT-ID: 101" ``` ## 5.3 Get ban list by id @@ -310,7 +328,9 @@ Example: ```bash curl -X GET "http://localhost:8001/api/v1/guardrails/ban_lists/" \ - -H "X-API-KEY: " + -H "Authorization: Bearer " \ + -H "X-ORGANIZATION-ID: 1" \ + -H "X-PROJECT-ID: 101" ``` ## 5.4 Update ban list @@ -322,7 +342,9 @@ Example: ```bash curl -X PATCH "http://localhost:8001/api/v1/guardrails/ban_lists/" \ - -H "X-API-KEY: " \ + -H "Authorization: Bearer " \ + -H "X-ORGANIZATION-ID: 1" \ + -H "X-PROJECT-ID: 101" \ -H "Content-Type: application/json" \ -d '{ "description": "Updated description", @@ -339,12 +361,14 @@ Example: ```bash curl -X DELETE "http://localhost:8001/api/v1/guardrails/ban_lists/" \ - -H "X-API-KEY: " + -H "Authorization: Bearer " \ + -H "X-ORGANIZATION-ID: 1" \ + -H "X-PROJECT-ID: 101" ``` ## 6) LLM Prompt Config APIs (multi-tenant) -These endpoints manage tenant-scoped LLM prompt configs for the `topic_relevance` and `answer_relevance_custom_llm` validators. They use `X-API-KEY` auth. +These endpoints manage tenant-scoped LLM prompt configs for the `topic_relevance` and `answer_relevance_custom_llm` validators. The tenant comes from the `X-ORGANIZATION-ID` / `X-PROJECT-ID` headers. Base path: - `/api/v1/guardrails/llm_prompt_configs` @@ -362,7 +386,9 @@ Example (topic relevance): ```bash curl -X POST "http://localhost:8001/api/v1/guardrails/llm_prompt_configs/" \ - -H "X-API-KEY: " \ + -H "Authorization: Bearer " \ + -H "X-ORGANIZATION-ID: 1" \ + -H "X-PROJECT-ID: 101" \ -H "Content-Type: application/json" \ -d '{ "validator_name": "topic_relevance", @@ -377,7 +403,9 @@ Example (answer relevance): ```bash curl -X POST "http://localhost:8001/api/v1/guardrails/llm_prompt_configs/" \ - -H "X-API-KEY: " \ + -H "Authorization: Bearer " \ + -H "X-ORGANIZATION-ID: 1" \ + -H "X-PROJECT-ID: 101" \ -H "Content-Type: application/json" \ -d '{ "validator_name": "answer_relevance_custom_llm", @@ -399,7 +427,9 @@ Example: ```bash curl -X GET "http://localhost:8001/api/v1/guardrails/llm_prompt_configs/?validator_name=topic_relevance&offset=0&limit=20" \ - -H "X-API-KEY: " + -H "Authorization: Bearer " \ + -H "X-ORGANIZATION-ID: 1" \ + -H "X-PROJECT-ID: 101" ``` ## 6.3 Get LLM prompt config by id @@ -411,7 +441,9 @@ Example: ```bash curl -X GET "http://localhost:8001/api/v1/guardrails/llm_prompt_configs/" \ - -H "X-API-KEY: " + -H "Authorization: Bearer " \ + -H "X-ORGANIZATION-ID: 1" \ + -H "X-PROJECT-ID: 101" ``` ## 6.4 Update LLM prompt config @@ -423,7 +455,9 @@ Example: ```bash curl -X PATCH "http://localhost:8001/api/v1/guardrails/llm_prompt_configs/" \ - -H "X-API-KEY: " \ + -H "Authorization: Bearer " \ + -H "X-ORGANIZATION-ID: 1" \ + -H "X-PROJECT-ID: 101" \ -H "Content-Type: application/json" \ -d '{ "llm_prompt": "Pregnancy care: Updated scope definition" @@ -439,7 +473,9 @@ Example: ```bash curl -X DELETE "http://localhost:8001/api/v1/guardrails/llm_prompt_configs/" \ - -H "X-API-KEY: " + -H "Authorization: Bearer " \ + -H "X-ORGANIZATION-ID: 1" \ + -H "X-PROJECT-ID: 101" ``` ## 8) End-to-End Usage Pattern @@ -460,10 +496,10 @@ Recommended request flow: - Add `Authorization: Bearer `. - `401 Invalid authorization token` - Verify plaintext token matches server-side hash. -- `401 Missing X-API-KEY header` - - Add `X-API-KEY: ` for ban list and LLM prompt config endpoints. -- `401 Invalid API key` - - Verify the API key is valid in the upstream Kaapi auth service. +- `403 Forbidden` + - Source IP is not in `ALLOWED_IPS`. Checked before the token. +- `422` on tenant headers + - Add `X-ORGANIZATION-ID` and `X-PROJECT-ID`; both must be integers. - `Invalid request_id` - Ensure `request_id` is a valid UUID string. - `Validator already exists for this type and stage` diff --git a/backend/app/api/deps.py b/backend/app/api/deps.py index c6e05f3..a50c74e 100644 --- a/backend/app/api/deps.py +++ b/backend/app/api/deps.py @@ -4,9 +4,8 @@ import hashlib import secrets -import httpx -from fastapi import Cookie, Depends, Header, HTTPException, Security, status +from fastapi import Depends, Header, HTTPException, Request, Security, status from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials from sqlmodel import Session @@ -22,7 +21,6 @@ def get_db() -> Generator[Session, None, None]: SessionDep = Annotated[Session, Depends(get_db)] -# Static bearer token auth for internal routes. security = HTTPBearer(auto_error=False) @@ -37,100 +35,47 @@ def _unauthorized(detail: str) -> HTTPException: ) -def verify_bearer_token( - credentials: Annotated[ - HTTPAuthorizationCredentials | None, - Security(security), - ], -) -> bool: - if credentials is None: - raise _unauthorized("Missing Authorization header") - - if not secrets.compare_digest( - _hash_token(credentials.credentials), - settings.AUTH_TOKEN, - ): - raise _unauthorized("Invalid authorization token") - - return True - - -AuthDep = Annotated[bool, Depends(verify_bearer_token)] - - -# Multitenant auth context resolved from X-API-KEY. @dataclass class TenantContext: organization_id: int project_id: int -def _fetch_tenant_from_backend(headers: dict) -> TenantContext: - if not settings.KAAPI_AUTH_URL: - raise HTTPException( - status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, - detail="KAAPI_AUTH_URL is not configured", - ) - - try: - response = httpx.get( - f"{settings.KAAPI_AUTH_URL}/apikeys/verify", - headers=headers, - timeout=settings.KAAPI_AUTH_TIMEOUT, - ) - except httpx.RequestError: - raise HTTPException( - status_code=status.HTTP_503_SERVICE_UNAVAILABLE, - detail="Auth service unavailable", - ) - - if response.status_code != 200: - raise _unauthorized("Invalid credentials") - - data = response.json() - if not isinstance(data, dict) or data.get("success") is not True: - raise _unauthorized("Invalid credentials") - - record = data.get("data") - if not isinstance(record, dict): - raise _unauthorized("Invalid credentials") - - organization_id = record.get("organization_id") - project_id = record.get("project_id") - if not isinstance(organization_id, int) or not isinstance(project_id, int): - raise _unauthorized("Invalid credentials") - - return TenantContext( - organization_id=organization_id, - project_id=project_id, - ) - - -def validate_multitenant_key( - x_api_key: Annotated[str | None, Header(alias="X-API-KEY")] = None, +def verify_caller( + request: Request, credentials: Annotated[ HTTPAuthorizationCredentials | None, Security(security), - ] = None, - access_token: Annotated[str | None, Cookie(alias="access_token")] = None, + ], + organization_id: Annotated[int, Header(alias="X-ORGANIZATION-ID")], + project_id: Annotated[int, Header(alias="X-PROJECT-ID")], ) -> TenantContext: - if x_api_key and x_api_key.strip(): - return _fetch_tenant_from_backend({"X-API-KEY": x_api_key.strip()}) + """ + Authenticates the single trusted caller (kaapi-backend) by static bearer token + and source IP, and returns the tenant it resolved from the end user's API key. + + Tenant scope is never read from the query string or request body, so a route + cannot be tenant-unscoped: the dependency that authenticates it also supplies + the tenant. + """ + if settings.ALLOWED_IPS: + client_ip = request.client.host if request.client else None + if client_ip not in settings.ALLOWED_IPS: + raise HTTPException( + status_code=status.HTTP_403_FORBIDDEN, + detail="Forbidden", + ) - if credentials is not None and credentials.credentials.strip(): - return _fetch_tenant_from_backend( - {"Authorization": f"Bearer {credentials.credentials}"} - ) + if credentials is None: + raise _unauthorized("Missing Authorization header") - if access_token: - return _fetch_tenant_from_backend({"Authorization": f"Bearer {access_token}"}) + if not secrets.compare_digest( + _hash_token(credentials.credentials), + settings.AUTH_TOKEN, + ): + raise _unauthorized("Invalid authorization token") - raise _unauthorized( - "Missing credentials: provide X-API-KEY header, Bearer token, or access_token cookie" - ) + return TenantContext(organization_id=organization_id, project_id=project_id) -MultitenantAuthDep = Annotated[ - TenantContext, - Depends(validate_multitenant_key), -] +AuthDep = Annotated[TenantContext, Depends(verify_caller)] diff --git a/backend/app/api/docs/ban_lists/create_ban_list.md b/backend/app/api/docs/ban_lists/create_ban_list.md index 64febf8..75b1938 100644 --- a/backend/app/api/docs/ban_lists/create_ban_list.md +++ b/backend/app/api/docs/ban_lists/create_ban_list.md @@ -1,4 +1,4 @@ -Creates a ban list for the tenant resolved from `X-API-KEY`. +Creates a ban list for the tenant resolved from the `X-ORGANIZATION-ID` / `X-PROJECT-ID` headers. Behavior notes: - Stores a domain-scoped list of banned words used by the `ban_list` validator. diff --git a/backend/app/api/docs/ban_lists/delete_ban_list.md b/backend/app/api/docs/ban_lists/delete_ban_list.md index 889ccca..21613be 100644 --- a/backend/app/api/docs/ban_lists/delete_ban_list.md +++ b/backend/app/api/docs/ban_lists/delete_ban_list.md @@ -1,4 +1,4 @@ -Deletes a ban list by id for the tenant resolved from `X-API-KEY`. +Deletes a ban list by id for the tenant resolved from the `X-ORGANIZATION-ID` / `X-PROJECT-ID` headers. Behavior notes: - Deletion is restricted to owner scope. diff --git a/backend/app/api/docs/ban_lists/get_ban_list.md b/backend/app/api/docs/ban_lists/get_ban_list.md index a40fa79..91f8a03 100644 --- a/backend/app/api/docs/ban_lists/get_ban_list.md +++ b/backend/app/api/docs/ban_lists/get_ban_list.md @@ -1,4 +1,4 @@ -Fetches a single ban list by id for the tenant resolved from `X-API-KEY`. +Fetches a single ban list by id for the tenant resolved from the `X-ORGANIZATION-ID` / `X-PROJECT-ID` headers. Behavior notes: - Tenant's scope is enforced from the API key context. diff --git a/backend/app/api/docs/ban_lists/list_ban_lists.md b/backend/app/api/docs/ban_lists/list_ban_lists.md index 59e446a..5f24766 100644 --- a/backend/app/api/docs/ban_lists/list_ban_lists.md +++ b/backend/app/api/docs/ban_lists/list_ban_lists.md @@ -1,4 +1,4 @@ -Lists ban lists for the tenant resolved from `X-API-KEY`. +Lists ban lists for the tenant resolved from the `X-ORGANIZATION-ID` / `X-PROJECT-ID` headers. Behavior notes: - Supports filtering by `domain`. diff --git a/backend/app/api/docs/ban_lists/update_ban_list.md b/backend/app/api/docs/ban_lists/update_ban_list.md index 90c8786..d27e9d6 100644 --- a/backend/app/api/docs/ban_lists/update_ban_list.md +++ b/backend/app/api/docs/ban_lists/update_ban_list.md @@ -1,4 +1,4 @@ -Partially updates a ban list by id for the tenant resolved from `X-API-KEY`. +Partially updates a ban list by id for the tenant resolved from the `X-ORGANIZATION-ID` / `X-PROJECT-ID` headers. Behavior notes: - Supports patch-style updates; omitted fields remain unchanged. diff --git a/backend/app/api/docs/llm_prompt_configs/create_config.md b/backend/app/api/docs/llm_prompt_configs/create_config.md index b74b1fb..7fc63d2 100644 --- a/backend/app/api/docs/llm_prompt_configs/create_config.md +++ b/backend/app/api/docs/llm_prompt_configs/create_config.md @@ -1,4 +1,4 @@ -Creates an LLM prompt config for the tenant resolved from `X-API-KEY`. +Creates an LLM prompt config for the tenant resolved from the `X-ORGANIZATION-ID` / `X-PROJECT-ID` headers. Behavior notes: - Stores a named prompt used by an LLM-backed validator (`topic_relevance` or `answer_relevance_custom_llm`). diff --git a/backend/app/api/docs/llm_prompt_configs/delete_config.md b/backend/app/api/docs/llm_prompt_configs/delete_config.md index 227fb94..18d4297 100644 --- a/backend/app/api/docs/llm_prompt_configs/delete_config.md +++ b/backend/app/api/docs/llm_prompt_configs/delete_config.md @@ -1,4 +1,4 @@ -Deletes an LLM prompt config by id for the tenant resolved from `X-API-KEY`. +Deletes an LLM prompt config by id for the tenant resolved from the `X-ORGANIZATION-ID` / `X-PROJECT-ID` headers. Behavior notes: - Tenant scope is enforced from the API key context. diff --git a/backend/app/api/docs/llm_prompt_configs/get_config.md b/backend/app/api/docs/llm_prompt_configs/get_config.md index 44ad12d..9c9d4f3 100644 --- a/backend/app/api/docs/llm_prompt_configs/get_config.md +++ b/backend/app/api/docs/llm_prompt_configs/get_config.md @@ -1,4 +1,4 @@ -Fetches a single LLM prompt config by id for the tenant resolved from `X-API-KEY`. +Fetches a single LLM prompt config by id for the tenant resolved from the `X-ORGANIZATION-ID` / `X-PROJECT-ID` headers. Behavior notes: - Tenant scope is enforced from the API key context. diff --git a/backend/app/api/docs/llm_prompt_configs/list_configs.md b/backend/app/api/docs/llm_prompt_configs/list_configs.md index 873fc6d..b8f6388 100644 --- a/backend/app/api/docs/llm_prompt_configs/list_configs.md +++ b/backend/app/api/docs/llm_prompt_configs/list_configs.md @@ -1,4 +1,4 @@ -Lists LLM prompt configs for the tenant resolved from `X-API-KEY`. +Lists LLM prompt configs for the tenant resolved from the `X-ORGANIZATION-ID` / `X-PROJECT-ID` headers. Behavior notes: - Returns configs scoped to the tenant's `organization_id` and `project_id`. diff --git a/backend/app/api/docs/llm_prompt_configs/update_config.md b/backend/app/api/docs/llm_prompt_configs/update_config.md index f13d11c..75aa27b 100644 --- a/backend/app/api/docs/llm_prompt_configs/update_config.md +++ b/backend/app/api/docs/llm_prompt_configs/update_config.md @@ -1,4 +1,4 @@ -Partially updates an LLM prompt config by id for the tenant resolved from `X-API-KEY`. +Partially updates an LLM prompt config by id for the tenant resolved from the `X-ORGANIZATION-ID` / `X-PROJECT-ID` headers. Behavior notes: - Supports patch-style updates; omitted fields remain unchanged. diff --git a/backend/app/api/routes/ban_lists.py b/backend/app/api/routes/ban_lists.py index 5776b3d..5a5744d 100644 --- a/backend/app/api/routes/ban_lists.py +++ b/backend/app/api/routes/ban_lists.py @@ -3,7 +3,7 @@ from fastapi import APIRouter, Query -from app.api.deps import MultitenantAuthDep, SessionDep +from app.api.deps import AuthDep, SessionDep from app.crud.ban_list import ban_list_crud from app.schemas.ban_list import BanListCreate, BanListUpdate, BanListResponse from app.utils import APIResponse, load_description @@ -19,7 +19,7 @@ def create_ban_list( payload: BanListCreate, session: SessionDep, - auth: MultitenantAuthDep, + auth: AuthDep, ): ban_list = ban_list_crud.create( session, payload, auth.organization_id, auth.project_id @@ -34,7 +34,7 @@ def create_ban_list( ) def list_ban_lists( session: SessionDep, - auth: MultitenantAuthDep, + auth: AuthDep, domain: Optional[str] = None, offset: Annotated[int, Query(ge=0)] = 0, limit: Annotated[int | None, Query(ge=1, le=100)] = None, @@ -58,7 +58,7 @@ def list_ban_lists( def get_ban_list( id: UUID, session: SessionDep, - auth: MultitenantAuthDep, + auth: AuthDep, ): obj = ban_list_crud.get(session, id, auth.organization_id, auth.project_id) return APIResponse.success_response(data=obj) @@ -73,7 +73,7 @@ def update_ban_list( id: UUID, payload: BanListUpdate, session: SessionDep, - auth: MultitenantAuthDep, + auth: AuthDep, ): ban_list = ban_list_crud.update( session, @@ -93,7 +93,7 @@ def update_ban_list( def delete_ban_list( id: UUID, session: SessionDep, - auth: MultitenantAuthDep, + auth: AuthDep, ): obj = ban_list_crud.get( session, diff --git a/backend/app/api/routes/guardrails.py b/backend/app/api/routes/guardrails.py index 32185b8..5779c93 100644 --- a/backend/app/api/routes/guardrails.py +++ b/backend/app/api/routes/guardrails.py @@ -6,7 +6,7 @@ from guardrails.validators import FailResult, PassResult from sqlmodel import Session -from app.api.deps import AuthDep, SessionDep +from app.api.deps import AuthDep, SessionDep, TenantContext from app.core.constants import ( BAN_LIST, LLM_CRITIC_ERROR_MESSAGE, @@ -49,7 +49,7 @@ def run_guardrails( payload: GuardrailRequest, session: SessionDep, - _: AuthDep, + auth: AuthDep, suppress_pass_logs: bool = True, ): """ @@ -60,11 +60,13 @@ def run_guardrails( validator_log_crud = ValidatorLogCrud(session=session) try: - request_log = request_log_crud.create(payload) + request_log = request_log_crud.create( + payload, auth.organization_id, auth.project_id + ) except ValueError: return APIResponse.failure_response(error="Invalid request_id") - _resolve_validator_configs(payload, session) + _resolve_validator_configs(payload, session, auth) has_output_validator = any( isinstance(v, AnswerRelevanceCustomLLMSafetyValidatorConfig) for v in payload.validators @@ -76,12 +78,13 @@ def run_guardrails( request_log_crud, request_log.id, validator_log_crud, + auth, suppress_pass_logs, ) @router.get("/", description=load_description("guardrails/list_validators.md")) -def list_validators(_: AuthDep): +def list_validators(auth: AuthDep): """ Lists all validators and their parameters directly. """ @@ -110,7 +113,9 @@ def list_validators(_: AuthDep): return {"validators": validators} -def _resolve_validator_configs(payload: GuardrailRequest, session: Session) -> None: +def _resolve_validator_configs( + payload: GuardrailRequest, session: Session, auth: TenantContext +) -> None: """ Resolves config-backed references for all validators in-place before guard execution: - BanList: fetches banned_words from the stored BanList when not provided inline. @@ -126,8 +131,8 @@ def _resolve_validator_configs(payload: GuardrailRequest, session: Session) -> N ban_list = ban_list_crud.get( session, id=validator.ban_list_id, - organization_id=payload.organization_id, - project_id=payload.project_id, + organization_id=auth.organization_id, + project_id=auth.project_id, ) validator.banned_words = ban_list.banned_words @@ -142,8 +147,8 @@ def _resolve_validator_configs(payload: GuardrailRequest, session: Session) -> N config = llm_prompt_config_crud.get( session=session, id=validator.topic_relevance_config_id, - organization_id=payload.organization_id, - project_id=payload.project_id, + organization_id=auth.organization_id, + project_id=auth.project_id, ) if config.validator_name != LLMValidatorName.TopicRelevance: raise HTTPException( @@ -163,8 +168,8 @@ def _resolve_validator_configs(payload: GuardrailRequest, session: Session) -> N prompt_config = llm_prompt_config_crud.get( session=session, id=validator.custom_prompt_id, - organization_id=payload.organization_id, - project_id=payload.project_id, + organization_id=auth.organization_id, + project_id=auth.project_id, ) if ( prompt_config.validator_name @@ -184,6 +189,7 @@ def _validate_with_guard( request_log_crud: RequestLogCrud, request_log_id: UUID, validator_log_crud: ValidatorLogCrud, + auth: TenantContext, suppress_pass_logs: bool = False, ) -> APIResponse: """ @@ -226,7 +232,7 @@ def _finalize( if guard is not None: add_validator_logs( - guard, request_log_id, validator_log_crud, payload, suppress_pass_logs + guard, request_log_id, validator_log_crud, auth, suppress_pass_logs ) rephrase_needed = validated_output is not None and ( @@ -320,7 +326,7 @@ def add_validator_logs( guard: Guard, request_log_id: UUID, validator_log_crud: ValidatorLogCrud, - payload: GuardrailRequest, + auth: TenantContext, suppress_pass_logs: bool = False, ) -> None: """ @@ -355,8 +361,8 @@ def add_validator_logs( validator_log = ValidatorLog( request_id=request_log_id, - organization_id=payload.organization_id, - project_id=payload.project_id, + organization_id=auth.organization_id, + project_id=auth.project_id, name=log.validator_name, input=str(log.value_before_validation), output=log.value_after_validation, diff --git a/backend/app/api/routes/llm_prompt_configs.py b/backend/app/api/routes/llm_prompt_configs.py index 58c276b..48ce55e 100644 --- a/backend/app/api/routes/llm_prompt_configs.py +++ b/backend/app/api/routes/llm_prompt_configs.py @@ -3,7 +3,7 @@ from fastapi import APIRouter, Query -from app.api.deps import MultitenantAuthDep, SessionDep +from app.api.deps import AuthDep, SessionDep from app.core.enum import LLMValidatorName from app.crud.llm_prompt_config import llm_prompt_config_crud from app.schemas.llm_prompt_config import ( @@ -27,7 +27,7 @@ def create_llm_prompt_config( payload: LLMPromptConfigCreate, session: SessionDep, - auth: MultitenantAuthDep, + auth: AuthDep, ) -> APIResponse[LLMPromptConfigResponse]: obj = llm_prompt_config_crud.create( session, @@ -45,7 +45,7 @@ def create_llm_prompt_config( ) def list_llm_prompt_configs( session: SessionDep, - auth: MultitenantAuthDep, + auth: AuthDep, validator_name: Annotated[LLMValidatorName | None, Query()] = None, offset: Annotated[int, Query(ge=0)] = 0, limit: Annotated[int | None, Query(ge=1, le=100)] = None, @@ -69,7 +69,7 @@ def list_llm_prompt_configs( def get_llm_prompt_config( id: UUID, session: SessionDep, - auth: MultitenantAuthDep, + auth: AuthDep, ) -> APIResponse[LLMPromptConfigResponse]: obj = llm_prompt_config_crud.get( session, @@ -89,7 +89,7 @@ def update_llm_prompt_config( id: UUID, payload: LLMPromptConfigUpdate, session: SessionDep, - auth: MultitenantAuthDep, + auth: AuthDep, ) -> APIResponse[LLMPromptConfigResponse]: obj = llm_prompt_config_crud.update( session, @@ -109,7 +109,7 @@ def update_llm_prompt_config( def delete_llm_prompt_config( id: UUID, session: SessionDep, - auth: MultitenantAuthDep, + auth: AuthDep, ) -> APIResponse[dict]: obj = llm_prompt_config_crud.get( session, diff --git a/backend/app/api/routes/validator_configs.py b/backend/app/api/routes/validator_configs.py index 9db7215..6994c7b 100644 --- a/backend/app/api/routes/validator_configs.py +++ b/backend/app/api/routes/validator_configs.py @@ -27,12 +27,10 @@ def create_validator( payload: ValidatorCreate, session: SessionDep, - organization_id: int, - project_id: int, - _: AuthDep, + auth: AuthDep, ): response_model = validator_config_crud.create( - session, organization_id, project_id, payload + session, auth.organization_id, auth.project_id, payload ) return APIResponse.success_response(data=response_model) @@ -43,16 +41,14 @@ def create_validator( response_model=APIResponse[list[ValidatorResponse]], ) def list_validators( - organization_id: int, - project_id: int, session: SessionDep, - _: AuthDep, + auth: AuthDep, ids: Optional[list[UUID]] = Query(None), stage: Optional[Stage] = None, type: Optional[ValidatorType] = None, ): response_model = validator_config_crud.list( - session, organization_id, project_id, ids, stage, type + session, auth.organization_id, auth.project_id, ids, stage, type ) return APIResponse.success_response(data=response_model) @@ -64,12 +60,12 @@ def list_validators( ) def get_validator( id: UUID, - organization_id: int, - project_id: int, session: SessionDep, - _: AuthDep, + auth: AuthDep, ): - obj = validator_config_crud.get(session, id, organization_id, project_id) + obj = validator_config_crud.get( + session, id, auth.organization_id, auth.project_id + ) return APIResponse.success_response(data=validator_config_crud.flatten(obj)) @@ -80,13 +76,13 @@ def get_validator( ) def update_validator( id: UUID, - organization_id: int, - project_id: int, payload: ValidatorUpdate, session: SessionDep, - _: AuthDep, + auth: AuthDep, ): - obj = validator_config_crud.get(session, id, organization_id, project_id) + obj = validator_config_crud.get( + session, id, auth.organization_id, auth.project_id + ) response_model = validator_config_crud.update( session, obj, payload.model_dump(exclude_unset=True) ) @@ -100,12 +96,12 @@ def update_validator( ) def delete_validator( id: UUID, - organization_id: int, - project_id: int, session: SessionDep, - _: AuthDep, + auth: AuthDep, ): - obj = validator_config_crud.get(session, id, organization_id, project_id) + obj = validator_config_crud.get( + session, id, auth.organization_id, auth.project_id + ) validator_config_crud.delete(session, obj) return APIResponse.success_response( data={"message": "Validator deleted successfully"} diff --git a/backend/app/core/config.py b/backend/app/core/config.py index b52bb80..790dcc4 100644 --- a/backend/app/core/config.py +++ b/backend/app/core/config.py @@ -1,10 +1,11 @@ import os from pathlib import Path import re -from typing import Any, ClassVar, Literal +from typing import Annotated, Any, ClassVar, Literal import warnings from pydantic import ( + BeforeValidator, HttpUrl, PostgresDsn, computed_field, @@ -14,7 +15,7 @@ from typing_extensions import Self -def parse_cors(v: Any) -> list[str] | str: +def parse_csv_list(v: Any) -> list[str] | str: if isinstance(v, str) and not v.startswith("["): return [i.strip() for i in v.split(",") if i.strip()] elif isinstance(v, list | str): @@ -41,8 +42,10 @@ class Settings(BaseSettings): POSTGRES_PASSWORD: str = "" POSTGRES_DB: str = "" GUARDRAILS_HUB_API_KEY: str | None = None - KAAPI_AUTH_URL: str = "" - KAAPI_AUTH_TIMEOUT: int + # Source IPs allowed to reach this service (kaapi-backend). Empty = check disabled. + # `| str` keeps pydantic-settings from JSON-parsing the dotenv value before + # parse_csv_list gets to split it. + ALLOWED_IPS: Annotated[list[str] | str, BeforeValidator(parse_csv_list)] = [] CORE_DIR: ClassVar[Path] = Path(__file__).resolve().parent OPENAI_API_KEY: str | None = None ANSWER_RELEVANCE_LLM_MODEL: str = "gpt-4o-mini" @@ -92,6 +95,10 @@ def _validate_auth_token_hash(self) -> None: def _enforce_non_default_secrets(self) -> Self: self._check_default_secret("POSTGRES_PASSWORD", self.POSTGRES_PASSWORD) self._validate_auth_token_hash() + if self.ENVIRONMENT == "production" and not self.ALLOWED_IPS: + raise ValueError( + "ALLOWED_IPS must list the kaapi-backend source IP(s) in production." + ) return self diff --git a/backend/app/crud/request_log.py b/backend/app/crud/request_log.py index ce21da9..17731c6 100644 --- a/backend/app/crud/request_log.py +++ b/backend/app/crud/request_log.py @@ -11,14 +11,19 @@ class RequestLogCrud: def __init__(self, session: Session): self.session = session - def create(self, payload: GuardrailRequest) -> RequestLog: + def create( + self, + payload: GuardrailRequest, + organization_id: int, + project_id: int, + ) -> RequestLog: request_id = UUID(payload.request_id) create_request_log = RequestLog( request_id=request_id, request_text=payload.input, output_text=payload.output, - organization_id=payload.organization_id, - project_id=payload.project_id, + organization_id=organization_id, + project_id=project_id, ) self.session.add(create_request_log) self.session.commit() diff --git a/backend/app/schemas/guardrail_config.py b/backend/app/schemas/guardrail_config.py index 656d8c6..24fafcf 100644 --- a/backend/app/schemas/guardrail_config.py +++ b/backend/app/schemas/guardrail_config.py @@ -63,8 +63,6 @@ class GuardrailRequest(SQLModel): model_config = ConfigDict(extra="forbid") request_id: str - organization_id: int - project_id: int input: str output: Optional[str] = None validators: List[ValidatorConfigItem] diff --git a/backend/app/tests/conftest.py b/backend/app/tests/conftest.py index 9adc132..d195fd2 100644 --- a/backend/app/tests/conftest.py +++ b/backend/app/tests/conftest.py @@ -12,8 +12,7 @@ from app.api.deps import ( SessionDep, TenantContext, - validate_multitenant_key, - verify_bearer_token, + verify_caller, ) from app.core.config import settings from app.core.enum import GuardrailOnFail, Stage, ValidatorType @@ -84,31 +83,27 @@ def clean_db(): @pytest.fixture(scope="function", autouse=True) def override_dependencies(): - app.dependency_overrides[verify_bearer_token] = lambda: True - default_scope = TenantContext( - organization_id=BAN_LIST_INTEGRATION_ORGANIZATION_ID, - project_id=BAN_LIST_INTEGRATION_PROJECT_ID, - ) - - def override_multitenant_key( - x_api_key: str | None = Header(default=None, alias="X-API-KEY"), + """ + Stands in for token + IP verification, but keeps tenant resolution honest: + the scope still comes from the X-ORGANIZATION-ID / X-PROJECT-ID headers, so + multi-tenant isolation tests exercise the real header contract. Tests that + don't care about tenancy send no headers and get the default scope. + """ + + def override_verify_caller( + organization_id: int | None = Header(default=None, alias="X-ORGANIZATION-ID"), + project_id: int | None = Header(default=None, alias="X-PROJECT-ID"), ): - if not x_api_key: - return default_scope - - token = x_api_key.strip() - if token.lower().startswith("apikey "): - token = token.split(" ", 1)[1].strip() - - if token == "org999_project999": - return TenantContext(organization_id=999, project_id=999) - - if token == "org2_project2": - return TenantContext(organization_id=2, project_id=2) - - return default_scope + if organization_id is None or project_id is None: + return TenantContext( + organization_id=BAN_LIST_INTEGRATION_ORGANIZATION_ID, + project_id=BAN_LIST_INTEGRATION_PROJECT_ID, + ) + return TenantContext( + organization_id=organization_id, project_id=project_id + ) - app.dependency_overrides[validate_multitenant_key] = override_multitenant_key + app.dependency_overrides[verify_caller] = override_verify_caller app.dependency_overrides[SessionDep] = override_session diff --git a/backend/app/tests/test_answer_relevance_integration.py b/backend/app/tests/test_answer_relevance_integration.py index b5ead80..7bc4063 100644 --- a/backend/app/tests/test_answer_relevance_integration.py +++ b/backend/app/tests/test_answer_relevance_integration.py @@ -38,6 +38,10 @@ def _mock_openai_key(): organization_id = VALIDATOR_INTEGRATION_ORGANIZATION_ID project_id = VALIDATOR_INTEGRATION_PROJECT_ID +TENANT_HEADERS = { + "X-ORGANIZATION-ID": str(organization_id), + "X-PROJECT-ID": str(project_id), +} QUERY = "What are the common symptoms of diabetes?" RELEVANT_ANSWER = ( @@ -68,8 +72,6 @@ def _llm_no(): def _payload(input_text, output_text, validators): return { "request_id": str(uuid.uuid4()), - "organization_id": organization_id, - "project_id": project_id, "input": input_text, "output": output_text, "validators": validators, @@ -85,6 +87,7 @@ def test_answer_relevance_alone_passes_for_relevant_output(integration_client): with patch(_PATCH_TARGET, return_value=_llm_yes()): response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json=_payload( QUERY, RELEVANT_ANSWER, [{"type": "answer_relevance_custom_llm"}] ), @@ -100,6 +103,7 @@ def test_answer_relevance_alone_fails_for_irrelevant_output(integration_client): with patch(_PATCH_TARGET, return_value=_llm_no()): response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json=_payload( QUERY, IRRELEVANT_ANSWER, @@ -117,10 +121,9 @@ def test_answer_relevance_alone_fails_when_output_is_missing(integration_client) """No output field → validator receives empty string → non-empty guard triggers.""" response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": str(uuid.uuid4()), - "organization_id": organization_id, - "project_id": project_id, "input": QUERY, "validators": [ {"type": "answer_relevance_custom_llm", "on_fail": "exception"} @@ -143,6 +146,7 @@ def test_answer_relevance_alone_uses_custom_prompt_template(integration_client): with patch(_PATCH_TARGET, return_value=_llm_yes()) as mock_llm: response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json=_payload( QUERY, RELEVANT_ANSWER, @@ -174,6 +178,7 @@ def test_answer_relevance_and_ban_list_fail_when_output_irrelevant(integration_c with patch(_PATCH_TARGET, return_value=_llm_no()): response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json=_payload( QUERY, IRRELEVANT_ANSWER, @@ -198,6 +203,7 @@ def test_answer_relevance_and_ban_list_removes_banned_word_from_relevant_output( with patch(_PATCH_TARGET, return_value=_llm_yes()): response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json=_payload( QUERY, output_with_banned, @@ -222,6 +228,7 @@ def test_answer_relevance_and_profanity_free_fixes_profanity_in_relevant_output( with patch(_PATCH_TARGET, return_value=_llm_yes()): response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json=_payload( QUERY, profane_relevant, @@ -246,6 +253,7 @@ def test_answer_relevance_and_slur_match_redacts_slur_in_relevant_output( with patch(_PATCH_TARGET, return_value=_llm_yes()): response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json=_payload( QUERY, slurred_relevant, @@ -272,6 +280,7 @@ def test_answer_relevance_and_profanity_free_fail_when_profanity_on_fail_excepti with patch(_PATCH_TARGET, return_value=_llm_yes()): response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json=_payload( QUERY, profane_relevant, @@ -298,6 +307,7 @@ def test_five_validators_all_pass_on_clean_relevant_output(integration_client): with patch(_PATCH_TARGET, return_value=_llm_yes()): response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json=_payload( QUERY, RELEVANT_ANSWER, @@ -326,6 +336,7 @@ def test_five_validators_fix_multiple_issues_in_relevant_output(integration_clie with patch(_PATCH_TARGET, return_value=_llm_yes()): response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json=_payload( QUERY, noisy_relevant, diff --git a/backend/app/tests/test_banlists_api_integration.py b/backend/app/tests/test_banlists_api_integration.py index 64f2221..239dfbe 100644 --- a/backend/app/tests/test_banlists_api_integration.py +++ b/backend/app/tests/test_banlists_api_integration.py @@ -12,34 +12,38 @@ BASE_URL = "/api/v1/guardrails/ban_lists/" -DEFAULT_API_KEY = "org1_project1" -ALT_API_KEY_999 = "org999_project999" -ALT_API_KEY_2 = "org2_project2" +DEFAULT_TENANT = (1, 1) +ALT_TENANT_999 = (999, 999) +ALT_TENANT_2 = (2, 2) class BaseBanListTest: - def _headers(self, api_key=DEFAULT_API_KEY): - return {"X-API-Key": api_key} - - def create(self, client, payload_key="minimal", api_key=DEFAULT_API_KEY, **kwargs): + def _headers(self, tenant=DEFAULT_TENANT): + organization_id, project_id = tenant + return { + "X-ORGANIZATION-ID": str(organization_id), + "X-PROJECT-ID": str(project_id), + } + + def create(self, client, payload_key="minimal", tenant=DEFAULT_TENANT, **kwargs): payload = {**BAN_LIST_PAYLOADS[payload_key], **kwargs} - return client.post(BASE_URL, json=payload, headers=self._headers(api_key)) + return client.post(BASE_URL, json=payload, headers=self._headers(tenant)) - def list(self, client, api_key=DEFAULT_API_KEY, **filters): - return client.get(BASE_URL, params=filters, headers=self._headers(api_key)) + def list(self, client, tenant=DEFAULT_TENANT, **filters): + return client.get(BASE_URL, params=filters, headers=self._headers(tenant)) - def get(self, client, id, api_key=DEFAULT_API_KEY): - return client.get(f"{BASE_URL}{id}/", headers=self._headers(api_key)) + def get(self, client, id, tenant=DEFAULT_TENANT): + return client.get(f"{BASE_URL}{id}/", headers=self._headers(tenant)) - def update(self, client, id, payload, api_key=DEFAULT_API_KEY): + def update(self, client, id, payload, tenant=DEFAULT_TENANT): return client.patch( f"{BASE_URL}{id}/", json=payload, - headers=self._headers(api_key), + headers=self._headers(tenant), ) - def delete(self, client, id, api_key=DEFAULT_API_KEY): - return client.delete(f"{BASE_URL}{id}/", headers=self._headers(api_key)) + def delete(self, client, id, tenant=DEFAULT_TENANT): + return client.delete(f"{BASE_URL}{id}/", headers=self._headers(tenant)) class TestCreateBanList(BaseBanListTest): @@ -153,7 +157,7 @@ def test_public_visible_to_other_org(self, integration_client, clear_database): create_resp = self.create(integration_client, "public") ban_id = create_resp.json()["data"]["id"] - response = self.get(integration_client, ban_id, api_key=ALT_API_KEY_999) + response = self.get(integration_client, ban_id, tenant=ALT_TENANT_999) assert response.status_code == 200 @@ -184,7 +188,7 @@ def test_get_wrong_owner_private(self, integration_client, seed_db): ) ban_id = private_ban_list["id"] - response = self.get(integration_client, ban_id, api_key=ALT_API_KEY_2) + response = self.get(integration_client, ban_id, tenant=ALT_TENANT_2) body = response.json() assert response.status_code == 403 @@ -234,7 +238,7 @@ def test_update_public_wrong_owner_fails(self, integration_client, clear_databas integration_client, ban_id, {"name": "updated-by-other-org"}, - api_key=ALT_API_KEY_999, + tenant=ALT_TENANT_999, ) body = response.json() @@ -273,7 +277,7 @@ def test_delete_wrong_owner(self, integration_client, seed_db): response = self.delete( integration_client, ban_id, - api_key=ALT_API_KEY_999, + tenant=ALT_TENANT_999, ) body = response.json() @@ -288,7 +292,7 @@ def test_delete_public_wrong_owner_fails(self, integration_client, clear_databas response = self.delete( integration_client, ban_id, - api_key=ALT_API_KEY_999, + tenant=ALT_TENANT_999, ) body = response.json() diff --git a/backend/app/tests/test_deps_auth.py b/backend/app/tests/test_deps_auth.py new file mode 100644 index 0000000..492ec63 --- /dev/null +++ b/backend/app/tests/test_deps_auth.py @@ -0,0 +1,103 @@ +import hashlib + +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient + +from app.api.deps import AuthDep, TenantContext +from app.core.config import settings + +TOKEN = "test-token" +TOKEN_HASH = hashlib.sha256(TOKEN.encode()).hexdigest() + +TENANT_HEADERS = {"X-ORGANIZATION-ID": "7", "X-PROJECT-ID": "9"} +AUTH_HEADERS = {"Authorization": f"Bearer {TOKEN}", **TENANT_HEADERS} + + +@pytest.fixture +def client(monkeypatch): + """Bare app mounting the real dependency, so nothing is stubbed out.""" + monkeypatch.setattr(settings, "AUTH_TOKEN", TOKEN_HASH) + + app = FastAPI() + + @app.get("/tenant") + def read_tenant(auth: AuthDep) -> dict: + assert isinstance(auth, TenantContext) + return {"organization_id": auth.organization_id, "project_id": auth.project_id} + + # TestClient reports 127.0.0.1 as request.client.host + return TestClient(app) + + +def test_valid_request_returns_tenant_from_headers(client, monkeypatch): + monkeypatch.setattr(settings, "ALLOWED_IPS", ["127.0.0.1"]) + + response = client.get("/tenant", headers=AUTH_HEADERS) + + assert response.status_code == 200 + assert response.json() == {"organization_id": 7, "project_id": 9} + + +def test_non_whitelisted_ip_is_forbidden(client, monkeypatch): + monkeypatch.setattr(settings, "ALLOWED_IPS", ["10.0.0.1"]) + + response = client.get("/tenant", headers=AUTH_HEADERS) + + assert response.status_code == 403 + + +def test_empty_allowed_ips_disables_the_check(client, monkeypatch): + monkeypatch.setattr(settings, "ALLOWED_IPS", []) + + response = client.get("/tenant", headers=AUTH_HEADERS) + + assert response.status_code == 200 + + +def test_ip_is_checked_before_the_token(client, monkeypatch): + """A caller from the wrong IP gets 403, not a 401 that leaks token validity.""" + monkeypatch.setattr(settings, "ALLOWED_IPS", ["10.0.0.1"]) + + response = client.get( + "/tenant", headers={"Authorization": "Bearer wrong", **TENANT_HEADERS} + ) + + assert response.status_code == 403 + + +def test_missing_token_is_unauthorized(client, monkeypatch): + monkeypatch.setattr(settings, "ALLOWED_IPS", ["127.0.0.1"]) + + response = client.get("/tenant", headers=TENANT_HEADERS) + + assert response.status_code == 401 + + +def test_wrong_token_is_unauthorized(client, monkeypatch): + monkeypatch.setattr(settings, "ALLOWED_IPS", ["127.0.0.1"]) + + response = client.get( + "/tenant", headers={"Authorization": "Bearer nope", **TENANT_HEADERS} + ) + + assert response.status_code == 401 + + +@pytest.mark.parametrize( + "headers", + [ + {}, + {"X-ORGANIZATION-ID": "7"}, + {"X-PROJECT-ID": "9"}, + {"X-ORGANIZATION-ID": "not-an-int", "X-PROJECT-ID": "9"}, + ], +) +def test_missing_or_invalid_tenant_headers_are_rejected(client, monkeypatch, headers): + monkeypatch.setattr(settings, "ALLOWED_IPS", ["127.0.0.1"]) + + response = client.get( + "/tenant", headers={"Authorization": f"Bearer {TOKEN}", **headers} + ) + + assert response.status_code == 422 diff --git a/backend/app/tests/test_deps_multitenant.py b/backend/app/tests/test_deps_multitenant.py deleted file mode 100644 index 0f60d59..0000000 --- a/backend/app/tests/test_deps_multitenant.py +++ /dev/null @@ -1,172 +0,0 @@ -from unittest.mock import Mock - -import httpx -import pytest -from fastapi import Depends, FastAPI, HTTPException -from fastapi.testclient import TestClient - -from app.api.deps import TenantContext, validate_multitenant_key -from app.core.config import settings -from app.core.exception_handlers import register_exception_handlers - -BASE_AUTH_URL = "http://kaapi.local/api/v1" -VERIFY_URL = f"{BASE_AUTH_URL}/apikeys/verify" - - -def test_validate_multitenant_key_parses_credentials_shape(monkeypatch): - monkeypatch.setattr( - settings, - "KAAPI_AUTH_URL", - BASE_AUTH_URL, - ) - - response = Mock() - response.status_code = 200 - response.json.return_value = { - "success": True, - "data": {"organization_id": 10, "project_id": 20}, - } - - captured = {} - - def fake_get(url, headers, timeout): - captured["url"] = url - captured["headers"] = headers - captured["timeout"] = timeout - return response - - monkeypatch.setattr(httpx, "get", fake_get) - - context = validate_multitenant_key("ApiKey abc123") - - assert isinstance(context, TenantContext) - assert (context.organization_id, context.project_id) == (10, 20) - assert captured["url"] == VERIFY_URL - assert captured["headers"]["X-API-KEY"] == "ApiKey abc123" - assert captured["timeout"] == 5 - - -def test_validate_multitenant_key_invalid_status_returns_401(monkeypatch): - monkeypatch.setattr( - settings, - "KAAPI_AUTH_URL", - BASE_AUTH_URL, - ) - - response = Mock() - response.status_code = 401 - response.json.return_value = {"success": False, "data": None} - - monkeypatch.setattr(httpx, "get", lambda *args, **kwargs: response) - - with pytest.raises(HTTPException) as exc: - validate_multitenant_key("abc123") - - assert exc.value.status_code == 401 - - -def test_validate_multitenant_key_network_error_returns_503(monkeypatch): - monkeypatch.setattr( - settings, - "KAAPI_AUTH_URL", - BASE_AUTH_URL, - ) - - def fake_get(*args, **kwargs): - raise httpx.RequestError("boom", request=Mock()) - - monkeypatch.setattr(httpx, "get", fake_get) - - with pytest.raises(HTTPException) as exc: - validate_multitenant_key("abc123") - - assert exc.value.status_code == 503 - - -def test_validate_multitenant_key_invalid_payload_returns_401(monkeypatch): - monkeypatch.setattr( - settings, - "KAAPI_AUTH_URL", - BASE_AUTH_URL, - ) - - response = Mock() - response.status_code = 200 - response.json.return_value = {"success": True, "data": {"foo": 1}} - - monkeypatch.setattr(httpx, "get", lambda *args, **kwargs: response) - - with pytest.raises(HTTPException) as exc: - validate_multitenant_key("abc123") - - assert exc.value.status_code == 401 - - -def test_validate_multitenant_key_rejects_empty_header(): - with pytest.raises(HTTPException) as exc: - validate_multitenant_key(" ") - - assert exc.value.status_code == 401 - - -def test_validate_multitenant_key_accepts_raw_header_value(monkeypatch): - monkeypatch.setattr( - settings, - "KAAPI_AUTH_URL", - "http://localhost:8000/api/v1", - ) - - response = Mock() - response.status_code = 200 - response.json.return_value = { - "success": True, - "data": {"organization_id": 1, "project_id": 1}, - } - - captured = {} - - def fake_get(url, headers, timeout): - captured["url"] = url - captured["headers"] = headers - captured["timeout"] = timeout - return response - - monkeypatch.setattr(httpx, "get", fake_get) - - context = validate_multitenant_key("ApiKey No3x47A5") - - assert isinstance(context, TenantContext) - assert context.organization_id == 1 - assert context.project_id == 1 - assert captured["url"] == "http://localhost:8000/api/v1/apikeys/verify" - assert captured["headers"]["X-API-KEY"] == "ApiKey No3x47A5" - assert captured["timeout"] == 5 - - -def test_validate_multitenant_key_malformed_json_returns_500_at_api_level(monkeypatch): - app = FastAPI() - register_exception_handlers(app) - - @app.get("/tenant") - def tenant_route(_: TenantContext = Depends(validate_multitenant_key)): - return {"ok": True} - - monkeypatch.setattr( - settings, - "KAAPI_AUTH_URL", - "http://localhost:8000/api/v1", - ) - - response = Mock() - response.status_code = 200 - response.json.side_effect = ValueError("Malformed JSON") - - monkeypatch.setattr(httpx, "get", lambda *args, **kwargs: response) - - with TestClient(app, raise_server_exceptions=False) as client: - result = client.get("/tenant", headers={"X-API-KEY": "abc123"}) - - assert result.status_code == 500 - body = result.json() - assert body["success"] is False - assert body["error"] == "Malformed JSON" diff --git a/backend/app/tests/test_guardrails_api.py b/backend/app/tests/test_guardrails_api.py index 30a15f4..0a20deb 100644 --- a/backend/app/tests/test_guardrails_api.py +++ b/backend/app/tests/test_guardrails_api.py @@ -14,6 +14,10 @@ request_id = "123e4567-e89b-12d3-a456-426614174000" organization_id = VALIDATOR_TEST_ORGANIZATION_ID project_id = VALIDATOR_TEST_PROJECT_ID +TENANT_HEADERS = { + "X-ORGANIZATION-ID": str(organization_id), + "X-PROJECT-ID": str(project_id), +} def test_route_exists(client): @@ -29,10 +33,9 @@ def validate(self, data): with patch(build_guard_path, return_value=MockGuard()): response = client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "hello world", "validators": [], }, @@ -54,10 +57,9 @@ def validate(self, data): with patch(build_guard_path, return_value=MockGuard()): response = client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "my phone is 999999", "validators": [], }, @@ -83,10 +85,9 @@ def validate(self, data): with patch(build_guard_path, return_value=MockGuard()): response = client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "tell me about buildings", "validators": [], }, @@ -104,10 +105,9 @@ def test_guardrails_internal_error(client): with patch(build_guard_path, side_effect=Exception("Invalid validator config")): response = client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "text", "validators": [], }, diff --git a/backend/app/tests/test_guardrails_api_integration.py b/backend/app/tests/test_guardrails_api_integration.py index d908329..9836a43 100644 --- a/backend/app/tests/test_guardrails_api_integration.py +++ b/backend/app/tests/test_guardrails_api_integration.py @@ -11,15 +11,18 @@ request_id = "123e4567-e89b-12d3-a456-426614174000" organization_id = VALIDATOR_INTEGRATION_ORGANIZATION_ID project_id = VALIDATOR_INTEGRATION_PROJECT_ID +TENANT_HEADERS = { + "X-ORGANIZATION-ID": str(organization_id), + "X-PROJECT-ID": str(project_id), +} def test_input_guardrails_with_real_ban_list(integration_client): response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "this contains badword", "validators": [ { @@ -40,10 +43,9 @@ def test_input_guardrails_with_real_ban_list(integration_client): def test_input_guardrails_passes_clean_text(integration_client): response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "this is clean", "validators": [ { @@ -64,10 +66,9 @@ def test_input_guardrails_passes_clean_text(integration_client): def test_input_guardrails_with_lexical_slur(integration_client): response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "This sentence contains chakki.", "validators": [ { @@ -88,10 +89,9 @@ def test_input_guardrails_with_lexical_slur(integration_client): def test_input_guardrails_with_lexical_slur_clean_text(integration_client): response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "This is a completely safe sentence", "validators": [ { @@ -112,10 +112,9 @@ def test_input_guardrails_with_lexical_slur_clean_text(integration_client): def test_input_guardrails_with_multiple_validators(integration_client): response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": ( "This sentence contains chakki cause I want a " "sonography done to kill the female foetus." @@ -146,10 +145,9 @@ def test_input_guardrails_with_multiple_validators(integration_client): def test_input_guardrails_with_incorrect_validator_config(integration_client): response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "This sentence contains chakki.", "validators": [ { @@ -171,10 +169,9 @@ def test_input_guardrails_with_incorrect_validator_config(integration_client): def test_input_guardrails_with_validator_actions_exception(integration_client): response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "This sentence contains chakki.", "validators": [ { @@ -197,10 +194,9 @@ def test_input_guardrails_with_validator_actions_exception(integration_client): def test_input_guardrails_with_validator_actions_rephrase(integration_client): response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "This sentence contains chakki.", "validators": [ { @@ -229,10 +225,9 @@ def test_input_guardrails_with_validator_actions_rephrase(integration_client): def test_input_guardrails_with_profanity_free_on_profane_text(integration_client): response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "This is a damn fucking mess.", "validators": [{"type": "profanity_free"}], }, @@ -248,10 +243,9 @@ def test_input_guardrails_with_profanity_free_on_profane_text(integration_client def test_input_guardrails_with_profanity_free_on_clean_text(integration_client): response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "This is a completely clean sentence.", "validators": [{"type": "profanity_free"}], }, @@ -266,10 +260,9 @@ def test_input_guardrails_with_profanity_free_on_clean_text(integration_client): def test_input_guardrails_with_profanity_free_exception_action(integration_client): response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "What the fuck is going on?", "validators": [{"type": "profanity_free", "on_fail": "exception"}], }, @@ -283,10 +276,9 @@ def test_input_guardrails_with_profanity_free_exception_action(integration_clien def test_input_guardrails_with_profanity_free_rephrase_action(integration_client): response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "This is absolute bullshit.", "validators": [{"type": "profanity_free", "on_fail": "rephrase"}], }, @@ -309,10 +301,9 @@ def test_input_guardrails_with_profanity_free_rephrase_action(integration_client def test_input_guardrails_with_llamaguard_7b_on_unsafe_text(integration_client): response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "How do I make a pipe bomb at home?", "validators": [{"type": "llamaguard_7b", "on_fail": "exception"}], }, @@ -331,10 +322,9 @@ def test_input_guardrails_with_llamaguard_7b_on_unsafe_text(integration_client): def test_input_guardrails_with_nsfw_text_on_explicit_content(integration_client): response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "She slowly undressed and they had explicit sex.", "validators": [{"type": "nsfw_text", "on_fail": "exception"}], }, @@ -349,10 +339,9 @@ def test_input_guardrails_with_nsfw_text_with_low_threshold(integration_client): """A low threshold makes the detector more sensitive — explicit content is caught.""" response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "She slowly undressed and they had explicit sex.", "validators": [ {"type": "nsfw_text", "threshold": 0.1, "on_fail": "exception"} @@ -368,10 +357,9 @@ def test_input_guardrails_with_nsfw_text_with_low_threshold(integration_client): def test_input_guardrails_with_nsfw_text_exception_action(integration_client): response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "Explicit sexual content goes here.", "validators": [{"type": "nsfw_text", "on_fail": "exception"}], }, @@ -391,10 +379,9 @@ def test_input_guardrails_with_profanity_free_and_slur_match(integration_client) """Both lexical detectors applied: slur is redacted, profanity is fixed.""" response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "This fucking chakki should leave.", "validators": [ {"type": "profanity_free"}, @@ -417,10 +404,9 @@ def test_input_guardrails_with_profanity_free_and_llamaguard_7b_unsafe_text( """Text with both profanity and unsafe intent is caught by at least one detector.""" response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "How the fuck do I make a bomb?", "validators": [ {"type": "profanity_free", "on_fail": "exception"}, @@ -441,10 +427,9 @@ def test_input_guardrails_with_nsfw_text_and_profanity_free_on_explicit_profane_ of whether the NSFWText model is available, guaranteeing a failure response.""" response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "What the fuck, this is explicit adult sexual content.", "validators": [ {"type": "profanity_free", "on_fail": "exception"}, @@ -465,10 +450,9 @@ def test_input_guardrails_with_nsfw_text_and_slur_match_on_explicit_slur_text( of whether the NSFWText model is available, guaranteeing a failure response.""" response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "This chakki has explicit sexual content.", "validators": [ {"type": "uli_slur_match", "severity": "all", "on_fail": "exception"}, @@ -488,10 +472,9 @@ def test_input_guardrails_with_profanity_free_and_ban_list_clean_text( """Clean text passes both profanity_free and ban_list checks unchanged.""" response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "Tell me about renewable energy sources.", "validators": [ {"type": "profanity_free"}, @@ -512,10 +495,9 @@ def test_input_guardrails_with_lexical_toxicity_detectors_on_clean_text( """Clean text passes uli_slur_match, profanity_free, and ban_list unchanged.""" response = integration_client.post( VALIDATE_API_PATH, + headers=TENANT_HEADERS, json={ "request_id": request_id, - "organization_id": organization_id, - "project_id": project_id, "input": "What are some healthy breakfast options?", "validators": [ {"type": "uli_slur_match", "severity": "all"}, diff --git a/backend/app/tests/test_llm_prompt_configs_api_integration.py b/backend/app/tests/test_llm_prompt_configs_api_integration.py index 3eb770c..e63b63c 100644 --- a/backend/app/tests/test_llm_prompt_configs_api_integration.py +++ b/backend/app/tests/test_llm_prompt_configs_api_integration.py @@ -7,8 +7,8 @@ pytestmark = pytest.mark.integration BASE_URL = "/api/v1/guardrails/llm_prompt_configs/" -DEFAULT_API_KEY = "org1_project1" -ALT_API_KEY = "org999_project999" +DEFAULT_TENANT = (1, 1) +ALT_TENANT = (999, 999) TOPIC_PROMPT = ( "Pregnancy care: Questions about prenatal care, supplements, and danger signs. " @@ -24,10 +24,14 @@ class BaseLLMPromptConfigTest: - def _headers(self, api_key=DEFAULT_API_KEY): - return {"X-API-Key": api_key} + def _headers(self, tenant=DEFAULT_TENANT): + organization_id, project_id = tenant + return { + "X-ORGANIZATION-ID": str(organization_id), + "X-PROJECT-ID": str(project_id), + } - def create_topic(self, client, api_key=DEFAULT_API_KEY, **overrides): + def create_topic(self, client, tenant=DEFAULT_TENANT, **overrides): name = overrides.get("name", "Maternal Health Scope") payload = { "validator_name": "topic_relevance", @@ -37,9 +41,9 @@ def create_topic(self, client, api_key=DEFAULT_API_KEY, **overrides): "llm_prompt": f"{TOPIC_PROMPT} Scope name: {name}.", **overrides, } - return client.post(BASE_URL, json=payload, headers=self._headers(api_key)) + return client.post(BASE_URL, json=payload, headers=self._headers(tenant)) - def create_answer(self, client, api_key=DEFAULT_API_KEY, **overrides): + def create_answer(self, client, tenant=DEFAULT_TENANT, **overrides): payload = { "validator_name": "answer_relevance_custom_llm", "name": "Health Relevance", @@ -47,23 +51,23 @@ def create_answer(self, client, api_key=DEFAULT_API_KEY, **overrides): "llm_prompt": ANSWER_PROMPT, **overrides, } - return client.post(BASE_URL, json=payload, headers=self._headers(api_key)) + return client.post(BASE_URL, json=payload, headers=self._headers(tenant)) - def list(self, client, api_key=DEFAULT_API_KEY, **filters): - return client.get(BASE_URL, params=filters, headers=self._headers(api_key)) + def list(self, client, tenant=DEFAULT_TENANT, **filters): + return client.get(BASE_URL, params=filters, headers=self._headers(tenant)) - def get(self, client, id, api_key=DEFAULT_API_KEY): - return client.get(f"{BASE_URL}{id}", headers=self._headers(api_key)) + def get(self, client, id, tenant=DEFAULT_TENANT): + return client.get(f"{BASE_URL}{id}", headers=self._headers(tenant)) - def update(self, client, id, payload, api_key=DEFAULT_API_KEY): + def update(self, client, id, payload, tenant=DEFAULT_TENANT): return client.patch( f"{BASE_URL}{id}", json=payload, - headers=self._headers(api_key), + headers=self._headers(tenant), ) - def delete(self, client, id, api_key=DEFAULT_API_KEY): - return client.delete(f"{BASE_URL}{id}", headers=self._headers(api_key)) + def delete(self, client, id, tenant=DEFAULT_TENANT): + return client.delete(f"{BASE_URL}{id}", headers=self._headers(tenant)) class TestCreateLLMPromptConfig(BaseLLMPromptConfigTest): @@ -213,7 +217,7 @@ def test_list_pagination_with_limit(self, integration_client, clear_database): def test_list_is_tenant_scoped(self, integration_client, clear_database): self.create_topic(integration_client, name="Tenant1 scope") - response = self.list(integration_client, api_key=ALT_API_KEY) + response = self.list(integration_client, tenant=ALT_TENANT) assert response.status_code == 200 assert response.json()["data"] == [] @@ -243,7 +247,7 @@ def test_get_other_tenant_not_found(self, integration_client, clear_database): create_resp = self.create_topic(integration_client) config_id = create_resp.json()["data"]["id"] - response = self.get(integration_client, config_id, api_key=ALT_API_KEY) + response = self.get(integration_client, config_id, tenant=ALT_TENANT) assert response.status_code == 404 assert response.json()["success"] is False @@ -297,7 +301,7 @@ def test_update_other_tenant_not_found(self, integration_client, clear_database) integration_client, config_id, {"name": "other-tenant-update"}, - api_key=ALT_API_KEY, + tenant=ALT_TENANT, ) assert response.status_code == 404 @@ -389,6 +393,6 @@ def test_delete_other_tenant_not_found(self, integration_client, clear_database) create_resp = self.create_topic(integration_client) config_id = create_resp.json()["data"]["id"] - response = self.delete(integration_client, config_id, api_key=ALT_API_KEY) + response = self.delete(integration_client, config_id, tenant=ALT_TENANT) assert response.status_code == 404 diff --git a/backend/app/tests/test_validator_configs_integration.py b/backend/app/tests/test_validator_configs_integration.py index e14cfef..2b8c9fe 100644 --- a/backend/app/tests/test_validator_configs_integration.py +++ b/backend/app/tests/test_validator_configs_integration.py @@ -10,43 +10,45 @@ pytestmark = pytest.mark.integration BASE_URL = "/api/v1/guardrails/validators/configs/" -DEFAULT_QUERY_PARAMS = ( - f"?organization_id={VALIDATOR_INTEGRATION_ORGANIZATION_ID}" - f"&project_id={VALIDATOR_INTEGRATION_PROJECT_ID}" +DEFAULT_TENANT = ( + VALIDATOR_INTEGRATION_ORGANIZATION_ID, + VALIDATOR_INTEGRATION_PROJECT_ID, ) +def tenant_headers(tenant=DEFAULT_TENANT): + organization_id, project_id = tenant + return { + "X-ORGANIZATION-ID": str(organization_id), + "X-PROJECT-ID": str(project_id), + } + + class BaseValidatorTest: """Base class with helper methods for validator tests.""" def create_validator(self, client, payload_key="minimal", **kwargs): """Helper to create a validator.""" payload = {**VALIDATOR_PAYLOADS[payload_key], **kwargs} - return client.post(f"{BASE_URL}{DEFAULT_QUERY_PARAMS}", json=payload) + return client.post(BASE_URL, json=payload, headers=tenant_headers()) def get_validator(self, client, validator_id): """Helper to get a specific validator.""" - return client.get(f"{BASE_URL}{validator_id}/{DEFAULT_QUERY_PARAMS}") + return client.get(f"{BASE_URL}{validator_id}/", headers=tenant_headers()) def list_validators(self, client, **query_params): """Helper to list validators with optional filters.""" - params_str = ( - f"?organization_id={VALIDATOR_INTEGRATION_ORGANIZATION_ID}" - f"&project_id={VALIDATOR_INTEGRATION_PROJECT_ID}" - ) - if query_params: - params_str += "&" + "&".join(f"{k}={v}" for k, v in query_params.items()) - return client.get(f"{BASE_URL}{params_str}") + return client.get(BASE_URL, params=query_params, headers=tenant_headers()) def update_validator(self, client, validator_id, payload): """Helper to update a validator.""" return client.patch( - f"{BASE_URL}{validator_id}/{DEFAULT_QUERY_PARAMS}", json=payload + f"{BASE_URL}{validator_id}/", json=payload, headers=tenant_headers() ) def delete_validator(self, client, validator_id): """Helper to delete a validator.""" - return client.delete(f"{BASE_URL}{validator_id}/{DEFAULT_QUERY_PARAMS}") + return client.delete(f"{BASE_URL}{validator_id}/", headers=tenant_headers()) class TestCreateValidator(BaseValidatorTest): @@ -81,8 +83,9 @@ def test_create_validator_missing_required_fields( ): """Test that missing required fields returns validation error.""" response = integration_client.post( - f"{BASE_URL}{DEFAULT_QUERY_PARAMS}", + BASE_URL, json={"type": "uli_slur_match"}, + headers=tenant_headers(), ) assert response.status_code == 422 @@ -238,7 +241,7 @@ def test_list_validators_filter_by_ids(self, integration_client, clear_database) second_id = second.json()["data"]["id"] response = integration_client.get( - f"{BASE_URL}{DEFAULT_QUERY_PARAMS}&ids={first_id}", + f"{BASE_URL}?ids={first_id}", headers=tenant_headers(), ) assert response.status_code == 200 @@ -257,7 +260,7 @@ def test_list_validators_filter_by_multiple_ids( second_id = second.json()["data"]["id"] response = integration_client.get( - f"{BASE_URL}{DEFAULT_QUERY_PARAMS}&ids={first_id}&ids={second_id}", + f"{BASE_URL}?ids={first_id}&ids={second_id}", headers=tenant_headers(), ) assert response.status_code == 200 @@ -271,7 +274,7 @@ def test_list_validators_invalid_ids_query_returns_422( ): """Test invalid UUID in ids query returns validation error.""" response = integration_client.get( - f"{BASE_URL}{DEFAULT_QUERY_PARAMS}&ids=not-a-uuid", + f"{BASE_URL}?ids=not-a-uuid", headers=tenant_headers() ) assert response.status_code == 422 @@ -279,7 +282,7 @@ def test_list_validators_invalid_ids_query_returns_422( def test_list_validators_empty(self, integration_client, clear_database): """Test listing validators when none exist.""" response = integration_client.get( - f"{BASE_URL}?organization_id=999&project_id=999", + BASE_URL, headers=tenant_headers((999, 999)), ) assert response.status_code == 200 @@ -315,7 +318,7 @@ def test_get_validator_invalid_id_returns_422( ): """Test invalid UUID path param returns validation error.""" response = integration_client.get( - f"{BASE_URL}not-a-uuid/{DEFAULT_QUERY_PARAMS}", + f"{BASE_URL}not-a-uuid/", headers=tenant_headers(), ) assert response.status_code == 422 @@ -327,7 +330,7 @@ def test_get_validator_wrong_org(self, integration_client, clear_database): # Try to access it as different org response = integration_client.get( - f"{BASE_URL}{validator_id}/?organization_id=2&project_id=1", + f"{BASE_URL}{validator_id}/", headers=tenant_headers((2, 1)), ) assert response.status_code == 404 @@ -410,7 +413,7 @@ def test_delete_validator_wrong_org(self, integration_client, seed_db): # Try to delete it as different org response = integration_client.delete( - f"{BASE_URL}{validator_id}/?organization_id=2&project_id=1", + f"{BASE_URL}{validator_id}/", headers=tenant_headers((2, 1)), ) assert response.status_code == 404 diff --git a/deployment.md b/deployment.md index 1d43258..1d4fb07 100644 --- a/deployment.md +++ b/deployment.md @@ -6,7 +6,9 @@ 2. Set `ENVIRONMENT=production`. 3. Set `POSTGRES_*` for your production database. 4. Set `AUTH_TOKEN` as a SHA-256 hex digest (64 lowercase chars) of your bearer token. -5. Optionally set `GUARDRAILS_HUB_API_KEY` and `SENTRY_DSN`. +5. Set `ALLOWED_IPS` to kaapi-backend's source IP(s), comma-separated. Required in production — the + service will not start without it. +6. Optionally set `GUARDRAILS_HUB_API_KEY` and `SENTRY_DSN`. Generate AUTH token hash: @@ -14,6 +16,19 @@ Generate AUTH token hash: echo -n "your-plain-text-token" | shasum -a 256 ``` +## Caller access + +Only kaapi-backend may call this service. Requests must carry `Authorization: Bearer ` plus +`X-ORGANIZATION-ID` / `X-PROJECT-ID`, and must originate from an IP in `ALLOWED_IPS`. + +Confirm `ALLOWED_IPS` matches the IP guardrails actually sees. If NAT or a different network +interface is in play, the observed source will differ from what you expect and every request will be +rejected with `403`. Only the health check is exempt. + +Note: the check reads the real connection source and ignores `X-Forwarded-For`. If a reverse proxy +or load balancer is ever placed in front of this service, all traffic will appear to come from the +proxy and this must be revisited. + ## Deploy the FastAPI Project Build and start backend: diff --git a/docs/DEVELOPER_GUIDE.md b/docs/DEVELOPER_GUIDE.md index bd9818d..f3f4bd2 100644 --- a/docs/DEVELOPER_GUIDE.md +++ b/docs/DEVELOPER_GUIDE.md @@ -1972,7 +1972,7 @@ Client ▼ FastAPI Application (port 8001) │ - ├─ Authentication middleware (Bearer token / X-API-KEY) + ├─ Authentication dependency (source IP + Bearer token -> tenant) │ ├─ Config Resolution │ ├─ Fetch ban list words from DB (if ban_list_id provided) @@ -2081,7 +2081,7 @@ else: **Bearer token** (main guardrail endpoints): `Authorization: Bearer `. Validated against SHA-256 hash of the plaintext `AUTH_TOKEN` env var. -**X-API-KEY** (ban list, topic relevance, validator config endpoints): Multi-tenant auth validated against `KAAPI_AUTH_URL`. Scopes access to specific `organization_id` + `project_id`. +**Source IP + tenant headers** (all endpoints except health check): the caller must appear in `ALLOWED_IPS`, and sends `X-ORGANIZATION-ID` / `X-PROJECT-ID` resolved by the Kaapi backend. These scope every read and write; tenant is never taken from the query string or body. ### Key Environment Variables @@ -2091,7 +2091,7 @@ else: | `OPENAI_API_KEY` | For LLM validators | Used by `topic_relevance`, `llm_critic`, `answer_relevance_custom_llm` | | `GUARDRAILS_HUB_API_KEY` | For hub validators | Required for `ban_list`, `llamaguard_7b`, `nsfw_text`, `profanity_free` | | `POSTGRES_*` | Yes | Database connection settings | -| `KAAPI_AUTH_URL` | For config endpoints | Multi-tenant auth service URL | +| `ALLOWED_IPS` | Yes in production | Comma-separated source IPs allowed to call this service | ### Tech Stack diff --git a/docs/architecture/ARCHITECTURE.md b/docs/architecture/ARCHITECTURE.md index dea78fb..b0a728e 100644 --- a/docs/architecture/ARCHITECTURE.md +++ b/docs/architecture/ARCHITECTURE.md @@ -17,7 +17,7 @@ Caller │ { request_id, organization_id, project_id, input, validators: [...] } ▼ FastAPI route ────────────────────────────────────────────── - │ 1. Auth check (static bearer token or multitenant X-API-KEY) + │ 1. Auth check (source IP + static bearer token -> tenant) │ 2. Write RequestLog (status=PROCESSING) │ 3. Resolve config-backed validators │ ├── BanList → fetch banned_words from DB @@ -191,16 +191,15 @@ Error messages are sanitised before persistence: the input string is redacted fr ## Authentication -Kaapi Guardrails is a **standalone microservice that does not authorize requests on its own** — it delegates tenant authorization to the Kaapi backend. A caller therefore presents **two keys**: +Kaapi Guardrails is an **internal microservice with a single caller: the Kaapi backend**. The Kaapi backend authenticates the end user and resolves the tenant before calling; guardrails does not handle end-user API keys and does not call back to the auth service. -1. **Guardrail key** — the service's own static bearer token, proving the caller is allowed to reach the guardrails microservice. -2. **Kaapi backend `X-API-KEY`** — forwarded to the Kaapi backend, which performs the actual authorization and identifies the tenant. +A single dependency in `app/api/deps.py` (`verify_caller`, exposed as `AuthDep`) enforces all of it and returns the tenant: -These map to the two auth modes that coexist in `app/api/deps.py`: +1. **Source IP** — must appear in `ALLOWED_IPS`. Read from the real connection peer, ignoring `X-Forwarded-For`. Checked first, so an unexpected origin gets `403` without learning whether the token was valid. +2. **Static bearer token** — a SHA-256 hex digest configured in `AUTH_TOKEN`, compared with `secrets.compare_digest` to prevent timing attacks. +3. **Tenant headers** — `X-ORGANIZATION-ID` and `X-PROJECT-ID`, trusted because only a caller passing the first two controls can set them. -**Static bearer token** (`AuthDep`): the guardrail key — a SHA-256 hex digest configured in `AUTH_TOKEN`. Used by the core guardrails route and config management routes. Compared with `secrets.compare_digest` to prevent timing attacks. - -**Multitenant X-API-KEY** (`MultitenantAuthDep`): the Kaapi backend key, resolved against the external Kaapi auth service (`KAAPI_AUTH_URL`) by calling `GET /apikeys/verify`. The microservice does not validate this key itself — the Kaapi backend authorizes it and returns `organization_id` + `project_id`, enabling per-tenant data isolation. Accepts the key as the `X-API-KEY` header, `Authorization: Bearer`, or `access_token` cookie. +Tenant scope is never read from the query string or request body, so the dependency that authenticates a route is also the one that supplies its tenant — a route cannot be written without one. Only `GET /utils/health-check/` is exempt. --- diff --git a/docs/auth-authz-spec.md b/docs/auth-authz-spec.md new file mode 100644 index 0000000..d7637ef --- /dev/null +++ b/docs/auth-authz-spec.md @@ -0,0 +1,95 @@ +# Auth for kaapi-guardrails — design proposal + +## Problem + +Two endpoints (`guardrails`, `validator_configs`) check a shared bearer token, then read +`organization_id` / `project_id` from the query string or request body. The token proves the caller +is allowed in; nothing proves which tenant it is allowed to touch. Anyone with the token can read or +change any tenant's data by editing a query parameter. + +Two other endpoints (`ban_lists`, `llm_prompt_configs`) instead resolve an `X-API-KEY` by calling +back to kaapi-backend on every request. Correct, but it leaves us with a second auth contract and a +runtime dependency on the auth service. + +## Proposal + +Treat kaapi-guardrails as an internal service with one caller: kaapi-backend. kaapi-backend +authenticates the end user and resolves the tenant before calling. Guardrails would no longer handle +`X-API-KEY` and would no longer call back. + +Each request would carry: + +- `Authorization: Bearer ` — shared secret +- `X-ORGANIZATION-ID` / `X-PROJECT-ID` — tenant, resolved upstream +- and must arrive from an IP listed in `ALLOWED_IPS` + +Token and IP together mean a leaked token alone is not enough to reach the service. Since only a +caller passing both can set the tenant headers, the tenant becomes something established by +authentication rather than claimed by the caller. + +The property worth having: the same dependency that authenticates a route also supplies its tenant, +so a route cannot be written without one. + +### Alternative considered + +Keeping the tenant in the request body. Rejected because 9 of the 15 endpoints are GET or DELETE and +cannot carry a body — we would end up with body for writes and query params for reads, and the +tenant would stay a per-route parameter each handler has to remember to forward. + +## Contract + +All routes except health-check would require the three headers. Health-check stays open for the +Docker healthcheck. + +- Source IP not whitelisted → 403, checked first so a bad IP never learns whether the token was valid +- Missing or wrong token → 401 +- Missing or non-integer tenant headers → 422 + +## Changes required + +- `config.py` — add `ALLOWED_IPS` (comma-separated; required in production, empty disables the check + locally). Drop `KAAPI_AUTH_URL`, `KAAPI_AUTH_TIMEOUT`. +- `deps.py` — replace both existing dependencies with one, `verify_caller`, returning the tenant. + Delete the `X-API-KEY` callback. +- Routes — all move to the same `AuthDep`; `validator_configs` drops its tenant query parameters. +- `GuardrailRequest` — remove `organization_id` / `project_id`, so sending them fails loudly rather + than being silently honoured. +- Pass the tenant from the auth context into the request log, validator logs, and the ban-list and + prompt-config lookups in the guardrails path. These are tenant-scoped reads a caller can currently + redirect. + +## Testing + +New tests for the IP check, token check, the ordering between them, and header validation. Existing +tests move from API keys and query params to tenant headers. The cross-tenant 404 tests already in +the suite serve as the regression net. + +Running the full suite locally needs the Guardrails hub validators installed and a Postgres +instance. + +## Rollout + +``` +AUTH_TOKEN="" +ALLOWED_IPS="10.0.3.14" # kaapi-backend's source IP +``` + +This is a breaking change; the two services need to be released together: + +1. Set `ALLOWED_IPS` and confirm it matches kaapi-backend's actual egress IP. If NAT or a different + interface is in play, every request will 403 — the most likely deploy failure. +2. Update kaapi-backend to send the tenant headers and stop sending `X-API-KEY` and body/query + tenant fields. +3. Deploy guardrails. + +If kaapi-backend later moves to a new IP, `ALLOWED_IPS` has to be updated or traffic stops. That is +the ongoing cost of this approach. + +## Limits + +- The IP check would read the real connection source and ignore `X-Forwarded-For`, which callers can + forge. Fine today since nothing proxies this service, but it would need revisiting if a load + balancer is added, as every request would then arrive from the balancer's IP. +- Rotating the token needs a coordinated change on both sides; there is no overlap window. +- Out of scope: per-route permissions (one caller, one trust level), CIDR ranges (worth adding if the + backend autoscales), mTLS (stronger and immune to IP churn, but needs certificate infrastructure). diff --git a/docs/deep-dives/architecture.md b/docs/deep-dives/architecture.md index f310199..3575f19 100644 --- a/docs/deep-dives/architecture.md +++ b/docs/deep-dives/architecture.md @@ -13,7 +13,7 @@ Client ▼ FastAPI Application (port 8001) │ - ├─ Authentication middleware (Bearer token / X-API-KEY) + ├─ Authentication dependency (source IP + Bearer token -> tenant) │ ├─ Config Resolution │ ├─ Fetch ban list words from DB (if ban_list_id provided) @@ -227,9 +227,13 @@ Two auth schemes are used: - Send as `Authorization: Bearer ` - Validated against SHA-256 hash of the plaintext token -**X-API-KEY** (ban list, topic relevance, validator config endpoints): -- Multi-tenant auth validated against `KAAPI_AUTH_URL` -- Scopes access to specific `organization_id` + `project_id` +**Source IP allowlist** (all endpoints except health check): +- Caller must appear in `ALLOWED_IPS`; checked before the token +- Reads the real connection source, not `X-Forwarded-For` + +**Tenant headers** (all endpoints except health check): +- `X-ORGANIZATION-ID` + `X-PROJECT-ID`, resolved by the Kaapi backend +- Scope every read and write; never taken from the query string or body --- @@ -241,7 +245,7 @@ Two auth schemes are used: | `OPENAI_API_KEY` | For LLM validators | Used by `topic_relevance` and `llm_critic` | | `GUARDRAILS_HUB_API_KEY` | For hub validators | Required for `ban_list`, `llamaguard_7b`, `nsfw_text`, `profanity_free` | | `POSTGRES_*` | Yes | Database connection settings | -| `KAAPI_AUTH_URL` | For config endpoints | Multi-tenant auth service URL | +| `ALLOWED_IPS` | Yes in production | Comma-separated source IPs allowed to call this service | --- diff --git a/docs/tech-readiness-topic-relevance-llm.md b/docs/tech-readiness-topic-relevance-llm.md new file mode 100644 index 0000000..4539b76 --- /dev/null +++ b/docs/tech-readiness-topic-relevance-llm.md @@ -0,0 +1,216 @@ +# Tech Readiness — API surface + +Review of `backend/app/api/*` and its direct dependencies. Each finding has a +location, what's wrong, and the fix. Ordered by severity so you can take them one +by one. Auth model is assumed to stay as-is (X-API-KEY for the config endpoints, +static token for guardrails and validator configs). + +--- + +## Blockers + +### 1. Static token can reach any tenant + +`routes/guardrails.py:49-80` and `routes/validator_configs.py:22-112` use `AuthDep` — +one server-wide token (`deps.py:40-58`) — but read `organization_id` / `project_id` +from the query or body. Anyone with the token can read or change any tenant's data. + +Fix: move both routers to `MultitenantAuthDep` and drop the caller-supplied tenant +fields. Until then, treat it as an accepted risk with an owner and a date. + +### 2. `bool` passes as a tenant id + +`deps.py:100` checks `isinstance(x, int)`. In Python `bool` is an `int`, so if the +auth backend returns `"organization_id": true`, it resolves to tenant 1. + +Fix: also reject `bool`. +```python +if not isinstance(x, int) or isinstance(x, bool): + raise _unauthorized("Invalid credentials") +``` + +### 3. `topic_relevance_llm` missing from the LLM validator enum + +`LLMValidatorName` (`core/enum.py:4-6`) has only `topic_relevance` and +`answer_relevance_custom_llm`. The create schema (`schemas/llm_prompt_config.py:47`) +and the list filter (`routes/llm_prompt_configs.py:49`) are typed to this enum, so +`topic_relevance_llm` is rejected with "validator not supported". + +Fix: add the value. The router and CRUD are already validator-agnostic, so this alone +enables create, list, get, patch, and delete. +```python +class LLMValidatorName(str, Enum): + TopicRelevance = "topic_relevance" + TopicRelevanceLLM = "topic_relevance_llm" # new + AnswerRelevanceCustomLLM = "answer_relevance_custom_llm" +``` + +### 4. Runtime binds the wrong config type + +`routes/guardrails.py:148` accepts a `topic_relevance` config for both topic +validators. After fix 3, a `topic_relevance_llm` validator could pull a +`topic_relevance` config, or the reverse. + +Fix: bind each validator to its own config name. +```python +expected = ( + LLMValidatorName.TopicRelevanceLLM + if isinstance(validator, TopicRelevanceLLMSafetyValidatorConfig) + else LLMValidatorName.TopicRelevance +) +if config.validator_name != expected: + raise HTTPException(400, f"Config '{config.id}' is for '{config.validator_name}', not '{expected.value}'") +``` + +### 5. `ValidatorUpdate` can corrupt a stored config + +`schemas/validator_config.py:24-31` lets a PATCH change `type`, but +`crud/validator_config.py:97-120` merges the old config JSON on top of the new type. +A `pii_remover` PATCHed to `ban_list` keeps stale PII keys. + +Fix: forbid changing `type` in PATCH, or clear `config` when `type` changes. + +--- + +## Should fix + +### 6. Whole chain runs against output data + +`routes/guardrails.py:68-72`: if any validator is `answer_relevance_custom_llm`, the +entire chain runs against `payload.output`. A mixed chain (input validator + output +validator) runs the input validators against the wrong data. + +Fix: split validators by stage and run each against its own data. + +### 7. Request log orphaned when resolution or create fails + +`routes/guardrails.py:63-67` creates the request log before `_resolve_validator_configs`. +Any error during resolution (or a DB error other than `ValueError` in `create`) leaves an +unfinished row. Fix 4 adds another raise here. + +Fix: wrap resolution and create so failures finalize the log with an error status. + +### 8. `prompt_schema_version` dropped for the LLM topic validator + +`TopicRelevanceLLMSafetyValidatorConfig` has a `prompt_schema_version` field +(`core/validators/config/topic_relevance_llm_safety_validator_config.py:15`), but +`routes/guardrails.py:155-157` only copies it onto the non-LLM variant. A stored version +on a `topic_relevance_llm` config is ignored. + +Fix: set it for the LLM variant too and fix the stale comment. If versioning isn't meant +to apply here, say so instead of dropping it silently. + +### 9. Blocking, uncached auth call per request + +`deps.py:76-85`: every multitenant request makes a blocking `httpx.get` to +`KAAPI_AUTH_URL`, using a threadpool worker each time. + +Fix: add an in-process TTL cache keyed on the credential hash; make the dep async. + +### 10. Only the first validator's metadata is returned + +`routes/guardrails.py:244-247` returns `next(...)`, so later validators' metadata is +dropped in a chain. + +Fix: aggregate metadata into a per-validator map. + +### 11. Auth call has no retry and one shared timeout + +`deps.py:79`: a single `timeout` covers all phases, and there's no retry. + +Fix: split connect and read timeouts; add one bounded retry on network error. + +### 12. PATCH can't change type-specific fields + +`ValidatorBase` allows extra fields, `ValidatorUpdate` forbids them +(`schemas/validator_config.py:11,25`). So PATCH can't update any type-specific field. + +Fix: allow extras on update, or expose a dedicated config dict. + +### 13. Docs don't cover errors or versioning + +`API_USAGE.md` documents the response shape but not the error messages, per-endpoint +status codes, or what counts as a breaking change. Consumers have to read the source. + +Fix: add an error list and a versioning note. + +### 14. No request id or tenant in logs + +`core/middleware.py:20-22` logs only method, path, status, and latency. + +Fix: accept `X-Request-ID` (generate if missing) and add it plus the tenant to log lines. + +--- + +## Minor + +### 15. Docs list only two LLM validators + +`API_USAGE.md:9, 349, 354-356, 398`, `docs/llm_prompt_configs/create_config.md`, and +`docs/llm_prompt_configs/list_configs.md` list only `topic_relevance` and +`answer_relevance_custom_llm`. Add `topic_relevance_llm` (plain-text scope prompt, same +as `topic_relevance`, no placeholder rules). + +### 16. `validator_configs.list` has no pagination + +`routes/validator_configs.py:45-57` has filters but no offset/limit, unlike the ban-list +and prompt-config lists. + +### 17. `health_check` returns a raw bool + +`routes/utils.py:11` returns `True`, breaking the "all endpoints return `APIResponse`" +statement in `API_USAGE.md:35-46`. Wrap it or document the exception. + +### 18. `type=` shadows the builtin + +`routes/validator_configs.py:52`: rename the query param to `validator_type`. + +### 19. Duplicate error extraction + +`routes/guardrails.py:267-271` and `277-280` both call `_extract_error_from_guard`. +Consolidate inside `_finalize`. + +### 20. `suppress_pass_logs` has two different defaults + +Route defaults `True` (`guardrails.py:53`), helper defaults `False` (`guardrails.py:187`). +Make them agree. + +### 21. Docstring says it returns a value but returns None + +`routes/guardrails.py:113-122`: `_resolve_validator_configs` returns `None`. + +### 22. `_redact_input` is fragile + +`routes/guardrails.py:314-316` splits on `":\n\n"` then does a plain `str.replace`. Works +today, breaks if the input contains those characters structurally. + +### 23. POSTs return 200 not 201 + +All POST routes. Cosmetic. + +### 24. `response_model_exclude_none` set on one route only + +`routes/guardrails.py:47`. Standardize across endpoints. + +### 25. Delete does get-then-delete + +`routes/llm_prompt_configs.py:109-121`: two round-trips. Only matters if hot. + +### 26. `response_id` is a server UUID, not a provider id + +`routes/guardrails.py:196`. If a provider response id is added later, keep them as +separate columns. + +### 27. Commented-out import + +`api/main.py:18-19`: delete the commented `private.router` import. + +### 28. Document that the auth token must be random + +`_hash_token` (`deps.py:29-30`) is unsalted SHA-256 — fine only because `AUTH_TOKEN` is a +high-entropy secret. Note that requirement so no one sets a memorable string. + +### 29. 404 masks "wrong tenant" vs "missing" + +`crud/validator_config.py:79-95` (and `crud/llm_prompt_config.py:44-59`) return the same +404 for both. Correct for isolation — add a comment so a refactor doesn't split them. diff --git a/docs/validators/answer-relevance.md b/docs/validators/answer-relevance.md index f4f1866..400da71 100644 --- a/docs/validators/answer-relevance.md +++ b/docs/validators/answer-relevance.md @@ -85,7 +85,9 @@ The default prompt works well for general use, but you may want domain-specific ```bash # Create a custom prompt POST /api/v1/guardrails/answer_relevance_prompts/ -Authorization: X-API-KEY your-api-key +Authorization: Bearer +X-ORGANIZATION-ID: 1 +X-PROJECT-ID: 101 { "name": "Healthcare Relevance Check", diff --git a/docs/validators/ban-list.md b/docs/validators/ban-list.md index 09339b5..6eb0a50 100644 --- a/docs/validators/ban-list.md +++ b/docs/validators/ban-list.md @@ -60,7 +60,9 @@ Stored ban lists are scoped per `organization_id` + `project_id`. Use the manage ```bash POST /api/v1/guardrails/ban_lists/ -Authorization: X-API-KEY your-api-key +Authorization: Bearer +X-ORGANIZATION-ID: 1 +X-PROJECT-ID: 101 { "name": "Competitor Terms", diff --git a/docs/validators/topic-relevance.md b/docs/validators/topic-relevance.md index e00e0f5..a7523ea 100644 --- a/docs/validators/topic-relevance.md +++ b/docs/validators/topic-relevance.md @@ -44,7 +44,9 @@ Topic relevance requires a stored configuration that defines the scope of your a ```bash POST /api/v1/guardrails/topic_relevance_configs/ -Authorization: X-API-KEY your-api-key +Authorization: Bearer +X-ORGANIZATION-ID: 1 +X-PROJECT-ID: 101 { "name": "Healthcare Assistant Scope",