Support plain SQL migrations

[alassek]

This introduces ROM::SQL::Migration::SQLMigration, which allows you to write migrations as a .sql file in the migration directory. This takes inspiration from my experience using pressly/goose.

SQL migrations have a very simple grammar:

1. Files are organized into two directions (up and down). Each direction may contain one or more begin..end sections; all sections for a direction execute sequentially in source order at apply time. Each section keeps its own split= setting, so a single direction may mix splitting strategies across sections.
2. Directives are written as SQL line comments using the -- @migrate sigil. The leading -- keeps directive lines valid SQL syntax (so editors, syntax highlighters, and linters treat them as comments); the @migrate sigil disambiguates them from author-written -- comments. Plain -- ... comments without the @migrate sigil are treated as part of the SQL body and pass through verbatim.
3. Optional settings may be provided in a pragma line before the up/down sections
4. Pragma and begin lines accept key=value settings separated by spaces or tabs
5. Each direction begins with a direction directive followed by one or more begin directives
6. The begin directive may itself carry settings (e.g. statement splitting strategy)
7. Each section concludes with an end directive
8. Down migrations are optional, but they always must follow the up migration
9. Empty sections (begin..end with no SQL body) are silently skipped

Rationale

For very simple cases, using a Ruby DSL is fine and makes your migrations user-friendly. But a major problem with migration DSLs is that they don't look very much like the SQL they intend to replace. This means when you start to do complex things in your DB, it gets increasingly difficult to figure out how to write it. Eventually you are bound to encounter situations where it makes sense to just shove a whole SQL string into execute.

This extension aims to solve this problem by giving users an escape hatch into plain SQL that can sit alongside their Ruby migrations.

Statement Splitting

Providing multiple ways to manage how your statements get split up is necessary, because different database engines have difference precise rules about it. There are three options demonstrated here: the whole body in one statement, split by line, and split by semicolon. You can also provide multiple statement sections per migration direction if you wish.

-- @migrate up

-- Extension + index can be split on `;` safely
-- @migrate begin split=semicolon
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE INDEX users_name_trgm_idx ON users USING gin (name gin_trgm_ops);
-- @migrate end

-- The function body is dollar-quoted and contains `;`, so it MUST stay in
-- default mode (whole body sent as one statement).
-- @migrate begin
CREATE OR REPLACE FUNCTION search_users(query text)
RETURNS SETOF users
LANGUAGE sql
STABLE
AS $$
  SELECT *
  FROM users
  WHERE name % query
  ORDER BY similarity(name, query) DESC, name ASC;
$$;
-- @migrate end

-- @migrate down
-- @migrate begin split=line
DROP FUNCTION IF EXISTS search_users(text);
DROP INDEX IF EXISTS users_name_trgm_idx;
DROP EXTENSION IF EXISTS pg_trgm;
-- @migrate end

Pragma: transaction

Set whether or not to use a DDL transaction via a top-level pragma:

-- @migrate transaction=false
-- @migrate up
-- @migrate begin
CREATE INDEX CONCURRENTLY users_name_trgm_idx ON users USING gin (name gin_trgm_ops);
-- @migrate end

-- @migrate down
-- @migrate begin
DROP INDEX CONCURRENTLY users_name_trgrm_idx ON users;
-- @migrate end

Pragma: env

You can opt into environment-substitution by passing the env=true pragma. This replaces instances of ${NAME} with ENV["NAME"] in your migration statements before execution.

-- @migrate env=true
-- @migrate up
-- @migrate begin split=line
GRANT ALL ON ALL TABLES IN SCHEMA public TO "${IAM_USER}";
GRANT ALL ON ALL SEQUENCES IN SCHEMA public TO "${IAM_USER}";
GRANT ALL ON ALL ROUTINES IN SCHEMA public TO "${IAM_USER}";
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO "${IAM_USER}";
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO "${IAM_USER}";
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON ROUTINES TO "${IAM_USER}";
-- @migrate end

-- @migrate down
-- @migrate begin split=line
ALTER DEFAULT PRIVILEGES IN SCHEMA public REVOKE ALL ON ROUTINES FROM "${IAM_USER}";
ALTER DEFAULT PRIVILEGES IN SCHEMA public REVOKE ALL ON SEQUENCES FROM "${IAM_USER}";
ALTER DEFAULT PRIVILEGES IN SCHEMA public REVOKE ALL ON TABLES FROM "${IAM_USER}";
REVOKE ALL ON ALL ROUTINES IN SCHEMA public FROM "${IAM_USER}";
REVOKE ALL ON ALL SEQUENCES IN SCHEMA public FROM "${IAM_USER}";
REVOKE ALL ON ALL TABLES IN SCHEMA public FROM "${IAM_USER}";
-- @migrate end

Comments for reviewers

1. This uses recent language features like Data, but the gemspec is outdated. We should probably bump the requirement to 3.3
2. The unfinished transition to ROM 6 is awkward, how should we deal with that in light of an added feature? This is almost entirely additive so it would not require very much effort to merge into main. Note that I am based on release-3.7 currently.
3. This doesn't include anything to generate a SQL migration file, but I think this would be a hanami-cli followon rather than live here.
4. rom-core requires dry-struct so I felt it was acceptable to use it here, but do we want to elevate the transitive dependency to a direct one?

AI Disclosure

The first pass at the tokenizer / compiler was crafted by hand, but I used Claude extensively here to track down edge cases. Everything I did not write has been carefully reviewed.