Test against new versions of Elixir/OTP that we're using on engage

[nathanbegbie]

@smn @santiagocardo @fedme Hoping to clarify something with this PR:

What versions of Elixir+OTP are we committing to supporting, given that this is a public repo, with dependents outside of Turn.io?

Should we just have a min and max version of elxiir and OTP?

I've taken a stab at how we might do this, but the number of runs will grow exponentially with this matrix approach.

[nathanbegbie]

Claude has some good ideas:

This is a common challenge in the Elixir ecosystem! Here are the typical strategies maintainers use:

Standard Approach: Test the Extremes + Current

Most Elixir library maintainers test a diagonal slice rather than the full matrix:

1. Minimum supported versions together (e.g., Elixir 1.11 + OTP 22)
2. Maximum/latest versions together (e.g., Elixir 1.17 + OTP 27)
3. A few strategic middle combinations that represent common production deployments

This gives you confidence in backward compatibility without explosion. If you support 4 Elixir versions × 4 OTP versions, you're testing ~4-6 combinations instead of 16.

Specific Patterns You'll See

Conservative approach (very common):

• Minimum Elixir + Minimum OTP
• Current Elixir + Current OTP
• Maybe one intermediate pair

Slightly broader (for popular libraries):

• Min Elixir + Min OTP
• Min Elixir + Max OTP (catches OTP breaking changes)
• Max Elixir + Min OTP (catches Elixir breaking changes)
• Max Elixir + Max OTP

Practical Considerations

Version compatibility is usually stable: Elixir maintains excellent backward compatibility, and OTP compatibility ranges are well-documented. Most issues arise at the boundaries, not in the middle.

Follow Elixir's own support policy: Elixir officially supports the last 5-6 minor versions. You can often safely support fewer. Many libraries only support the last 3-4 Elixir versions.

OTP is more stable: OTP breaking changes are rarer. Testing OTP boundaries (min and max) is usually sufficient.

Examples from Popular Libraries

• Phoenix: Tests ~3-4 combinations
• Ecto: Tests current + one older Elixir/OTP pair
• Many libraries only test latest + minimum supported

My Recommendation

Start with this simple 3-combination approach:

# .github/workflows/test.yml
strategy:
  matrix:
    include:
      - elixir: '1.11'
        otp: '22'
      - elixir: '1.14'
        otp: '25'
      - elixir: '1.17'
        otp: '27'

You can always expand if you discover compatibility issues, but most maintainers find this sufficient. The key is being explicit in your README about which versions you support, then testing the boundaries of that support.

[nathanbegbie]

Test failures are due to a formatting issue in the V2 code - this is removed in #291 , so I'll wait till this lands before moving forward here.

[smn]

Happy with this

[nathanbegbie]

Thanks @smn - you'll have to approve, as I cannot mark my own homework (not sure why I didn't think of this as a solution - thanks!)