Pypy testing no longer xfails

[mattip]

Description

Working through pypy/pypy#5481 and #6049. Fixes to PyPy no longer fail some tests. The xfail tests were marked with xfail(strict=True) (by default strict value), so now that they pass they are marked as failing. I run weekly testing of pybind11 HEAD + pypy HEAD at https://github.com/pypy/binary-testing, you can see the three unexpected tests pass here

Suggested changelog entry:

No changlog entry needed, these are really minor changes in multi-inheritance behavior that I doubt many users hit in practice.

---

📚 Documentation preview 📚: https://pybind11--6077.org.readthedocs.build/

[rwgk]

Hi @mattip, this seems useful for pypy/binary-testing, but I'd prefer to avoid strict=False in the final PR. Instead, we should query the PyPy version in the xfail condition, something like:

@pytest.mark.xfail(
    "env.PYPY and sys.pypy_version_info < fixed_version",
    reason="Fixed in PyPy ...",
)

Or cleaner, add a PYPY_VERSION helper to tests/env.py and use that in the marker.

[mattip]

I will do whatever the project wants, but it seems the pattern elsewhere is to add strict=False and not probe sys.pypy_version_info.

 git grep "env.PYPY.*strict=False"
tests/test_buffers.py:    env.PYPY, reason="PyPy 7.3.7 doesn't clear this anymore", strict=False
tests/test_call_policies.py:@pytest.mark.xfail("env.PYPY", reason="sometimes comes out 1 off on PyPy", strict=False)
tests/test_class_sh_disowning_mi.py:@pytest.mark.xfail("env.PYPY", strict=False)
tests/test_class_sh_disowning_mi.py:@pytest.mark.xfail("env.PYPY", strict=False)
tests/test_class_sh_disowning_mi.py:@pytest.mark.xfail("env.PYPY", strict=False)
tests/test_exceptions.py:@pytest.mark.xfail(env.PYPY, reason="Failure on PyPy 3.8 (7.3.7)", strict=False)
tests/test_methods_and_attributes.py:@pytest.mark.xfail("env.PYPY", strict=False)

[rwgk]

it seems the pattern elsewhere is to add strict=False and not probe sys.pypy_version_info.

Thanks for pointing out, I wasn't aware. We should probably fix what you found when we get a chance.

strict=False is useful for genuinely flaky tests, but here the behavior appears to have changed deterministically in PyPy. Leaving a broad env.PYPY, strict=False xfail means both "still broken" and "now fixed" remain acceptable indefinitely, so the test stops telling us when the workaround is obsolete or when a real regression appears.

What I'm suggesting is really simple, a few minutes worth of effort or so. It'll help us keep the project healthy. A version-gated xfail keeps the expected-failure window explicit and lets newer PyPy versions exercise the test normally.

[mattip]

A version-gated xfail keeps the expected-failure window explicit

Done

[mattip]

PyPy 3.11 v7.3.23 was released, so I updated CI to remove the pin. I also removed some CI jobs: pypy2.10 is no longer supported, and macos-x86_64 is a platform that sees little usage.

[mattip]

The action is not yet picking up 7.3.23, it is picking up 7.3.22. Maybe later today I will try again.

[rwgk]

I pushed a fix that should resolve the CIBW Android wheel ubuntu-latest failure: commit 92c47c5

[rwgk]

Awesome to see the PyPy pinning go. Thanks @mattip!