Skip to content

Default to loky executor on all platforms (fixes forkserver breakage on Python 3.14)#497

Merged
basnijholt merged 1 commit into
mainfrom
loky-default-executor
Jun 10, 2026
Merged

Default to loky executor on all platforms (fixes forkserver breakage on Python 3.14)#497
basnijholt merged 1 commit into
mainfrom
loky-default-executor

Conversation

@basnijholt

Copy link
Copy Markdown
Member

What

Make loky.get_reusable_executor the default executor on all platforms, instead of only macOS/Windows. Removes the Linux-only concurrent.futures.ProcessPoolExecutor default and the now-dead lambda-pickling check in AsyncRunner that only applied to it. Updates the parallelism tutorial accordingly.

Why

Python 3.14 changed the default multiprocessing start method on Linux from fork to forkserver (python/cpython#84559). Forkserver workers re-import __main__ instead of inheriting the parent's memory, so functions defined interactively (notebooks, executed doc pages) — which pickle stores by reference as __main__.<name> — fail to unpickle in the workers:

File ".../python3.14/multiprocessing/queues.py", line 120, in get
    return _ForkingPickler.loads(res)
AttributeError: module '__main__' has no attribute 'sphere'

(e.g. when running docs/source/tutorial/tutorial.LearnerND.md on Linux + Python 3.14.)

The old comment in runner.py justified the Linux fast-path with "On Linux the whole process is forked, so the issue does not appear" — that assumption no longer holds. loky serializes functions by value via cloudpickle, so it works everywhere, and it is already a required dependency (loky >= 2.9 in core dependencies, imported unconditionally in runner.py).

Testing

  • Full suite passes locally on Python 3.14.5 (macOS), except 3 pre-existing test_to_dataframe failures (KeyError: -1) that also fail on unmodified main under 3.14 — unrelated to this change, likely a pandas-on-3.14 issue; will open a separate issue.
  • CI nox matrix already includes 3.14, so this PR's CI exercises the new default there.
  • pre-commit (ruff, mypy, etc.) passes.

Python 3.14 changed the default multiprocessing start method on Linux
from "fork" to "forkserver" (python/cpython#84559), so workers of the
stdlib ProcessPoolExecutor now re-import __main__ instead of inheriting
it. Functions defined interactively (notebooks, executed doc pages) are
pickled by reference and fail to unpickle in the workers with e.g.:

    AttributeError: module '__main__' has no attribute 'sphere'

loky's reusable executor serializes functions by value via cloudpickle
and was already the default on macOS and Windows (and loky is already a
required dependency), so use it on Linux too. This also removes the
now-dead lambda-pickling check in AsyncRunner that only applied when the
default executor was ProcessPoolExecutor.
@basnijholt basnijholt merged commit 1bb8618 into main Jun 10, 2026
17 of 18 checks passed
@basnijholt basnijholt deleted the loky-default-executor branch June 10, 2026 21:49
@basnijholt basnijholt restored the loky-default-executor branch June 10, 2026 22:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant