Default to loky executor on all platforms (fixes forkserver breakage on Python 3.14)#497
Merged
Conversation
Python 3.14 changed the default multiprocessing start method on Linux from "fork" to "forkserver" (python/cpython#84559), so workers of the stdlib ProcessPoolExecutor now re-import __main__ instead of inheriting it. Functions defined interactively (notebooks, executed doc pages) are pickled by reference and fail to unpickle in the workers with e.g.: AttributeError: module '__main__' has no attribute 'sphere' loky's reusable executor serializes functions by value via cloudpickle and was already the default on macOS and Windows (and loky is already a required dependency), so use it on Linux too. This also removes the now-dead lambda-pickling check in AsyncRunner that only applied when the default executor was ProcessPoolExecutor.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Make
loky.get_reusable_executorthe default executor on all platforms, instead of only macOS/Windows. Removes the Linux-onlyconcurrent.futures.ProcessPoolExecutordefault and the now-dead lambda-pickling check inAsyncRunnerthat only applied to it. Updates the parallelism tutorial accordingly.Why
Python 3.14 changed the default multiprocessing start method on Linux from
forktoforkserver(python/cpython#84559). Forkserver workers re-import__main__instead of inheriting the parent's memory, so functions defined interactively (notebooks, executed doc pages) — which pickle stores by reference as__main__.<name>— fail to unpickle in the workers:(e.g. when running
docs/source/tutorial/tutorial.LearnerND.mdon Linux + Python 3.14.)The old comment in
runner.pyjustified the Linux fast-path with "On Linux the whole process is forked, so the issue does not appear" — that assumption no longer holds. loky serializes functions by value via cloudpickle, so it works everywhere, and it is already a required dependency (loky >= 2.9in coredependencies, imported unconditionally inrunner.py).Testing
test_to_dataframefailures (KeyError: -1) that also fail on unmodifiedmainunder 3.14 — unrelated to this change, likely a pandas-on-3.14 issue; will open a separate issue.