Parallelize locale merging in mix gettext.merge#436
Merged
josevalim merged 1 commit intoJun 10, 2026
Conversation
mix gettext.merge processes locales sequentially, but each locale's
merge is fully independent: every locale task writes only to its own
<pot_dir>/<locale>/LC_MESSAGES/*.po files, the POT files at the root of
pot_dir are only read, and Gettext.Merger is stateless (pure functions
over Expo structs). Switch merge_all_locale_dirs/3 to Task.async_stream
over locales, mirroring the existing Task.async_stream over POT files
within a locale in merge_dirs/5.
The return value change (list -> :ok via Stream.run/1) is safe: the
only caller is merge_messages_dir/3, whose result is discarded in
run/1. The --locale single-locale path is untouched. A crashed locale
task raises out of Stream.run/1, matching the sequential version's
failure behavior. Mix.shell() output is GenServer-backed, so concurrent
lines serialize correctly; the only observable change is that progress
lines from different locales can interleave, as already happens in
Gettext.Compiler's parallel paths.
Benchmark on a 16-locale / 11-domain Phoenix app (160 PO files, ~4,700
msgids, Elixir 1.20.1 / OTP 29, Apple Silicon):
mix gettext.merge priv/gettext --no-fuzzy
before: 9.4s wall (55% CPU)
after: 1.9-2.4s wall (341-381% CPU)
~4.9x speedup with byte-identical output files (clean git status over
committed PO state after both runs). The win scales with locale count.
Contributor
|
💚 💙 💜 💛 ❤️ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi,
this is not huge change and it can help. Following text is generated by Claude Code based on my tests.
mix gettext.mergeprocesses locales sequentially, but each locale's merge is fully independent: every locale task writes only to its own<pot_dir>/<locale>/LC_MESSAGES/*.pofiles, the POT files at the root of pot_dir are only read, andGettext.Mergeris stateless. Switchmerge_all_locale_dirs/3toTask.async_streamover locales, mirroring the existingTask.async_streamover POT files within a locale inmerge_dirs/5.The return value change (list ->
:ok) is safe because the only caller ismerge_messages_dir/3, whose result is discarded in run. The--locale single-localepath is untouched. A crashed locale task raises out ofStream.run/1, matching the sequential version's failure behavior.Mix.shell()output isGenServer-backed, so concurrent lines serialize correctly. The only observable change is that progress lines from different locales can interleave, as already happens inGettext.Compiler's parallel paths.Benchmark on a 16-locale / 11-domain Phoenix app (160 PO files, ~4700 msgids, Elixir 1.20.1 / OTP 29, M4 Pro):
~4.9x speedup with byte-identical output files (clean git status over committed PO state after both runs). The win scales with locale count.