sched/arch/libc: give fork(), vfork() and task_fork() their real, separate semantics

[casaroli]

Summary

This is step 1 of the plan agreed in #19540 (ordering, go-ahead): the core semantics, plus the arch/Kconfig change that withdraws fork() from every architecture so that per-architecture patches can restore it, one at a time, with real POSIX semantics.

Where this stands, and what I am asking for

apache/nuttx-apps#3673 needs to merge first. NuttX PRs build against apps master, and three things there call fork() unconditionally, so until the companion lands this PR's CI cannot go green — Linux (sim-01) fails and the rest of the matrix is cancelled behind it. The companion is deliberately written to work against NuttX with or without this change, so merging it early costs nothing and loses no coverage: the fork test ostest runs today keeps running against master, under the name of the primitive it has always actually tested. If reviewers are content with it, getting it in soon is what would let this PR show real numbers instead of a red ✗.

A green CI here would not mean this is ready to merge, and I would rather it waited. CI builds; it does not run fork() on your board. This touches the lowest-level machinery in the system — address environments, stack setup, the architecture's register context — and it is a breaking change for anyone who calls fork(). I have tested what I can reach, but almost all of it is QEMU, and only one board was real silicon — an RP2350 (Cortex-M33), where task_fork() and vfork() both pass. My matrix is not a substitute for other people's hardware, and I would sooner this sat open for a while and collected evidence than merged on the strength of it.

So: please try it on a board you have. Anything at all is useful.

1. Check out this branch together with testing/ostest: split the fork test into task_fork, vfork and fork nuttx-apps#3673.

2. Build and run ostest. The three fork tests now run first, so a fault shows up in seconds rather than after everything else has passed:

   user_main: task_fork() test
   task_fork_test: Child 6 ran successfully
   
   user_main: vfork() test
   vfork_test: Child 7 ran and exited before the parent resumed
   ...
   ostest_main: Exiting with status 0
   

3. Build and run your own application — especially if it calls fork() or vfork(). That is the case I cannot test for you, and it is the one that matters.

Useful to report: board and defconfig, build mode (FLAT / PROTECTED / KERNEL), the final ostest status, and whether your application still behaves. If it calls fork() you will get a build error naming the function — that is the intended signal, and CONFIG_FORK_IS_TASK_FORK=y is meant to restore the previous behaviour exactly; please tell me if it does not. Failures are more useful to me than successes, and so is the opinion that some part of this is the wrong shape.

Real fork() already works — it is just not in this PR

fork() is implemented and passing on most of the MMU-capable architectures already, in a follow-up branch: armv7-a, arm64, risc-v (Sv32 and Sv39), x86_64 and Xtensa LX7. In a twenty-configuration ostest sweep there, POSIX fork() passes on all five BUILD_KERNEL configurations, and the Xtensa row is on real ESP32-S3 silicon. Each of those becomes its own PR once this one has settled, because each is a separate up_addrenv_fork() and deserves separate review. Splitting them that way is also the reason this PR temporarily leaves fork() unavailable rather than shipping one architecture's implementation inline — but if reviewers would prefer to see one MMU architecture land alongside the core so that the result is demonstrable in-tree, I am happy to do that instead (discussion).

The problem

NuttX implements fork() and vfork() as the same function. Both are libc wrappers around a single up_fork(); vfork() differs only by a trailing waitpid(). Underneath, the child joins the parent's address environment — the same addrenv_join() that pthread_create() uses — and gets a private copy of the stack. The child therefore shares .data, .bss and the heap with its parent and runs concurrently with it.

That is not fork(). It is vfork()-with-a-private-stack published under fork()'s name, and the history says so: today's fork() is NuttX's old vfork(), renamed in c33d1c9c97 (2023) with no change of behaviour. NuttX's own comment on nxtask_setup_fork() documents fork() by quoting the POSIX definition of vfork(), word for word.

The failure mode is the worst one available: silent. A program written against POSIX fork() compiles, links, runs — and has its child's writes land in the parent's variables. No diagnostic, no error.

What this does

Three primitives, chosen by which function the caller called rather than by what the hardware happens to be:

	memory	parent	Kconfig
fork()	child gets its own copy at the same virtual addresses	runs concurrently	ARCH_HAVE_FORK
vfork()	child shares the parent's memory	suspended until _exit()/exec()	ARCH_HAVE_VFORK
task_fork()	child shares memory, private stack copy	runs concurrently	ARCH_HAVE_TASK_FORK

Below libc there are now three syscalls — up_task_fork(), up_vfork() and up_fork(). The per-architecture register snapshot is common to all three: each architecture's three entry points share one sequence and differ only in a FORK_TYPE_* selector (include/nuttx/fork.h) handed to nxtask_setup_fork(), which is the single place the memory semantics are decided. Per architecture the plumbing is +5…+42 lines; nothing is rewritten.

task_fork() is today's behaviour under an honest name. Not fork(), not vfork(): a task cloned at the call site with the memory relationship of a thread. The nearest precedent is Plan 9's rfork(RFPROC|RFMEM) — data and bss shared, stack copied. The name deliberately avoids "vfork", because the defining property of vfork() is the suspension, which this primitive does not have; something like stack_vfork() would recreate the very confusion this removes.

The vfork() parent suspension moves out of libc into the kernel — nxtask_start_vfork(), released from nxsched_release_tcb(). Two things follow. The parent is now resumed at exec() as POSIX requires, because exec_swap() has already handed the child's pid to the loaded program by the time the vfork stub exits. And vfork() no longer depends on CONFIG_SCHED_WAITPID — until now, a configuration without that option had no vfork() at all.

That release point needs one fix in nxtask_exit(). It raises rtcb->lockcount directly rather than through sched_lock() while it tears the TCB down, so the nxsem_post() that wakes the vfork() parent queues it on g_pendingtasks — and the matching raw lockcount-- does not merge that list the way sched_unlock() would. The parent is left stranded with nothing to move it off. A nxsched_merge_pending() after the decrement publishes it; the call is a no-op while pre-emption is still disabled, and up_exit() re-reads this_task() afterwards. Without it, vfork() deadlocks on rv-virt:nsh64 and rv-virt:pnsh64, where NSH is blocked in waitpid() holding the lock and nothing else calls sched_unlock().

fork() is built on a new addrenv_fork(), backed by an up_addrenv_fork() hook that duplicates an address environment into freshly allocated pages mapped at the same virtual addresses — unlike up_addrenv_clone(), which copies only the representation and leaves both pointing at the same page tables. The child then adopts the parent's stack geometry rather than being given a relocated copy: a pointer to a stack local taken before fork() must name the same object in the child that it named in the parent, and the parent's stack is already in the duplicate, at the parent's address, with its contents. Nothing is allocated and nothing is copied for it.

There is no copy-on-write — NuttX has no demand paging to build it on — so the copy is eager and can fail with ENOMEM. That is the nature of the primitive, not a defect: fork()'s value here is correctness for portable code. Spawn-heavy code should prefer posix_spawn() or vfork(), on NuttX as anywhere.

Why fork() ends up unavailable everywhere

No architecture implements up_addrenv_fork() in this PR, so CONFIG_ARCH_HAVE_FORK is unset in every configuration and fork() is not declared. That is deliberate, and it is step 3 of the agreed ordering folded into step 1 — the two cannot be cleanly separated, because step 1 is what redefines CONFIG_ARCH_HAVE_FORK to mean "can provide POSIX semantics", and splitting them would require an intermediate release in which the symbol means two things at once.

The generic machinery is complete and needs no further work: an architecture implements the hook, adds one default y if line, and fork() becomes live.

Impact

This is a breaking change, and it breaks loudly rather than quietly.

fork() disappears from ARCH_ARM, flat ARCH_ARM64, ARCH_RISCV, ARCH_SIM and ARCH_X86_64. Code that calls it fails to build, with an error naming the function. That is the point: a build error is strictly better than the silent runtime wrongness it replaces, and it is the only signal portable code can act on — there is no feature-test macro or sysconf() query by which an application can discover that a fork() does not copy.

CONFIG_FORK_IS_TASK_FORK=y restores the previous behaviour exactly — same sharing, same concurrency, no new suspension — on precisely the configurations that had fork() before. default n, so the honest behaviour is what you get unless you ask otherwise, and the Kconfig help states plainly what you are opting into. Existing users flip one switch and are where they were.

If withdrawing fork() in a single release is judged too abrupt, the switch can ship default y with a deprecation warning for a cycle. The end state should still be default n; the schedule is negotiable, the destination shouldn't be.

vfork() changes behaviour. The parent is now suspended in the kernel and released when the child's TCB is torn down, so by the time it runs the child is completely gone. Where the child called exec() this makes no difference: exec_swap() has already given the loaded program the child's pid and that program is still running, so waitpid() behaves normally. Where the child called _exit(), waitpid() can only return its status if CONFIG_SCHED_CHILD_STATUS is enabled; otherwise it returns ECHILD. That is a pre-existing property of that configuration rather than a change — the previous implementation blocked in a libc waitpid(WNOWAIT) and an application's own waitpid() afterwards hit the same wall.

vfork() gains availability: it no longer depends on CONFIG_SCHED_WAITPID.

ARM kernel builds lose vfork()/task_fork(), and that is a fix. ARCH_ARM selected the fork family unconditionally, BUILD_KERNEL included, and it has never worked there: on a kernel build the architecture's fork entry point sees the kernel's return address and stack pointer rather than the caller's, so the child resumes at a kernel address. On qemu-armv7a:knsh, unmodified master faults in ostest's task_fork case with "Child did not run" and then a data abort. ARCH_ARM64 and ARCH_X86_64 already carried if !BUILD_KERNEL for exactly this reason — ARM was the outlier. Conditioning it turns a latent runtime fault into an honest absence, and qemu-armv7a:knsh now runs ostest to status 0. arch/arm takes the condition off again in the patch that adds its saved-syscall-frame path. Cortex-M is unaffected, since it cannot build BUILD_KERNEL at all.

Nothing is deleted. Every line of the existing machinery survives as task_fork().

task_fork() is optional, and on by default. CONFIG_TASK_FORK depends on ARCH_HAVE_TASK_FORK and defaults to y, so it is enabled exactly where fork() existed before and no configuration loses the primitive by upgrading. The two symbols say different things: ARCH_HAVE_TASK_FORK is the architecture's capability, TASK_FORK is whether this build asked for it. Turning it off drops the architecture entry point, the system call and the libc stub — worth having because task_fork() is a NuttX extension that a configuration calling only vfork() need not carry. vfork() and fork() are unaffected; each is still selected on its own.

apps: companion PR, must merge first, as above. The full audit, for reviewers who want the blast radius up front:

	what it needs
testing/ostest	the three-way test split (the substance of the companion PR)
testing/ltp	278 of 1943 open_posix_testsuite files dropped via LTP's existing BLACKWORDS; no-op while fork() exists
system/libuv	two test files dropped from the glob — already dead on NuttX, the port disabled their TEST_ENTRYs long ago
testing/drivers/nand_sim	wanted task_fork() all along
interpreters/python, netutils/libwebsockets, testing/fs/fdsantest	feature macros re-pointed at vfork() vs fork()
games/NXDoom, system/syslogd	nothing — one is inside #if 0, the other already uses posix_spawn()
interpreters/bas	deferred: bas_statement.c has 1681 pre-existing style findings, so CI rejects any patch touching it

In-tree, libs/libbuiltin/libgcc/gcov.c was the only NuttX-side caller; __gcov_fork() is now compiled under the same condition unistd.h uses to declare fork().

Documentation: Documentation/guides/fork_vfork_migration.rst is new and answers "which replacement do I want?" from the reader's own reason for having called fork(). reference/user/01_task_control.rst gains fork() and task_fork() entries and rewrites vfork()'s. standards/posix.rst moves fork() from "No" to "Cond." and vfork() from "Yes" to "Cond.".

Testing

Host: macOS 15 (Darwin 25.5.0) on Apple Silicon. Toolchains: xPack riscv-none-elf-gcc 14.2.0-3; Arm GNU arm-none-eabi-gcc and aarch64-none-elf-gcc 14.2.Rel1 (darwin-arm64); Homebrew x86_64-elf-gcc; QEMU 11.0.3. apps at the companion PR's branch throughout.

Run under QEMU — full ostest suite, exit status 0 on every one

config	build model	task_fork()	vfork()	fork()	ostest
rv-virt:nsh64	FLAT	PASS	PASS	absent ✓	status 0
rv-virt:pnsh64	PROTECTED	PASS	PASS	absent ✓	status 0
rv-virt:knsh64	KERNEL + ARCH_ADDRENV	PASS	PASS	absent ✓	status 0
qemu-armv7a:nsh	FLAT	PASS	PASS	absent ✓	status 0
qemu-armv8a:nsh	FLAT	PASS	PASS	absent ✓	status 0
qemu-intel64:nsh	FLAT	PASS	PASS	absent ✓	status 0

rv-virt:knsh64 is the interesting row: it is a BUILD_KERNEL + CONFIG_ARCH_ADDRENV configuration, and it is where fork() visibly goes away. vfork() there also exercises the kernel-side suspension across a system call.

"absent" was checked, not merely inferred from the config. nm shows up_task_fork, up_vfork, task_fork and vfork, and no fork, up_fork or fork_test. For example, rv-virt:nsh64:

0000000080000424 T up_task_fork      0000000080023eda T task_fork
0000000080000428 T up_vfork          0000000080023ede T vfork
0000000080002786 T nxtask_setup_fork 000000008000290c T nxtask_start_vfork

Run under real hardware — full ostest suite, exit status 0 on every one

config	build model	task_fork()	vfork()	fork()	ostest
pimoroni-pico-2-plus:nsh (RP2350, Cortex-M33)	FLAT	PASS	PASS	absent ✓	status 0
pimoroni-pico-2-plus:pnsh (RP2350, Cortex-M33)	PROTECTED	PASS	PASS	absent ✓	status 0

Both flashed over a Raspberry Pi Debug Probe. pnsh is the row worth noting: a protected build on an MPU, where fork() cannot exist at all because a Cortex-M33 has no MMU to duplicate an address environment.

Built clean

stm32f4discovery:nsh (armv7-m), mps2-an521:nsh (armv8-m), sabre-6quad:nsh (armv7-a), sim:ostest — the last with CONFIG_MM_KASAN disabled, see the note below. On sim, task_fork_test and vfork_test also pass at run time.

The break, and the escape hatch, both verified

Without CONFIG_FORK_IS_TASK_FORK (qemu-armv8a:nsh), a call to fork() is a build error naming the function, while the honest spellings still compile:

forkchk.c:3:28: error: implicit declaration of function 'fork'
   3 | pid_t probe(void) { return fork(); }
     |                            ^~~~

pid_t a(void) { return vfork(); }      /* compiles */
pid_t b(void) { return task_fork(); }  /* compiles */

With CONFIG_FORK_IS_TASK_FORK=y (rv-virt:nsh64), the same translation unit compiles, unistd.h declares fork() again, and staging/libc.a provides it:

0000000000000000 T fork
0000000000000000 T task_fork
0000000000000000 T vfork

fork_test correctly stays out of that build: the alias sets FORK_IS_TASK_FORK, not ARCH_HAVE_FORK, so no POSIX fork() test is advertised for a configuration that does not have POSIX fork().

Style and docs

tools/checkpatch.sh -c -u -m -g 7fd17c9d7c..HEAD, the exact command .github/workflows/check.yml runs — ✔️ All checks pass, with codespell, cvt2utf, cmake-format and nxstyle all installed. Note that codespell checks the whole of any file a patch touches, which is why the docs commit also corrects three long-standing typos in implementation/memory_configurations.rst.

sphinx-build -b html over Documentation/ — build succeeded, 1 warning, and that warning is the pre-existing plantuml command cannot be run in guides/port_bootsequence.rst, unrelated to this change.

[github-actions]

MemBrowse Memory Report

arduino-mega2560

• flash: .text +6 B (+0.0%, 67,436 B / 262,144 B, total: 26% used)

esp32-devkitc

• ROM: .flash.text +8 B (+0.0%, 124,380 B / 4,194,272 B, total: 3% used)
• irom0_0_seg: .flash.text +8 B (+0.0%, 88,588 B / 3,342,304 B, total: 3% used)

hifive1-revb

• flash: .text +16 B (+0.0%, 83,368 B / 4,194,304 B, total: 2% used)
• sram: .bss +32 B (+0.8%, 3,964 B / 16,384 B, total: 24% used)

mirtoo

• kseg0_progmem: .text +68 B (+0.1%, 67,504 B / 131,072 B, total: 52% used)

qemu-armv8a

• Code: .rodata -314 B, .text.nxsched_release_tcb +8 B, .text.nxtask_exit +12 B, .text.pthread_start -4 B, .text.up_initial_state -4 B, .text.user_main -16 B (-0.6%, 316,128 B)
• Data: .bss.g_idletcb +8 B (+0.0%, 76,977 B)

qemu-intel64

• Code: .text -1,546 B (-0.0%, 8,656,502 B)
• Data: .bss +64 B, .rodata -278 B (-0.2%, 120,433 B)

rx65n-rsk2mb

• ROM: .text +16 B (+0.0%, 87,264 B / 2,097,152 B, total: 4% used)

s698pm-dkit

• Code: .text +16 B (+0.0%, 363,264 B)

stm32-nucleo-f103rb

• flash: .text +40 B (+0.1%, 33,968 B / 131,072 B, total: 26% used)

[casaroli]

I was planning on having one follow up PR for each relevant architecture, however I now think maybe we should keep one of the architectures that supports fork() (an MMU one) in this PR as a sample and that anyone can replicate the results. I am happy to follow any direction..

these are MMU capable, in order of complexity (low to high):

• armv8-a
• armv7-a
• riscv
• xtensa LX7 (esp32s3)
• sim (could be simpler but I did not investigate enough
• x86_64
• x86 - pending sampething, no?

[jerpelea]

please replace
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
with
Assisted-by: Claude Opus 5 (1M context) noreply@anthropic.com