Pyroxide

Python tasks, isolated processes, WebAssembly, and C ABI shared libraries.
One embedded engine. One task API.

Read the user manual »
API reference · Examples · Report a bug · Request a feature

---

Run work in the background without Redis, worker daemons, or another service to operate. Start with a Python decorator; choose threads, isolation, WASM, or native execution for each workload.

Why Pyroxide?

• Nothing else to deploy. The task engine lives in your application. There is no broker, separate worker service, or network hop for local work.
• Choose the boundary per task. Keep ordinary work lightweight, move CPU-bound Python into another interpreter, run portable plugins in Wasmtime, or call a reviewed C ABI shared library without holding the GIL.
• One lifecycle to operate. Bounded admission, batches, async results, cancellation rules, statistics, fork safety, and explicit shutdown are part of the same engine.
• Move toward native speed without redesigning the caller. A task can start as Python and later move behind a WASM or native boundary while keeping the submit-and-result workflow.

Free-threaded CPython can also run pure-Python @task work in parallel. On regular CPython, use isolation for parallel CPU-bound Python.

Four execution modes

Mode	Reach for it when you need
@task	Lightweight background work inside the application
@task(isolated=True)	CPU-bound Python or process crash containment
@wasm_task	Portable guest code with memory and execution-time limits
@dylib_task	GIL-free calls into trusted native libraries

Where it fits

Choose	When
Pyroxide	Work belongs to one application and benefits from different execution boundaries
ThreadPoolExecutor or ProcessPoolExecutor	A basic local thread or process pool is enough
Celery, RQ, or another durable queue	Jobs must survive application failure, run on schedules, retry durably, or move across hosts
Ray or Dask	The workload needs a distributed compute runtime

Pyroxide does not try to turn local work into a distributed system. Its strength is putting several useful local execution models behind one small API. The comparison guide covers the trade-offs, and the benchmark study publishes reproducible results, including workloads where the standard library wins.

Quick start

Install the pyro3 package and import it as pyroxide:

pip install pyro3

Python task

from pyroxide import task

@task
def square(value: int) -> int:
    return value * value

handle = square(12)
print(handle.result())  # 144

Inside an event loop, use await handle.result_async() instead of blocking the loop. See Concurrency and asyncio.

Isolated Python

Add isolated=True when CPU-bound Python needs another interpreter or when a trusted native crash must not take down the main application. Workers are reused, bounded, and recycled rather than spawned for every task.

tasks.py:

from pyroxide import task

@task(isolated=True)
def calculate(value: int) -> int:
    return sum(i * i for i in range(value))

app.py:

from tasks import calculate

print(calculate(1_000_000).result())

WebAssembly

Register a precompiled .wasm module when plugin code needs a portable application boundary. Pyroxide supplies no host imports and applies memory and epoch-time limits to each call.

from pathlib import Path
from pyroxide import load_wasm, register_wasm, wasm_task

register_wasm("codec", Path("codec.wasm").read_bytes())

@wasm_task("codec", "compress")
def compress(payload: bytes) -> bytes:
    pass

@wasm_task("codec", "decompress")
def decompress(payload: bytes) -> bytes:
    pass

compressed = compress(b"data").result()

codec = load_wasm("codec")
restored = codec.decompress(compressed).result()
handles = codec.compress.batch([b"first", b"second"])

A module can export many functions. Bind each export with its own @wasm_task decorator, or use one load_wasm() proxy and call exports as methods such as codec.compress() and codec.decompress(). Proxy methods also support .batch(...).

Native shared library

Compatible shared libraries may be written in C, Rust, Zig, or another language using Pyroxide's supported byte-buffer C ABI. Register a reviewed precompiled library:

from pyroxide import dylib_task, load_dylib, register_dylib

register_dylib("codec", "./libcodec.so")

@dylib_task("codec", "compress")
def compress(payload: bytes) -> bytes:
    pass

@dylib_task("codec", "decompress")
def decompress(payload: bytes) -> bytes:
    pass

compressed = compress(b"data").result()

codec = load_dylib("codec")
restored = codec.decompress(compressed).result()
handles = codec.compress.batch([b"first", b"second"])

A library can export many functions. Bind each export with its own @dylib_task decorator, or use one load_dylib() proxy and call exports as methods. Decorators and proxy methods both support .batch(...).

No custom Python extension wrapper is required. During development, compile_c(), compile_rust(), and compile_zig() can build and register trusted source. Production can load a reviewed .so, .dylib, or .dll.

The manual documents serialization, ABI ownership, guest limits, and failure semantics before you cross any of these boundaries.

Performance Benchmarks

These Apple M1 Pro reference runs report median complete batch time. The Python executor table used four workers. Lower is better.

Python tasks and isolation

CPython and workload	ThreadPoolExecutor	Pyroxide @task	ProcessPoolExecutor	Pyroxide isolated
3.14, 32 CPU tasks	65.20 ms	55.49 ms	17.79 ms	19.22 ms
3.14t, 32 CPU tasks	18.91 ms	18.03 ms	13.31 ms	15.88 ms
3.14, 1,000 trivial tasks	6.29 ms	18.25 ms	157.46 ms	52.35 ms

Native, WebAssembly, and application workloads

Workload	Pyroxide	Comparison
Scheduled native call, 1 KiB Rust workload	22.56 µs	Direct PyO3, nanobind, CFFI, and ctypes: 7.26-8.59 µs
Warm WebAssembly call, same 1 KiB workload	47.57 µs	Direct wasmtime-py host: 80.24 µs
Odoo 19 compute-only, Python 3.13, 8 payloads, 2 workers	30.85 ms	ProcessPoolExecutor: 31.77 ms; inline: 60.37 ms

What to expect:

• @task: Competitive with ThreadPoolExecutor for substantial work. The standard thread pool wins for extremely small tasks.
• Isolated Python: Close to ProcessPoolExecutor on CPU work, with lower overhead in the small-task batch and the same Pyroxide task workflow.
• Free-threaded Python: @task can run Python across cores while retaining Pyroxide handles, batching, statistics, and lifecycle controls.
• Native and WASM: Native scheduling adds overhead compared with a direct binding. In return, it joins compiled code to the task system. WASM provides a portable, resource-limited guest boundary through that same system.

The five-minute RC1 run accounted for all 3,080 accepted operations and recovered after 300 deliberate worker crashes. The full benchmark study contains setup details, confidence intervals, memory results, distributed systems, and workloads where other tools win.

Know the boundaries

1.0.0rc1 is a release candidate. Its API is intended to become 1.0. Start production adoption with a canary and representative failure testing.

• Pyroxide is embedded, not durable or distributed. Queued work is lost if the application process exits.
• Pending tasks can be cancelled. Running isolated work can be terminated; running in-process Python, WASM, or native work cannot be safely interrupted.
• Native libraries have unrestricted access to the host process. Isolation can contain a native crash to a worker process, but it is not an OS security sandbox.
• Pyroxide gives WASM guests no host imports and applies memory and epoch-time limits. Treat that as an application isolation boundary, not an absolute security promise.

Explore

• User manual - guided workflows and technical contracts
• Production operations
  • capacity, telemetry, shutdown, and fork safety
• Benchmark evidence
  • reproducible measurements and saved study metadata
• Examples - runnable task, isolation, WASM, and native integrations
• API reference - public Python interface
• Security policy
  • supported releases and private reporting
• Contributing
  • development workflow and change requirements

Development

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[dev]'
maturin develop
pytest -q

Read CONTRIBUTING.md before submitting a change. Report security issues through SECURITY.md, not a public issue.

License

Choose either the MIT or Apache-2.0 license.