Open4D

Open4D is a research repository for representing, compressing, evaluating, and playing time-varying 3D geometry. It brings several mesh-compression systems, an experimental Open4D container format, viewers, and benchmark tooling into one workspace for XR, teleoperation, digital-twin, robotics, and graphics research.

Project status: Open4D is under active development. The individual research modules contain working pipelines, while the shared 4D data model, common metrics API, and repository-wide benchmark suite are not yet stable.

Repository layout

Open4D/
├── open4d/
│   ├── core/          planned shared sequence and metadata abstractions
│   ├── io/            experimental .o4d mesh and point-cloud containers
│   ├── player/        PyQt/OpenGL sequence viewers
│   ├── tools/         conversion utilities for .o4d files
│   └── modules/
│       ├── 4d-reconstruction/
│       ├── Draco/
│       ├── KLT/
│       ├── N4MC/
│       ├── Quantized-Neural-Displacement-Fields/
│       ├── mpeg-vdmc-tm/
│       ├── tsmc/
│       ├── tvmc/
│       └── unity_decoder/
├── benchmarks/        benchmark scaffolding and research baselines
├── examples/          minimal .o4d playback examples
├── scripts/           repository-level setup utilities
├── tests/             repository test-suite placeholder
└── docs/              architecture and repository policies

There are currently no top-level cpp/, python/, or docker/ directories. Native C++, C#, and build files are owned by the research modules that require them.

Components

Open4D IO and players

open4d/io implements version-1 chunked containers for mesh sequences, raw point-cloud sequences, and Draco-compressed point-cloud sequences. The matching tools convert folders of geometry files into .o4d containers, and the players provide local desktop playback.

open4d/core now contains the first shared temporal mesh model: a validated NumPy-backed TriangleMesh, temporal Frame, lazy Sequence, and provider contract. The existing codecs have not yet migrated to it, and point clouds, volumes, transforms, and a stable codec API remain planned work. See docs/sequence-design.md for the architecture and staged migration plan.

Research modules

• N4MC — neural TSDF-based mesh compression, including a newer modular codec under its data, models, losses, training, and evaluation packages.
• Quantized Neural Displacement Fields (QNDF) — static mesh compression using an SSP coarse mesh and an implicit displacement decoder.
• TVMC — a Python, .NET, and Draco pipeline for tracked time-varying mesh compression. It includes setup and resumable pipeline scripts.
• TSMC — scene-mesh compression with optional SAM-based static/dynamic separation, ARAP volume tracking, deformation, displacement compression, and evaluation.
• Unity decoder — a C++ decoder backend and C# Unity front end for playback on XR targets.
• Draco — Google Draco mesh-compression baseline. Wraps the vendored draco_encoder/draco_decoder binaries into a per-frame encode/decode/eval pipeline for benchmarking against the neural codecs.
• KLT — Karhunen–Loève Transform baseline that compresses TSDF voxel blocks with a learned linear basis and quantized coefficients, reconstructing meshes via marching cubes.
• 4D reconstruction — synchronized multi-camera RGB-D ingestion, calibrated point-cloud fusion, CUDA TSDF mesh reconstruction, and live browser playback. It includes both the original native reconstruction code and the Python two-camera streaming pipeline.
• MPEG V-DMC test model — the pinned MPEG reference implementation for video-based dynamic mesh coding. The mpeg-vdmc-tm submodule provides the standard's reference encoder, decoder, metric tools, and unit tests; it is separate from Open4D's TVMC research pipeline.

Each module has its own README and environment requirements. TVMC currently targets Python 3.10; TSMC targets Python 3.12. Treat module environments as independent until a shared environment is documented.

Basketball sequence comparison

Side-by-side playback of the N4MC, QNDF, TVMC, and TSMC decodes of the basketball_player sequence, with a toggle for a shared-scale surface-error heatmap.

Installation

Clone with submodules to obtain the pinned Draco, SAM3, and MPEG V-DMC source:

git clone --recurse-submodules https://github.com/SINRG-Lab/Open4D.git
cd Open4D

For the lightweight .o4d IO package:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e .

Optional local tooling is available through extras:

python -m pip install -e ".[tools]"   # trimesh conversion tools
python -m pip install -e ".[draco]"  # Draco point-cloud support
python -m pip install -e ".[player]" # desktop viewers
python -m pip install -e ".[all]"

These extras do not install the heavyweight research-module environments. Use the setup instructions inside the selected module before running a codec.

If an existing clone is missing Draco, initialize and build both copies with:

./scripts/setup_draco.sh

Reproducibility and artifacts

Do not commit local datasets, virtual environments, benchmark jobs, training runs, checkpoints, logs, or decoded outputs. The expected local directories, publication-manifest requirements, and policy for existing historical fixtures are documented in docs/artifacts.md.

The repository-wide benchmarks/ and tests/ directories are currently scaffolding rather than a complete validation suite. Results should identify the exact module revision, configuration, dataset/frame range, encoded byte count, runtime environment, and metric implementation.

Contributing

Contributions are welcome, especially around shared data abstractions, common metrics, reproducible benchmark fixtures, module adapters, tests, documentation, and performance. Keep research-module dependencies isolated and document any new binary fixture or external artifact alongside the code that consumes it.

Please contact the Open4D maintainers before adding a large dataset, checkpoint, or third-party source tree.