Layered Mesh-Gaussian (4D)

This repository extends Layered Mesh-Gaussian (LMG) — itself built on "GaMeS: Mesh-Based Adapting and Modification of Gaussian Splatting" and the original 3D Gaussian Splatting — from static scenes to 4D dynamic scenes.

On top of LMG's hybrid mesh-Gaussian representation and distortion-based splat allocation, this fork adds:

• Dynamic mesh sequences. Train and render over a sequence of per-frame meshes (--mesh_start / --mesh_end) that share topology, with a canonical frame plus per-frame fine-tuning.
• Variable-topology sequences (--variable_topology). When each frame is meshed independently (different vertex/face counts), keep a single persistent set of Gaussians and re-bind them to every frame's mesh by register-then-snap tracking (--track_method ∈ {laplacian (default, robust), nricp_amberg, nricp_sumner, closest_point}). Gaussian identity and the per-Gaussian temporal residuals persist across the topology changes; the input meshes are used as a fixed per-frame scaffold (not optimized). No consistent-topology preprocessing required.
• Sequence-aware allocation. A single splat budget is allocated once across the whole sequence by reducing per-frame budgeting weights (--sequence_weight_reduction ∈ {mean, max, mean_max}), so the layout stays stable over time.
• Compact temporal attribute module. A small neural network (--temporal_attributes) predicts per-Gaussian residuals (uvw / scaling / opacity / color) as a function of time, instead of storing a full set of Gaussian attributes per frame.

Installation

See INSTALL.md for environment setup. The pinned environment (see environment.yml) is load-bearing for the custom CUDA submodules under submodules/ (diff-gaussian-rasterization, simple-knn).

Quick start: the 4D pipeline (dancer example)

The four shell wrappers run the full dynamic pipeline and default to the bundled data/dancer sequence. Each is configured by environment variables at the top of the file (GPU, dataset, mesh dir/prefix, frame range, budget, policy, temporal-module settings).

# 0) (Optional) Warmup: pre-render mesh backgrounds/depth for all cameras and
#    precompute the sequence-aware allocation policy.
bash warmup.sh

# 1) Train: canonical frame + per-frame fine-tuning, with the compact temporal
#    attribute module enabled by default.
bash train.sh

# 2) Render: writes per-frame test renders (compact temporal render by default).
bash render.sh

# 3) Evaluate: PSNR / SSIM / LPIPS over the rendered test views.
bash evaluation.sh

Common overrides (same variables across scripts):

GPU_ID=1                       # CUDA device
DATASET=data/dancer            # source dataset (cameras + images)
MESH_DIR=data/dancer/mesh_dynamic
MESH_PREFIX=dancer_            # frame files look like dancer_0001.obj ...
START_FRAME=1 END_FRAME=10     # inclusive frame range
TOTAL_SPLATS=100000            # global splat budget
ALLOC_POLICY=distortion        # uniform|random|area|planarity|distortion
SEQUENCE_WEIGHT_REDUCTION=max  # mean|max|mean_max
TEMPORAL_ATTRIBUTES=1          # enable the compact temporal module

Individual stages (raw commands)

Warmup (optional)

Pre-renders mesh RGB/depth backgrounds for every camera and generates/validates the allocation policy .npy, then exits before the training loop.

CUDA_VISIBLE_DEVICES=1 python train.py --eval --warmup_only \
  -s data/dancer -m output/dancer_seq \
  --texture_obj_path data/dancer/mesh_dynamic/dancer_0001.obj \
  --mesh_start 1 --mesh_end 10 \
  --mesh_type sugar --gs_type gs_mesh \
  --total_splats 100000 --alloc_policy distortion \
  --sequence_weight_reduction max \
  --precaptured_mesh_img_path data/dancer/mesh -w --iteration 10

Training

4D (mesh sequence + temporal module). When --mesh_start/--mesh_end span more than one frame and --gs_type gs_mesh, training runs the sequence path: it computes a sequence-aware policy, trains the canonical frame, then fine-tunes each remaining frame.

CUDA_VISIBLE_DEVICES=1 python train.py --eval \
  -s data/dancer -m output/dancer_network \
  --texture_obj_path data/dancer/mesh_dynamic/dancer_0001.obj \
  --mesh_start 1 --mesh_end 10 --canonical_frame 1 \
  --mesh_type sugar --gs_type gs_mesh --occlusion \
  --total_splats 100000 --alloc_policy distortion \
  --sequence_weight_reduction max \
  --temporal_attributes --temporal_attr_width 64 --temporal_attr_depth 3 \
  --temporal_attr_latent_dim 8 --temporal_start_iter 100 \
  --precaptured_mesh_img_path data/dancer/mesh -w --iteration 1000

Variable-topology 4D (per-frame meshes differ). Add --variable_topology (optionally --track_method). Training keeps a persistent Gaussian set, re-binds it to each frame by register-then-snap tracking, and writes a compact per-frame binding cache to <model_path>/bindings/frame_XXXX.pt. To render, point the compact path at that cache (the persistent base lives in the canonical frame dir):

# train
... python train.py ... --mesh_start 1 --mesh_end 10 --canonical_frame 1 \
  --variable_topology --track_method nricp_amberg \
  --temporal_attributes ...

# render (per frame): base checkpoint + cached binding + temporal residuals
... python render_mesh_splat.py -s data/dancer -m output/dancer/frame_0005 \
  --load_model_path output/dancer/frame_0001 \
  --binding_cache_dir output/dancer/bindings \
  --texture_obj_path data/dancer/mesh_dynamic/dancer_0005.obj \
  --temporal_attributes --temporal_attr_checkpoint output/dancer/temporal_attr_model.pth \
  --mesh_start 1 --mesh_end 10 --gs_type gs_mesh -w

Or drive the whole sequence with VARIABLE_TOPOLOGY=1 bash render.sh.

Static (single mesh). Omit --mesh_start/--mesh_end (and the temporal flags). Use --total_splats for an absolute budget or --budget_per_tri for a per-triangle multiplier.

CUDA_VISIBLE_DEVICES=1 python train.py --eval \
  -s /path/to/dataset -m output/exp_name \
  --texture_obj_path /path/to/mesh.ply --mesh_type colmap \
  --gs_type gs_mesh --occlusion \
  --budget_per_tri 1.5 --alloc_policy planarity \
  -w --iteration 5000

Rendering

python render_mesh_splat.py \
  -s data/dancer -m output/dancer_network \
  --gs_type gs_mesh --skip_train --occlusion \
  --total_splats 100000 --alloc_policy distortion \
  --texture_obj_path data/dancer/mesh_dynamic/dancer_0001.obj \
  --mesh_type sugar --policy_path /path/to/policy.npy \
  --temporal_attributes --temporal_attr_checkpoint output/dancer_network/temporal_attr_model.pth \
  --mesh_start 1 --mesh_end 10 -w

Train-free point-light relighting

render_relight.py relights an existing gs_mesh checkpoint without optimizing materials or training another network. The mesh supplies normals, inverse-square point-light transport, and cube-map shadows. The same transport field is interpolated at every mesh-bound Gaussian, while the trained Gaussian DC color and a conservative fraction of its view-dependent SH detail are retained.

python render_relight.py \
  -s data/dancer -m output/dancer/frame_0001 \
  --iteration 5000 --skip_train -w \
  --texture_obj_path data/dancer/mesh_dynamic/dancer_0001.obj \
  --mesh_type sugar --gs_type gs_mesh \
  --total_splats 100000 --alloc_policy distortion \
  --light_position 1.0 2.0 0.5 \
  --light_color 1.0 0.85 0.7 \
  --light_intensity 1.5 --ambient 0.25 \
  --shadow_strength 0.9 --shadow_resolution 512

The default --hybrid_shadows mode combines mesh depth shadows with approximate light-space Gaussian opacity. It costs twelve extra Gaussian renders per light (opacity and a depth moment for six cube faces); use --no-hybrid_shadows for the faster mesh-only shadow path.

If the original capture light can be estimated, pass --reference_light_position, --reference_light_color, and --reference_light_intensity. The renderer then applies a target/reference shading ratio; choosing the same target and reference light gives an exact identity transfer. Without a reference, the baked checkpoint is treated as neutral unit illumination. Since the input texture and SH appearance may contain baked shadows and highlights, this is controllable lighting transfer rather than full material recovery.

Multiple manually authored lights or animation keyframes can be supplied with --light_path:

{
  "lights": [
    {
      "name": "left_warm",
      "position": [-1.5, 1.0, 1.5],
      "color": [1.0, 0.75, 0.55],
      "intensity": 1.2,
      "ambient": 0.25
    },
    {
      "name": "right_cool",
      "position": [1.5, 1.0, 1.5],
      "color": [0.55, 0.75, 1.0],
      "intensity": 1.2,
      "ambient": 0.25
    }
  ]
}

Relit previews are written below <model_path>/<train|test>/ours_<iteration>/renders_relight/<light-name>/. Use --output_format pt for unclamped float tensors, or exr when the local OpenCV build has OpenEXR enabled. PNG output clamps by default so an identity transfer preserves the checkpoint appearance; add --tone_map reinhard for high-intensity previews. The relighting path currently requires the PyTorch3D mesh backend.

Interactive relighting viewer

viewer_relight.py keeps one trained checkpoint and camera loaded while exposing OpenCV sliders for point-light position, color, intensity, ambient illumination, shadow strength, and exposure:

python viewer_relight.py \
  -s data/dancer \
  -m output/dancer_test/frame_0001 \
  --iteration 5000 -w \
  --texture_obj_path data/dancer/meshes_distorted/dancer_0001.obj \
  --mesh_type sugar --total_splats 100000 \
  --light_position 1.0 2.0 0.5 \
  --light_color 1.0 0.85 0.7 \
  --light_intensity 1.5 --ambient 0.25 \
  --view_set test --view_index 0

The Show mesh and Show Gaussians controls independently hide either compositing layer. In the Gaussian-only view the lighting is still computed from the Gaussian's mesh binding, which makes it useful for verifying that mesh transport is actually driving the splats. Keyboard shortcuts are:

• M: toggle mesh visibility.
• G: toggle Gaussian visibility.
• S: toggle shadows.
• T: toggle Reinhard preview tone mapping.
• , / .: previous or next dataset camera.
• P: save the current annotated frame under viewer_captures/.
• Q or Esc: close the viewer.

Direct lighting updates immediately while a slider is moving. If shadows are enabled, their light-space maps rebuild after 350 ms of inactivity; adjust this with --shadow_idle_ms. The viewer defaults to faster mesh-only shadows at resolution 256. Add --hybrid_shadows to include Gaussian opacity shadows and --viewer_shadows to start with shadows enabled. Hybrid shadow rebuilding requires twelve additional Gaussian renders and will be noticeably slower.

The viewer requires a desktop session with DISPLAY available and an OpenCV build with GUI support.

Evaluation metrics

python metrics.py -m output/dancer_network/frame_0001 ... --gs_type gs_mesh

Key command-line arguments

Dataset & mesh

Argument	Description	Type
-s, --source_path	Path to dataset directory	str (required)
-m, --model_path	Output model directory	str (required)
--texture_obj_path	Path to mesh file (.obj or .ply)	str
--mesh_type	Mesh source type: sugar or colmap	str
-w, --white_background	Use white background (not black)	flag

Mesh-splat configuration

Argument	Description	Default
--gs_type	Renderer/model type (see Model types below)	gs_mesh
--total_splats	Total number of splats for the entire scene	None
--budget_per_tri	Splats per triangle (multiplier); used if --total_splats unset	1.0
--alloc_policy	uniform, random, area, planarity, distortion	area
--occlusion	Enable occlusion-aware (mesh-depth) compositing	Disabled
--policy_path	Path to a pre-computed policy .npy	None
--precaptured_mesh_img_path	Dir with mesh_texture/ and mesh_depth/ (from warmup)	None
--mesh_rasterizer_type	Mesh background backend: pytorch3d or nvdiffrast	pytorch3d

4D / sequence & temporal module

Argument	Description	Default
--mesh_start / --mesh_end	Inclusive frame range of a numbered mesh sequence	None
--canonical_frame	Frame trained first and reused as the base	first
--canonical_iterations	Iterations for the canonical frame	None
--temporal_iterations	Iterations per fine-tuned frame	500
--sequence_weight_reduction	mean, max, or mean_max over per-frame weights	max
--recompute_sequence_policy	Recompute the sequence policy even if cached	False
--strict_sequence_topology	Require identical face ordering across frames	False
--variable_topology	Allow per-frame topology changes; persistent Gaussians re-bound by tracking	False
--track_method	Re-binding tracker: laplacian, nricp_amberg, nricp_sumner, closest_point, tvm	laplacian
--track_rigid_prealign	Rigid ICP pre-align before non-rigid registration	True
--tvm_arap_dir/--tvm_editor_exe/--tvm_config_template	External ARAP+TVM tracker paths (when --track_method tvm); auto-runs the pipeline per frame, falls back to laplacian on failure	submodules/…
--temporal_attributes	Enable the compact temporal attribute module	False
--temporal_attr_width/_depth/_latent_dim	MLP width / depth / per-triangle latent size	64 / 3 / 8
--temporal_attr_lr	Temporal module learning rate	1e-3
--temporal_start_iter	Iteration at which temporal training begins	100
--temporal_predict_{uvw,scaling,opacity,color}	Which residuals the module predicts	flags
--temporal_max_d_{uvw,scaling,opacity,color}	Clamp bounds on predicted residuals	per-attr
--temporal_attr_checkpoint	(render) temporal module weights to load	None

Training / debugging

Argument	Description	Default
--iteration	Number of training iterations	1000
--eval	Hold out test views during training	False
--warmup_only	Run only the warmup stage and exit	False
--debugging	Save debug visualizations	False
--debug_freq	Frequency of saving debug images	1

Model types (--gs_type)

• gs — standard Gaussian Splatting.
• gs_mesh — Gaussians parameterized on a mesh surface (the LMG path; requires a mesh).
• gs_flat — flat Gaussians (one scale ≈ epsilon).
• gs_multi_mesh — multi-mesh variant of gs_mesh (use --meshes).
• gs_flame — FLAME-model parameterization (requires FLAME files under scene/flame/).
• gs_points — render-only parameterization of the flat model.

Allocation policies (--alloc_policy)

uniform, random, area (face-area proportional), planarity (mean resultant length of the face-normal neighborhood), and distortion (per-triangle rendered distortion vs. ground truth). See scene/budgeting.py.

Mesh format notes

• SuGaR meshes: --mesh_type sugar --texture_obj_path /path/to/mesh.obj
• Colmap meshes: --mesh_type colmap --texture_obj_path /path/to/mesh.ply

Output structure

output/
└── EXP_NAME/
    ├── frame_0001/                 # per-frame model + renders (sequence mode)
    │   ├── point_cloud/iteration_*/…
    │   └── test/                    # rendered test views (input to metrics)
    ├── temporal_attr_model.pth      # compact temporal module weights
    ├── sequence_policy/             # sequence-aware allocation .npy + weights
    └── results_gs_mesh.json         # metrics

Repository layout

Entry points live at the root (train.py, render_mesh_splat.py, metrics.py, full_eval.py, convert.py); arguments/ + arguments_games/ hold parameter groups; scene/ holds cameras / dataset readers / the Gaussian model / budgeting.py plus every gs_type model class, its readers, the model_zoo.py dispatch registries, and the flame/ package; renderer/ holds the rasterizer front ends; utils/ holds shared helpers (including mesh_utils.py and sequence_utils.py). Dormant experimental code is kept under archive/.