Vision Embodied End-to-end Robot Architecture (VEERA)

An open robotics project building a closed-loop manipulation pipeline using NVIDIA Isaac Sim, ROS 2 Jazzy, and OpenVLA. Designed to run in simulation with cloud-based VLA inference, and architected to extend to real robot hardware.

---

Project Goal

Build a closed-loop manipulation system where:

• A robot arm runs inside Isaac Sim 6.0.1 (native install)
• A wrist camera publishes images over ROS 2 Jazzy
• OpenVLA (7B VLA model) runs inference remotely via a FastAPI endpoint on Modal
• Joint commands are published back to the simulated arm via ROS 2

The target task is pick-and-place. No physical hardware required.

---

System Architecture

Isaac Sim (native install, local)
  └── Robot arm + wrist camera
        └── /camera/image_raw [ROS 2 topic]
              └── ROS 2 bridge node (Python)
                    └── POST /infer → FastAPI endpoint (Modal cloud GPU)
                          └── OpenVLA inference (4-bit quantised, ~10GB VRAM)
                                └── joint delta response
                                      └── /joint_commands [ROS 2 topic]
                                            └── Isaac Sim articulation controller

---

Stack

Component	Version / Tool
Simulation	NVIDIA Isaac Sim 6.0.1 (native install)
ROS 2	Jazzy (Ubuntu 24.04, native system install)
VLA Model	OpenVLA (7B, 4-bit quantised via bitsandbytes)
Inference serving	FastAPI on Modal (cloud GPU)
HTTP client	httpx (async, inside ROS 2 node)
OS	Ubuntu 24.04 LTS
CPU	Intel Core Ultra 9 275HX
RAM	32 GB
GPU	NVIDIA RTX 5070 Ti Laptop (12GB VRAM, driver 580.95, CUDA 13.0)

Note: local VRAM only needs to carry Isaac Sim's rendering/physics — OpenVLA inference stays on Modal, so there's no local VRAM contention between sim and inference.

---

Environment Setup

Prerequisites

• Ubuntu 24.04 LTS
• NVIDIA GPU (12GB+ VRAM recommended)
• NVIDIA driver + CUDA installed

1. Install ROS 2 Jazzy (system-wide, before Isaac Sim)

sudo apt update && sudo apt install -y curl gnupg lsb-release
sudo curl -sSL https://raw.githubusercontent.com/ros/rosdistro/master/ros.key -o /usr/share/keyrings/ros-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/ros-archive-keyring.gpg] http://packages.ros.org/ros2/ubuntu $(. /etc/os-release && echo $UBUNTU_CODENAME) main" | sudo tee /etc/apt/sources.list.d/ros2.list > /dev/null
sudo apt update
sudo apt install -y ros-jazzy-desktop
source /opt/ros/jazzy/setup.bash

Install ROS 2 before installing Isaac Sim.

2. Install Isaac Sim 6.0.1 (native binary)

Download the Isaac Sim binaries from NVIDIA (requires a free NVIDIA account), then:

mkdir -p ~/isaacsim
# unzip the downloaded package into ~/isaacsim

export ISAACSIM_PATH="${HOME}/isaacsim"
export ISAACSIM_PYTHON_EXE="${ISAACSIM_PATH}/python.sh"

3. Launch Isaac Sim with ROS 2 sourced

source /opt/ros/jazzy/setup.bash
cd ~/isaacsim
./isaac-sim.sh

Sourcing ROS 2 Jazzy before launching, in the same terminal, lets Isaac Sim auto-load the matching internal Jazzy bridge libraries — no ROS_DISTRO override needed. If ROS 2 isn't sourced, Isaac Sim falls back to its own bundled Jazzy libraries automatically on Ubuntu 24.04.

Project Status

• Environment Setup — Native Isaac Sim 6.0.1 install on Ubuntu 24.04 + native ROS 2 Jazzy
• Scene & Robot Setup — Robot URDF, scene setup, wrist camera publishing to ROS 2
  • Franka robot + table + pick object loaded in scene
  • ROS 2 joint states publishing verified (/joint_states)
  • Wrist camera → ROS 2 topic publish (Action Graph)
  • Verify camera topic externally with ros2 topic echo
  • Confirm ROS 2 Jazzy bridge stays stable across a full sim run
• RO2-cuMotion Robot Control - For GPU accelerated robot control using ROS2
• VLA Inference Service — OpenVLA loaded with 4-bit quantisation, FastAPI endpoint on Modal
• Inference Loop Integration — ROS 2 bridge node closing the inference loop
• Evaluation — Evaluation and prompt tuning
• Demo & Writeup — Demo video, architecture writeup, results

---