Skip to content

Sync tutorials with 6.2 #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sureshgupta-amd wants to merge 1 commit into
base: main
Choose a base branch
from
+705 −433
Open

Sync tutorials with 6.2 #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions versal_2ve/examples/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@ examples/
├── tutorials/ # Guided tutorials (see tutorials/)
│ ├── resnet18_bf16/ # ResNet-18 BF16 flow
│ ├── resnet50_quark/ # ResNet50 INT8 with AMD Quark
│ ├── resnet50Cpp/ # ResNet50 FP32 (BF16); Python sources; run with C++ (ONNX-RT)
│ ├── yolov8m/ # YOLOv8m detection: Quark VINT8, compile, ORT on target
│ └── README.md
├── python_examples/ # Python ORT + VitisAI EP on the embedded target (see python_examples/)
Expand Down Expand Up @@ -58,7 +57,6 @@ The tables below are the reference overview:
|---------|----------|------|---------------|---------------|-------|--------------|
| [`resnet18_bf16`](tutorials/resnet18_bf16/) | Python | ResNet-18: export ONNX → Vitis AI compile → deploy; `runmodel.py` compares CPU vs NPU (e.g. RMSE) | Yes (`compile.py` in Docker) | Yes (`runmodel.py` on board) | BF16 (compiler from FP32 ONNX) | ImageNet-style validation; ONNX under `models/` |
| [`resnet50_quark`](tutorials/resnet50_quark/) | Python | ResNet50: Quark INT8 quant → compile → accuracy on CPU/NPU → on-target inference | Yes (`compile.py` in Docker) | Yes (`runmodel.py`; `runmodel_pre_cpu.py` for host checks) | INT8 (AMD Quark calibration) | ImageNet val / calibration JPEGs; ONNX under `models/` |
| [`resnet50Cpp`](tutorials/resnet50Cpp/) | Python and C++ | ResNet50 FP32 (BF16); Python sources; run with C++ (ONNX-RT) | Yes (`compile.py` in Docker) | Yes (`runmodel.cpp` on board) | FP32 (BF16) | ImageNet val / calibration JPEGs; ONNX under `models/` |
| [`yolov8m`](tutorials/yolov8m/) | Python | YOLOv8m: Quark VINT8 (skip-nodes), compile, latency tuning, ORT EP on board | Yes (`compile.py` in Docker) | Yes (`run_inference.py` on board) | INT8 VINT8 + BF16 tail (per tutorial) | Calibration / val images; COCO-style labels |

### Python samples (`python_examples`)
Expand Down
12 changes: 5 additions & 7 deletions versal_2ve/examples/tutorials/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,16 +20,14 @@ tutorials/
├── README.md # This file
├── resnet18_bf16/ # ResNet-18 end-to-end (BF16); Python sources + README
├── resnet50_quark/ # ResNet50 INT8 with AMD Quark; Python sources + README
├── resnet50Cpp/ # ResNet50 FP32 (BF16); Python sources (compile); run with C++ (ONNX-RT)
└── yolov8m/ # YOLOv8m object detection (Quark VINT8 → compile → ORT); sources + README
```

## Tutorials (summary)

| Tutorial | Role | Quant | Main scripts |
|----------|------|-------|----------------|
| [**resnet18_bf16**](resnet18_bf16/) | Export ONNX, compile with Vitis AI, deploy and run ORT on the board; optional CPU vs NPU comparison | BF16 via compiler | `export_to_onnx.py`, `compile.py`, `runmodel.py` |
| [**resnet50_quark**](resnet50_quark/) | Download ONNX, Quark INT8 quantization, compile, evaluate accuracy, on-target inference | INT8 (Quark) | `quantize.py`, `compile.py`, `evaluate.py`, `runmodel.py`, `runmodel_pre_cpu.py` |
| [**yolov8m**](yolov8m/) | YOLOv8m detection: export, Quark VINT8 (with skip-nodes), compile, NPU timing / config tuning, on-target ORT inference | INT8 VINT8 (Quark); compiler BF16 tail per tutorial | `models/export_to_onnx.py`, `quantize.py`, `compile.py`, `evaluate.py`, `run_inference.py` |

| Tutorial | description | Datatype |
|----------|-------------|----------|
| [**resnet18_bf16**](resnet18_bf16/) | Export ONNX, compile with Vitis AI, deploy and run ORT on the board; optional CPU vs NPU comparison | BF16 via compiler |
| [**resnet50_quark**](resnet50_quark/) | Download ONNX, Quark INT8 quantization, compile, evaluate accuracy, on-target inference | INT8 (Quark) |
| [**resnet50Cpp**](resnet50Cpp/) | Download ONNX, compile model, compile C++ ONNX-RT run program, on-target inference with C++ based executable | BF16 via compiler |
| [**yolov8m**](yolov8m/) | YOLOv8m detection: export, Quark VINT8 (with skip-nodes), compile, NPU timing / config tuning, on-target ORT inference | INT8 VINT8 (Quark); compiler BF16 tail per tutorial |
188 changes: 31 additions & 157 deletions versal_2ve/examples/tutorials/resnet18_bf16/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,10 @@
<table class="sphinxhide" width="100%">
<tr width="100%">
<td align="center"><img src="https://raw.githubusercontent.com/Xilinx/Image-Collateral/main/xilinx-logo.png" width="30%"/><h1>Vitis AI Development</h1>
<td align="center"><img src="https://raw.githubusercontent.com/Xilinx/Image-Collateral/main/xilinx-logo.png" width="30%"/><h1> Getting Started with Vitis AI: ResNet-18 End-to-End Flow</h1>
</td>
</tr>
</table>

# Getting Started with Vitis AI: ResNet-18 End-to-End Flow

## Introduction

The Vitis AI toolchain supports compiling and deploying AI models in the ONNX format for efficient execution on Versal AI Edge Series Gen 2 devices. By using the Vitis AI Execution Provider (EP) within ONNX Runtime, developers can seamlessly run ONNX models and leverage hardware acceleration provided by the NPU.
Expand All @@ -17,7 +15,7 @@ This tutorial shows how to compile an ONNX model with the Vitis AI flow and depl

To build the example and deploy it on board, the following software and hardware are required:

* Latest Vitis AI Docker for Versal AI Edge Series Gen 2:
* Vitis AI 6.2 Docker for Versal AI Edge Series Gen 2:
* Instructions for installation and startup are in the Vitis AI User Guide for Versal AI Edge Series Gen 2.
* VEK385 evaluation kit:
* Setup instructions are available in the Vitis AI User Guide for Versal AI Edge Series Gen 2.
Expand All @@ -35,15 +33,15 @@ Before starting Docker, get the tutorial repository and adjust the access permis
chmod -R a+w <path/to/resnet18_bf16>
```

Load the docker image:
Refer to Vitis AI User Guide for Versal AI Edge Series Gen 2 to load and start docker:

```
docker load -i <docker_image_file>.tgz
docker run -it --network host \
-v /path/to/your/license:/usr/licenses \
-v /<host_path>:/<path_in_docker> \
--rm <REPOSITORY>:<TAG> "bash"
```

Load the latest docker image and launch it as explained in the Vitis AI User Guide for Versal AI Edge Series Gen 2.


## Vitis AI Compilation & Deployment Flow

1. Inside the docker, change directory to the tutorial folder, install python packages required by the example, and export the ResNet-18 ONNX model:
Expand Down Expand Up @@ -88,10 +86,10 @@ To get more details about compilation results you can display the content of the
```
--------- Final Summary of VAIML Pass ----------
OS: Linux X64
VAIP commit: 744227ab2a0fddec1eccdfe04ca222afd339f53f
VAIP commit: ......
Model: ....../models/resnet18.a1_in1k.onnx
Model signature: 41d764d4ef1d716a260bc7b2b4e07ff1
Device: ve2-xc2ve3858
Device: ve2
Model data type: float32
Device data type: bfloat16
Number of operators in the model: 49
Expand All @@ -110,17 +108,13 @@ Subgraph vaiml_par_0 stats:
Type: npu
Operators: 49 (100.000%)
GOPs : 3.644 (100.000%) OPs: 3,643,881,552
fp32 ops %: 99.731
```

3. Refer to Vitis AI User Guide for Versal AI Edge Series Gen 2, boot up the AIE-ML_v2 board, and run following commands in the board to setup environment:
3. Refer to Vitis AI User Guide for Versal AI Edge Series Gen 2, boot up the AIE-ML_v2 board, and setup environment:

```
sudo su # To avoid permission issues while creating the hw context
echo 1 > /sys/module/rcupdate/parameters/rcu_cpu_stall_suppress
export XRT_AIARM=true
export LD_LIBRARY_PATH=/usr/lib/python3.12/site-packages/voe/lib/:/usr/lib/python3.12/site-packages/flexmlrt/lib/
export XLNX_ENABLE_CACHE=0
export XRT_ELF_FLOW=1
export LD_LIBRARY_PATH=/usr/lib/python3.12/site-packages/voe/lib:/usr/lib/python3.12/site-packages/flexmlrt/lib:/usr/lib/python3.12/site-packages/onnxruntime/capi
```

4. Run the inference on the board. The working directory can be mounted on the board or copied to the board by scp:
Expand All @@ -139,10 +133,10 @@ The script runs four inferences of the model and displays messages similar to th

```
Running 4 inferences, comparing CPU and NPU outputs
Iteration 1: Max absolute difference = 0.198444, Root mean squared error = 0.081654
Iteration 2: Max absolute difference = 0.154286, Root mean squared error = 0.068371
Iteration 3: Max absolute difference = 0.210051, Root mean squared error = 0.081457
Iteration 4: Max absolute difference = 0.196577, Root mean squared error = 0.063051
Iteration 1: Max absolute difference = 0.228190, Root mean squared error = 0.085001
Iteration 2: Max absolute difference = 0.231800, Root mean squared error = 0.086444
Iteration 3: Max absolute difference = 0.248591, Root mean squared error = 0.092887
Iteration 4: Max absolute difference = 0.177171, Root mean squared error = 0.069008
Inference Done!
```

Expand All @@ -156,20 +150,22 @@ python runmodel.py
The output includes the number of operators offloaded to the NPU and the number of NPU-executed subgraphs:

```
I20250529 19:23:43.187124 1265 stat.cpp:193] [Vitis AI EP] No. of Operators :
I20250529 19:23:43.187179 1265 stat.cpp:204] VAIML 49
I20250529 19:23:43.187194 1265 stat.cpp:213]
I20250529 19:23:43.187206 1265 stat.cpp:218] [Vitis AI EP] No. of Subgraphs :
I20250529 19:23:43.187219 1265 stat.cpp:226] NPU 1
I20250529 19:23:43.187227 1265 stat.cpp:229] Actually running on NPU 1
I20250529 19:23:43.188418 1265 vitisai_compile_model.cpp:1477] AVG CPU Usage 95.4545%
I20250529 19:23:43.188459 1265 vitisai_compile_model.cpp:1478] Peak Working Set size 213.195 MB
[2025-05-29 19:23:43.261] [console] [info] [FLEXMLRT] FlexMLClient.cpp:1269 FlexMLRT Git Hash: 512d4e65
I20260608 01:57:50.007778 1212 stat.cpp:198] [Vitis AI EP] No. of Operators :
I20260608 01:57:50.007839 1212 stat.cpp:198] VAIML 49
I20260608 01:57:50.007958 1212 stat.cpp:198]
I20260608 01:57:50.007978 1212 stat.cpp:198] [Vitis AI EP] No. of Subgraphs :
I20260608 01:57:50.007992 1212 stat.cpp:198] NPU 1
I20260608 01:57:50.008001 1212 stat.cpp:198] Actually running on NPU 1
......
Running 4 inferences, comparing CPU and NPU outputs
Iteration 1: Max absolute difference = 0.193388, Root mean squared error = 0.083394
Iteration 2: Max absolute difference = 0.241203, Root mean squared error = 0.090799
Iteration 3: Max absolute difference = 0.190464, Root mean squared error = 0.080506
Iteration 4: Max absolute difference = 0.217875, Root mean squared error = 0.087615
......
Iteration 1: Max absolute difference = 0.211893, Root mean squared error = 0.075283
......
Iteration 2: Max absolute difference = 0.220449, Root mean squared error = 0.082772
......
Iteration 3: Max absolute difference = 0.169577, Root mean squared error = 0.055290
......
Iteration 4: Max absolute difference = 0.223119, Root mean squared error = 0.077367
Inference Done!
```

Expand All @@ -186,128 +182,6 @@ And then run the inference. The output contains information as follows:
[xrt_xdna] DEBUG: Partition Created with start_col 0 num_columns 4 partition_id 1024
```

## Vitis AI Flow Essential

This section covers some essential concepts in Vitis AI model compilation and inference. By learning the concepts and example codes, the flow can be extended to other ONNX models.

1. Onnx model is used as input to the model compilation, which tries to accelerate the operators in NPU. So, prepare the ONNX model in ML frameworks.

2. Models are compiled for the NPU by creating an ONNX inference session using the Vitis AI Execution Provider (VAI EP). The example python code can be found in `compile.py`.

```
import onnxruntime

provider_options_dict = {
"config_file": 'vitisai_config.json',
"cache_dir": 'my_cache_dir',
"cache_key": 'resnet18.a1_in1k',
"log_level": 'info',
"target": 'VAIML'
}

print(f"Creating ORT inference session for model models/resnet18.a1_in1k.onnx")
session = onnxruntime.InferenceSession(
'models/resnet18.a1_in1k.onnx',
providers=["VitisAIExecutionProvider"],
provider_options=[provider_options_dict]
)
```

The example configuration file `vitisai_config.json` contains options for Vitis AI compiler:

```
{
"passes": [
{
"name": "init",
"plugin": "vaip-pass_init"
},
{
"name": "vaiml_partition",
"plugin": "vaip-pass_vaiml_partition",
"vaiml_config":
{
"device": "ve2-xc2ve3858",
"optimize_level": 2,
"logging_level": "info",
"keep_outputs": true,
"threshold_gops_percent": 20
}
}
],
"target": "VAIML",
"targets": [
{
"name": "VAIML",
"pass": [
"init",
"vaiml_partition"
]
}
]
}
```

The value `ve2-xc2ve3858` for the `device` option selects the VEK385 part on Versal AI Edge Series Gen 2 (AIE-ML_v2) for Vitis AI 6.2 compilation.

3. To execute the compiled model on hardware, transfer the compiled model artifacts and the original ONNX model file to the target board. The compiled ONNX graph is automatically partitioned into multiple subgraphs by the VitisAI Execution Provider (EP). The subgraph(s) containing operators supported by the NPU are executed on the NPU. The remaining subgraph(s) are executed on the CPU. This graph partitioning and deployment technique across CPU and NPU is fully automated by the VAI EP and is totally transparent to the end-user.

Model execution is performed using a Python script that establishes an ONNX Runtime (ORT) inference session. This session is initialized with the target ONNX model and configured to utilize the Vitis AI Execution Provider (EP). Upon execution, the ORT session leverages the Vitis AI EP, which utilizes the compiled model binaries in the specified directory and deploys the ONNX subgraph(s) on the NPU and the CPU.

The example python code for deploying on the hardware can be found in `runmodel.py`. It creates `InferenceSession` for CPU and NPU and runs inferences. And then compute the RMSE (Root Mean Square Error) between the CPU and NPU results:

```
import numpy as np
import onnxruntime as ort

provider_options_dict = {
"config_file": 'vitisai_config.json',
"cache_dir": 'my_cache_dir',
"cache_key": 'resnet18.a1_in1k',
"log_level": 'info',
"target": 'VAIML',
}

print(f"Creating ORT inference session for model models/resnet18.a1_in1k.onnx")

onnx_model="models/resnet18.a1_in1k.onnx"
# CPU session to compute reference values
cpu_session = ort.InferenceSession(
onnx_model,
)
# NPU session
npu_session = ort.InferenceSession(
onnx_model,
providers=["VitisAIExecutionProvider"],
provider_options=[provider_options_dict]
)

num_iter = 4
print(f"Running {num_iter} inferences, comparing CPU and NPU outputs")
for i in range(num_iter):
# Generate random data
input_data = {}
for input in npu_session.get_inputs():
fixed_shape = [1 if isinstance(dim, str) else dim for dim in input.shape]
input_data[input.name] = np.random.rand(*fixed_shape).astype(np.float32)

# Compute CPU results (reference values)
cpu_outputs = cpu_session.run(None, input_data)
# Compute NPU results
try:
npu_outputs = npu_session.run(None, input_data)
except Exception as e:
print(f"Failed to run on NPU: {e}")
sys.exit(1)

# Compare CPU and NPU results
max_diff = np.max(np.abs(cpu_outputs[0] - npu_outputs[0]))
rmse = np.sqrt(np.mean((cpu_outputs[0] - npu_outputs[0]) ** 2))
print(f'Iteration {i+1:3d}: Max absolute difference = {max_diff:.6f}, Root mean squared error = {rmse:.6f}')

print("Inference Done!")
```

## Summary

By completing this tutorial, you learned:
Expand Down
6 changes: 4 additions & 2 deletions versal_2ve/examples/tutorials/resnet18_bf16/vitisai_config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,11 @@
{
"device": "ve2-xc2ve3858",
"optimize_level": 2,
"logging_level": "info",
"keep_outputs": true,
"threshold_gops_percent": 20
"logging_level": "info",
"threshold_gops_percent": 20,
"dp_size": 1,
"tp_size": 1
}
}
],
Expand Down
22 changes: 20 additions & 2 deletions versal_2ve/examples/tutorials/resnet50Cpp/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ The process begins with getting the resnet50 model from HuggingFace ONNX Model Z

This tutorial requires:

* Latest Vitis AI Docker for Versal AI Edge Series Gen 2:
* Vitis AI 6.2 Docker for Versal AI Edge Series Gen 2:
* Instructions for installation and startup are in the Vitis AI User Guide for Versal AI Edge Series Gen 2.
* VEK385 evaluation kit:
* Setup instructions are available in the Vitis AI User Guide for Versal AI Edge Series Gen 2.
Expand Down Expand Up @@ -62,8 +62,26 @@ cd ..
chmod -R a+w resnet50Cpp
```

Load the latest docker image and launch it as explained in the Vitis AI User Guide for Versal AI Edge Series Gen 2.
Load the docker image:

```bash
docker load -i <docker_image_file>.tgz
```

Run `docker images` to verify docker REPOSITORY, IMAGEID and TAG information.

|REPOSITORY | TAG | IMAGE ID | CREATED | SIZE |
|--------------------|-------------------|-------------|---------------|--------|
|vitis_ai_2ve_docker | release_v6.2 | ?????? | xx hours ago | 39.1GB |

Start the docker:

```bash
docker run -it --network host \
-v /path/to/your/license:/usr/licenses \
-v $PWD/resnet50Cpp:/resnet50Cpp \
--rm vitis_ai_2ve_docker:release_v6.2 "bash"
```
### Model Compilation

Compile the resnet50 ONNX model `models/resnet50-v1-12.onnx` for the NPU.
Expand Down
24 changes: 21 additions & 3 deletions versal_2ve/examples/tutorials/resnet50_quark/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ In addition, this tutorial includes scripts for evaluating model accuracy on bot

To build the example and deploy it on board, the following software and hardware are required:

* Latest Vitis AI Docker for Versal AI Edge Series Gen 2:
* Instructions for installation and startup are in the Vitis AI User Guide for Versal AI Edge Series Gen 2.
* Vitis AI 6.2 Docker for Versal AI Edge Series Gen 2:
* Instructions for installation and startup are in the Vitis AI User Guide for Versal AI Edge Series Gen 2.
* VEK385 evaluation kit:
* Setup instructions are available in the Vitis AI User Guide for Versal AI Edge Series Gen 2.
* Internet access:
Expand All @@ -46,8 +46,26 @@ Adjust the access permissions of the working directories on the host machine:
chmod -R a+w <path/to/resnet50_quark>
```

Load the latest docker image and launch it as explained in the Vitis AI User Guide for Versal AI Edge Series Gen 2.
Load the docker image:

```
docker load -i <docker_image_file>.tgz
```

Run `docker images` to verify docker REPOSITORY, IMAGEID and TAG information.

|REPOSITORY | TAG | IMAGE ID | CREATED | SIZE |
|--------------------|-------------------|-------------|---------------|--------|
|vitis_ai_2ve_docker | release_v6.2 | ?????? | xx hours ago | 39.1GB |

Star the docker:

```
docker run -it --network host \
-v /path/to/your/license:/usr/licenses \
-v $PWD/resnet50_quark:/resnet50_quark \
--rm vitis_ai_2ve_docker:release_v6.2 "bash"
```
## Evaluate the Float Model Accuracy with ImageNet Dataset

Evaluate the float model accuracy before quantization and deployment:
Expand Down
Loading