From d5697b00c79127cbcd446de45553f20bcbdda750 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 2 Jun 2026 01:04:14 +0800 Subject: [PATCH 01/37] Create tutorial for Apptainer on Torch Added a comprehensive tutorial on using Apptainer with Torch, covering container basics, file access, command execution, and using Docker images. --- .../11_apptainer_on_torch | 223 ++++++++++++++++++ 1 file changed, 223 insertions(+) create mode 100644 docs/hpc/13_tutorial_intro_hpc/11_apptainer_on_torch diff --git a/docs/hpc/13_tutorial_intro_hpc/11_apptainer_on_torch b/docs/hpc/13_tutorial_intro_hpc/11_apptainer_on_torch new file mode 100644 index 0000000000..372d82e2ff --- /dev/null +++ b/docs/hpc/13_tutorial_intro_hpc/11_apptainer_on_torch @@ -0,0 +1,223 @@ +# Introduction to Apptainer on Torch + +## Why containers? +Researchers often rely on complex software stacks that include programming languages and libraries. Installing and maintaining these dependencies can be challenging when different projects require different software versions. + +Containers provide a way to package software together with the environment it needs to run. Instead of manually installing every dependency on a system, users can run software inside a container that already includes the required libraries and tools. + +## What is Apptainer? +Apptainer is a container platform that allows users to run software inside isolated environments. It allows users to run software inside isolated environments without requiring administrator privileges on the host system. + +Apptainer is the continuation of the Singularity project. The open-source Singularity project was renamed to Apptainer and continues to be developed under the Linux Foundation. + +Unlike Docker, Apptainer does not require a privileged daemon running on the system. This makes it well suited for shared HPC environments where security and multi-user access are important considerations. + +Torch uses Apptainer as its supported container platform. Many container images distributed through Docker registries can be used directly with Apptainer, allowing researchers to take advantage of existing software environments while working on the cluster. + +# Files in Apptainer Containers +Apptainer is designed to work closely with the host filesystem. In most cases, your home directory and current working directory remain accessible from within the container. + +## Accessing Your Files + +While inside a container, check your current directory: + +```bash +pwd +``` + +You can also list files in your home directory: + +```bash +ls ~ +``` + +The files and directories you see should match those available outside the container. + +Files created in these mounted directories remain available after the container exits. + +## Binding Additional Directories + +Sometimes you may need access to additional directories that are not automatically available inside the container. + +Apptainer allows additional directories to be mounted using the `-B` option: + +```bash +apptainer shell \ + -B /scratch:/scratch \ + /share/apps/images/ubuntu-24.04.3.sif +``` + +This makes `/scratch` available inside the container. + +You can also mount a directory at a different location: + +```bash +apptainer shell \ + -B /scratch:/data \ + /share/apps/images/ubuntu-24.04.3.sif +``` + +In this example, files stored in `/scratch` on the host system are accessible through `/data` inside the container. + +## Why This Matters + +Container images are typically read-only. Research data, scripts, notebooks, and output files usually remain outside the container. + +By making host directories available inside the container, Apptainer allows applications to access data stored on Torch while maintaining a reproducible software environment. + +# Using Apptainer to Run Commands on Torch + +:::warning + +Container workloads should be run on compute nodes rather than login nodes. + +While simple commands may work on a login node, pulling images, launching software, installing packages, or building environments can consume significant CPU, memory, and storage resources. These activities should be performed within an interactive Slurm allocation or a batch job. +::: + +Torch provides many prebuilt container images under: + +```bash +ls /share/apps/images/ +``` + +:::note + +Torch provides container images with both `.sif` and `.sqf` extensions. + +`.sif` is the standard Apptainer image format. Some Torch-provided application images use `.sqf` and may be intended to be launched through wrapper scripts such as `run-anaconda3-2024.10-1.bash`. + +When available, use the wrapper script documented for that application. For general Apptainer examples in this tutorial, we use `.sif` images because they work directly with `apptainer exec`, `apptainer run`, and `apptainer shell`. + +::: + +For this tutorial, we will use the Ubuntu 24.04 image that is already available on the cluster. + +## Running Your First Container + +Apptainer images can define a default action that runs when the container starts. + +To launch a container and execute its default action, use `apptainer run`: + +```bash +apptainer run /share/apps/images/ubuntu-24.04.3.sif +``` + +Depending on how the image was built, this command may produce output, launch an application, or simply start and exit. + +The important point is that with `apptainer run`, Apptainer executes the default action defined by the image creator. + +Sometimes, however, we want to run a specific command instead of the image's default action. In those cases, we use `apptainer exec`. + +## Running Specific Commands Within a Container + +Unlike `apptainer run`, which executes the image's default action, `apptainer exec` allows us to specify exactly what command should run inside the container. + +For example: + +```bash +apptainer exec /share/apps/images/ubuntu-24.04.3.sif /bin/echo "Hello World!" +``` + +Output: + +```text +Hello World! +``` + +## The Difference Between `apptainer run` and `apptainer exec` + +Both `apptainer run` and `apptainer exec` start a container, but they serve different purposes. + +`apptainer run` executes the default action defined by the image creator. Depending on how the image was built, this may launch an application, run a script, or perform another predefined task. + +`apptainer exec` allows you to specify exactly which command should run inside the container. Rather than relying on the image's default behavior, you provide the command directly. + +In practice, `apptainer exec` is often used when working on HPC systems because it provides more control over what is executed inside the container environment. + +## Opening an Interactive Shell Within a Container + +Sometimes it is useful to explore a container interactively. Apptainer provides the `apptainer shell` command for this purpose. + +Launch a shell inside the Ubuntu container: + +```bash +apptainer shell /share/apps/images/ubuntu-24.04.3.sif +``` + +You should see a prompt similar to: + +```text +Singularity> +``` + +You can now run commands inside the container: + +```bash +whoami +pwd +cat /etc/os-release +``` + +Example output: + +```text +PRETTY_NAME="Ubuntu 24.04.3 LTS" +NAME="Ubuntu" +VERSION_ID="24.04" +... +``` + +Notice that the prompt changes to indicate that you are working inside the container environment. + +When you are finished, leave the container with: + +```bash +exit +``` + +This returns you to your normal shell on Torch. + +## Using Docker images with Apptainer + +So far, we have used container images that are already available on Torch under `/share/apps/images`. + +In practice, you may also want to run software that is not provided by the cluster. Apptainer can pull images directly from Docker registries and convert them into the Apptainer SIF format. + +For example, we can pull an official PyTorch image from Docker Hub: + +```bash +apptainer pull pytorch.sif docker://pytorch/pytorch:latest +``` + +During the pull process, Apptainer downloads the Docker image layers and converts them into a single SIF image: + +```text +INFO: Converting OCI blobs to SIF format +INFO: Starting build... +INFO: Fetching OCI image... +... +INFO: Creating SIF file... +``` + +The output shows that Apptainer is downloading the Docker image layers and converting them into a single SIF image. Once the conversion completes, the resulting SIF file can be used without Docker. +When the command completes, a new image named `pytorch.sif` will be created in the current directory. + +You can verify that the image exists: + +```bash +ls -lh pytorch.sif +``` + +The image can now be used like any other Apptainer image. + +For example: + +```bash +apptainer exec pytorch.sif python --version +``` + +or + +```bash +apptainer exec pytorch.sif python -c "import torch; print(torch.__version__)" +``` From 45c5dee911f0c753e19c330c82d81646addfb08a Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 2 Jun 2026 12:00:51 +0800 Subject: [PATCH 02/37] Rename 11_apptainer_on_torch to 11_apptainer_on_torch.mdx --- .../{11_apptainer_on_torch => 11_apptainer_on_torch.mdx} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/hpc/13_tutorial_intro_hpc/{11_apptainer_on_torch => 11_apptainer_on_torch.mdx} (100%) diff --git a/docs/hpc/13_tutorial_intro_hpc/11_apptainer_on_torch b/docs/hpc/13_tutorial_intro_hpc/11_apptainer_on_torch.mdx similarity index 100% rename from docs/hpc/13_tutorial_intro_hpc/11_apptainer_on_torch rename to docs/hpc/13_tutorial_intro_hpc/11_apptainer_on_torch.mdx From 2ae3afcd4fd7da2c2c8025924c867360cd339d85 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 2 Jun 2026 12:21:09 +0800 Subject: [PATCH 03/37] Update 11_apptainer_on_torch.mdx --- .../11_apptainer_on_torch.mdx | 106 +++++++++--------- 1 file changed, 53 insertions(+), 53 deletions(-) diff --git a/docs/hpc/13_tutorial_intro_hpc/11_apptainer_on_torch.mdx b/docs/hpc/13_tutorial_intro_hpc/11_apptainer_on_torch.mdx index 372d82e2ff..c7f84f9649 100644 --- a/docs/hpc/13_tutorial_intro_hpc/11_apptainer_on_torch.mdx +++ b/docs/hpc/13_tutorial_intro_hpc/11_apptainer_on_torch.mdx @@ -6,7 +6,7 @@ Researchers often rely on complex software stacks that include programming langu Containers provide a way to package software together with the environment it needs to run. Instead of manually installing every dependency on a system, users can run software inside a container that already includes the required libraries and tools. ## What is Apptainer? -Apptainer is a container platform that allows users to run software inside isolated environments. It allows users to run software inside isolated environments without requiring administrator privileges on the host system. +Apptainer is a container platform that allows users to run software inside isolated environments without requiring administrator privileges on the host system. Apptainer is the continuation of the Singularity project. The open-source Singularity project was renamed to Apptainer and continues to be developed under the Linux Foundation. @@ -14,57 +14,6 @@ Unlike Docker, Apptainer does not require a privileged daemon running on the sys Torch uses Apptainer as its supported container platform. Many container images distributed through Docker registries can be used directly with Apptainer, allowing researchers to take advantage of existing software environments while working on the cluster. -# Files in Apptainer Containers -Apptainer is designed to work closely with the host filesystem. In most cases, your home directory and current working directory remain accessible from within the container. - -## Accessing Your Files - -While inside a container, check your current directory: - -```bash -pwd -``` - -You can also list files in your home directory: - -```bash -ls ~ -``` - -The files and directories you see should match those available outside the container. - -Files created in these mounted directories remain available after the container exits. - -## Binding Additional Directories - -Sometimes you may need access to additional directories that are not automatically available inside the container. - -Apptainer allows additional directories to be mounted using the `-B` option: - -```bash -apptainer shell \ - -B /scratch:/scratch \ - /share/apps/images/ubuntu-24.04.3.sif -``` - -This makes `/scratch` available inside the container. - -You can also mount a directory at a different location: - -```bash -apptainer shell \ - -B /scratch:/data \ - /share/apps/images/ubuntu-24.04.3.sif -``` - -In this example, files stored in `/scratch` on the host system are accessible through `/data` inside the container. - -## Why This Matters - -Container images are typically read-only. Research data, scripts, notebooks, and output files usually remain outside the container. - -By making host directories available inside the container, Apptainer allows applications to access data stored on Torch while maintaining a reproducible software environment. - # Using Apptainer to Run Commands on Torch :::warning @@ -177,7 +126,58 @@ exit This returns you to your normal shell on Torch. -## Using Docker images with Apptainer +# Files in Apptainer Containers +Apptainer is designed to work closely with the host filesystem. In most cases, your home directory and current working directory remain accessible from within the container. + +## Accessing Your Files + +While inside a container, check your current directory: + +```bash +pwd +``` + +You can also list files in your home directory: + +```bash +ls ~ +``` + +The files and directories you see should match those available outside the container. + +Files created in these mounted directories remain available after the container exits. + +## Binding Additional Directories + +Sometimes you may need access to additional directories that are not automatically available inside the container. + +Apptainer allows additional directories to be mounted using the `-B` option: + +```bash +apptainer shell \ + -B /scratch:/scratch \ + /share/apps/images/ubuntu-24.04.3.sif +``` + +This makes `/scratch` available inside the container. + +You can also mount a directory at a different location: + +```bash +apptainer shell \ + -B /scratch:/data \ + /share/apps/images/ubuntu-24.04.3.sif +``` + +In this example, files stored in `/scratch` on the host system are accessible through `/data` inside the container. + +## Why This Matters + +Container images are typically read-only. Research data, scripts, notebooks, and output files usually remain outside the container. + +By making host directories available inside the container, Apptainer allows applications to access data stored on Torch while maintaining a reproducible software environment. + +# Using Docker images with Apptainer So far, we have used container images that are already available on Torch under `/share/apps/images`. From 61544002f626e7bb64896261ee6fa26e315644ea Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 2 Jun 2026 12:25:33 +0800 Subject: [PATCH 04/37] Correct capitalization in Docker images section --- docs/hpc/13_tutorial_intro_hpc/11_apptainer_on_torch.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/13_tutorial_intro_hpc/11_apptainer_on_torch.mdx b/docs/hpc/13_tutorial_intro_hpc/11_apptainer_on_torch.mdx index c7f84f9649..9c2ba97b66 100644 --- a/docs/hpc/13_tutorial_intro_hpc/11_apptainer_on_torch.mdx +++ b/docs/hpc/13_tutorial_intro_hpc/11_apptainer_on_torch.mdx @@ -177,7 +177,7 @@ Container images are typically read-only. Research data, scripts, notebooks, and By making host directories available inside the container, Apptainer allows applications to access data stored on Torch while maintaining a reproducible software environment. -# Using Docker images with Apptainer +# Using Docker Images with Apptainer So far, we have used container images that are already available on Torch under `/share/apps/images`. From 6cd11bf4217a7a56310301d798cce0ba421ff419 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 2 Jun 2026 12:38:59 +0800 Subject: [PATCH 05/37] Update 11_apptainer_on_torch.mdx From 1760d5546ae55cb42a53457191a9fa96d7d1d9c0 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Thu, 4 Jun 2026 20:31:59 +0800 Subject: [PATCH 06/37] Update generate.ts --- .../storage-finder-data-generator/generate.ts | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/scripts/storage-finder-data-generator/generate.ts b/scripts/storage-finder-data-generator/generate.ts index 81f5965c6d..c3200cbe50 100644 --- a/scripts/storage-finder-data-generator/generate.ts +++ b/scripts/storage-finder-data-generator/generate.ts @@ -307,6 +307,28 @@ async function writeJson( await writeFile(path, content + "\n", "utf8"); } +function stripHtml(value: string | undefined): string { + return (value ?? "").replaceAll(/<[^>]+>/g, " ").replaceAll(/\s+/g, " ").trim(); +} + +function buildSearchMdx(services: ServiceRecord[]): string { + return [ + "# Storage Finder Data", + "", + "This page is generated from the Storage Finder data so the search index can crawl service details.", + "", + ...services.map((service) => + [ + `## ${service.title}`, + "", + ...Object.values(service.field_data).map((field) => + [`### ${field.label}`, "", stripHtml(field.value), ""].join("\n"), + ), + ].join("\n"), + ), + ].join("\n"); +} + function slugify(value: string): string { const normalized = value.toLowerCase().replaceAll(/[^a-z0-9]+/g, "-"); const trimmed = normalized.replaceAll(/^-+|-+$/g, ""); @@ -336,6 +358,11 @@ async function main(): Promise { const facetOutputPath = `${outputDirectory}/${FACET_TREE_FILENAME}`; await writeJson(serviceOutputPath, services, options.pretty); await writeJson(facetOutputPath, facetTree, options.pretty); + await writeFile( + "src/pages/storage-finder-data.mdx", + buildSearchMdx(services), + "utf8", +); logger.log(`Wrote ${services.length} services to ${serviceOutputPath}`); logger.log(`Wrote ${facetTree.length} facets to ${facetOutputPath}`); } From db7ef9ea968badb67638a37e9935f4376e68af21 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Thu, 4 Jun 2026 21:37:03 +0800 Subject: [PATCH 07/37] Remove unused functions from generate.ts Removed unused functions for HTML stripping and MDX building. --- .../storage-finder-data-generator/generate.ts | 27 ------------------- 1 file changed, 27 deletions(-) diff --git a/scripts/storage-finder-data-generator/generate.ts b/scripts/storage-finder-data-generator/generate.ts index c3200cbe50..81f5965c6d 100644 --- a/scripts/storage-finder-data-generator/generate.ts +++ b/scripts/storage-finder-data-generator/generate.ts @@ -307,28 +307,6 @@ async function writeJson( await writeFile(path, content + "\n", "utf8"); } -function stripHtml(value: string | undefined): string { - return (value ?? "").replaceAll(/<[^>]+>/g, " ").replaceAll(/\s+/g, " ").trim(); -} - -function buildSearchMdx(services: ServiceRecord[]): string { - return [ - "# Storage Finder Data", - "", - "This page is generated from the Storage Finder data so the search index can crawl service details.", - "", - ...services.map((service) => - [ - `## ${service.title}`, - "", - ...Object.values(service.field_data).map((field) => - [`### ${field.label}`, "", stripHtml(field.value), ""].join("\n"), - ), - ].join("\n"), - ), - ].join("\n"); -} - function slugify(value: string): string { const normalized = value.toLowerCase().replaceAll(/[^a-z0-9]+/g, "-"); const trimmed = normalized.replaceAll(/^-+|-+$/g, ""); @@ -358,11 +336,6 @@ async function main(): Promise { const facetOutputPath = `${outputDirectory}/${FACET_TREE_FILENAME}`; await writeJson(serviceOutputPath, services, options.pretty); await writeJson(facetOutputPath, facetTree, options.pretty); - await writeFile( - "src/pages/storage-finder-data.mdx", - buildSearchMdx(services), - "utf8", -); logger.log(`Wrote ${services.length} services to ${serviceOutputPath}`); logger.log(`Wrote ${facetTree.length} facets to ${facetOutputPath}`); } From dc7c3b5d5794af5a70c1dc82c7925bcf184f489b Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 9 Jun 2026 22:51:25 +0800 Subject: [PATCH 08/37] Rename docs/hpc/13_tutorial_intro_hpc/11_apptainer_on_torch.mdx to docs/hpc/15_tutorial_apptainer.mdx --- .../11_apptainer_on_torch.mdx => 15_tutorial_apptainer.mdx} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/hpc/{13_tutorial_intro_hpc/11_apptainer_on_torch.mdx => 15_tutorial_apptainer.mdx} (100%) diff --git a/docs/hpc/13_tutorial_intro_hpc/11_apptainer_on_torch.mdx b/docs/hpc/15_tutorial_apptainer.mdx similarity index 100% rename from docs/hpc/13_tutorial_intro_hpc/11_apptainer_on_torch.mdx rename to docs/hpc/15_tutorial_apptainer.mdx From d344e658f881e9c94e347e506573adb3efaf6368 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 9 Jun 2026 22:53:24 +0800 Subject: [PATCH 09/37] Rename docs/hpc/15_tutorial_apptainer.mdx to docs/hpc/15_tutorial_apptainer/01_intro_apptainer.mdx --- .../01_intro_apptainer.mdx} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/hpc/{15_tutorial_apptainer.mdx => 15_tutorial_apptainer/01_intro_apptainer.mdx} (100%) diff --git a/docs/hpc/15_tutorial_apptainer.mdx b/docs/hpc/15_tutorial_apptainer/01_intro_apptainer.mdx similarity index 100% rename from docs/hpc/15_tutorial_apptainer.mdx rename to docs/hpc/15_tutorial_apptainer/01_intro_apptainer.mdx From 694a2dfe1c129e10b2180f6781ba37fbb403c07e Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 9 Jun 2026 23:02:37 +0800 Subject: [PATCH 10/37] Create 02_Running Containers --- .../02_Running Containers | 111 ++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100644 docs/hpc/15_tutorial_apptainer/02_Running Containers diff --git a/docs/hpc/15_tutorial_apptainer/02_Running Containers b/docs/hpc/15_tutorial_apptainer/02_Running Containers new file mode 100644 index 0000000000..2fae89e098 --- /dev/null +++ b/docs/hpc/15_tutorial_apptainer/02_Running Containers @@ -0,0 +1,111 @@ +# Using Apptainer to Run Commands on Torch + +:::warning + +Container workloads should be run on compute nodes rather than login nodes. + +While simple commands may work on a login node, pulling images, launching software, installing packages, or building environments can consume significant CPU, memory, and storage resources. These activities should be performed within an interactive Slurm allocation or a batch job. +::: + +Torch provides many prebuilt container images under: + +```bash +ls /share/apps/images/ +``` + +:::note + +Torch provides container images with both `.sif` and `.sqf` extensions. + +`.sif` is the standard Apptainer image format. Some Torch-provided application images use `.sqf` and may be intended to be launched through wrapper scripts such as `run-anaconda3-2024.10-1.bash`. + +When available, use the wrapper script documented for that application. For general Apptainer examples in this tutorial, we use `.sif` images because they work directly with `apptainer exec`, `apptainer run`, and `apptainer shell`. + +::: + +For this tutorial, we will use the Ubuntu 24.04 image that is already available on the cluster. + +## Running Your First Container + +Apptainer images can define a default action that runs when the container starts. + +To launch a container and execute its default action, use `apptainer run`: + +```bash +apptainer run /share/apps/images/ubuntu-24.04.3.sif +``` + +Depending on how the image was built, this command may produce output, launch an application, or simply start and exit. + +The important point is that with `apptainer run`, Apptainer executes the default action defined by the image creator. + +Sometimes, however, we want to run a specific command instead of the image's default action. In those cases, we use `apptainer exec`. + +## Running Specific Commands Within a Container + +Unlike `apptainer run`, which executes the image's default action, `apptainer exec` allows us to specify exactly what command should run inside the container. + +For example: + +```bash +apptainer exec /share/apps/images/ubuntu-24.04.3.sif /bin/echo "Hello World!" +``` + +Output: + +```text +Hello World! +``` + +## The Difference Between `apptainer run` and `apptainer exec` + +Both `apptainer run` and `apptainer exec` start a container, but they serve different purposes. + +`apptainer run` executes the default action defined by the image creator. Depending on how the image was built, this may launch an application, run a script, or perform another predefined task. + +`apptainer exec` allows you to specify exactly which command should run inside the container. Rather than relying on the image's default behavior, you provide the command directly. + +In practice, `apptainer exec` is often used when working on HPC systems because it provides more control over what is executed inside the container environment. + +## Opening an Interactive Shell Within a Container + +Sometimes it is useful to explore a container interactively. Apptainer provides the `apptainer shell` command for this purpose. + +Launch a shell inside the Ubuntu container: + +```bash +apptainer shell /share/apps/images/ubuntu-24.04.3.sif +``` + +You should see a prompt similar to: + +```text +Singularity> +``` + +You can now run commands inside the container: + +```bash +whoami +pwd +cat /etc/os-release +``` + +Example output: + +```text +PRETTY_NAME="Ubuntu 24.04.3 LTS" +NAME="Ubuntu" +VERSION_ID="24.04" +... +``` + +Notice that the prompt changes to indicate that you are working inside the container environment. + +When you are finished, leave the container with: + +```bash +exit +``` + +This returns you to your normal shell on Torch. From 7932456f2fe332859ff2c918000622ab150c15a6 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 9 Jun 2026 23:03:14 +0800 Subject: [PATCH 11/37] Rename 02_Running Containers to 02_running_containers.mdx --- .../{02_Running Containers => 02_running_containers.mdx} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/hpc/15_tutorial_apptainer/{02_Running Containers => 02_running_containers.mdx} (100%) diff --git a/docs/hpc/15_tutorial_apptainer/02_Running Containers b/docs/hpc/15_tutorial_apptainer/02_running_containers.mdx similarity index 100% rename from docs/hpc/15_tutorial_apptainer/02_Running Containers rename to docs/hpc/15_tutorial_apptainer/02_running_containers.mdx From c30858057576b2dc667766922e7a35dd288c9e31 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 9 Jun 2026 23:05:43 +0800 Subject: [PATCH 12/37] Create 03_apptainer_files.mdx --- .../03_apptainer_files.mdx | 50 +++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 docs/hpc/15_tutorial_apptainer/03_apptainer_files.mdx diff --git a/docs/hpc/15_tutorial_apptainer/03_apptainer_files.mdx b/docs/hpc/15_tutorial_apptainer/03_apptainer_files.mdx new file mode 100644 index 0000000000..01c45fe789 --- /dev/null +++ b/docs/hpc/15_tutorial_apptainer/03_apptainer_files.mdx @@ -0,0 +1,50 @@ +# Files in Apptainer Containers +Apptainer is designed to work closely with the host filesystem. In most cases, your home directory and current working directory remain accessible from within the container. + +## Accessing Your Files + +While inside a container, check your current directory: + +```bash +pwd +``` + +You can also list files in your home directory: + +```bash +ls ~ +``` + +The files and directories you see should match those available outside the container. + +Files created in these mounted directories remain available after the container exits. + +## Binding Additional Directories + +Sometimes you may need access to additional directories that are not automatically available inside the container. + +Apptainer allows additional directories to be mounted using the `-B` option: + +```bash +apptainer shell \ + -B /scratch:/scratch \ + /share/apps/images/ubuntu-24.04.3.sif +``` + +This makes `/scratch` available inside the container. + +You can also mount a directory at a different location: + +```bash +apptainer shell \ + -B /scratch:/data \ + /share/apps/images/ubuntu-24.04.3.sif +``` + +In this example, files stored in `/scratch` on the host system are accessible through `/data` inside the container. + +## Why This Matters + +Container images are typically read-only. Research data, scripts, notebooks, and output files usually remain outside the container. + +By making host directories available inside the container, Apptainer allows applications to access data stored on Torch while maintaining a reproducible software environment. From 7c113e3750a52e108ac6d68777da7fe425c936de Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 9 Jun 2026 23:07:00 +0800 Subject: [PATCH 13/37] Create 04_docker_images.mdx --- .../04_docker_images.mdx | 44 +++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100644 docs/hpc/15_tutorial_apptainer/04_docker_images.mdx diff --git a/docs/hpc/15_tutorial_apptainer/04_docker_images.mdx b/docs/hpc/15_tutorial_apptainer/04_docker_images.mdx new file mode 100644 index 0000000000..f995042f8b --- /dev/null +++ b/docs/hpc/15_tutorial_apptainer/04_docker_images.mdx @@ -0,0 +1,44 @@ +# Using Docker Images with Apptainer + +So far, we have used container images that are already available on Torch under `/share/apps/images`. + +In practice, you may also want to run software that is not provided by the cluster. Apptainer can pull images directly from Docker registries and convert them into the Apptainer SIF format. + +For example, we can pull an official PyTorch image from Docker Hub: + +```bash +apptainer pull pytorch.sif docker://pytorch/pytorch:latest +``` + +During the pull process, Apptainer downloads the Docker image layers and converts them into a single SIF image: + +```text +INFO: Converting OCI blobs to SIF format +INFO: Starting build... +INFO: Fetching OCI image... +... +INFO: Creating SIF file... +``` + +The output shows that Apptainer is downloading the Docker image layers and converting them into a single SIF image. Once the conversion completes, the resulting SIF file can be used without Docker. +When the command completes, a new image named `pytorch.sif` will be created in the current directory. + +You can verify that the image exists: + +```bash +ls -lh pytorch.sif +``` + +The image can now be used like any other Apptainer image. + +For example: + +```bash +apptainer exec pytorch.sif python --version +``` + +or + +```bash +apptainer exec pytorch.sif python -c "import torch; print(torch.__version__)" +``` From c041d0a2d26daf965d5b51387d456d89ee56e726 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 10 Jun 2026 08:57:51 +0800 Subject: [PATCH 14/37] Update 01_intro_apptainer.mdx --- .../01_intro_apptainer.mdx | 203 +----------------- 1 file changed, 6 insertions(+), 197 deletions(-) diff --git a/docs/hpc/15_tutorial_apptainer/01_intro_apptainer.mdx b/docs/hpc/15_tutorial_apptainer/01_intro_apptainer.mdx index 9c2ba97b66..8b9d5d7395 100644 --- a/docs/hpc/15_tutorial_apptainer/01_intro_apptainer.mdx +++ b/docs/hpc/15_tutorial_apptainer/01_intro_apptainer.mdx @@ -14,210 +14,19 @@ Unlike Docker, Apptainer does not require a privileged daemon running on the sys Torch uses Apptainer as its supported container platform. Many container images distributed through Docker registries can be used directly with Apptainer, allowing researchers to take advantage of existing software environments while working on the cluster. -# Using Apptainer to Run Commands on Torch - -:::warning - -Container workloads should be run on compute nodes rather than login nodes. - -While simple commands may work on a login node, pulling images, launching software, installing packages, or building environments can consume significant CPU, memory, and storage resources. These activities should be performed within an interactive Slurm allocation or a batch job. -::: - -Torch provides many prebuilt container images under: - -```bash -ls /share/apps/images/ -``` - -:::note +## What are .sif and .sqf files Torch provides container images with both `.sif` and `.sqf` extensions. -`.sif` is the standard Apptainer image format. Some Torch-provided application images use `.sqf` and may be intended to be launched through wrapper scripts such as `run-anaconda3-2024.10-1.bash`. - -When available, use the wrapper script documented for that application. For general Apptainer examples in this tutorial, we use `.sif` images because they work directly with `apptainer exec`, `apptainer run`, and `apptainer shell`. - -::: - -For this tutorial, we will use the Ubuntu 24.04 image that is already available on the cluster. - -## Running Your First Container - -Apptainer images can define a default action that runs when the container starts. - -To launch a container and execute its default action, use `apptainer run`: - -```bash -apptainer run /share/apps/images/ubuntu-24.04.3.sif -``` - -Depending on how the image was built, this command may produce output, launch an application, or simply start and exit. - -The important point is that with `apptainer run`, Apptainer executes the default action defined by the image creator. - -Sometimes, however, we want to run a specific command instead of the image's default action. In those cases, we use `apptainer exec`. - -## Running Specific Commands Within a Container - -Unlike `apptainer run`, which executes the image's default action, `apptainer exec` allows us to specify exactly what command should run inside the container. - -For example: - -```bash -apptainer exec /share/apps/images/ubuntu-24.04.3.sif /bin/echo "Hello World!" -``` - -Output: - -```text -Hello World! -``` - -## The Difference Between `apptainer run` and `apptainer exec` - -Both `apptainer run` and `apptainer exec` start a container, but they serve different purposes. - -`apptainer run` executes the default action defined by the image creator. Depending on how the image was built, this may launch an application, run a script, or perform another predefined task. - -`apptainer exec` allows you to specify exactly which command should run inside the container. Rather than relying on the image's default behavior, you provide the command directly. - -In practice, `apptainer exec` is often used when working on HPC systems because it provides more control over what is executed inside the container environment. - -## Opening an Interactive Shell Within a Container - -Sometimes it is useful to explore a container interactively. Apptainer provides the `apptainer shell` command for this purpose. - -Launch a shell inside the Ubuntu container: - -```bash -apptainer shell /share/apps/images/ubuntu-24.04.3.sif -``` - -You should see a prompt similar to: - -```text -Singularity> -``` - -You can now run commands inside the container: - -```bash -whoami -pwd -cat /etc/os-release -``` - -Example output: - -```text -PRETTY_NAME="Ubuntu 24.04.3 LTS" -NAME="Ubuntu" -VERSION_ID="24.04" -... -``` - -Notice that the prompt changes to indicate that you are working inside the container environment. - -When you are finished, leave the container with: - -```bash -exit -``` - -This returns you to your normal shell on Torch. - -# Files in Apptainer Containers -Apptainer is designed to work closely with the host filesystem. In most cases, your home directory and current working directory remain accessible from within the container. - -## Accessing Your Files - -While inside a container, check your current directory: - -```bash -pwd -``` - -You can also list files in your home directory: - -```bash -ls ~ -``` - -The files and directories you see should match those available outside the container. - -Files created in these mounted directories remain available after the container exits. - -## Binding Additional Directories - -Sometimes you may need access to additional directories that are not automatically available inside the container. - -Apptainer allows additional directories to be mounted using the `-B` option: - -```bash -apptainer shell \ - -B /scratch:/scratch \ - /share/apps/images/ubuntu-24.04.3.sif -``` - -This makes `/scratch` available inside the container. - -You can also mount a directory at a different location: - -```bash -apptainer shell \ - -B /scratch:/data \ - /share/apps/images/ubuntu-24.04.3.sif -``` - -In this example, files stored in `/scratch` on the host system are accessible through `/data` inside the container. - -## Why This Matters - -Container images are typically read-only. Research data, scripts, notebooks, and output files usually remain outside the container. - -By making host directories available inside the container, Apptainer allows applications to access data stored on Torch while maintaining a reproducible software environment. - -# Using Docker Images with Apptainer - -So far, we have used container images that are already available on Torch under `/share/apps/images`. - -In practice, you may also want to run software that is not provided by the cluster. Apptainer can pull images directly from Docker registries and convert them into the Apptainer SIF format. - -For example, we can pull an official PyTorch image from Docker Hub: - -```bash -apptainer pull pytorch.sif docker://pytorch/pytorch:latest -``` - -During the pull process, Apptainer downloads the Docker image layers and converts them into a single SIF image: - -```text -INFO: Converting OCI blobs to SIF format -INFO: Starting build... -INFO: Fetching OCI image... -... -INFO: Creating SIF file... -``` - -The output shows that Apptainer is downloading the Docker image layers and converting them into a single SIF image. Once the conversion completes, the resulting SIF file can be used without Docker. -When the command completes, a new image named `pytorch.sif` will be created in the current directory. +`.sif` is the standard Apptainer image format and can be used directly with commands such as `apptainer exec`, `apptainer run`, and `apptainer shell`. -You can verify that the image exists: +Some Torch-provided application images use `.sqf` files and are typically accessed through wrapper scripts such as `run-anaconda3-2024.10-1.bash`. These scripts handle the details of launching the application environment and are often the recommended interface for users. -```bash -ls -lh pytorch.sif -``` +When available, use the wrapper script documented for that application. For general Apptainer examples in this tutorial, we use `.sif` images because they can be launched directly with standard Apptainer commands. -The image can now be used like any other Apptainer image. +## Apptainer Cache -For example: +When Apptainer pulls an image from a remote registry, it stores downloaded image layers in a local cache. If the same image is requested again and has not changed, Apptainer can reuse the cached data instead of downloading it again. This reduces network transfers and can significantly speed up repeated image pulls. -```bash -apptainer exec pytorch.sif python --version -``` -or -```bash -apptainer exec pytorch.sif python -c "import torch; print(torch.__version__)" -``` From 25b75ec902c03eb8c99f00438cfe03f746b69368 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 10 Jun 2026 11:19:13 +0800 Subject: [PATCH 15/37] Update 01_intro_apptainer.mdx --- docs/hpc/15_tutorial_apptainer/01_intro_apptainer.mdx | 7 ------- 1 file changed, 7 deletions(-) diff --git a/docs/hpc/15_tutorial_apptainer/01_intro_apptainer.mdx b/docs/hpc/15_tutorial_apptainer/01_intro_apptainer.mdx index 8b9d5d7395..2585918311 100644 --- a/docs/hpc/15_tutorial_apptainer/01_intro_apptainer.mdx +++ b/docs/hpc/15_tutorial_apptainer/01_intro_apptainer.mdx @@ -23,10 +23,3 @@ Torch provides container images with both `.sif` and `.sqf` extensions. Some Torch-provided application images use `.sqf` files and are typically accessed through wrapper scripts such as `run-anaconda3-2024.10-1.bash`. These scripts handle the details of launching the application environment and are often the recommended interface for users. When available, use the wrapper script documented for that application. For general Apptainer examples in this tutorial, we use `.sif` images because they can be launched directly with standard Apptainer commands. - -## Apptainer Cache - -When Apptainer pulls an image from a remote registry, it stores downloaded image layers in a local cache. If the same image is requested again and has not changed, Apptainer can reuse the cached data instead of downloading it again. This reduces network transfers and can significantly speed up repeated image pulls. - - - From cef5efffb4c54cc38af8cee788b88da459dcd2f4 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 10 Jun 2026 14:33:37 +0800 Subject: [PATCH 16/37] Introduce Apptainer's image cache explanation Added section on Apptainer's image cache and its benefits. --- docs/hpc/15_tutorial_apptainer/01_intro_apptainer.mdx | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/hpc/15_tutorial_apptainer/01_intro_apptainer.mdx b/docs/hpc/15_tutorial_apptainer/01_intro_apptainer.mdx index 2585918311..ff66116f22 100644 --- a/docs/hpc/15_tutorial_apptainer/01_intro_apptainer.mdx +++ b/docs/hpc/15_tutorial_apptainer/01_intro_apptainer.mdx @@ -5,6 +5,7 @@ Researchers often rely on complex software stacks that include programming langu Containers provide a way to package software together with the environment it needs to run. Instead of manually installing every dependency on a system, users can run software inside a container that already includes the required libraries and tools. + ## What is Apptainer? Apptainer is a container platform that allows users to run software inside isolated environments without requiring administrator privileges on the host system. @@ -23,3 +24,9 @@ Torch provides container images with both `.sif` and `.sqf` extensions. Some Torch-provided application images use `.sqf` files and are typically accessed through wrapper scripts such as `run-anaconda3-2024.10-1.bash`. These scripts handle the details of launching the application environment and are often the recommended interface for users. When available, use the wrapper script documented for that application. For general Apptainer examples in this tutorial, we use `.sif` images because they can be launched directly with standard Apptainer commands. + +## Apptainer’s image cache + +While Apptainer doesn’t have a local image repository in the same way as Docker, it does cache downloaded image files. As we saw in the previous episode, images are simply `.sif` files stored on your local disk. + +If you delete a local `.sif` image that you have pulled from a remote image repository and then pull it again, if the image is unchanged from the version you previously pulled, you will be given a copy of the image file from your local cache rather than the image being downloaded again from the remote source. This removes unnecessary network transfers and is particularly useful for large images which may take some time to transfer over the network. From 01f637ac9eab1806f496874b8b2fb53b1df45bbb Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 16 Jun 2026 14:13:21 +0800 Subject: [PATCH 17/37] Update 03_apptainer_files.mdx --- .../15_tutorial_apptainer/03_apptainer_files.mdx | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/hpc/15_tutorial_apptainer/03_apptainer_files.mdx b/docs/hpc/15_tutorial_apptainer/03_apptainer_files.mdx index 01c45fe789..16d6fa6ced 100644 --- a/docs/hpc/15_tutorial_apptainer/03_apptainer_files.mdx +++ b/docs/hpc/15_tutorial_apptainer/03_apptainer_files.mdx @@ -21,27 +21,27 @@ Files created in these mounted directories remain available after the container ## Binding Additional Directories -Sometimes you may need access to additional directories that are not automatically available inside the container. +Sometimes you may need access to directories through a different path inside the container. Apptainer allows additional directories to be mounted using the `-B` option: ```bash apptainer shell \ - -B /scratch:/scratch \ + -B /scratch/(NetID):/data \ /share/apps/images/ubuntu-24.04.3.sif ``` -This makes `/scratch` available inside the container. +In this example, your scratch directory on the host system is made available as `/data` inside the container. -You can also mount a directory at a different location: +After entering the container, you can verify the mount: ```bash -apptainer shell \ - -B /scratch:/data \ - /share/apps/images/ubuntu-24.04.3.sif +ls /data ``` -In this example, files stored in `/scratch` on the host system are accessible through `/data` inside the container. +Any files stored in `/scratch ` on the host system will be accessible through `/data` inside the container. + +The source and destination paths do not need to be the same. This can be useful when an application expects data to be located in a specific directory within the container. ## Why This Matters From 7a5f2d730f69bae8a8490ccac67e865a5b097c35 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 16 Jun 2026 20:03:51 +0800 Subject: [PATCH 18/37] Update 04_docker_images.mdx --- docs/hpc/15_tutorial_apptainer/04_docker_images.mdx | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/hpc/15_tutorial_apptainer/04_docker_images.mdx b/docs/hpc/15_tutorial_apptainer/04_docker_images.mdx index f995042f8b..fb1474c1b4 100644 --- a/docs/hpc/15_tutorial_apptainer/04_docker_images.mdx +++ b/docs/hpc/15_tutorial_apptainer/04_docker_images.mdx @@ -4,6 +4,12 @@ So far, we have used container images that are already available on Torch under In practice, you may also want to run software that is not provided by the cluster. Apptainer can pull images directly from Docker registries and convert them into the Apptainer SIF format. +:::warning + +Pulling container images can require significant network bandwidth and disk space. On Torch, image downloads should be performed on compute nodes rather than login nodes. + +::: + For example, we can pull an official PyTorch image from Docker Hub: ```bash From 02bb9dcd615a013c75563616ef46e542ea738395 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 16 Jun 2026 22:29:44 +0800 Subject: [PATCH 19/37] Create 05_customize_container_environments.mdx --- .../05_customize_container_environments.mdx | 67 +++++++++++++++++++ 1 file changed, 67 insertions(+) create mode 100644 docs/hpc/15_tutorial_apptainer/05_customize_container_environments.mdx diff --git a/docs/hpc/15_tutorial_apptainer/05_customize_container_environments.mdx b/docs/hpc/15_tutorial_apptainer/05_customize_container_environments.mdx new file mode 100644 index 0000000000..da132b7575 --- /dev/null +++ b/docs/hpc/15_tutorial_apptainer/05_customize_container_environments.mdx @@ -0,0 +1,67 @@ +## Why Customize a Container Environment? +Prebuilt container images provide a convenient starting point, but they may not contain all of the software required for a particular project. + +Most Apptainer images are distributed as `.sif` files, which are typically read-only. If you need to install additional software or make persistent changes to a container, Apptainer provides overlays, which add a writable layer on top of the original image while leaving the `.sif` file unchanged. + +## Example: Creating a Custom Directory with an Overlay + + +Pull a Python container image from Docker Hub: + +```bash +apptainer pull python.sif docker://python:3.10 +``` + +Create a writable overlay: + +```bash +apptainer overlay create --size 1024 python_overlay.ext3 +``` + +Launch the container with the overlay attached: + +```bash +apptainer shell \ + --overlay python_overlay.ext3:rw \ + python.sif +``` + +Verify that numpy is not installed in the base image: + +```bash +python -c "import numpy" +``` + +You should see: + +```text +ModuleNotFoundError: No module named 'numpy' +``` + +Install numpy: + +```bash +pip install numpy +``` + +Verify that the packages are available: + +```bash +python -c "import numpy; print(numpy.__version__)" +``` + +Exit the container: + +```bash +exit +``` + +The next time you want to use the customized environment, launch the container using the same overlay: + +```bash +apptainer shell \ + --overlay python_overlay.ext3:rw \ + python.sif +``` + +This demonstrates that software installed within the overlay persists across sessions, while the original `.sif` image remains unchanged. From 2193024bbe5b7f9b4e67112f961d3888bfcef91e Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 16 Jun 2026 22:35:48 +0800 Subject: [PATCH 20/37] Update 02_running_containers.mdx --- docs/hpc/15_tutorial_apptainer/02_running_containers.mdx | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/hpc/15_tutorial_apptainer/02_running_containers.mdx b/docs/hpc/15_tutorial_apptainer/02_running_containers.mdx index 2fae89e098..55aa1a10ea 100644 --- a/docs/hpc/15_tutorial_apptainer/02_running_containers.mdx +++ b/docs/hpc/15_tutorial_apptainer/02_running_containers.mdx @@ -39,11 +39,11 @@ Depending on how the image was built, this command may produce output, launch an The important point is that with `apptainer run`, Apptainer executes the default action defined by the image creator. -Sometimes, however, we want to run a specific command instead of the image's default action. In those cases, we use `apptainer exec`. +In cases we want to run a specific command, use `apptainer exec`. ## Running Specific Commands Within a Container -Unlike `apptainer run`, which executes the image's default action, `apptainer exec` allows us to specify exactly what command should run inside the container. +apptainer exec` allows us to specify exactly what command should run inside the container. For example: @@ -65,11 +65,9 @@ Both `apptainer run` and `apptainer exec` start a container, but they serve diff `apptainer exec` allows you to specify exactly which command should run inside the container. Rather than relying on the image's default behavior, you provide the command directly. -In practice, `apptainer exec` is often used when working on HPC systems because it provides more control over what is executed inside the container environment. - ## Opening an Interactive Shell Within a Container -Sometimes it is useful to explore a container interactively. Apptainer provides the `apptainer shell` command for this purpose. +It is useful to explore a container interactively. Apptainer provides the `apptainer shell` command for this purpose. Launch a shell inside the Ubuntu container: From b0ee4a5a1b020c973d721893a2d77ec406eb90b5 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 17 Jun 2026 10:58:26 +0800 Subject: [PATCH 21/37] Update 04_docker_images.mdx --- docs/hpc/15_tutorial_apptainer/04_docker_images.mdx | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/hpc/15_tutorial_apptainer/04_docker_images.mdx b/docs/hpc/15_tutorial_apptainer/04_docker_images.mdx index fb1474c1b4..e768478ab0 100644 --- a/docs/hpc/15_tutorial_apptainer/04_docker_images.mdx +++ b/docs/hpc/15_tutorial_apptainer/04_docker_images.mdx @@ -6,9 +6,7 @@ In practice, you may also want to run software that is not provided by the clust :::warning -Pulling container images can require significant network bandwidth and disk space. On Torch, image downloads should be performed on compute nodes rather than login nodes. - -::: +Pulling container images can require significant network bandwidth and disk space. On Torch, image downloads should be performed on compute nodes rather than login nodes.::: For example, we can pull an official PyTorch image from Docker Hub: From 8cc8de995d945e0fbe134aaec054acd7dbdccd55 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Thu, 18 Jun 2026 23:28:17 +0800 Subject: [PATCH 22/37] Update 05_customize_container_environments.mdx --- .../05_customize_container_environments.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/15_tutorial_apptainer/05_customize_container_environments.mdx b/docs/hpc/15_tutorial_apptainer/05_customize_container_environments.mdx index da132b7575..84d2e24eb3 100644 --- a/docs/hpc/15_tutorial_apptainer/05_customize_container_environments.mdx +++ b/docs/hpc/15_tutorial_apptainer/05_customize_container_environments.mdx @@ -1,4 +1,4 @@ -## Why Customize a Container Environment? +# Why Customize a Container Environment? Prebuilt container images provide a convenient starting point, but they may not contain all of the software required for a particular project. Most Apptainer images are distributed as `.sif` files, which are typically read-only. If you need to install additional software or make persistent changes to a container, Apptainer provides overlays, which add a writable layer on top of the original image while leaving the `.sif` file unchanged. From 986352fa0d57442465863897e06f5928b90a2ba1 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Thu, 18 Jun 2026 23:35:17 +0800 Subject: [PATCH 23/37] Add category label for Apptainer tutorial --- docs/hpc/15_tutorial_apptainer/_category_.json | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 docs/hpc/15_tutorial_apptainer/_category_.json diff --git a/docs/hpc/15_tutorial_apptainer/_category_.json b/docs/hpc/15_tutorial_apptainer/_category_.json new file mode 100644 index 0000000000..df501c11cf --- /dev/null +++ b/docs/hpc/15_tutorial_apptainer/_category_.json @@ -0,0 +1,3 @@ +{ + "label": "Tutorial: Apptainer on Torch" +} From baecba4c6b0e7eda4a9a118e581fe676de7bb618 Mon Sep 17 00:00:00 2001 From: Sajid Ali Date: Wed, 24 Jun 2026 12:08:19 -0400 Subject: [PATCH 24/37] reorder section number for tutorial --- .../01_intro_apptainer.mdx | 0 .../02_running_containers.mdx | 0 .../03_apptainer_files.mdx | 0 .../04_docker_images.mdx | 0 .../05_customize_container_environments.mdx | 0 .../_category_.json | 0 docs/hpc/{14_support => 15_support}/01_support.md | 0 docs/hpc/{14_support => 15_support}/_category_.json | 0 8 files changed, 0 insertions(+), 0 deletions(-) rename docs/hpc/{15_tutorial_apptainer => 14_tutorial_apptainer}/01_intro_apptainer.mdx (100%) rename docs/hpc/{15_tutorial_apptainer => 14_tutorial_apptainer}/02_running_containers.mdx (100%) rename docs/hpc/{15_tutorial_apptainer => 14_tutorial_apptainer}/03_apptainer_files.mdx (100%) rename docs/hpc/{15_tutorial_apptainer => 14_tutorial_apptainer}/04_docker_images.mdx (100%) rename docs/hpc/{15_tutorial_apptainer => 14_tutorial_apptainer}/05_customize_container_environments.mdx (100%) rename docs/hpc/{15_tutorial_apptainer => 14_tutorial_apptainer}/_category_.json (100%) rename docs/hpc/{14_support => 15_support}/01_support.md (100%) rename docs/hpc/{14_support => 15_support}/_category_.json (100%) diff --git a/docs/hpc/15_tutorial_apptainer/01_intro_apptainer.mdx b/docs/hpc/14_tutorial_apptainer/01_intro_apptainer.mdx similarity index 100% rename from docs/hpc/15_tutorial_apptainer/01_intro_apptainer.mdx rename to docs/hpc/14_tutorial_apptainer/01_intro_apptainer.mdx diff --git a/docs/hpc/15_tutorial_apptainer/02_running_containers.mdx b/docs/hpc/14_tutorial_apptainer/02_running_containers.mdx similarity index 100% rename from docs/hpc/15_tutorial_apptainer/02_running_containers.mdx rename to docs/hpc/14_tutorial_apptainer/02_running_containers.mdx diff --git a/docs/hpc/15_tutorial_apptainer/03_apptainer_files.mdx b/docs/hpc/14_tutorial_apptainer/03_apptainer_files.mdx similarity index 100% rename from docs/hpc/15_tutorial_apptainer/03_apptainer_files.mdx rename to docs/hpc/14_tutorial_apptainer/03_apptainer_files.mdx diff --git a/docs/hpc/15_tutorial_apptainer/04_docker_images.mdx b/docs/hpc/14_tutorial_apptainer/04_docker_images.mdx similarity index 100% rename from docs/hpc/15_tutorial_apptainer/04_docker_images.mdx rename to docs/hpc/14_tutorial_apptainer/04_docker_images.mdx diff --git a/docs/hpc/15_tutorial_apptainer/05_customize_container_environments.mdx b/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx similarity index 100% rename from docs/hpc/15_tutorial_apptainer/05_customize_container_environments.mdx rename to docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx diff --git a/docs/hpc/15_tutorial_apptainer/_category_.json b/docs/hpc/14_tutorial_apptainer/_category_.json similarity index 100% rename from docs/hpc/15_tutorial_apptainer/_category_.json rename to docs/hpc/14_tutorial_apptainer/_category_.json diff --git a/docs/hpc/14_support/01_support.md b/docs/hpc/15_support/01_support.md similarity index 100% rename from docs/hpc/14_support/01_support.md rename to docs/hpc/15_support/01_support.md diff --git a/docs/hpc/14_support/_category_.json b/docs/hpc/15_support/_category_.json similarity index 100% rename from docs/hpc/14_support/_category_.json rename to docs/hpc/15_support/_category_.json From d076557ff28b16f6a3f78a5915aad71d73b47ce7 Mon Sep 17 00:00:00 2001 From: Sajid Ali Date: Wed, 24 Jun 2026 12:26:42 -0400 Subject: [PATCH 25/37] minor polish --- docs/hpc/14_tutorial_apptainer/03_apptainer_files.mdx | 7 ++++--- docs/hpc/14_tutorial_apptainer/04_docker_images.mdx | 2 +- .../05_customize_container_environments.mdx | 4 ++-- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/hpc/14_tutorial_apptainer/03_apptainer_files.mdx b/docs/hpc/14_tutorial_apptainer/03_apptainer_files.mdx index 16d6fa6ced..64a43ff000 100644 --- a/docs/hpc/14_tutorial_apptainer/03_apptainer_files.mdx +++ b/docs/hpc/14_tutorial_apptainer/03_apptainer_files.mdx @@ -27,7 +27,7 @@ Apptainer allows additional directories to be mounted using the `-B` option: ```bash apptainer shell \ - -B /scratch/(NetID):/data \ + -B /scratch/:/data \ /share/apps/images/ubuntu-24.04.3.sif ``` @@ -39,12 +39,13 @@ After entering the container, you can verify the mount: ls /data ``` -Any files stored in `/scratch ` on the host system will be accessible through `/data` inside the container. +Any files stored in `/scratch/` on the host system will be accessible through `/data` inside the container. The source and destination paths do not need to be the same. This can be useful when an application expects data to be located in a specific directory within the container. -## Why This Matters +::: info Container images are typically read-only. Research data, scripts, notebooks, and output files usually remain outside the container. By making host directories available inside the container, Apptainer allows applications to access data stored on Torch while maintaining a reproducible software environment. +::: diff --git a/docs/hpc/14_tutorial_apptainer/04_docker_images.mdx b/docs/hpc/14_tutorial_apptainer/04_docker_images.mdx index e768478ab0..b01b998429 100644 --- a/docs/hpc/14_tutorial_apptainer/04_docker_images.mdx +++ b/docs/hpc/14_tutorial_apptainer/04_docker_images.mdx @@ -16,7 +16,7 @@ apptainer pull pytorch.sif docker://pytorch/pytorch:latest During the pull process, Apptainer downloads the Docker image layers and converts them into a single SIF image: -```text +```bash INFO: Converting OCI blobs to SIF format INFO: Starting build... INFO: Fetching OCI image... diff --git a/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx b/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx index 84d2e24eb3..38e659a1d3 100644 --- a/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx +++ b/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx @@ -26,7 +26,7 @@ apptainer shell \ python.sif ``` -Verify that numpy is not installed in the base image: +Verify that `numpy` is not installed in the base image: ```bash python -c "import numpy" @@ -38,7 +38,7 @@ You should see: ModuleNotFoundError: No module named 'numpy' ``` -Install numpy: +Install `numpy`: ```bash pip install numpy From c439dec92da2106463c3cd73bb054d562ba5805f Mon Sep 17 00:00:00 2001 From: Sajid Ali Date: Wed, 24 Jun 2026 12:36:43 -0400 Subject: [PATCH 26/37] minor polish --- .gitignore | 3 +++ .memsearch/memory/2026-05-14.md | 3 --- docs/hpc/14_tutorial_apptainer/04_docker_images.mdx | 11 +++++------ 3 files changed, 8 insertions(+), 9 deletions(-) delete mode 100644 .memsearch/memory/2026-05-14.md diff --git a/.gitignore b/.gitignore index 34e23272b2..77254e2567 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,6 @@ +# agent memory +.memsearch/* + # pixi environments .pixi *.egg-info diff --git a/.memsearch/memory/2026-05-14.md b/.memsearch/memory/2026-05-14.md deleted file mode 100644 index 2c35cd48be..0000000000 --- a/.memsearch/memory/2026-05-14.md +++ /dev/null @@ -1,3 +0,0 @@ - -## Session 17:45 - diff --git a/docs/hpc/14_tutorial_apptainer/04_docker_images.mdx b/docs/hpc/14_tutorial_apptainer/04_docker_images.mdx index b01b998429..9896f07333 100644 --- a/docs/hpc/14_tutorial_apptainer/04_docker_images.mdx +++ b/docs/hpc/14_tutorial_apptainer/04_docker_images.mdx @@ -2,11 +2,11 @@ So far, we have used container images that are already available on Torch under `/share/apps/images`. -In practice, you may also want to run software that is not provided by the cluster. Apptainer can pull images directly from Docker registries and convert them into the Apptainer SIF format. +In practice, you may also want to run software that is not provided by the cluster. Apptainer can pull images directly from Docker registries and convert them into the Apptainer `SIF` format. :::warning - -Pulling container images can require significant network bandwidth and disk space. On Torch, image downloads should be performed on compute nodes rather than login nodes.::: +Pulling container images can require significant network bandwidth and disk space. On Torch, image downloads should be performed on compute nodes rather than login nodes. +::: For example, we can pull an official PyTorch image from Docker Hub: @@ -14,7 +14,7 @@ For example, we can pull an official PyTorch image from Docker Hub: apptainer pull pytorch.sif docker://pytorch/pytorch:latest ``` -During the pull process, Apptainer downloads the Docker image layers and converts them into a single SIF image: +During the pull process, Apptainer downloads the Docker image layers and converts them into a single `SIF` image: ```bash INFO: Converting OCI blobs to SIF format @@ -24,8 +24,7 @@ INFO: Fetching OCI image... INFO: Creating SIF file... ``` -The output shows that Apptainer is downloading the Docker image layers and converting them into a single SIF image. Once the conversion completes, the resulting SIF file can be used without Docker. -When the command completes, a new image named `pytorch.sif` will be created in the current directory. +The output shows that Apptainer is downloading the Docker image layers and converting them into a single `SIF` image. Once the conversion completes, the resulting `SIF` file can be used without Docker. When the command completes, a new image named `pytorch.sif` will be created in the current directory. You can verify that the image exists: From 2814506c636e58fb3c46476b565205f4053743a6 Mon Sep 17 00:00:00 2001 From: Sajid Ali Date: Wed, 24 Jun 2026 12:43:23 -0400 Subject: [PATCH 27/37] minor polish --- docs/hpc/14_tutorial_apptainer/01_intro_apptainer.mdx | 10 +++++----- .../14_tutorial_apptainer/02_running_containers.mdx | 2 +- docs/hpc/15_support/01_support.md | 4 ++-- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/hpc/14_tutorial_apptainer/01_intro_apptainer.mdx b/docs/hpc/14_tutorial_apptainer/01_intro_apptainer.mdx index ff66116f22..0cf1e8471b 100644 --- a/docs/hpc/14_tutorial_apptainer/01_intro_apptainer.mdx +++ b/docs/hpc/14_tutorial_apptainer/01_intro_apptainer.mdx @@ -1,12 +1,12 @@ # Introduction to Apptainer on Torch -## Why containers? +## Why Containers? Researchers often rely on complex software stacks that include programming languages and libraries. Installing and maintaining these dependencies can be challenging when different projects require different software versions. Containers provide a way to package software together with the environment it needs to run. Instead of manually installing every dependency on a system, users can run software inside a container that already includes the required libraries and tools. -## What is Apptainer? +## What Is Apptainer? Apptainer is a container platform that allows users to run software inside isolated environments without requiring administrator privileges on the host system. Apptainer is the continuation of the Singularity project. The open-source Singularity project was renamed to Apptainer and continues to be developed under the Linux Foundation. @@ -15,7 +15,7 @@ Unlike Docker, Apptainer does not require a privileged daemon running on the sys Torch uses Apptainer as its supported container platform. Many container images distributed through Docker registries can be used directly with Apptainer, allowing researchers to take advantage of existing software environments while working on the cluster. -## What are .sif and .sqf files +## What Are `.sif` And `.sqf` Files Torch provides container images with both `.sif` and `.sqf` extensions. @@ -25,8 +25,8 @@ Some Torch-provided application images use `.sqf` files and are typically access When available, use the wrapper script documented for that application. For general Apptainer examples in this tutorial, we use `.sif` images because they can be launched directly with standard Apptainer commands. -## Apptainer’s image cache +## Apptainer Image Cache -While Apptainer doesn’t have a local image repository in the same way as Docker, it does cache downloaded image files. As we saw in the previous episode, images are simply `.sif` files stored on your local disk. +While Apptainer doesn’t have a local image repository in the same way as Docker, it does cache downloaded image files. Images are simply `.sif` files stored on your local disk. If you delete a local `.sif` image that you have pulled from a remote image repository and then pull it again, if the image is unchanged from the version you previously pulled, you will be given a copy of the image file from your local cache rather than the image being downloaded again from the remote source. This removes unnecessary network transfers and is particularly useful for large images which may take some time to transfer over the network. diff --git a/docs/hpc/14_tutorial_apptainer/02_running_containers.mdx b/docs/hpc/14_tutorial_apptainer/02_running_containers.mdx index 55aa1a10ea..9886edae57 100644 --- a/docs/hpc/14_tutorial_apptainer/02_running_containers.mdx +++ b/docs/hpc/14_tutorial_apptainer/02_running_containers.mdx @@ -43,7 +43,7 @@ In cases we want to run a specific command, use `apptainer exec`. ## Running Specific Commands Within a Container -apptainer exec` allows us to specify exactly what command should run inside the container. +`apptainer exec` allows us to specify exactly what command should run inside the container. For example: diff --git a/docs/hpc/15_support/01_support.md b/docs/hpc/15_support/01_support.md index fe100daa67..5769d2bf96 100644 --- a/docs/hpc/15_support/01_support.md +++ b/docs/hpc/15_support/01_support.md @@ -3,8 +3,8 @@ - Some of your questions may be already answered here - [Tutorial: Introduction to Using the Shell on Torch](../12_tutorial_intro_shell_hpc/01_intro.mdx) - [Tutorial: Introduction to High-Performance Computing](../13_tutorial_intro_hpc/01_intro_hpc.mdx) - - Consider to sign up for Training and Workshop. You can find the list of available HPC coruses [can be viewed at nyu.libcal.com](https://nyu.libcal.com/calendar?cid=1564&t=d&d=0000-00-00&cal=1564&ct=6016). - - Consider signing up for ACCESS workshops. As part of the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support program, NSF provides tutorials for HPC, OpenOnDemand, etc. Here's a list of upcoming workshops: [link](https://support.access-ci.org/events). + - Consider to sign up for Training and Workshop. You can find the list of available HPC courses [can be viewed at nyu.libcal.com](https://nyu.libcal.com/calendar?cid=1564&t=d&d=0000-00-00&cal=1564&ct=6016). + - Consider signing up for ACCESS workshops. As part of the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support program, NSF provides tutorials for HPC, `OpenOnDemand`, etc. Here's a list of upcoming workshops: [link](https://support.access-ci.org/events). - [Introductory HPC Video Playlist](https://www.youtube.com/watch?v=0pP_TeKH1MI&list=PL5l6Qz3Xhfi9Jn9-iMKJisYsSW5tRzPSd&t=3s). - NYU HPC offers personalized help through **personal consultations** for simple and advanced cases: From 2de61635f50a4f874e9f4f781057b20f8781460d Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Sun, 28 Jun 2026 23:30:24 +0800 Subject: [PATCH 28/37] Update docs/hpc/14_tutorial_apptainer/03_apptainer_files.mdx Co-authored-by: Robert Young --- docs/hpc/14_tutorial_apptainer/03_apptainer_files.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/14_tutorial_apptainer/03_apptainer_files.mdx b/docs/hpc/14_tutorial_apptainer/03_apptainer_files.mdx index 64a43ff000..43569d7c3b 100644 --- a/docs/hpc/14_tutorial_apptainer/03_apptainer_files.mdx +++ b/docs/hpc/14_tutorial_apptainer/03_apptainer_files.mdx @@ -43,7 +43,7 @@ Any files stored in `/scratch/` on the host system will be accessible thr The source and destination paths do not need to be the same. This can be useful when an application expects data to be located in a specific directory within the container. -::: info +:::info Container images are typically read-only. Research data, scripts, notebooks, and output files usually remain outside the container. From 31dfb2fa5d23ed67ebac919d1515c31265d9e89d Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Sun, 28 Jun 2026 23:30:56 +0800 Subject: [PATCH 29/37] Update docs/hpc/14_tutorial_apptainer/03_apptainer_files.mdx Co-authored-by: Robert Young --- docs/hpc/14_tutorial_apptainer/03_apptainer_files.mdx | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/hpc/14_tutorial_apptainer/03_apptainer_files.mdx b/docs/hpc/14_tutorial_apptainer/03_apptainer_files.mdx index 43569d7c3b..8361ae17e8 100644 --- a/docs/hpc/14_tutorial_apptainer/03_apptainer_files.mdx +++ b/docs/hpc/14_tutorial_apptainer/03_apptainer_files.mdx @@ -17,7 +17,9 @@ ls ~ The files and directories you see should match those available outside the container. +:::note Files created in these mounted directories remain available after the container exits. +::: ## Binding Additional Directories From 6e8f80875429921dcc03876d0dd79558f835ac03 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Sun, 28 Jun 2026 23:31:43 +0800 Subject: [PATCH 30/37] Update docs/hpc/14_tutorial_apptainer/02_running_containers.mdx Co-authored-by: Robert Young --- docs/hpc/14_tutorial_apptainer/02_running_containers.mdx | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/hpc/14_tutorial_apptainer/02_running_containers.mdx b/docs/hpc/14_tutorial_apptainer/02_running_containers.mdx index 9886edae57..63d1efcc59 100644 --- a/docs/hpc/14_tutorial_apptainer/02_running_containers.mdx +++ b/docs/hpc/14_tutorial_apptainer/02_running_containers.mdx @@ -67,7 +67,13 @@ Both `apptainer run` and `apptainer exec` start a container, but they serve diff ## Opening an Interactive Shell Within a Container -It is useful to explore a container interactively. Apptainer provides the `apptainer shell` command for this purpose. +There are many reasons why you might want to use a container interactively. +- debugging (software, bind mounts, hardware integrations, etc.) +- testing software upgrades +- rapid Software Prototyping +- data exploration + +Apptainer provides the `apptainer shell` command for this purpose. Launch a shell inside the Ubuntu container: From 135c0bee25000febf82dd857c1220ce84f5bd629 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Sun, 28 Jun 2026 23:32:11 +0800 Subject: [PATCH 31/37] Update docs/hpc/14_tutorial_apptainer/02_running_containers.mdx Co-authored-by: Robert Young --- docs/hpc/14_tutorial_apptainer/02_running_containers.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/14_tutorial_apptainer/02_running_containers.mdx b/docs/hpc/14_tutorial_apptainer/02_running_containers.mdx index 63d1efcc59..2628e27dcb 100644 --- a/docs/hpc/14_tutorial_apptainer/02_running_containers.mdx +++ b/docs/hpc/14_tutorial_apptainer/02_running_containers.mdx @@ -39,7 +39,7 @@ Depending on how the image was built, this command may produce output, launch an The important point is that with `apptainer run`, Apptainer executes the default action defined by the image creator. -In cases we want to run a specific command, use `apptainer exec`. +To run a specific command, use `apptainer exec`. ## Running Specific Commands Within a Container From 6865d1e21caf85a1844e2cb1da661a2674a193c7 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Sun, 28 Jun 2026 23:38:13 +0800 Subject: [PATCH 32/37] Update docs/hpc/14_tutorial_apptainer/01_intro_apptainer.mdx Co-authored-by: Robert Young --- docs/hpc/14_tutorial_apptainer/01_intro_apptainer.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/14_tutorial_apptainer/01_intro_apptainer.mdx b/docs/hpc/14_tutorial_apptainer/01_intro_apptainer.mdx index 0cf1e8471b..8cb30c204d 100644 --- a/docs/hpc/14_tutorial_apptainer/01_intro_apptainer.mdx +++ b/docs/hpc/14_tutorial_apptainer/01_intro_apptainer.mdx @@ -21,7 +21,7 @@ Torch provides container images with both `.sif` and `.sqf` extensions. `.sif` is the standard Apptainer image format and can be used directly with commands such as `apptainer exec`, `apptainer run`, and `apptainer shell`. -Some Torch-provided application images use `.sqf` files and are typically accessed through wrapper scripts such as `run-anaconda3-2024.10-1.bash`. These scripts handle the details of launching the application environment and are often the recommended interface for users. +Some Torch-provided application images use `.sqf` (Squash File System) files and are typically accessed through wrapper scripts such as `run-anaconda3-2024.10-1.bash`. These scripts handle the details of launching the application environment and are often the recommended interface for users. When available, use the wrapper script documented for that application. For general Apptainer examples in this tutorial, we use `.sif` images because they can be launched directly with standard Apptainer commands. From 991bba1b440d917e74533485d82585b82eefad95 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Sun, 28 Jun 2026 23:45:01 +0800 Subject: [PATCH 33/37] Update 05_customize_container_environments.mdx --- .../05_customize_container_environments.mdx | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx b/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx index 38e659a1d3..813799cff0 100644 --- a/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx +++ b/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx @@ -65,3 +65,19 @@ apptainer shell \ ``` This demonstrates that software installed within the overlay persists across sessions, while the original `.sif` image remains unchanged. + +:::info + +In the examples above, the overlay is mounted in read-write mode (`:rw`) so that changes can be made to the container environment. + +Once you have finished installing packages or making modifications, it is recommended to mount the overlay in read-only mode (`:ro`). + +:::warning + +Only use read-write mode (`:rw`) when you need to modify the overlay. While an overlay is mounted in read-write mode, it cannot be opened by other processes. + +Read-only mode (`:ro`) allows multiple processes to use the same overlay simultaneously, making it the recommended choice for running applications and parallel workloads. + +::: + +::: From ceb260b981a451393d56a1af4cc4c9b4e46c72a0 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Sun, 28 Jun 2026 23:46:26 +0800 Subject: [PATCH 34/37] Update 05_customize_container_environments.mdx --- .../05_customize_container_environments.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx b/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx index 813799cff0..3468151840 100644 --- a/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx +++ b/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx @@ -72,12 +72,12 @@ In the examples above, the overlay is mounted in read-write mode (`:rw`) so that Once you have finished installing packages or making modifications, it is recommended to mount the overlay in read-only mode (`:ro`). -:::warning +::::warning Only use read-write mode (`:rw`) when you need to modify the overlay. While an overlay is mounted in read-write mode, it cannot be opened by other processes. Read-only mode (`:ro`) allows multiple processes to use the same overlay simultaneously, making it the recommended choice for running applications and parallel workloads. -::: +:::: ::: From 0c5b3c3d602081c81789304f1f6def34146104c0 Mon Sep 17 00:00:00 2001 From: Sajid Ali Date: Mon, 29 Jun 2026 14:12:33 -0400 Subject: [PATCH 35/37] Update docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx Co-authored-by: Robert Young --- .../05_customize_container_environments.mdx | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx b/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx index 3468151840..3d9bbc0b48 100644 --- a/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx +++ b/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx @@ -66,6 +66,16 @@ apptainer shell \ This demonstrates that software installed within the overlay persists across sessions, while the original `.sif` image remains unchanged. +:::info +In the examples above we started a shell session with the the overlay in read-write mode (`:rw`). +::::warning +Only open the overlay in writeable mode when you are adding packages to it. When not loading packages you should use read-only mode (`:ro`). + +If you leave the overlay open in read-write mode no other process will be able to open it in either read-write or read-only mode. +:::: +read-only overlays can be used by multiple processes which is why they are useful for parallel processing. +::: + :::info In the examples above, the overlay is mounted in read-write mode (`:rw`) so that changes can be made to the container environment. From e835a4dcfc84786438f69e973efd2b197509f536 Mon Sep 17 00:00:00 2001 From: Sajid Ali Date: Mon, 29 Jun 2026 14:15:16 -0400 Subject: [PATCH 36/37] fix typo --- .../05_customize_container_environments.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx b/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx index 3d9bbc0b48..177120e1ad 100644 --- a/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx +++ b/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx @@ -67,9 +67,9 @@ apptainer shell \ This demonstrates that software installed within the overlay persists across sessions, while the original `.sif` image remains unchanged. :::info -In the examples above we started a shell session with the the overlay in read-write mode (`:rw`). +In the examples above we started a shell session with the overlay in read-write mode (`:rw`). ::::warning -Only open the overlay in writeable mode when you are adding packages to it. When not loading packages you should use read-only mode (`:ro`). +Only open the overlay in writable mode when you are adding packages to it. When not loading packages you should use read-only mode (`:ro`). If you leave the overlay open in read-write mode no other process will be able to open it in either read-write or read-only mode. :::: From 74e301a228b5fd1bdb4d59dd5e6ab170b067b45c Mon Sep 17 00:00:00 2001 From: Sajid Ali Date: Mon, 29 Jun 2026 14:16:31 -0400 Subject: [PATCH 37/37] fix duplicate text --- .../05_customize_container_environments.mdx | 16 ---------------- 1 file changed, 16 deletions(-) diff --git a/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx b/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx index 177120e1ad..5e666d11af 100644 --- a/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx +++ b/docs/hpc/14_tutorial_apptainer/05_customize_container_environments.mdx @@ -75,19 +75,3 @@ If you leave the overlay open in read-write mode no other process will be able t :::: read-only overlays can be used by multiple processes which is why they are useful for parallel processing. ::: - -:::info - -In the examples above, the overlay is mounted in read-write mode (`:rw`) so that changes can be made to the container environment. - -Once you have finished installing packages or making modifications, it is recommended to mount the overlay in read-only mode (`:ro`). - -::::warning - -Only use read-write mode (`:rw`) when you need to modify the overlay. While an overlay is mounted in read-write mode, it cannot be opened by other processes. - -Read-only mode (`:ro`) allows multiple processes to use the same overlay simultaneously, making it the recommended choice for running applications and parallel workloads. - -:::: - -:::