Skip to content

debugging.md

Latest commit

 

History

History
119 lines (87 loc) · 4.68 KB

debugging.md

File metadata and controls

119 lines (87 loc) · 4.68 KB
Raw

Debugging

WARNING: This guide assumes that you have already logged in to a login node successfully. Running an IDE directly without a first login will not work as the first login will force you to change your password.

Running IDE as Text Editor

To run your IDE (e.g., PyCharm, VSCode) on the login node, simply enter your credentials using the remote SSH functionality of your IDE.

When you need to run your script with GPU, you may run Slurm in your IDE's terminal to start a job. More information on how to use srun and sbatch can be found in the Slurm guide.

Note: You should avoid running too many extensions as there is a RAM limit enforced on login nodes that may cause your IDE to be killed due to Out of Memory. Ensure you exit your IDEs properly as we have been informed that some users run into issues when they do not do so and two instances of their IDE run simultaneously.

Please take note of the following:

  1. Each and every user has a CPU and memory limit on the each login node.
  2. IDEs beyond VSCode and PyCharm can be more hungry on CPU and memory. This means either it initialize very slow, or it directly triggers OOM-killer.
  3. Most IDEs deploy a backend process on the login node so you can edit your code in real time. This also means this backend eats up your memory quota. Backends not closed gracefully can stay in the memory and eat your memory quota as well.
  4. IDE-related issues are difficult to reproduce remotely. We may not be able to resolve them without being able to replicate the problem ourselves.

Running IDE on Compute Node to Debug

IMPORTANT NOTE: This guide should only be used if you specifically need to be on the compute node running your processes. Following these steps will hog up resources until you close your terminal. You are recommended to run only your script most of the time.

⚠ Hard cap on salloc / srun: interactive allocations are limited to 2 hours and 1 GPU total. The IDE-on-compute-node flow below uses salloc, so the same cap applies — plan accordingly. For longer or multi-GPU work, use sbatch and connect your IDE to the login node only.

We are fully aware that users might want to run debugging sessions, either checking outputs in shell sessions or running a Python debugger and debug your code line-by-line.

To debug directly on the cluster using IDEs (e.g., PyCharm, VSCode), set up an SSH tunnel. This tunnel runs on the login node and relays traffic to a GPU node, allowing you to “directly” interact with it.

💡 Tip: If you're unfamiliar with configuring remote connections in PyCharm or VSCode, refer to their official documentation.

Step-by-Step Instructions

  1. Allocate a GPU node with the resources you need:

    salloc <resource_request>
    • Example: 
      salloc --gpus=a40:1 --time=0:30:00
    • This starts a shell. Do not exit this shell — it will cancel the job.

  2. Open a new terminal window.

  3. SSH into your allocated GPU node.

    1. Via SSH command (VSCode)

      This is the command to provide your IDE if you want to use an IDE. If you are a PyCharm user

      ssh -J <username>@<login_node_ip> <username>@<allocated_node_name>
      • Example: 
        ssh -J user@0.0.0.0 user@gpu-example-1

      This tells SSH to go through the login node (-J) and connect directly to the compute node.

    2. Create an SSH tunnel (PyCharm)

      For IDEs where you are unable to customize the SSH command, please see below.

      Run the following command on your computer's terminal (NOT any terminals that already has SSH running).

      ssh -L <port>:<allocated_node_name>:22 <username>@<login_node_ip>

      This will start an SSH session. Do not close this shell either. This SSH session maps a local port on your computer to the SSH port on your allocated node. In this way, you can put localhost:<port> instead in your IDE as your remote host, i.e., your IDE is connecting to this local port, which is being forwarded to the allocated node. This forwarding process is transparent to your IDE.

My IDE doesn't want to connect.

We have a section dedicated to IDEs in the FAQ here.

After You're Done

  • Close all terminals and SSH sessions properly.
  • ⚠️ Do not leave SSH sessions hanging or unattended. This will cause your priority to become lower AND other users to be unable to use the resources you are using.