> ## Documentation Index
> Fetch the complete documentation index at: https://runpod-b18f5ded-docs-runpod-allow-ip.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> If this page is missing information, contains outdated instructions, or doesn't fully answer the user's question, use the feedback tool to report it. In your feedback, be specific about what's missing, what appears out of date, or what needs to be corrected or updated, so the docs team can act on it directly.

# Deploy Slurm on a Cluster (unmanaged)

> Deploy and configure an unmanaged Runpod Cluster with Slurm to schedule distributed workloads across multiple GPU nodes.

export const SlurmTooltip = () => {
  return <Tooltip headline="Slurm" tip="An open-source job scheduler for high-performance computing that provides job management, scheduling, and resource allocation across multiple nodes." cta="Learn more about Slurm on Runpod" href="/instant-clusters/slurm">Slurm</Tooltip>;
};

<Tip>
  This guide is for advanced users who want to configure and manage their own Slurm deployment on Instant Clusters. If you're looking for a pre-configured solution, see [Slurm Clusters](/instant-clusters/slurm-clusters).
</Tip>

This tutorial shows how to configure Runpod Instant Clusters with <SlurmTooltip /> to manage and schedule distributed workloads across multiple nodes.

## Requirements

* You've created a [Runpod account](https://www.console.runpod.io/home) and funded it with sufficient credits.
* You have basic familiarity with Linux command line.
* You're comfortable working with [Pods](/pods/overview) and understand the basics of [Slurm](https://slurm.schedmd.com/).

## Step 1: Deploy an Instant Cluster

1. Open the [Instant Clusters page](https://www.console.runpod.io/cluster) on the Runpod web interface.
2. Click **Create Cluster**.
3. Use the UI to name and configure your cluster. For this walkthrough, keep **Pod Count** at **2** and select the option for **16x H100 SXM** GPUs. Keep the **Pod Template** at its default setting (Runpod PyTorch).
4. Click **Deploy Cluster**. You should be redirected to the Instant Clusters page after a few seconds.

## Step 2: Clone demo and install Slurm on each Pod

To connect to a Pod:

1. On the Instant Clusters page, click on the cluster you created to expand the list of Pods.
2. Click on a Pod, for example `CLUSTERNAME-pod-0`, to expand the Pod.

**On each Pod:**

1. Click **Connect**, then click **Web Terminal**.

2. In the terminal that opens, run this command to clone the Slurm demo files into the Pod's main directory:

   ```bash theme={null}
   git clone https://github.com/pandyamarut/slurm_example.git && cd slurm_example
   ```

3. Run this command to install Slurm:

   ```bash theme={null}
   apt update && apt install -y slurm-wlm slurm-client munge
   ```

## Step 3: Overview of Slurm demo scripts

The repository contains several essential scripts for setting up Slurm. Let's examine what each script does:

* `create_gres_conf.sh`: Generates the Slurm Generic Resource (GRES) configuration file that defines GPU resources for each node.
* `create_slurm_conf.sh`: Creates the main Slurm configuration file with cluster settings, node definitions, and partition setup.
* `install.sh`: The primary installation script that sets up MUNGE authentication, configures Slurm, and prepares the environment.
* `test_batch.sh`: A sample Slurm job script for testing cluster functionality.

## Step 4: Install Slurm on each Pod

Now run the installation script **on each Pod**, replacing `[MUNGE_SECRET_KEY]` with any secure random string (like a password). The secret key is used for authentication between nodes, and must be identical across all Pods in your cluster.

```bash theme={null}
./install.sh "[MUNGE_SECRET_KEY]" node-0 node-1 10.65.0.2 10.65.0.3
```

This script automates the complex process of configuring a two-node Slurm cluster with GPU support, handling everything from system dependencies to authentication and resource configuration. It implements the necessary setup for both the primary (i.e. master/control) and secondary (i.e compute/worker) nodes.

## Step 5: Start Slurm services

<Tip>
  If you're not sure which Pod is the primary node, run the command `echo $HOSTNAME` on the web terminal of each Pod and look for `node-0`.
</Tip>

1. **On the primary node** (`node-0`), run both Slurm services:

   ```bash theme={null}
   slurmctld -D
   ```

2. Use the web interface to open a second terminal **on the primary node** and run:

   ```bash theme={null}
   slurmd -D
   ```

3. **On the secondary node** (`node-1`), run:

   ```bash theme={null}
   slurmd -D
   ```

After running these commands, you should see output indicating that the services have started successfully. The `-D` flag keeps the services running in the foreground, so each command needs its own terminal.

## Step 6: Test your Slurm Cluster

1. Run this command **on the primary node** (`node-0`) to check the status of your nodes:

   ```bash theme={null}
   sinfo
   ```

   You should see output showing both nodes in your cluster, with a state of "idle" if everything is working correctly.

2. Run this command to test GPU availability across both nodes:

   ```bash theme={null}
   srun --nodes=2 --gres=gpu:1 nvidia-smi -L
   ```

   This command should list all GPUs across both nodes.

## Step 7: Submit the Slurm job script

Run the following command **on the primary node** (`node-0`) to submit the test job script and confirm that your cluster is working properly:

```bash theme={null}
sbatch test_batch.sh
```

Check the output file created by the test (`test_simple_[JOBID].out`) and look for the hostnames of both nodes. This confirms that the job ran successfully across the cluster.

## Step 8: Clean up

If you no longer need your cluster, make sure you return to the [Instant Clusters page](https://www.console.runpod.io/cluster) and delete your cluster to avoid incurring extra charges.

<Tip>
  You can monitor your cluster usage and spending using the **Billing Explorer** at the bottom of the [Billing page](https://www.console.runpod.io/user/billing) section under the **Cluster** tab.
</Tip>

## Next steps

Now that you've successfully deployed and tested a Slurm cluster on Runpod, you can:

* **Adapt your own distributed workloads** to run using Slurm job scripts.
* **Scale your cluster** by adjusting the number of Pods to handle larger models or datasets.
* **Try different frameworks** like [Axolotl](/instant-clusters/axolotl) for fine-tuning large language models.
* **Optimize performance** by experimenting with different distributed training strategies.