> ## Documentation Index
> Fetch the complete documentation index at: https://runpod-b18f5ded-docs-runpod-allow-ip.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> If this page is missing information, contains outdated instructions, or doesn't fully answer the user's question, use the feedback tool to report it. In your feedback, be specific about what's missing, what appears out of date, or what needs to be corrected or updated, so the docs team can act on it directly.

# Deploy an Instant Cluster with Axolotl

> Deploy a Runpod Cluster with Axolotl and configure multi-GPU distributed training for fine-tuning large language models.

export const InferenceTooltip = () => {
  return <Tooltip headline="AI inference" tip="The execution phase where a trained model makes predictions on new data. When you prompt a model and it responds, that's inference.">inference</Tooltip>;
};

export const TrainingTooltip = () => {
  return <Tooltip headline="AI training" tip="The initial phase of AI model development, in which a model analyzes a dataset to learn patterns and relationships.">training</Tooltip>;
};

This tutorial demonstrates how to use Instant Clusters with [Axolotl](https://axolotl.ai/) to fine-tune large language models (LLMs) across multiple GPUs. By leveraging PyTorch's distributed <TrainingTooltip /> capabilities and Runpod's high-speed networking infrastructure, you can significantly accelerate your training process compared to single-GPU setups.

Follow the steps below to deploy a cluster and start training your models efficiently.

## Step 1: Deploy an Instant Cluster

1. Open the [Instant Clusters page](https://www.console.runpod.io/cluster) on the Runpod web interface.
2. Click **Create Cluster**.
3. Use the UI to name and configure your Cluster. For this walkthrough, keep **Pod Count** at **2** and select the option for **16x H100 SXM** GPUs. Keep the **Pod Template** at its default setting (Runpod PyTorch).
4. Click **Deploy Cluster**. You should be redirected to the Instant Clusters page after a few seconds.

## Step 2: Set up Axolotl on each Pod

1. Click your cluster to expand the list of Pods.

2. Click on a Pod, for example `CLUSTERNAME-pod-0`, to expand the Pod.

3. Click **Connect**, then click **Web Terminal**.

4. In the terminal that opens, run this command to clone the Axolotl repository into the Pod's main directory:

   ```sh theme={null}
   git clone https://github.com/axolotl-ai-cloud/axolotl
   ```

5. Navigate to the `axolotl` directory:

   ```sh theme={null}
   cd axolotl
   ```

6. Install the required packages:

   ```sh theme={null}
   pip3 install -U packaging setuptools wheel ninja
   pip3 install --no-build-isolation -e '.[flash-attn,deepspeed]'
   ```

7. Navigate to the `examples/llama-3` directory:

   ```sh theme={null}
   cd examples/llama-3
   ```

Repeat these steps for **each Pod** in your cluster.

## Step 3: Start the training process on each Pod

Run this command in the web terminal of **each Pod**:

```php theme={null}
torchrun \
    --nnodes $NUM_NODES \
    --node_rank $NODE_RANK \
    --nproc_per_node $NUM_TRAINERS \
    --rdzv_id "myjob" \
    --rdzv_backend static \
    --rdzv_endpoint "$PRIMARY_ADDR:$PRIMARY_PORT" -m axolotl.cli.train lora-1b.yml
```

<Info>
  Currently, the dynamic `c10d` backend is not supported. Please keep the `rdzv_backend` flag set to `static`.
</Info>

After running the command on the last Pod, you should see output similar to this after the training process is complete:

```csharp theme={null}
...
{'loss': 1.2569, 'grad_norm': 0.11112671345472336, 'learning_rate': 5.418275829936537e-06, 'epoch': 0.9}
{'loss': 1.2091, 'grad_norm': 0.11100614815950394, 'learning_rate': 3.7731999690749585e-06, 'epoch': 0.92}
{'loss': 1.2216, 'grad_norm': 0.10450132936239243, 'learning_rate': 2.420361737256438e-06, 'epoch': 0.93}
{'loss': 1.223, 'grad_norm': 0.10873789340257645, 'learning_rate': 1.3638696597277679e-06, 'epoch': 0.95}
{'loss': 1.2529, 'grad_norm': 0.1063728854060173, 'learning_rate': 6.069322682050516e-07, 'epoch': 0.96}
{'loss': 1.2304, 'grad_norm': 0.10996092110872269, 'learning_rate': 1.518483566683826e-07, 'epoch': 0.98}
{'loss': 1.2334, 'grad_norm': 0.10642101615667343, 'learning_rate': 0.0, 'epoch': 0.99}
{'train_runtime': 61.7602, 'train_samples_per_second': 795.189, 'train_steps_per_second': 1.085, 'train_loss': 1.255359119443751, 'epoch': 0.99}

100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 67/67 [01:00<00:00,  1.11it/s]
[2025-04-01 19:24:22,603] [INFO] [axolotl.train.save_trained_model:211] [PID:1009] [RANK:0] Training completed! Saving pre-trained model to ./outputs/lora-out.
```

<Check>
  Congratulations! You've successfully trained a model using Axolotl on an Instant Cluster. Your fine-tuned model has been saved to the `./outputs/lora-out` directory. You can now use this model for <InferenceTooltip /> or continue training with different parameters.
</Check>

## Step 4: Clean up

If you no longer need your cluster, make sure you return to the [Instant Clusters page](https://www.console.runpod.io/cluster) and delete your cluster to avoid incurring extra charges.

<Info>
  You can monitor your cluster usage and spending using the **Billing Explorer** at the bottom of the [Billing page](https://www.console.runpod.io/user/billing) section under the **Cluster** tab.
</Info>

## Next steps

Now that you've successfully deployed and tested an Axolotl distributed training job on an Instant Cluster, you can:

* Review the [fine-tuning guide](/fine-tune) for a complete Axolotl workflow and deployment path.
* **Fine-tune your own models** by modifying the configuration files in Axolotl to suit your specific requirements.
* **Scale your training** by adjusting the number of Pods in your cluster (and the size of their containers and volumes) to handle larger models or datasets.
* **Try different optimization techniques** such as DeepSpeed, FSDP (Fully Sharded Data Parallel), or other distributed training strategies.

For more information on fine-tuning with Axolotl, refer to the [Axolotl documentation](https://github.com/axolotl-ai-cloud/axolotl).