This document provides recommendations for the accelerators, consumption options, and deployment tools that are best suited for different artificial intelligence (AI), machine learning (ML), and high performance computing (HPC) workloads. Use this document to help you identify the best deployment for your workload.
For information and recommendations about infrastructure pillars for AI, ML, and HPC workloads, see the following documents:
Workloads overview
AI Hypercomputer architecture supports the following use cases:
| Workload | Description | Recommendation |
|---|---|---|
| Pre-training foundation models | This involves building a language model using a large dataset. The
result of pre-training foundation models is a new model that is good
at performing general tasks. Models are categorized based on their size as follows:
|
See recommendations for pre-training models |
| Fine-tuning | This involves taking a trained model and adapting it to perform specific tasks by using specialized data sets or other techniques. Fine-tuning is generally performed on large models. | See recommendations for fine-tuning models |
| Inference or serving | This involves taking a trained or fine-tuned model and making it
available for consumption by users or applications. Inference workloads are categorized based on the size of the models as follows:
|
See recommendations for inference |
| ML for small or medium-sized models | This involves training and serving ML models that are smaller in size and complexity, typically for more specialized tasks. | See recommendations for ML for small or medium-sized models |
| HPC | This is the practice of aggregating computing resources to gain performance greater than that of a single workstation, server, or computer. HPC is used to solve problems in academic research, science, design, simulation, and business intelligence. | See recommendations for HPC |
Recommendations for pre-training models
To pre-train foundation models, large clusters of accelerators continuously read large volumes of data. These accelerators adjust weights through forward and backward passes to learn from the data. These training jobs run for weeks or even months at a time.
The following sections outline the accelerators and consumption options that we recommend you use to pre-train foundation models.
Recommended accelerators
To pre-train foundation models on Google Cloud, we recommend that you use A4X Max (NVIDIA GB300), A4X (NVIDIA GB200), A4 (NVIDIA B200), A3 Ultra (NVIDIA H200 141GB), A3 Mega (NVIDIA H100 80GB), or A3 High (NVIDIA H100 80GB) accelerator-optimized machine types, and use an orchestrator to deploy the cluster.
To deploy these clusters of accelerators, we also recommend that you use Cluster Director or Cluster Toolkit. For more information, see the respective cluster deployment guide for your machine type of choice in the following table.
| Workloads | Recommendations | Cluster deployment guide | |
|---|---|---|---|
| Machine type | Orchestrator | ||
| Pre-training foundation models |
|
GKE | Create an AI-optimized GKE cluster with default configuration |
| Slurm | |||
| Large model training | A3 Ultra (NVIDIA H200 141GB) | GKE | Create an AI-optimized GKE cluster with default configuration |
| Slurm | |||
| Large model training |
|
GKE | Maximize GPU network bandwidth in Standard mode clusters |
| Slurm | |||
Recommended consumption option
For a high level of assurance in obtaining large clusters of accelerators, we recommend using a reservation. Specifically, to minimize the costs of reserved resources, we recommend that you request a reservation duration that's long enough to receive committed use discounts. For more information about consumption options, see Choose a consumption option.
Recommendations for fine-tuning models
To fine-tune large foundation models, smaller clusters of accelerators read moderate volumes of data. These accelerators adjust the model to perform specific tasks. These fine-tuning jobs run for days or even weeks.
The following sections outline the recommended accelerators and consumption option to use when fine-tuning models.
Recommended accelerators
To fine-tune models on Google Cloud, we recommend that you use A3 Ultra (NVIDIA H200 141GB), A3 Mega (NVIDIA H100 80GB), or A3 High (NVIDIA H100 80GB) accelerator-optimized machine types, and use an orchestrator to deploy the cluster.
To deploy these clusters of accelerators, we also recommend that you use Cluster Director or Cluster Toolkit. For more information, see the respective cluster deployment guide for your machine type of choice in the following table.
| Workloads | Recommendations | Cluster deployment guide | |
|---|---|---|---|
| Machine type | Orchestrator | ||
| Fine-tuning large models | A3 Ultra (NVIDIA H200 141GB) | GKE | Create an AI-optimized GKE cluster with default configuration |
| Slurm | |||
| Fine-tuning large models |
|
GKE | Maximize GPU network bandwidth in Standard mode clusters |
| Slurm | |||
Recommended consumption option
For fine-tuning workloads, use a future reservation in calendar mode to provision resources. For more information about consumption options, see Choose a consumption option.
Recommendations for inference
The following sections outline the recommended accelerators and consumption options to use when performing inference.
Recommended accelerators
The recommended accelerators for inference depend on whether you're performing multi-host frontier or large model inference, or single-host frontier inference.
Recommended accelerators (multi-host)
To perform multi-host frontier or large model inference on Google Cloud, use an A4X Max (NVIDIA GB300), A4X (NVIDIA GB200), A4 (NVIDIA B200), A3 Ultra (NVIDIA H200 141GB), A3 Mega (NVIDIA H100 80GB), or A3 High (NVIDIA H100 80GB) accelerator-optimized machine types, and deploy the machine by using an orchestrator. To deploy these clusters of accelerators, we also recommend that you use Cluster Director or Cluster Toolkit. The table provides links to cluster deployment guides for each recommended machine type.
| Workloads | Recommendations | Cluster deployment guide | |
|---|---|---|---|
| Machine type | Orchestrator | ||
| Multi-host frontier inference |
|
GKE | Create an AI-optimized GKE cluster with default configuration |
| Slurm | |||
| Large model inference | A3 Ultra (NVIDIA H200 141GB) | GKE | Create an AI-optimized GKE cluster with default configuration |
| Slurm | |||
| Large model inference |
|
GKE | Maximize GPU network bandwidth in Standard mode clusters |
| Slurm | |||
Recommended accelerators (single host)
The table outlines the accelerators that we recommend you use to perform single-host frontier inference. The table provides links to VM deployment guides for each recommended machine type.
| Workloads | Recommendations | VM deployment guide | |
|---|---|---|---|
| Machine type | Orchestrator | ||
| Single-host frontier and mainstream inference |
|
N/A | Create an A4 or A3 Ultra instance |
|
Create an A3 High or A3 Edge instance | ||
| G4 (NVIDIA RTX PRO 6000) | Create a G4 instance | ||
| A2 (NVIDIA A100) | Create an A2 instance | ||
| G2 (NVIDIA L4) | Create a G2 instance | ||
| N1 (NVIDIA T4 or V100) | Create an N1 instance | ||
Recommended consumption option
For inference, use either a long-running reservation or a future reservation in calendar mode. For more information about consumption options, see Choose a consumption option.
Recommendations for small or medium-sized models
For ML workloads that involve small to medium-sized models, achieving an optimal balance between price and performance is a primary consideration.
Recommended accelerators
The following table outlines the recommended accelerators to use for small to medium-sized model ML workloads.
| Workloads | Recommendations | VM deployment guide | |
|---|---|---|---|
| Machine type | Orchestrator | ||
| ML for small or medium-sized models | G4 (NVIDIA RTX PRO 6000) | N/A | Create a G4 instance |
| G2 (NVIDIA L4) | Create a G2 instance | ||
| A2 (NVIDIA A100) | Create an A2 instance | ||
| A3 Edge (NVIDIA H100 80GB) | Create an A3 Edge instance | ||
| N1 (NVIDIA T4 or V100) | Create an N1 instance | ||
Recommendations for HPC
For HPC workloads, any accelerator-optimized machine series or compute-optimized machine series works well. If using an accelerator-optimized machine series, the best fit depends on the amount of computation that must be offloaded to the GPU. For a detailed list of recommendations for HPC workloads, see Best practices for running HPC workloads.
Summary of recommendations
The following table summarizes the recommended accelerators and consumption options for each workload.
| Resource | Recommendation |
|---|---|
| Model pre-training | |
| Machine family | Use one of the following accelerator-optimized machine types: A4X Max (NVIDIA GB300), A4X (NVIDIA GB200), A4 (NVIDIA B200), A3 Ultra (NVIDIA H200 141GB), A3 Mega (NVIDIA H100 80GB), or A3 High (NVIDIA H100 80GB) |
| Consumption option | 'Use standard future reservations |
| Model fine-tuning | |
| Machine family | Use A3 Ultra (NVIDIA H200 141GB), A3 Mega (NVIDIA H100 80GB), or A3 High (NVIDIA H100 80GB) accelerator-optimized machine types |
| Consumption option | 'Use standard future reservations |
| Inference | |
| Machine family | Use one of the following machine types: A4X Max (NVIDIA GB300), A4X (NVIDIA GB200), A4 (NVIDIA B200), A3 Ultra (NVIDIA H200 141GB), A3 Mega (NVIDIA H100 80GB), A3 High (NVIDIA H100 80GB), G2 (NVIDIA L4), G4 (NVIDIA RTX PRO 6000), A2 (NVIDIA A100), A3 Edge (NVIDIA H100 80GB), or N1 (NVIDIA T4 or V100) |
| Consumption option | Use reservations, on-demand, or Spot |
| ML for small or medium-sized models | |
| Machine family | Use one of the following machine types: G2 (NVIDIA L4), G4 (NVIDIA RTX PRO 6000), A2 (NVIDIA A100), A3 Edge (NVIDIA H100 80GB), or N1 (NVIDIA T4 or V100) |
| Consumption option | Use on-demand, Spot, or standard reservations |
| Storage | Use Cloud Storage FUSE |
| HPC | |
| See the summary section of the best practices for running HPC workloads | |
What's next
- Learn how to Create an AI-optimized GKE cluster.
- Learn how to Create a fully managed Slurm cluster.
- Learn how to Create an AI-optimized instance.