GPU dedicated servers: choose a configuration and get started

A GPU dedicated server is a bare-metal server with its own NVIDIA graphics card, for workloads that run far faster on a GPU than on a CPU: training and fine-tuning models, LLM inference, rendering, video transcoding and other CUDA computing. The GPU, CPU, RAM and storage are yours alone, with full root access and no shared cloud GPU scheduler.

Configurations

Stock of each build changes. The GPU servers page shows what is available now, and custom configurations can be quoted. Provisioning takes about 24 hours.

How to choose

• GPU memory (VRAM) decides what fits. A model, its activations and the batch must fit in VRAM. The RTX 5090's 32 GB suits larger models and bigger batches; 16 GB cards suit smaller models, inference and quantized models.
• RTX 5090: the fastest option, for training, fine-tuning and heavy rendering, with fast DDR5 system memory and large NVMe storage for datasets and checkpoints.
• RTX 4070 Ti SUPER: strong value for CUDA work, inference, rendering and development.
• Tesla T4: a data-center inference card with low power use, many CPU cores and extra IPs; a budget choice for serving models rather than training them.
• Location: pick the one closest to your users or data, especially for interactive inference.

Software on delivery

Servers are prepared with Ubuntu 22.04 LTS and NVIDIA drivers, with the AI stack available: CUDA and cuDNN, PyTorch, TensorFlow, JAX, JupyterLab, Docker with the NVIDIA Container Toolkit, and common Python data tools. Tell the team at order time which parts you want set up, or if you prefer a clean system to install your own versions.

Check the GPU and CUDA

After you receive your login (see SSH key authentication to switch to keys), confirm the GPU works:

# GPU model, memory, driver and CUDA version
nvidia-smi

# CUDA compiler, if the toolkit is installed
nvcc --version

# PyTorch sees the GPU?
python3 -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"

# TensorFlow sees the GPU?
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

If nvidia-smi fails after a kernel update, the driver module may need rebuilding for the new kernel. Reboot once, and if it still fails, reinstall the driver package or open a ticket.

GPU containers with Docker

Containers keep each project's CUDA and Python versions separate. With the NVIDIA Container Toolkit installed:

# quick test: the container sees the GPU
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

# run a PyTorch container with your project folder mounted
docker run --rm -it --gpus all -v "$PWD":/work -w /work pytorch/pytorch:latest python train.py

Pick image tags whose CUDA version is supported by the installed driver; nvidia-smi shows the highest CUDA version the driver supports.

Use JupyterLab safely

Do not expose JupyterLab directly to the internet. Run it bound to localhost and reach it through an SSH tunnel:

# on the server
jupyter lab --no-browser --ip=127.0.0.1 --port=8888

# on your computer: forward the port, then open http://localhost:8888
ssh -N -L 8888:127.0.0.1:8888 user@your-server-ip

Use the token JupyterLab prints, and keep the firewall closed to everything except SSH and the services you deliberately publish.

Keep the GPU healthy

# live use, memory, temperature and power, every 5 seconds
nvidia-smi --query-gpu=utilization.gpu,memory.used,temperature.gpu,power.draw --format=csv -l 5

# which processes use the GPU
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv

• Save checkpoints often, so a reboot or crash does not cost hours of training.
• Keep datasets and checkpoints backed up off the server. See Business Backup.
• Update the driver deliberately, not mid-project: test new drivers before long runs.
• On the RTX 5090 build, IPMI remote management is included for console access. See IPMI and remote console.

FAQ

Related: IPMI and remote console | SSH key authentication | Linux server basics | Business Backup

Related articles