GPU Pool

This quick guide will guide you on how to use a GPU pool.

CUDA

https://developer.nvidia.com/cuda-downloads

Runtime

1nvcc -V

Driver

1nvidia-smi

PyTorch

https://pytorch.org/get-started/locally/

install

1pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 --trusted-host mirrors.aliyun.com

version

1pip show torch

torch-script

1import torch
2print(torch.__version__)

cuda-script

1import torch
2print(torch.cuda.is_available())
3print(torch.version.cuda)

update

1pip install --upgrade pytorch torchvision --trusted-host mirrors.aliyun.com

uninstall

1pip uninstall torch

Hardware

Hardware Requirement

	Bits	7B	13B	30B	65B	8X7B
Full	16 (2 bytes)	160GB	320GB	600GB	1200GB	900GB
Freeze	16 (2 bytes)	20GB	40GB	120GB	240GB	200GB
LoRA	16 (2 bytes)	16GB	32GB	80GB	160GB	120GB
QLoRA	8 (1 bytes)	10GB	16GB	40GB	80GB	80GB
QLoRA	4 (0.5 bytes)	6GB	12GB	24GB	48GB	32GB

Calculation formula (estimation): Memory = Number of parameters × Number of bytes (QLoRA 4) = 30,000,000,000 × 0.5 = 15,000,000,000 bytes 15,000,000,000 bytes / 1024 / 1024 / 1024 = 13.97G Simplified: 15,000,000,000 bytes / 1000 / 1000 / 1000 = 15G

CUDA

PyTorch

Hardware