PyTorch became the dominant deep learning framework in research and is now winning in production too — TorchServe, ONNX export, and PyTorch 2.x's compiler stack make it a credible choice end-to-end. If you are learning machine learning in 2026, learning PyTorch is not optional; it is the language of the field.
What changed in 2026
- PyTorch 2.4 (stable in 2025) made
torch.compile production-safe — it compiles model graphs to optimized kernels using Triton, delivering 30–50% training speedups with one line of code.
torch.compile now supports AMD ROCm in addition to NVIDIA CUDA — hardware choice is less constrained.- PEFT (Parameter-Efficient Fine-Tuning) via Hugging Face became the standard way to fine-tune LLMs — LoRA, QLoRA, and DoRA are beginner-accessible.
torch.distributed with FSDP2 replaced DDP for large model training — fully sharded data parallelism is now the default for multi-GPU.- Flash Attention 3 landed as a PyTorch-native implementation, making attention computation ~2× faster on H100-class GPUs.
Learning roadmap
| Phase |
Topics |
Weeks |
| 1. Tensors |
Creation, indexing, math, GPU transfer |
1 |
| 2. Autograd |
Computational graph, backward(), gradients |
1 |
3. nn.Module |
Layers, forward pass, parameter registration |
1–2 |
| 4. Training loop |
DataLoader, optimizer, loss, eval mode |
1–2 |
| 5. CNN / RNN |
Image classification, sequence modeling |
2–3 |
| 6. Transformers |
Attention, BERT fine-tuning via HF |
2–3 |
| 7. torch.compile |
Graph compilation, Triton backends |
1 |
| 8. LLM fine-tuning |
LoRA, QLoRA, PEFT, TRL |
2–3 |
Phase 1: tensors
Tensors are PyTorch's N-dimensional arrays. The essential operations:
import torch
# Create
x = torch.randn(3, 4) # shape (3, 4), float32 by default
y = torch.zeros(3, 4, dtype=torch.float16)
# GPU transfer (CUDA or MPS on Apple Silicon)
device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"
x = x.to(device)
# Indexing — same as NumPy
print(x[:, 0]) # first column
print(x[0, :2]) # first row, first two elements
# Matrix ops
z = x @ x.T # (3, 4) @ (4, 3) = (3, 3)
Phase 2: autograd and the training loop
PyTorch tracks operations on tensors to build a computational graph; .backward() computes gradients.
import torch
import torch.nn as nn
# A minimal two-layer MLP
model = nn.Sequential(
nn.Linear(784, 256),
nn.ReLU(),
nn.Linear(256, 10)
)
model = model.to(device)
# torch.compile — one line, 30–50% faster training
model = torch.compile(model)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)
loss_fn = nn.CrossEntropyLoss()
def train_epoch(dataloader):
model.train()
for X, y in dataloader:
X, y = X.to(device), y.to(device)
optimizer.zero_grad()
logits = model(X)
loss = loss_fn(logits, y)
loss.backward()
optimizer.step()
Phase 3: Hugging Face fine-tuning
In 2026, training a Transformer from scratch is rare. Fine-tuning a pre-trained model on your data is the standard approach.
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from transformers import Trainer, TrainingArguments
model_name = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2)
training_args = TrainingArguments(
output_dir="./results",
num_train_epochs=3,
per_device_train_batch_size=16,
eval_strategy="epoch",
bf16=True, # bfloat16 on Ampere+ GPUs
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
)
trainer.train()
Phase 4: LoRA fine-tuning for LLMs
Full fine-tuning a 7B model requires ~112 GB VRAM at fp16. LoRA (Low-Rank Adaptation) fine-tunes ~1% of parameters while preserving most of the base model's knowledge.
from peft import LoraConfig, get_peft_model, TaskType
from transformers import AutoModelForCausalLM
import torch
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3-8B",
torch_dtype=torch.bfloat16,
device_map="auto",
)
lora_config = LoraConfig(
r=16, # rank — higher = more params but better fit
lora_alpha=32,
target_modules=["q_proj", "v_proj"],
lora_dropout=0.05,
task_type=TaskType.CAUSAL_LM,
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 6.8M || all params: 8.03B (0.08% trainable)
QLoRA adds 4-bit NF4 quantization, reducing VRAM to ~10 GB for an 8B model.
How to start
- Work through the official PyTorch tutorials — "60 Minute Blitz" first, then the custom dataset and DataLoader tutorial.
- Implement a linear regression model by hand (no
nn.Linear) — derive backprop yourself once.
- Train an MLP on MNIST — classification benchmark that runs in minutes on CPU.
- Fine-tune a DistilBERT sentiment classifier on a dataset from Hugging Face Hub.
- Run a LoRA fine-tune with
trl's SFTTrainer on a small 1–3B model.
Common mistakes
Keeping data on CPU when running on GPU. Always move tensors to the same device as the model. A missing .to(device) call produces a cryptic device mismatch error.
Calling optimizer.zero_grad() after loss.backward(). Zero gradients before computing the new backward pass, not after — otherwise you accumulate gradients from the previous batch.
Not switching between model.train() and model.eval(). Dropout and BatchNorm behave differently in each mode. Always set the mode explicitly before training and inference loops.
Using float64 by default. Deep learning works in float32 or bfloat16. float64 is 2× the memory for no accuracy benefit on most tasks.
What to skip
- TensorFlow for new deep learning projects in 2026 — PyTorch dominates research and is closing the production gap. Learn one framework deeply.
- Raw CUDA extensions before you understand the Python training loop — optimize Python first, then profile.
- Training LLMs from scratch on a single GPU — it is not feasible; use fine-tuning and PEFT.
FAQ
PyTorch vs TensorFlow in 2026?
PyTorch leads in research (85%+ of papers) and is closing in on production. TensorFlow/Keras is still viable but the community momentum is with PyTorch.
Do I need a GPU to learn PyTorch?
No — small models train on CPU. Google Colab and Kaggle both offer free T4 and P100 GPU notebooks for larger experiments.
What is the difference between torch.compile and TorchScript?
torch.compile uses torchdynamo to trace and compile eager Python PyTorch code. TorchScript is an older serialization format. Use torch.compile for training speedups; use ONNX or torch.export for deployment.
How long does it take to learn PyTorch?
Basic training loops: 2–4 weeks. Comfortable fine-tuning pre-trained models: 2–3 months. Deep understanding of internals and custom CUDA: 1+ year.
Where to go next