Click any module to expand. The module progresses from classical image processing (still essential for preprocessing and edge cases) through deep learning classification and detection, to the 2025–26 frontier: foundation vision models, generative AI, and multimodal vision-language understanding.
How computers represent images and the classical image processing toolkit
Images as NumPy arrays: (height, width, channels). Pixel values: 0–255 (uint8) or 0.0–1.0 (float32). Colour spaces: RGB (display), BGR (OpenCV default), HSV (colour-based filtering), Grayscale (edge detection). Converting between spaces: cv2.cvtColor(). Why understanding image representation matters — every DL model expects a specific input format (normalised, resized, channel order).
cv2.imread(), cv2.imshow(), cv2.imwrite(). Drawing: rectangles (bounding boxes), circles, text, polylines — used to visualise model predictions. Video capture: cv2.VideoCapture() for webcam/video file. Frame-by-frame processing. Resizing, cropping, rotating. PIL/Pillow as alternative. OpenCV is the image I/O library for every CV pipeline — preprocessing before the model, visualisation after.
Blurring: Gaussian blur (noise reduction), median blur (salt-and-pepper noise). Sharpening with kernels. Edge detection: Canny (the standard), Sobel (directional gradients), Laplacian. Thresholding: binary, Otsu's adaptive. Morphological operations: erosion, dilation, opening, closing. Contour detection: findContours(). These classical techniques STILL power industrial inspection, document processing, and preprocessing for DL models.
Keypoint detection: identify distinctive points in an image. SIFT (Scale-Invariant Feature Transform): robust to rotation, scale changes. ORB (Oriented FAST and Rotated BRIEF): fast, open-source alternative. Feature matching: BFMatcher, FLANN. Use cases: image stitching (panoramas), visual SLAM, template matching. When classical features beat DL: few training images, explainability required, resource-constrained. The foundation before CNNs existed.
Convolutional neural networks — the architecture that made deep learning visual
Convolution: learnable filters slide over image → extract features (edges → textures → shapes → objects). Parameters: kernel size, stride, padding (same/valid). Feature maps: output of convolution layer. Receptive field: how deep layers "see" larger regions. Pooling: MaxPool (take max — keep strongest activation), AvgPool, GlobalAvgPool (entire map → single value). Architecture: Conv-ReLU-Pool stacks → Flatten → Dense → Softmax.
nn.Conv2d(in_channels, out_channels, kernel_size). nn.MaxPool2d(). nn.BatchNorm2d(). Building custom CNN: define layers in __init__, connect in forward(). Training on CIFAR-10: 10 classes, 50K images. Training loop with GPU: zero_grad → forward → loss → backward → step. Evaluation: accuracy, confusion matrix. Data augmentation during training. From-scratch CNN → understand what ResNet automates.
LeNet (1998): the pioneer — 5 layers. AlexNet (2012): deep + ReLU + dropout → won ImageNet. VGG (2014): simple 3×3 convolutions stacked deep. GoogLeNet/Inception (2014): parallel filter sizes. ResNet (2015): skip connections → train 100+ layers without vanishing gradients. EfficientNet (2019): compound scaling (depth × width × resolution). MobileNet: depthwise separable convolutions for mobile. Understanding WHY each innovation mattered.
Pretrained on ImageNet: 14M images, 1000 classes — features transfer to new tasks. Transfer strategy: (1) freeze backbone → train new head, (2) unfreeze top layers → fine-tune with small LR, (3) unfreeze all → full fine-tune. torchvision.models: resnet50(weights='IMAGENET1K_V2'). timm library: 700+ pretrained models. Practical: classify custom dataset (50 training images per class) with 95%+ accuracy using transfer learning. The most practical DL skill.
Why augment: more training data → better generalisation. Albumentations: fast, flexible, GPU-friendly augmentation library. Geometric: flip, rotate, crop, resize, affine. Colour: brightness, contrast, hue-saturation, RGB shift. Advanced: CoarseDropout (Cutout), MixUp, CutMix, Mosaic (used in YOLO). Augmentation pipeline: A.Compose([A.HorizontalFlip(), A.RandomBrightnessContrast(), A.Normalize()]). Train vs validation: NEVER augment validation data.
The architecture that's replacing CNNs — self-attention for images
ViT: split image into patches (16×16) → linear projection → add positional encoding → Transformer encoder → classification head. Self-attention on patches: every patch attends to every other patch (global context from layer 1 — CNNs need many layers for this). Why ViT works: captures long-range dependencies, scales better with compute and data. ViT vs CNN: ViT needs more data/pretraining, but matches or beats CNNs when scaled.
DeiT (Data-efficient ViT): distillation token enables training on smaller datasets. Swin Transformer: hierarchical windows → efficient for high-resolution images (used in detection and segmentation). BEiT: BERT-style pretraining for vision (masked image modelling). ConvNeXt: CNN modernised with Transformer tricks (competitive with ViT). EVA, DINOv2: self-supervised ViTs with incredible feature representations. The 2025 vision backbone landscape.
Hugging Face: ViTForImageClassification.from_pretrained("google/vit-base-patch16-224"). PEFT/LoRA: fine-tune only 0.5% of parameters → save GPU memory, train faster. LoraConfig(r=16, target_modules=["query","value"]). Fine-tuning on custom dataset: 100 images per class, 15 minutes on Colab GPU, 97%+ accuracy. The hero code pattern. When ViT + LoRA beats full CNN fine-tuning — and when it doesn't.
Grad-CAM for CNNs: gradient-weighted class activation maps — WHERE is the model looking? Attention maps for ViTs: visualise which patches the model attends to. "The model classified this as a dog because it focused on the face region." Debugging: model looking at background instead of subject → data problem. Essential for medical imaging, autonomous driving, and any safety-critical application.
Finding and localising objects in images — the most deployed CV task
Classification: "This image contains a cat." Detection: "There's a cat at (x1,y1,x2,y2) with 94% confidence." Bounding boxes: (x, y, width, height) or (x1, y1, x2, y2). IoU (Intersection over Union): measure of box overlap. Non-Maximum Suppression (NMS): remove duplicate detections. Anchor boxes. Evaluation: mAP (mean Average Precision) at IoU thresholds (mAP@50, mAP@50:95). The metrics every detection engineer must understand.
YOLO philosophy: single-pass detection (look at the image ONCE). YOLO evolution: YOLOv1 (2016) → YOLOv5 (production standard) → YOLOv8 (2023) → YOLO11 (2024, Ultralytics). Ultralytics API: model = YOLO("yolo11n.pt"); results = model("image.jpg") — detection in 4 lines. Training custom YOLO: annotate with Roboflow/LabelImg, train with model.train(data="custom.yaml"). YOLO for video: real-time detection at 30–100+ FPS. The most deployed detection model in production.
Faster R-CNN: Region Proposal Network → classification. More accurate but slower than YOLO. When to use: accuracy > speed (medical, satellite). DETR (Detection Transformer): end-to-end detection with Transformers — no anchors, no NMS. RT-DETR: real-time Transformer detection. Grounding DINO: open-vocabulary detection (detect ANY object from text description — "find all red cars"). The detector landscape: YOLO (speed) vs DETR (elegance) vs Grounding DINO (zero-shot).
The complete pipeline: collect images → annotate (Roboflow, CVAT, LabelImg) → augment → train YOLO/DETR → evaluate (mAP) → deploy (FastAPI/TorchServe) → monitor. Data annotation: bounding box format (COCO, Pascal VOC, YOLO). Handling class imbalance: oversample rare classes. Small object detection: tile images, higher resolution. The end-to-end workflow for production detection systems.
Pixel-level understanding — classifying every pixel in the image
Semantic segmentation: every pixel gets a class label (road, sky, car, person) — same class instances are NOT distinguished. Instance segmentation: separate each object instance (car_1, car_2, person_1). Panoptic segmentation: semantic + instance combined. Use cases: autonomous driving (where is the road?), medical imaging (tumour boundary), satellite imagery (land use mapping). The three segmentation levels and when each is needed.
U-Net: encoder-decoder with skip connections — the medical imaging standard. DeepLabv3+: atrous convolutions for multi-scale context. Mask R-CNN: Faster R-CNN + pixel mask branch → instance segmentation. SegFormer: Transformer-based segmentation. Training U-Net on custom data: binary masks, Dice loss. Evaluation: IoU (Intersection over Union) per class, mean IoU. Building a medical image segmentation model from scratch.
SAM (Meta, 2023): foundation model for segmentation — segment ANY object with a point click, bounding box, or text prompt. SAM 2: extends to video segmentation. Zero-shot: no training on your specific objects — it just works. SAM as annotation tool: click to get perfect masks (replaces hours of manual annotation). Grounded-SAM: Grounding DINO (detect) + SAM (segment) → text-prompted segmentation. The foundation model that changed segmentation forever.
Creating images — from GANs to Stable Diffusion
GAN: generator creates images from noise, discriminator judges real vs fake — adversarial training. Training: discriminator gets better → generator must improve. DCGAN: convolutional GAN. StyleGAN: high-quality face generation with controllable features (age, expression, hair). Pix2Pix: paired image translation (sketch → photo). CycleGAN: unpaired translation (horse → zebra). Conditional GAN: generate based on class label.
Forward process: gradually add Gaussian noise until image is pure noise. Reverse process: learn to denoise step by step → generate images from noise. DDPM: the foundational paper. Stable Diffusion architecture: U-Net denoiser + CLIP text encoder + VAE decoder. Text-to-image: describe → generate. Image-to-image: modify existing images. Inpainting: fill masked regions. ControlNet: guide generation with poses, edges, depth maps. The technology behind DALL-E 3, Midjourney, and Stable Diffusion.
LoRA for Stable Diffusion: fine-tune on your style/domain with 20–50 images. DreamBooth: personalise models (teach "my face" or "my product"). Textual Inversion: learn new concepts via text embeddings. Training with diffusers library: Hugging Face's diffusion framework. Use cases: product photography, architectural visualisation, custom art styles, synthetic data generation for training other models.
Models that understand images AND text — the frontier of computer vision
CLIP (OpenAI): jointly train image encoder + text encoder to match images with descriptions. Zero-shot image classification: describe categories in text → CLIP matches — NO training on your categories needed. SigLIP, EVA-CLIP: improved variants. Use cases: image search by text, content moderation, zero-shot classification. CLIP embeddings as feature extractors for downstream tasks. The model that connected vision and language.
VQA: ask questions about images → model answers in natural language. LLaVA (Large Language and Vision Assistant): connect a vision encoder (CLIP) to an LLM (LLaMA) → multimodal understanding. "What's happening in this image?" "Count the red objects." GPT-4o, Gemini, Claude: commercial multimodal models. Building a VQA application: upload image → ask question → model answers. The multimodal interface paradigm of 2025.
Florence-2 (Microsoft): unified vision foundation model — caption, detect, segment, OCR in one model. Grounding DINO: detect objects from text descriptions ("find all traffic signs"). Open-vocabulary: detect/segment ANY category without training on it. OWLv2: open-world detection. The shift from closed-set (train on 80 COCO classes) to open-set (detect anything described in text). The future of CV is foundation models + text prompts.
Beyond single images — video analysis, text recognition, and domain-specific applications
Video = sequence of frames. Frame extraction with OpenCV. Object tracking: follow a detected object across frames. SORT, DeepSORT: tracking-by-detection (detect → associate → track). ByteTrack: state-of-the-art multi-object tracking. SAM 2: segment and track objects in video. Action recognition: classify activities (walking, running, falling). Use cases: surveillance, sports analytics, traffic monitoring, retail analytics.
OCR (Optical Character Recognition): extract text from images. Tesseract: open-source OCR engine. EasyOCR: deep learning OCR (80+ languages). PaddleOCR: industrial-grade. Document AI: layout analysis + OCR + entity extraction from invoices, receipts, contracts. LayoutLM / DocTR: Transformer-based document understanding. Use cases: invoice processing, license plate recognition, digitising handwritten forms.
Medical imaging: X-ray classification, tumour segmentation with U-Net, retinal disease detection. Autonomous driving: lane detection, traffic sign recognition, depth estimation (MiDaS). Agriculture: crop disease detection, yield estimation from drone imagery. Manufacturing: defect detection on production lines, surface inspection. Satellite/remote sensing: land use classification, change detection. Each domain has unique challenges — understanding them opens specialised career paths.
Professional CV engineering — experiment tracking, reproducibility, and scaling
W&B: log training curves, learning rates, augmentation examples, prediction visualisations, confusion matrices — all in one dashboard. Compare runs: which augmentation + LR schedule gave best mAP? Hyperparameter sweeps. Model checkpointing. Image logging: see model predictions on val set during training. W&B is the experiment tracking standard at AI labs — OpenAI, Google DeepMind, NVIDIA all use it.
Learning rate: use LR Finder, then OneCycleLR or cosine annealing. Mixed precision (FP16): 2× speed, half memory. Gradient accumulation: simulate larger batch sizes. Progressive resizing: train on small images first, then increase. Test-Time Augmentation (TTA): augment at inference, average predictions. Ensemble: average predictions from multiple models. The bag of tricks that improves accuracy by 2–5% without changing the model architecture.
Data quality > model quality: clean, well-annotated data beats a better architecture on bad data. Annotation tools: Roboflow (hosted, easiest), CVAT (open-source, powerful), Label Studio (multi-modal). Active learning: model identifies uncertain samples → human annotates those → model improves. SAM-assisted annotation: click → perfect mask in seconds. Label quality: inter-annotator agreement, review pipelines. "Garbage in, garbage out" — the most important CV engineering lesson.
Serving CV models — cloud APIs, edge devices, and building a portfolio
ONNX export: framework-agnostic model format (PyTorch → ONNX → any runtime). Quantisation: FP32 → INT8 (4× smaller, 2–4× faster inference). TensorRT (NVIDIA): optimised inference for NVIDIA GPUs (2–6× speedup). Pruning: remove unimportant weights. Knowledge distillation: train smaller model from larger teacher. These optimisations transform "runs on a V100" into "runs on a Jetson Nano."
FastAPI: REST API for CV inference (upload image → return predictions as JSON). Gradio: interactive demo (drag-and-drop image → see detection/classification results). Hugging Face Spaces: free hosting for Gradio demos. Streamlit for dashboards. Batch prediction pipelines. The deployment stack: train (Colab) → optimise (ONNX) → serve (FastAPI) → demo (Gradio) → host (HF Spaces).
Edge: run models on mobile, embedded, cameras — not cloud. NVIDIA Jetson: GPU-powered edge device for real-time CV. TensorFlow Lite: mobile deployment (Android/iOS). ONNX Runtime Mobile. OpenVINO (Intel edge devices). Coral Edge TPU (Google). Model constraints: ≤10MB for mobile, ≤50ms inference. Use cases: smart cameras, drone vision, factory inspection, mobile apps. The market where CV meets hardware.
Portfolio structure: 4–5 projects showing progression (classification → detection → segmentation → generative → multimodal). Each: GitHub repo + Gradio demo + W&B experiment logs + Roboflow dataset. Kaggle CV competitions: participate even without winning. Write-ups: explain architecture choices, failure modes, what you'd improve. "Try my model" demos beat "look at my code" every time. The portfolio that gets CV Engineer interviews.
Fine-tune Vision Transformer on custom dataset (medical/food/wildlife), Grad-CAM visualisation, W&B logging, deployed on HF Spaces with Gradio.
Annotate custom dataset (Roboflow), train YOLO11, evaluate mAP, real-time video detection, deploy via FastAPI with live demo.
U-Net on medical imaging dataset (skin lesion/retinal), Dice + IoU evaluation, attention visualisation, Gradio demo for clinician testing.
Embed image database with CLIP, search by text query ("sunset over mountains"), FAISS for fast retrieval, Streamlit interface.
LoRA fine-tune Stable Diffusion on custom art style (20–50 images), text-to-image generation, deployed on HF Spaces.
Trainers build CV pipelines live — from OpenCV preprocessing to YOLO training to Gradio deployment. Students train on their own GPU alongside.
Each week: detect, classify, or segment on a new dataset. Leaderboard competition. Post-challenge: review winning approaches and why they worked.
Trainers review: architecture choice, augmentation strategy, evaluation metrics, Grad-CAM analysis, deployment readiness. The feedback loop that builds CV engineering intuition.
Students deploy CV models as Gradio demos on HF Spaces. Upload an image → see detection/segmentation results live. The portfolio that makes interviewers say "let me try it."
Tesla, Waymo, Ola Electric, Ather — autonomous driving runs on CV: lane detection, object detection, depth estimation, tracking. Robotics: pick-and-place, navigation, visual SLAM. The highest-paying CV specialisation with massive growth ahead.
X-ray analysis, tumour segmentation, retinal disease detection, pathology slide analysis. Medical CV roles at Qure.ai, Predible Health, and global healthtech companies offer the highest AI salaries. U-Net + domain expertise = the most valuable CV specialisation in healthcare.
Camera filters, barcode/QR scanning, document scanning, AR effects, visual search — mobile CV is everywhere. Edge deployment skills (TFLite, ONNX Mobile) are needed at every app company. CV + mobile = massive consumer market.
Manufacturing defect detection, quality control, warehouse automation, retail shelf monitoring, surveillance analytics. Industrial CV is the fastest-growing enterprise CV market in India. Companies pay premium for engineers who can deploy YOLO on edge devices for real-time factory inspection.
E-305, Industrial Area, Sahibzada Ajit Singh Nagar, Punjab 160055, IN
