AI & Machine Learning Engineering Notes
In-depth engineering references covering the full ML development stack: from statistical foundations through large-scale distributed training and transformer architectures, to production fine-tuning strategies for geospatial foundation models.
The articles follow a deliberate learning arc — building from mathematical fundamentals through architectural deep-dives to the practical challenges of adapting foundation models to domain-specific targets.
Foundations
-
The conceptual bridge from classical statistics to neural network optimization — written for domain scientists transitioning from probabilistic modelling to deep learning.
-
The central tension between model simplicity and flexibility: overfitting, underfitting, and the strategies (regularization, cross-validation, ensembles) that keep models generalizable.
-
Building intuition for the confusion matrix from first principles, bridging ML vocabulary (false positive/negative) with remote-sensing accuracy assessment (omission/commission error, producer's/user's accuracy).
-
Deep learning mechanics from first principles: tensors, gradient descent, training loops, and production model architectures in PyTorch.
Architectures
-
Inside self-attention, multi-head attention, key-query-value dimensions, positional encodings, and the structural foundations of large language and vision models.
-
How raw text and spatial signals are encoded into dense vector representations: BPE, SentencePiece, and geospatial embedding strategies.
-
How a single vector representation and one formula (cosine similarity) make satellite pixels, biomedical records, and crop stress directly comparable — from geospatial foundation models (Prithvi, Clay, AlphaEarth, OlmoEarth) to the same pattern showing up far outside Earth science.
-
Engineering lifecycle of transformer inference: embedding dimensions, QKV projection math, multi-head attention mechanics, and GPU memory footprint of the KV-cache at scale.
-
How temperature controls token entropy via the Boltzmann distribution, the difference between standard and reasoning models, and how MCTS-based models like Kimi K2.6 use high entropy productively.
Scale & Infrastructure
Training-side scaling lives here. For the cluster itself — distributed execution, on-premise model serving, and environment configuration — see HPC & Infrastructure.
-
GPU memory management, single- and multi-GPU optimization, Distributed Data Parallel (DDP) configuration, and SLURM HPC integration for deep learning workloads.
Adaptation & Fine-Tuning
-
Full fine-tuning vs. linear probing vs. feature extraction — when to use each strategy and the computational trade-offs involved.
Computer Vision
-
How ViT adapts the Transformer to image understanding: patch tokenization, positional encodings, and the conceptual shift from CNNs to attention-based vision models.
-
Why Vision Transformers are more data-hungry than CNNs — the role of inductive bias, what CNNs encode by design vs. what ViTs must learn from data, and implications for transfer learning.