GPU Architecture Deep Dive: Understanding Core Components
The modern Graphics Processing Unit (GPU) stands as a marvel of parallel computing, designed from the ground up to execute thousands of operations simultaneously. This architecture, fundamentally different from a CPU, prioritizes throughput over low-latency single-thread performance, making it indispensable for tasks ranging from real-time graphics rendering and scientific simulations to artificial intelligence and deep learning. Understanding its core components reveals the ingenuity behind its immense computational power.
Streaming Multiprocessors (SMs) / Compute Units (CUs): The Heart of Parallelism
At the very core of a GPU’s processing capability are its Streaming Multiprocessors (NVIDIA nomenclature) or Compute Units (AMD nomenclature). These are the fundamental building blocks responsible for executing parallel workloads. A single GPU chip contains dozens, sometimes hundreds, of these SMs/CUs, each capable of independently running multiple threads concurrently. Each SM/CU is a self-contained processing unit, equipped with a suite of functional units and memory resources.
Within each SM/CU, the primary computational engines are the CUDA Cores (NVIDIA) or Stream Processors (AMD). These are highly specialized Arithmetic Logic Units (ALUs) designed for single-instruction, multiple-data (SIMD) operations. A single CUDA core can execute a floating-point or integer operation in parallel with many others. Modern GPUs feature thousands of these cores across all SMs/CUs, enabling massive data parallelism. Alongside these general-purpose cores, SMs also house Special Function Units (SFUs), which are optimized for complex mathematical operations like transcendental functions (sine, cosine, reciprocal square root) and interpolation, critical for graphics and scientific computing.
Memory access within an SM/CU is facilitated by Load/Store Units (LSUs), which handle data movement between registers, shared memory, and caches. Each SM/CU also contains a dedicated set of registers for each active thread. These are the fastest memory locations available to a thread, offering extremely low-latency access to frequently used data.
Crucially, each SM/CU features a small, extremely fast, on-chip memory known as Shared Memory (NVIDIA) or Local Data Share (AMD). This memory is shared among threads within the same thread block, allowing for efficient data exchange and reuse without incurring the high latency of off-chip global memory. Its programmable nature allows developers to optimize data access patterns for significant performance gains. Additionally, an L1 Data Cache and an Instruction Cache are present within each SM/CU, designed to reduce latency for frequently accessed data and instructions, respectively.
The orchestration of thread execution within an SM/CU is handled by Warp Schedulers (NVIDIA) or Workgroup Processors (AMD). These schedulers manage groups of threads (warps in NVIDIA, wavefronts in AMD) that execute the same instruction in lockstep. When one warp encounters a memory latency bottleneck, the scheduler can instantly switch to another ready warp, effectively hiding latency and keeping the computational units busy, a technique known as latency hiding.
Memory Subsystem: Fueling the Cores
The immense computational power of the SMs/CUs requires a high-bandwidth, low-latency memory subsystem to feed them data. The primary memory resource is Global Memory, typically implemented using high-speed GDDR (Graphics Double Data Rate) memory, such as GDDR6 or GDDR6X, or increasingly, High Bandwidth Memory (HBM). This VRAM (Video RAM) is off-chip, connected to the GPU die via wide memory buses, providing hundreds of gigabytes per second, or even terabytes per second, of bandwidth. Its large capacity (from several gigabytes to tens of gigabytes) allows it to store complex textures, frame buffers, and large datasets for AI models.
Memory Controllers act as the interface between the GPU’s processing units and the VRAM. These