內在維度假說與低秩矩陣分解數學機理
Agent 輔助工程檢查
前置要求
第 48 天線性層與矩陣乘法代數基礎。
學習目標
透徹理解內在維度假說（Intrinsic Dimensionality），將全量微調增量 Delta W 精確因子化為低秩分解式 Delta W = (alpha/r) * B @ A，其中 r << min(d, k)。
操作指引
1. 研讀 Aghajanyan 等人（2020）內在維度理論：超參數化大模型在特定下游任務微調時，其有效參數變化實質收斂於極低維度的子空間。
2. 傳統全量微調更新為 W = W0 + Delta W，Delta W 為全秩實數矩陣 R^{d x k}，更新開銷隨維度呈二次方膨脹。
3. LoRA（Hu 等人 2021）將增量約束為兩個低秩小矩陣相乘：B in R^{d x r}，A in R^{r x k}，秩 r 通常取 4, 8, 16 即可滿足需求。
4. 當 d=4096, k=4096, r=8 時：全量權重需 16,777,216 參數，而 LoRA 僅需 8*(4096+4096) = 65,536 參數，壓縮率高達 99.6%。
自主實作與記錄
給定輸入維度 d_in=1024，輸出維度 d_out=2048 的線性層。計算全量矩陣 W0 與秩 r=8 的 LoRA 矩陣 A、B 的參數量，並求出參數削減百分比。
全量參數 = 1024 * 2048 = 2,097,152。LoRA 參數 = 8 * (1024 + 2048)。
全量權重參數 = 1024 * 2048 = 2,097,152 個。
LoRA 適配參數 = 8 * (1024 + 2048) = 24,576 個。
參數量削減幅度 = 1 - (24,576 / 2,097,152) = 98.83%。
能準確寫出低秩矩陣因子化公式並推算任意維度下的參數削減賬目。
權威來源資源與權利證明
LoRA: Low-Rank Adaptation of Large Language Models
對應章節 / 時間戳: Section 4.1 Low-Rank-Parametrized Update Matrices; Section 7.2 What Is the Optimal Rank r for LoRA?
驗證版本與日期: ICLR 2022 / arXiv:2106.09685v2 (Verified: 2026-10-30)
Primary literature introducing Low-Rank Adaptation, establishing parameter decomposition W = W0 + (alpha/r)*BA, initialization strategies, and zero inference latency merging.
Matrix factorization, rank, linear layers, and transformer attention.
Complete matrix derivations, tensor dimensions, and pure PyTorch LoRALinear modules are implemented directly in lesson code.
對比候選資源與選型分析 (2)
Introduces bottleneck adapter layers in series, which introduces unavoidable inference latency overhead; LoRA's parallel formulation merges with zero latency.
Prepends virtual prompt tokens; reduces available context length, unlike LoRA which preserves full sequence capacity.
Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning
對應章節 / 時間戳: Section 3 Intrinsic Dimensionality of Finetuning
驗證版本與日期: ACL 2021 / arXiv:2012.13255v1 (Verified: 2026-10-30)
Key theoretical foundation demonstrating that overparameterized pre-trained language models reside on a low-dimensional manifold, proving why low rank r (e.g. 4 to 16) is sufficient for task adaptation.
Linear algebra, matrix subspace projection, and pretraining representations.
The intrinsic dimensionality concept, subspace projections, and parameter reduction trade-offs are fully explained in lesson text.
對比候選資源與選型分析 (2)
General neural network optimization landscapes; Aghajanyan focuses explicitly on foundation language models.
Measures how low-rank LoRA trades target-domain learning for less forgetting; Aghajanyan provides the direct theoretical motivation for LoRA's low-rank assumption.