Add momentum and the momentum coefficient

_subfiles/_sec_optimization_gradient_descent.qmd

@@ -153,3 +153,19 @@
 {{< slidebreak >}}
 
 {{< include mds/_subfiles/vector-calc/_sol-best-step-count.qmd >}}
+
+{{< slidebreak >}}
+
+{{< include mds/_subfiles/vector-calc/_def-momentum-coefficient.qmd >}}
+
+{{< slidebreak >}}
+
+{{< include mds/_subfiles/vector-calc/_exm-momentum-coefficient.qmd >}}
+
+{{< slidebreak >}}
+
+{{< include mds/_subfiles/vector-calc/_def-momentum.qmd >}}
+
+{{< slidebreak >}}
+
+{{< include mds/_subfiles/vector-calc/_exm-momentum.qmd >}}

_subfiles/vector-calc/_def-momentum-coefficient.qmd

@@ -0,0 +1,14 @@
+:::{#def-momentum-coefficient}
+#### Momentum coefficient
+
+Let $\vg^{(1)}, \vg^{(2)}, \ldots$ be vectors in $\reals^p$,
+and let $\vv^{(t)}$ be a running **velocity** built from them.
+The **momentum coefficient** $\beta \in [0, 1)$ is the factor that multiplies the previous velocity in
+
+$$
+\vv^{(t)} \eqdef \beta\,\vv^{(t-1)} + \vg^{(t)},
+$$
+
+for $t = 1, 2, \ldots$, with $\vv^{(0)} \eqdef \vzero$.
+
+:::

_subfiles/vector-calc/_def-momentum.qmd

@@ -0,0 +1,19 @@
+:::{#def-momentum}
+#### Momentum (heavy-ball method)
+
+Let $f : \reals^p \to \reals$ have first [partial derivatives](vector-calculus.qmd#def-partial-derivative),
+let $\vx^{(0)}$ be a starting point,
+let $\steplen > 0$ be a [step length](#def-gradient-descent),
+and let $\beta \in [0, 1)$ be a [momentum coefficient](#def-momentum-coefficient).
+**Momentum** (the **heavy-ball method**) updates, for $t = 1, 2, \ldots$, by
+
+$$
+\begin{aligned}
+\vv^{(t)} &\eqdef \beta\,\vv^{(t-1)} + \deriv{\vx} f(\vx^{(t-1)}) \\
+\vx^{(t)} &\eqdef \vx^{(t-1)} - \steplen\,\vv^{(t)},
+\end{aligned}
+$$
+
+with $\vv^{(0)} \eqdef \vzero$.
+
+:::

_subfiles/vector-calc/_exm-momentum-coefficient.qmd

@@ -0,0 +1,43 @@
+:::{#exm-momentum-coefficient}
+#### How far back the velocity remembers
+
+Unrolling the recursion, the vector $\vg^{(t-k)}$ enters $\vv^{(t)}$ with factor $\beta^k$, for $k < t$.
+These factors add up to at most $\sum_{k=0}^{\infty} \beta^k = 1 / (1 - \beta)$.
+As a rule of thumb, the [velocity](#def-momentum-coefficient) is therefore dominated by the last $1 / (1 - \beta)$ vectors.
+
+::: {.panel-tabset group="language"}
+
+### R
+
+```{r}
+#| label: momentum-coefficient-r
+#| code-fold: false
+beta_mc <- c(0, 0.5, 0.9, 0.99)
+k_mc <- 10
+weight_mc <- beta_mc^k_mc
+memory_mc <- 1 / (1 - beta_mc)
+weight_rounded_mc <- signif(weight_mc, 2)
+# nolint start: undesirable_function_linter.
+rbind(beta = beta_mc, weight = weight_rounded_mc, memory = memory_mc)
+# nolint end
+```
+
+### Python
+
+```{python}
+#| label: momentum-coefficient-python
+#| code-fold: false
+beta_mc = [0, 0.5, 0.9, 0.99]
+k_mc = 10
+for b_mc in beta_mc:
+    print(b_mc, float(f"{b_mc**k_mc:.2g}"), round(1 / (1 - b_mc)));
+```
+
+:::
+
+At $\beta = 0$ the [velocity](#def-momentum-coefficient) is the current vector alone,
+and at $\beta = 0.99$ it combines about `r round(memory_mc[4])` vectors.
+
+<!-- no-figure: the example tabulates weights -->
+
+:::

_subfiles/vector-calc/_exm-momentum.qmd

@@ -0,0 +1,59 @@
+:::{#exm-momentum}
+#### Momentum on a one-dimensional quadratic
+
+Let $f(x) = \tfrac{1}{2} x^2$,
+so $\deriv{x} f(x) = x$.
+Start at $x^{(0)} = 1$ with $\steplen = 0.1$,
+and take $20$ updates with $\beta = 0$, which is plain [gradient descent](#def-gradient-descent), and with $\beta = 0.9$:
+
+::: {.panel-tabset group="language"}
+
+### R
+
+```{r}
+#| label: momentum-r
+#| code-fold: false
+x0_mo <- 1
+s_mo <- 0.1
+beta_mo <- c(plain = 0, momentum = 0.9)
+n_mo <- 20
+x_mo <- rep(x0_mo, 2)
+v_mo <- c(0, 0)
+path_mo <- matrix(NA_real_, nrow = n_mo, ncol = 2)
+for (k_mo in seq_len(n_mo)) {
+  v_mo <- beta_mo * v_mo + x_mo
+  x_mo <- x_mo - s_mo * v_mo
+  path_mo[k_mo, ] <- x_mo
+}
+last_mo <- path_mo[n_mo, ]
+lowest_mo <- apply(path_mo, 2, min)
+# nolint start: undesirable_function_linter.
+summary_mo <- rbind(last = last_mo, lowest = lowest_mo)
+# nolint end
+round(summary_mo, 4)
+```
+
+### Python
+
+```{python}
+#| label: momentum-python
+#| code-fold: false
+s_mo, n_mo = 0.1, 20
+for beta_mo in (0.0, 0.9):
+    x_mo, v_mo, path_mo = 1.0, 0.0, []
+    for _ in range(n_mo):
+        v_mo = beta_mo * v_mo + x_mo
+        x_mo = x_mo - s_mo * v_mo
+        path_mo.append(x_mo)
+    print(beta_mo, round(path_mo[-1], 4), round(min(path_mo), 4));
+```
+
+:::
+
+After $20$ updates, plain descent has reached `r signif(last_mo[1], 3)` and [momentum](#def-momentum) `r signif(last_mo[2], 3)`.
+Momentum overshoots the [minimizer](algebra.qmd#def-global-minimizer) $0$, down to `r signif(lowest_mo[2], 3)` at update `r which.min(path_mo[, 2])`, and then swings back.
+In this run, with a small [step length](#def-gradient-descent), the swing leaves momentum behind plain descent after $20$ updates.
+
+<!-- no-figure: a one-dimensional run summarized in two numbers per method -->
+
+:::