mds/optimization.qmd at main · Morrison-Lab/mds · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
---
title: "Optimization"
format:
  html: default
  revealjs:
    output-file: optimization-slides.html
  pdf:
    output-file: optimization-handout.pdf
  docx:
    output-file: optimization.docx
resources:
  - data/auto.csv
---

{{< include latex-macros/macros.qmd >}}

```{r}
#| include: false
reticulate::py_require(c("numpy", "pandas"))
```

{{< include mds/_subfiles/_sec_optimization.qmd >}}

## Further reading {#sec-additional-resources}

- @boyd2004convex
  is the standard reference on convex optimization, covering convex sets, convex functions, convex optimization problems, duality, unconstrained minimization, and interior-point methods.
- @nocedal2006numerical
  is a comprehensive reference on continuous numerical optimization, covering line search methods, trust-region methods, conjugate gradient methods, quasi-Newton methods, and nonlinear equations.
- [Hua Zhou](https://hua-zhou.github.io/)'s [lecture notes for "UCLA Biostat 216 - Mathematical Methods for Biostatistics" (2024 Fall)](https://ucla-biostat-216.github.io/2024fall/slides/13-optim/13-optim.html)
- [Neural Networks](https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi)
  is a YouTube playlist by Grant Sanderson (3Blue1Brown);
  its chapters on gradient descent and backpropagation calculus visually illustrate
  how gradients of multivariate cost functions are computed and used for optimization.

-   @wikipedia_fermats_theorem_stationary_points --- Fermat's theorem on stationary points

-   @wikipedia_taylors_theorem --- Taylor's theorem and its remainders

-   @wikipedia_saddle_point --- saddle points

-   @wikipedia_convex_set --- convex sets

-   @wikipedia_epigraph_mathematics --- epigraphs

-   @wikipedia_tangent_space --- tangent spaces and planes

-   @wikipedia_feasible_region --- feasible regions

-   @wikipedia_mathematical_optimization --- mathematical optimization

-   @wikipedia_lagrange_multiplier --- Lagrange multipliers

-   @wikipedia_iterative_method --- iterative methods

-   @wikipedia_newtons_method_in_optimization --- Newton's method in optimization

-   @wikipedia_rate_of_convergence --- rates of convergence

-   @wikipedia_gradient_descent --- gradient descent

-   @wikipedia_line_search --- line search

-   @wikipedia_backtracking_line_search --- backtracking line search and the Armijo condition

## References {.unnumbered}

::: {#refs}
:::