DRP (IAP 2019)

This is the webpage for a directed reading program (DRP) I am leading during MIT's Independent Activities Period (IAP 2019). The subject is high-dimensional statistics.

References

  • [Ko] Koltchinskii, Oracle Inequalities in Empirical Risk Minimization and Sparse Recovery Problems (2011).
  • [BvdG] B├╝hlmann and van de Geer, Statistics for High-Dimensional Data (2011).
  • [Ke] Keener, Theoretical Statistics (2010).
  • [vdVW] van der Vaart and Wellner, Weak Convergence and Empirical Processes (1996).
  • [vH] van Handel, Probability in High Dimension (2016).
  • [V] Vershynin, High-Dimensional Probability (2018).

Readings

  • (preparation) The basic language of measure theory, and a preview of the topics in the reading group. [Ke] 1.1-1.6, [Ko] 1.1-1.7.
  • (1/7) Concentration inequalities: Hoeffding, bounded difference, Talagrand, Bosquet. Sub-Gaussian random variables and Orlicz norms. [V] 2.1-2.2, 2.5, 2.7.1, [Ko] 2.3.
  • (1/9) Covering/packing and the volumetric argument. Chaining and Dudley's entropy integral. [V] 4.2, 8.1 (skip 8.1.2), [Ko] 3.1.
  • (1/11) Symmetrization and contraction principle. Rademacher complexity of finite classes of functions. [vH] 7.1, [Ko] 2.1-2.2, 3.2.
  • (1/14) Vapnik-Chervonenkis (VC) dimension: Pajor's lemma, Sauer-Shelah lemma. Uniform Glivenko-Cantelli classes. [V] 8.3, [Ko] 3.3.
  • (1/16) VC subgraph classes. Fat-shattering dimension and the Mendelson-Vershynin bound on covering numbers. [vdVW] 2.6.2, [vH] 7.3.
  • (1/18) Distribution-dependent excess risk bounds via fixed-point argument. Excess risk bounds for regression with quadratic loss. [Ko] 4.1, 5.1.
  • (1/22) Upper bound for random metric entropy. Excess risk bounds for empirical risk minimization with convex loss. [Ko] 3.4, 5.2.
  • (1/23) Data-dependent bounds on excess risk. Model selection via penalized empirical risk minimization for monotone families. [Ko] 4.2, 6.1.
  • (1/25) Statistical guarantees for least absolute shrinkage and selection operator (LASSO) in sparse linear regression with fixed design. LASSO for general convex loss functions. Compatibility condition and the restricted isometry property (RIP). [BvdG] 6.1-6.2.2, 6.3, [V] 10.5.2.
  • (1/28) Statistical guarantees for L2Boosting with componentwise linear least squares base procedure for sparse linear regression. [BvdG] 12.1-12.2, 12.4.4.1, 12.5.1, 12.5.4-12.5.5, 12.6.2.2, 12.8.2.
  • (1/30) Implicit regularization. Gunasekar et al., Characterizing implicit bias in terms of optimization geometry (2018).

Exercises