Applied Machine Learning 2024 - Useful ML links

The field of Machine Learning (ML) is developing at a very fast pace and by an expanding number of practitioners. For this reason, text books on ML are typically few and slightly depricated, though they of course describe the general concepts very well. Research papers have partially filled the gap, as these are more versatile and frequently updated. However, they typically only deal with a very small and specific part of the generel ML "phase space".
But whereas "classic" literature (i.e. books and papers) only covers partially, blogs and github repositories seem to fill out the rest, and typically in a much more accessible fashion with great illustrations and example code. For this reason, we have tried to gather some of the more useful links to such blogs and repositories below. It is simply our (slightly random) selection of webpages that we have come across and found illustrative and useful. So use these at will, and also build your own list of reference sites.

Books:
  • Applied Machine Learning by David Forsyth. Great book, at times even mildly entertaining, from one of the masters of ML vision.
  • Elements of Statistical Learning II by Trevor Hastie et al. Second eddition of standard book on fundamentals.
  • Machine Learning (from PDG) by Kyle Cranmer et al. (2021). A concise (60 page) overview
  • Deep Learning by Ian Goodfellow et al. (2016). A short and good general introduction to ML can be found in Chapter 5 of Part 1.
  • Pattern Recognition and Machine Learning by Christopher M. Bishop (2006).
  • "Interpretable Machine Learning" by Christoph Molnar (2020). A Guide for Making Black Box Models Explainable.
  • "Convolutional Neural Networks for Visual Recognition" by Andrej Karpathy (2017?). Used for teaching CNN in Stanford's cn231 class.
  • "Deep Learning with Python" by author of Keras, Francois Chollet, now at Google AI (2017). Especially chapter 4 is a good overview of the fundamentals of Machine Learning.
  • Book on neural networks and deep learning. Online book only!
  • Pattern Recognition and Machine Learning (2006) by Christopher M. Bishop is a classic (if older) standard reference.

    Papers:
  • XGBoost paper (2016). Highly readable paper showing the innovations of the XGBoost algorithm.
  • LightGBM paper (2017). Explaining the great speedup and showing examples of execution times.
  • More general paper (100 pages) on Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges (2021).
  • ROCKET: Time series classification using random convolutional kernels


  • Blogs/Links/Tutorials - supervised learning:
  • Introduction to tree based learning. Very good introduction to the basics of tree based learning.
  • Introduction to neural net based learning. Very good introduction to the basics of Neural Net (NN) based learning.
  • SciKit Learn tutorial. Gives a quick introduction to ML in general and has code examples for SciKit Learn.
  • XGBoost vs. LightGBM. Discussion of differences, with code examples.
  • XGBoost, LightGBM, and CatBoost. Discussion of differences and hyperparameters.
  • Introduction to NGBoost, which is a tree based algorithm, which makes a probabilistic predictions (i.e. uncertainties).
  • PyTorch Geometric Graph Neural Network Tutorial (2019). Reasonably good guide with code examples.
  • Survey of the performance of BDTs and NNs on tabulated data, essentially showing that tree based methods simply are very good at this!
  • Great introduction with code examples to the Verstack package, which combines the powers of LightGBM and Optuna.
  • Andrej Kaparthy's Recipe for Training Neural Networks (a grand master of this art).


  • Blogs/Links/Tutorials - unsupervised learning:
  • SciKit-Learn manual for and discussion of (unsupervised) clustering.
  • Two (food & RNA) great examples of PCA applied to simple data.
  • Overview of t-SNE algorithm with papers and implementations.
  • Comparison of PCA, t-SNE, UMAP, and LDA in a short visually nice writeup with code examples.
  • Isolation Forests (Wiki) and their implementation (Towards Data Science) for anomaly detection.
  • Unsupervised projector using TensorFlow, which includes different datasets and methods.


  • Blogs/Links/Tutorials - time series analysis:
  • Blog on using XGBoost for time series analysis, which includes dataset and code.
  • Do We Really Need Deep Learning Models for Time Series Forecasting?, using BDTs for time series forecasting.


  • Blogs/Links/Tutorials - feature importance, re-weighting, optimisation, etc.:
  • Permutation importance described in the SciKit-Learn implementation.
  • Permutation importance and vs Random Forest Feature Importance (MDI) with code examples of usage.
  • GitHub repository for SHAPley value calculation, which gives game theory based variable rankings.
  • Shapley Values explained in chapter 5.9 of C. Molnar's book.
  • Great non-technical introduction to SHAP values.
  • Re-weighting data samples using GBReweighter.
  • Description of the Adam optimiser. Three page overview of the ADAM optimiser (probably currently best Stochastic Gradient Descent algorithm).
  • Oversampling for imbalanced classification with SMOTE, to improve on situations with little data in one class.
  • Great overview of Stochastic Gradient Descent methods, with illustrative animations.


  • Code:
  • Oneliners for ML mostly to display performance.
  • Plotting boundaries for ML classification.
  • Example of using UMAP for clustering in an artistic way!.


  • On Loss functions:
  • Cross Entropy (Wiki) (log loss), which is one of the most used loss functions for classification.
  • Cross Entropy illustrated.
  • Kullback–Leibler divergence (Wiki) or "relative entropy" used for loss functions.


  • Other:
  • "So Long Sucker" game description (Wiki), developed by Lloyd S. Shapley (behind SHAP values) and other.
  • Illustrative animations of ML inner workings, as a great explainer.
  • Illustrative webpage showing how a CNN works on the MNIST data (great!) and general images.

    Last updated: 11th of March 2024 by Troels Petersen.