Netlab: Algorithms for Pattern Recognition

• Chapter 1: Introduction
  Introduction to Matlab: Matlab Basics; Matlab Plotting; Programming in Matlab; Programming Facilities.
  The Netlab Toolbox: Overview of Netlab; Generic Functions.
  Worked Example: Data Normalisation

• Chapter 2: Parameter Optimisation Algorithms

Controlling the Algorithms: Information Display; Termination Criteria; Extra Arguments and Return Values; Demonstration Program.
Quadratic Approximation at a Minimum
Line Search: Precision; Line Minimisation Algorithms; Bracketing the Minimum.
Batch Gradient Descent
Conjugate Gradients
Scaled Conjugate Gradients
Quasi-Newton Methods
Optimisation and Neural Networks: General Purpose Optimisation Algorithms; On-line Gradient Descent.
Worked Example: Constrained Optimisation
• Chapter 3: Density Modelling and Clustering
  Gaussian Mixture Models: Theory; Implementation.
  Computing Probabilities: GMM Activations; GMM Probabilities; GMM Posteriors.
  EM Training Algorithm: Algorithm Theory; Initialisation; Netlab EM Implementation.
  Demonstrations of GMM: EM Algorithm; Density Modelling.
  K-means Clustering: Algorithm and Netlab Implementation; Demonstration Program.
  K-nearest-neighbour: Algorithm and Netlab Implementation; Demonstration Program.
  Worked Examples: Classification with Density Models; Novelty Detection.

• Chapter 4: Single Layer Networks

The Single Layer Feed-forward Network: Netlab Implementation; Forward Propagation.
Error Functions
Error Gradient Calculation
Evaluating Other Derivatives: Network Activation Derivatives; The Hessian Matrix.
Iterated Re-weighted Least Squares Training
Demonstration Programs: Two-class Problem; Three-class Problem.
Worked Example: Training Regularised Models
• Chapter 5: The Multi-layer Perceptron

The Two-layer Feed-forward Network: Definition; Network Creation and Initialisation; Manipulating Weights; Forward Propagation.
Error Functions and Network Training
Error Gradient Calculation
Evaluating other Derivatives
The Hessian Matrix: Fast Multiplication by the Hessian; Netlab Implementation.
Demonstration Programs: Regression Demonstration; Classification Demonstration.
Mixture Density Networks: Model Structure; Network Creation and Initialisation; Forward Propagation; Conditional Probabilities; Error and Gradient Calculation; Demonstration Program.
Worked Example: Adding Direct Connections
• Chapter 6: Radial Basis Functions

The RBF Network: Theory; Netlab}\ Implementation.
Special Purpose Training Algorithms: Basis Function Optimisation; Output Weight Optimisation; Netlab Implementation.
Error and Error Gradient Calculation
Evaluating Other Derivatives: Output Derivatives; Network Jacobian; Network Hessian.
Demonstration Program
Worked Examples: Linear Smoothing; Dual Basis Functions; Characterising Network Complexity.
• Chapter 7: Visualisation and Latent Variable Models

Principal Component Analysis: Theory; Netlab Implementation; Examples.
Probabilistic Principal Component Analysis: Probabilistic PCA; PPCA Implementation; Mixture of PPCA; Mixture of PPCA Implementation.
Generative Topographic Mapping: The GTM Model; Model Creation and Initialisation; Computing Probabilities; EM Algorithm; Magnification Factors; Demonstration Programs.
Topographic Projection: Neuroscale Algorithm; Neuroscale Implementation; Demonstration of Neuroscale.
Worked Example: Canonical Variates.
• Chapter 8: Sampling

Monte Carlo Integration
Basic Sampling: Random Number Generators; Transformation Methods; Rejection Sampling; Importance Sampling.
Markov Chain Sampling: Markov Chain Fundamentals; Gibbs Sampling; Metropolis-Hastings Algorithm; Hybrid Monte Carlo.
Demonstration Programs: Metropolis-Hastings Sampling; Hybrid Monte Carlo Sampling.
Worked Example: Convergence Diagnostics
• Chapter 9: Bayesian Techniques

Principles of Bayesian Inference
Priors for Neural Networks: Theory of Priors; Netlab Implementation of Priors; Demonstration of Gaussian Priors; Masks and Weight Manipulation.
Computing Error and Gradient Functions
The Evidence Procedure: Theory; Netlab Implementation.
Predictions and Error Bars: Predictions for Regression; Predictions for Classification; Netlab Implementation.
Demonstrations of Evidence Procedure: The Evidence Procedure for Regression; The Evidence Procedure for Classification; Automatic Relevance Determination.
Monte Carlo Methods
Demonstration of Hybrid Monte Carlo for MLPs
Worked Example: Improved Classification Approximation
• Chapter 10: Gaussian Processes

Bayesian Regression: Weight Space View; Function Space View.
Theory of Gaussian Processes: Model Definition; Learning Hyperparameters.
Netlab Implementation: Model Creation and Initialisation; Making Predictions; Model Training.
Demonstration Programs: Regression Demonstration; ARD Demonstration.
Worked Example: GPs for Classification
• Appendices: Linear Algebra and Matrices; Algorithm Error Analysis.