MA40189: Topics in Bayesian statistics

 Lectures and timetable information

 Lecturer: Simon Shaw; s.shaw at bath.ac.uk Timetable: Lectures: Monday 14:15 (3W4.7) and Tuesday 09:15 (3W4.7). Problems classes: Thursday 16:15 (3W4.7).

The full unit timetable is available here. A schedule for the course is available here.

 Syllabus

 Credits: 6 Level: Masters Period: Semester 2 Assessment: EX 100% Other work: There will be weekly question sheets. These will be set and handed in during problems classes. Any work submitted by the hand-in deadline will be marked and returned to you. Full solutions to all exercises and general feedback sheets will be made available. Requisites: Before taking this unit you must take MA40092 (home-page). Description: Aims & Learning Objectives: Aims: To introduce students to the ideas and techniques that underpin the theory and practice of the Bayesian approach to statistics. Objectives: Students should be able to formulate the Bayesian treatment and analysis of many familiar statistical problems. Content: Bayesian methods provide an alternative approach to data analysis, which has the ability to incorporate prior knowledge about a parameter of interest into the statistical model. The prior knowledge takes the form of a prior (to sampling) distribution on the parameter space, which is updated to a posterior distribution via Bayes' Theorem, using the data. Summaries about the parameter are described using the posterior distribution. The Bayesian Paradigm; decision theory; utility theory; exchangeability; Representation Theorem; prior, posterior and predictive distributions; conjugate priors. Tools to undertake a Bayesian statistical analysis will also be introduced. Simulation based methods such as Markov Chain Monte Carlo and importance sampling for use when analytical methods fail.

 Some useful books

We won't follow a book as such but useful references, in ascending order of difficulty, include:

1. Peter M. Lee, Bayesian Statistics: an introduction, Fourth Edition, 2012.
Very readable, introductory text. This edition is not in the library but the full text is available as an e-book here. The Third Edition is available in the library. Further details about the book can be found on Peter Lee's pages here. This includes all the exercises in the book and their solutions.
2. Andrew Gelman, John B. Carlin, Hal S. Stern and Donald B. Rubin, Bayesian Data Analysis, Second Edition, 2004. 512.795 GEL
A slightly more advanced introductory text with a focus upon practical applications. The full text is available as an e-book, either by following the link from the library here or directly here. Further details about the book can be found on Andrew Gelman's pages here. This includes some of the solutions to exercises in the book. Andrew Gelman also has a blog which often raises some interesting statistical topics, frequently related to current news topics.
3. Christian P. Robert, The Bayesian Choice From Decision-Theoretic Foundations to Computational Implementation, Second Edition, 2007.
This book is not in the library but the full text is available here. A really nice book; Christian Robert also has a wide-ranging blog.
4. Anthony O'Hagan, Kendall's Advanced Theory of Statistics Volume 2B Bayesian Inference, 1994. 512.795 KEN
A harder, more advanced, book than the previous two but a rewarding and insightful one. It has a good mix of theory and foundations: a personal favourite.
5. Jose M. Bernardo and Adrian F.M. Smith, Bayesian Theory, 1994. 512.795 BER
The classic graduate text. Develops the Bayesian view from a foundational standpoint. A very readable short overview to Bayesian statistics written by Jose Bernardo can be downloaded from here.

The International Society for Bayesian Analysis (ISBA) is a good starting point for a number of Bayesian resources.
 Lecture notes and summaries

Lecture notes: pdf.
Table of useful distributions: pdf (Handed out in Problems Class of 16 Feb 17)

Material covered:
 Lecture 1 (06 Feb 17): Introduction: working definitions of classical and Bayesian approaches to inference about parameters. Lecture overview: pdf. Handwritten notes: pdf. Online notes: p4-5 (middle of Example 3). Lecture 2 (07 Feb 17): §1 The Bayesian method: Bayes' theorem, using Bayes' theorem for parametric inference. Lecture overview: pdf. Handwritten notes: pdf. Online notes: p5 (middle of Example 3)-7 (prior to equation (1.6)). Lecture 3 (13 Feb 17): Sequential data updates, conjugate Bayesian updates, Beta-Binomial example. Lecture overview: pdf. Handwritten notes: pdf. Online notes: p7 (prior to equation (1.6))-9 (equation (1.12)). Lecture 4 (14 Feb 17): Definition of conjugate family, role of prior (weak and strong) and likelihood in the posterior. Handout of beta distributions: pdf. Lecture overview: pdf. Handwritten notes: pdf. Online notes: p9 (equation (1.12))-10 (Example 5). Lecture 5 (20 Feb 17): Example of weak/strong prior finished, kernel of a density, conjugate Normal example. Handout of weak/strong prior example: pdf. Lecture overview: pdf. Handwritten notes: pdf. Online notes: p10 (Example 5)-13 (equation (1.16)). Lecture 6 (21 Feb 17): Using the posterior for inference, credible interval, highest density regions. Lecture overview: pdf. Handwritten notes: pdf. Online notes: p13 (equation (1.16))-15 (end of Example 9). Lecture 7 (27 Feb 17): §2 Modelling: predictive distribution, Binomial-Beta example. Lecture overview: pdf. Handwritten notes: pdf. Online notes: p15 (end of Example 9)-19 (equation (2.4)). Lecture 8 (28 Feb 17): Predictive summaries, finite exchangeability. Lecture overview: pdf. Handwritten notes: pdf. Online notes: p19 (equation (2.4))-21 (end of Example 14). Lecture 9 (06 Mar 17): Infinite exchangeability, example of non-extendibility of finitely exchangeable sequence, general representation theorem for infinitely exchangeable events and random quantities. Lecture overview: pdf. Handwritten notes: pdf. Online notes: p21 (end of Example 14)-23 (prior to Theorem 2). Lecture 10 (07 Mar 17): Example of exchangeable Normal random quantities, sufficiency. Lecture overview: pdf. Handwritten notes: pdf. Online notes: p23 (prior to Theorem 2)-25 (start of Section 2.3). Lecture 11 (13 Mar 17): k-parameter exponential family, sufficient statistics, conjugate priors for exchangeable k-parameter exponential family random quantities. Lecture overview: pdf. Handwritten notes: pdf. Online notes: p25 (start of Section 2.3)-27 (equation (2.23)). Lecture 12 (14 Mar 17): Hyperparameters, usefulness of conjugate priors. Lecture overview: pdf. Handwritten notes: pdf. Online notes: p27 (equation (2.23))-29 (start of Section 2.4). Lecture 13 (20 Mar 17): Improper priors, Fisher information matrix, Jeffreys' prior. Lecture overview: pdf. Handwritten notes: pdf. Online notes: p29 (start of Section 2.4)-32 (after Example 25). Lecture 14 (21 Mar 17): Invariance property under transformation of the Jeffreys prior, final remarks about noninformative priors, §3 Computation: preliminary issues. Lecture overview: pdf. Handwritten notes: pdf. Online notes: p32 (after Example 25)-36 (prior to Section 3.1). Lecture 15 (27 Mar 17): Normal approximation, expansion about the mode, Monte Carlo integration. Lecture overview: pdf. Handwritten notes: pdf. Online notes: p36 (prior to Section 3.1)-38 (prior to Section 3.2.2). Lecture 16 (28 Mar 17): Importance sampling. Basic idea of Markov chain Monte Carlo (MCMC): transition kernel. Handout: pdf. Lecture overview: pdf. Handwritten notes: pdf. Online notes: p38 (prior to Section 3.2.2)-40 (equation (3.6)). Lecture 17 (03 Apr 17): Basic definitions (irreducible, periodic, recurrent, ergodic, stationary) and theorems (existence/uniqueness, convergence, ergodic) of Markov chains and their consequences for MCMC techniques. The Metropolis-Hastings algorithm. Lecture overview: pdf. Handwritten notes: pdf. Online notes: p40 (equation (3.6))-43 (prior to equation (3.9)). Lecture 18 (04 Apr 17): Example of the Metropolis-Hastings algorithm. Handout of example: pdf. Lecture overview: pdf. Handwritten notes: pdf. Online notes: p43 (prior to equation (3.9))-52 (start of Section 3.3.3). Lecture 19 (06 Apr 17): The Gibbs sampler algorithm and example. Lecture overview: pdf. Handwritten notes: pdf. Online notes: p52 (start of Section 3.3.3)-53 (prior to Example 32). Lecture 19A (24 Apr 17): Gibbs sampler example concluded. Handout of example: pdf. Lecture overview: pdf. Handwritten notes: pdf. Online notes: p53 (prior to Example 32)-59 (prior to Section 3.3.4). Lecture 20 (25 Apr 17): Overview of why the Metropolis-Hastings algorithm works, efficiency of MCMC algorithms. Lecture overview: pdf. Handwritten notes: pdf. Online notes: p59 (prior to Section 3.3.4)-61 (after bullet point 2). Lecture 21 (27 Apr 17): §4 Decision theory: Statistical decision theory: loss, risk, Bayes risk and Bayes rule. Lecture overview: pdf. Handwritten notes: pdf. Online notes: p61 (after bullet point 2)-65 (beginning of Example 36). Note: Section 4.1 Utility was omitted and is not required for the exam. Lecture 22 (02 May 17): Quadratic loss, Bayes risk of the sampling procedure, worked example . Lecture overview: pdf. Handwritten notes: pdf. Online notes: p65 (beginning of Example 36)-67 (after equation (4.7)). Lecture 23 (04 May 17): Worked example finished. Lecture overview: pdf. Handwritten notes: pdf. Online notes: p67 (after equation (4.7))-69 (end of notes).

Forthcoming material:
This course is now completed.

 R functions

 gibbs2 Gibbs sampler for θ1|θ2 ~ Bin(n, θ2), θ2|θ1 ~ Beta(θ1+α, n-θ1+β), illustration of Example 32 (p54) of the lecture notes. Sample plots for n=10, α=β=1 and n=10, α=2, β=3: pdf.

 gibbs.update Step by step illustration of the Gibbs sampler for bivariate normal, X, Y standard normal with Cov(X, Y) = rho; press return to advance. gibbs Long run version of the Gibbs sampler for bivariate normal, X, Y standard normal with Cov(X, Y) = rho. metropolis.update Step by step illustration of Metropolis-Hastings for sampling from N(mu.p,sig.p^2) with proposal N(theta[t-1],sig.q^2); press return to advance. metropolis Long run version of Metropolis-Hastings for sampling from N(mu.p,sig.p^2) with proposal N(theta[t-1],sig.q^2). Illustration of Example 30 (p45) of the lecture notes using metropolis.update and metropolis with mu.p=0, sig.p=1 and firstly sig.q=1 and secondly sig.q=0.6: pdf. plot.mcmc Plot time series summaries of output from a Markov chain. Allows you to specify burn-in and thinning. f Function for plotting bivariate Normal distribution in gibbs.update. All above All of the above functions in one file for easy reading into R; thanks to Ruth Salway for these functions.

The following functions are for sampling from bivariate normals, with thanks to Merrilee Hurn

 gibbs1 Gibbs sampler (arguments: n the number of iterations, rho the correlation coefficient of the bivariate normal, start1 and start2 the initial values for the sampler). metropolis1 Metropolis-Hastings (arguments: n the number of iterations, rho the correlation coefficient of the bivariate normal, start1 and start2 the initial values for the sampler, tau the standard deviation of the Normal proposal). metropolis2 Metropolis-Hastings for sampling from a mixture of bivariate normals (arguments: n the number of iterations, rho the correlation coefficient of the bivariate normal, start1 and start2 the initial values for the sampler, tau the standard deviation of the Normal proposal, sigma2 the variance of the normal mixtures).

 Question sheets and solutions

Question sheets will be set in the Thursday problems class. They will appear here with full worked solutions available shortly after the submission date.

 Problems 0 (09 Feb 17): Question Sheet Zero: pdf. Solution Sheet Zero: pdf. Handwritten notes: pdf. Problems 1 (16 Feb 17): Question Sheet One: pdf. Solution Sheet One: pdf. Handwritten notes: pdf. Problems 2 (23 Feb 17): Question Sheet Two: pdf. Solution Sheet Two: pdf. Handwritten notes: pdf. Problems 3 (02 Mar 17): Question Sheet Three: pdf. Solution Sheet Three: pdf. Handwritten notes: pdf. Problems 4 (09 Mar 17): Question Sheet Four: pdf. Solution Sheet Four: pdf. Handwritten notes: pdf. Problems 5 (16 Mar 17): Question Sheet Five: pdf. Solution Sheet Five: pdf. Handwritten notes: pdf. Problems 6 (23 Mar 17): Question Sheet Six: pdf. Solution Sheet Six: pdf. Handwritten notes: pdf. Problems 7 (30 Mar 17): Question Sheet Seven: pdf. Solution Sheet Seven: pdf. Handwritten notes: pdf. Problems 8 (24 Apr 17): Question Sheet Eight: pdf. Solution Sheet Eight: pdf. Handwritten notes: pdf. Problems 9 (04 May 17): Question Sheet Nine: pdf. Solution Sheet Nine: pdf.

 Past exam papers and solutions

The exam is two hours long and contains four questions, each worth 20 marks. Full marks will be given for correct answers to three questions. Only the best three answers will contribute towards the assessment. The exam is thus marked out of 60. In the exam you will be given the table of useful distributions: pdf.

 Exams: 2015/16 2014/15 2013/14 2012/13 2011/12 Paper: pdf pdf pdf pdf pdf Solutions: pdf pdf pdf pdf pdf

 Last revision: 08/05/17 University home Mathematics home Mathematics staff Top of page