Applied Statistics - Week 3

Monday the 5th - Friday the 9th of December 2022

The following is a description of what we will go through during this week of the course. The chapter references and computer exercises are considered read, understood, and solved by the beginning of the following class, where I'll shortly go through the exercise solution.

General notes, links, and comments:
The project groups (Version 1.0, Friday the 25th of November) can be found here: AS2022_LABscheduleV4_30nov.pdf.
The overview of where to show up (and later do experiments) can be found here: AS2022_NBIoverview_ExperimentalLocations.pdf.

Monday:
Experiments for project: (Group A)
We will be working on the experiments for Project in First Lab.
This project should be handed in (on Absalon) by 22:00 on Wednesday the 14th of December 2022 (please, don't sit up all night!).
I would be very happy, if you would give the file the logical name "Project_GroupX_Name1Name2Name3Name4Name5.pdf", where NameX is the first name of the group members.

Lectures and exercises: (Group B)
Real data almost never follows theoretical PDFs, as the real world contains dirty wires, unknown biases, and mismeasurements. We will devote the day to discussion of real data analysis and systematic errors, and apply this to our "Table Measurements" from Aud. A.

Reading:
  • Barlow, chapter 4.2.3 (short and very good!) and 4.4
  • Note on rejecting data using Chauvenet's criteria.
  • Optionally, Wikipedia also covers Chauvenet's Criterion and the more modern Peirce's Criterion.
    Podcast:
  • Systematic Uncertainties.
    Lecture(s):
  • Systematic Errors.
  • Trial Factors.
    Zoom: Link to lecture.
  • Recording of Lecture video.
    Computer Exercise(s):
  • TableMeasurements: TableMeasurement_original.ipynb,
    data_TableMeasurements2009.txt
    data_TableMeasurements2010.txt
    data_TableMeasurements2011.txt
    data_TableMeasurements2012.txt
    data_TableMeasurements2013.txt
    data_TableMeasurements2014.txt
    data_TableMeasurements2015.txt
    data_TableMeasurements2016.txt
    data_TableMeasurements2017.txt
    data_TableMeasurements2018.txt
    data_TableMeasurements2019.txt
    data_TableMeasurements2020.txt
    data_TableMeasurements2021.txt
    data_TableMeasurements2022.txt



    Tuesday:
    We will consider Monte Carlo Techniques, which is a ubiquitious tool in statistics. The central point is to be able to generate random numbers according to any given distribution, and subsequently use this.
    I'll also shortly comment on calculating a ChiSquare between two histograms and the Minuit output.

    Reading:
    Interestingly, Barlow does not cover this important area, but there are fortunately plenty of other references:
  • Glen Cowan: Chapter 3 (highly recommended!).
  • Wiki on transformation method.
  • Wiki on rejection sampling (Von Neumann) method.
  • Particle Data Group (PDG) note on Monte Carlo generators (optional - extends Cowan's chapter 3).
    Lecture(s):
  • Monte Carlo methods.
  • ChiSquare between histograms
    Zoom: Link to lecture.
  • Recording of Lecture video.
    Computer Exercise(s):
  • Making Random Numbers according to any distribution:
             For your illustration (with linear function): TransformationAcceptReject_simple.ipynb
             For your testing (with 3rd degree polynomial): TransformationAcceptReject_pol3.ipynb
             For your exercise (various functions): TransformationAcceptReject_general.ipynb


    Friday:
    This Friday will be the last morning lecture (for all) starting 8:15, and it will fall into two parts. The first part will be a summary of curriculum until now, where I will re-iterate on subjects, and answer questions about them. The second part will be on fitting strategies when faced with real data and more complicated functions including discontinuities.
    Finally, I'll be lecturing on types of data and ways of plotting, and we'll shortly discuss Simpson's Paradox, for which there is also an exercise (for those who want).

    Reading:
  • Barlow, chapter 5.3 to 5.7 (but not 5.5 and the proofs).
    Lecture(s):
  • Fitting and significance
  • Minuit output explained
  • Simpson's Paradox
    Zoom: Link to lecture.
  • Recording of Summary video.
  • Recording of Lecture video.
    Computer Exercise(s):
  • Fitting Danish Company Sizes: CompanySizes_original.ipynb
  • NBI Coffee Usage and Xmas vacation problem (discontinuous fitting): CoffeeUsage_original.ipynb (empty version)
  • Weighted Mean - and relation to ChiSquare: WeightedMeanSigmaChi2.ipynb (previously posted small exercise in preparation for project)
    Last updated: 7th of December 2022.