Applied Statistics - Week 4
Monday the 12th - Friday the 16th of December 2022
The following is a description of what we will go through during this
week of the course. The chapter references and computer exercises are
considered read, understood, and solved by the beginning of the
following class, where I'll shortly go through the exercise
solution.
General notes, links, and comments:
Louis Lyons discussing discovery levels::
1310.1284v1_LouisLyons_Why5sigma.pdf
Comparison between different tests for normality::
Power_Comparisons_of_Shapiro-Wilk_Kolmogorov-Smirn.pdf
Illustration of ROC curves: ROCcurves_GaussianSeparations.pdf
Animation of ROC curves: basic_animation.mp4
Comment on multiple hypothesis testing p-values::
p-value histogram
Paper of George Marsaglia on testing random numbers::
Random Number Generators
"Extraordinary claims require extraordinary evidence"
[P.S. Laplace (1814), Marcello Truzzi (1978), Carl Sagan (1980)]
This statement refers to the fact, that the business of hypothesis testing is to assign a probability of one hypothesis compared to an alternative. Whether or not the value of this probabilty is "enough" to make any claims, is up to circumstances and the individual case, as statistics does not provide any exact decision boundary... only guidelines (See Louis Lyons above).
“I believe in evidence. I believe in observation, measurement, and reasoning, confirmed by independent observers. I'll believe anything, no matter how wild and ridiculous, if there is evidence for it. The wilder and more ridiculous something is, however, the firmer and more solid the evidence will have to be.”
[Isaac Asimov, The Roving Mind]
Monday:
In the lecture, we will mainly focus on discussion of the TableMeasurement (in Aud. A),
which covers both the philosophy of data handling and analysis, and actually also the
construction of fits.
In the exercises we will work on the project experiment data, doing our best to answer
any final questions and consider possible experimental discrepancies.
Reading:
Barlow, chapter 7.2
Lecture(s):
Table Measurement Solution/Discussion
ChiSquare between histograms
Zoom: Link to lecture.
Recording of Lecture video.
Computer Exercise(s):
Work on the project data analysis (or previous exercises).
Tuesday:
The main theme of this week will be Hypothesis testing.
In addition to the ChiSquare test, there are several other tests, some
simple (one/two sample tests) and some more conceptually challenging
(Kolmogorov test, Wald-Wolfowitz runs test, and Anderson Darling's test).
We will start with an exercise gently introducing the subject.
Reading:
Barlow, chapter 8 on hypothesis testing (in particular 8.1-8.3).
Cohen, chapter 4 on hypothesis testing (perhaps omitting 4.2-4.4).
Lecture(s):
Coincidences
Hypothesis Testing
On p-value histograms
Zoom: Link to lecture.
Recording of Lecture video.
Computer Exercise(s):
Hypothesis testing: HypothesisTests_original.ipynb
Producing a ROC curve: MakeROCfigure.ipynb (for illustration)
Illustration/Animation of ROC curve (requires additional Python packages): MakeROCfigure_animation.ipynb
Friday:
The lecture will discuss how to concretely implement a hypothesis test.
The example data considered are the "random" numbers (most of) you gave
in the course questionnaire. Are these really random, and more importantly:
How would you test this?.
Following this lecture and discussion, the exercise we will exactly to implement
different tests for your own random (?) data, and determining which sample is
human.
Reading:
Same as for Tuesday (hypothesis testing).
Test inspiration from the Diehard Tests.
Lecture(s):
Testing random numbers
Confidence Intervals And Limits
Zoom: Link to lecture.
Recording of Lecture video.
Computer Exercise(s):
Random Digits Test: RandomDigitsTest_original.ipynb,
data_RandomDigits2022_A.txt,
data_RandomDigits2022_B.txt,
data_RandomDigits2022_C.txt,
data_RandomDigits2022_D.txt,
data_RandomDigits2022_E.txt,
data_RandomDigits2022_F.txt, and
data_RandomDigits2022_G.txt
For a large scale test, try one million digits of pi: pi1000000.txt
In order to see, if you can test individuals ability to produce randon numbers,
consider this data file (from 2017 - just to keep people anonymous):
PersonsDigitsForTest2017.txt
Last updated: 16th of December 2022.