Professional Documents
Culture Documents
What is Data?
Data is a collection of facts, such as values or measurements. It can be numbers, words, measurements, observations or even just descriptions of things.
Qualitative vs Quantitative
Data can be qualitative or quantitative. Qualitative data is descriptive information (it describes something) Quantitative data, is numerical information (numbers).
And Quantitative data can also be Discrete or Continuous: Discrete data can only take certain values (like whole numbers) Continuous data can take any value (within a range)
Put simply: Discrete data is counted, Continuous data is measured To help you remember think "Quantitative is about Quantity"
Collecting
Data can be collected in many ways. The simplest way is direct observation. Example: you want to find how many cars pass by a certain point on a road in a 10minute interval. So: simply stand at that point on the road, and count the cars that pass by in that interval. You collect data by doing a Survey.
Census or Sample
A Census is when you collect data for every member of the group (the whole "population"). A Sample is when you collect data just for selected members of the group. Example: there are 120 people in your local football club. You can ask everyone (all 120) what their age is. That is a census. Or you could just choose the people that are there this afternoon. That is a sample. A census is accurate, but hard to do. A sample is not as accurate, but may be good enough, and is a lot easier.
Language
Data or Datum?
The singular form is "datum", so we would say "that datum is very high". "Data" is the plural so we can say "the data are available", but it is also a collection of facts, so "the data is available" is fine too.
Discrete Data
Discrete Data can only take certain values. Example: the number of students in a class (you can't have half a student).
Continuous Data
Continuous Data is data that can take any value (within a range) Examples: A person's height: could be any value (within the range of human heights), not just certain fixed heights, Time in a race: you could even measure it to fractions of a second, A dog's weight, The length of a leaf, Lots more!
2 Numbers
With just 2 numbers the answer is easy: go half-way in-between.
You can calculate it by adding 3 and 7 and then dividing the result by 2:
(3+7) / 2 = 10/2 = 5
3 or More Numbers
You can use the same idea when you have 3 or more numbers:
(3+7+8) / 3 = 18/3 = 6
The Mean
So far we have been calculating the Mean (or the Average):
(13+13+13+13+13+13+1+1+1+1+1) / 11 = 7.5...
The 13 year olds are embarrassed, and the 1-year olds can't jump!
The Mean was accurate, but in this case it was not useful.
The Median
But you could also use the Median: simply list all numbers in order and choose the middle one:
3, 4, 7, 9, 12, 15
So we average them:
(7+9) / 2 = 16/2 = 8
The Median is 8
The Mode
The Mode is the value that occurs most often:
Conclusion
There are other ways of measuring central values, but Mean, Median and Mode are the most common. Use the one that best suits your data. Or better still, use all three!
In other words it is the sum divided by the count. Example 1: What is the Mean of these numbers?
6, 11, 7
Add the numbers: 6 + 11 + 7 = 24 Divide by how many numbers (there are 3 numbers): 24 / 3 = 8
The Mean is 8
3, 7, 5, 13, 20, 23, 39, 23, 40, 23, 14, 12, 56, 23, 29
The sum of these numbers is 330 There are fifteen numbers. The mean is equal to 330 / 15 = 22
Negative Numbers
How do you handle negative numbers? Adding a negative number is the same as subtracting the number (without the negative). For example 3 + (-2) = 3-2 = 1. Knowing this, let us try an example:
3, -7, 5, 13, -2
The sum of these numbers is 3 - 7 + 5 + 13 - 2 = 12 There are 5 numbers. The mean is equal to 12 5 = 2.4
Median Value
The Median is the "middle number" (in a sorted list of numbers).
3, 5, 12
The middle number is 5, so the median is 5.
Example 2
Look at these numbers:
3, 13, 7, 5, 21, 23, 39, 23, 40, 23, 14, 12, 56, 23, 29
If we put those numbers in order we have:
3, 5, 7, 12, 13, 14, 21, 23, 23, 23, 23, 29, 39, 40, 56
There are fifteen numbers. Our middle number will be the eighth number:
3, 5, 7, 12, 13, 14, 21, 23, 23, 23, 23, 29, 39, 40, 56
The median value of this set of numbers is 23.
(Note that it didn't matter if we had some numbers the same in the list)
3, 13, 7, 5, 21, 23, 23, 40, 23, 14, 12, 56, 23, 29
If we put those numbers in order we have:
3, 5, 7, 12, 13, 14, 21, 23, 23, 23, 23, 29, 40, 56
There are now fourteen numbers and so we don't have just one middle number, we have a pair of middle numbers:
3, 5, 7, 12, 13, 14, 21, 23, 23, 23, 23, 29, 40, 56
In this example the middle numbers are 21 and 23. To find the value half-way between them, add them together and divide by 2:
21 + 23 = 44 44 2 = 22
Your Turn
Remember: sort them first (by dragging them left or right) !
View Larger
Example: 3, 7, 5, 13, 20, 23, 39, 23, 40, 23, 14, 12, 56, 23, 29
In order these numbers are:
3, 5, 7, 12, 13, 14, 20, 23, 23, 23, 23, 29, 39, 40, 56
This makes it easy to see which numbers appear most often.
In this case the mode is 23. Another Example: {19, 8, 29, 35, 19, 28, 15}
Arrange them in order: {8, 15, 19, 19, 28, 29, 35} 19 appears twice, all the rest appear only once, so 19 is the mode. How to remember? Think "mode is most"
Example: {1, 3, 3, 3, 4, 4, 6, 6, 6, 9}
3 appears three times, as does 6. So there are two modes: at 3 and 6 Having two modes is called "bimodal". Having more than two modes is called "multimodal".
Grouping
When all values appear the same number of times the idea of a mode is not useful. But you could group them to see if one group has more than the others.
6, 11, 7
Add the numbers: 6 + 11 + 7 = 24 Divide by how many numbers (there are 3 numbers): 24 3 = 8
The Mean is 8
But sometimes you won't have a simple list of numbers, you might have a frequency table like this (the "frequency" says how often they occur): Score 1 2 3 4 5 Frequency 2 5 4 2 1
(it says that score 1 occurred 2 times, score 2 occurred 5 times, etc) You could list all the numbers like this:
Mean =
But rather than do lots of adds (like 3+3+3+3) it is often easier to use multiplication: 21 + 52 + 43 + 24 + 15 (how many numbers)
Mean =
And rather than count how many numbers there are, we can add up the frequencies: 21 + 52 + 43 + 24 + 15 2+5+4+2+1
Mean =
So let's calculate: 2 + 10 + 12 + 8 + 5 14 37 14
Mean =
= 2.64...
And that is how to calculate the mean from a frequency table! Here is another example:
Parking Spaces
1 2 3 4
Frequency
15 27 8 5
What is the mean number of Parking Spaces? Answer: 151 + 272 + 83 + 54 Mean = 15+27+8+5 = 55 15+54+24+20 = 2.05...
Notation
Now you know how to do it, let's do that last example again, but using formulas. This symbol (called Sigma) means "sum up" (read more at Sigma Notation) So we can say "add up all frequencies" this way:
f is frequency)
(where
And the formula for calculating the mean from a frequency table is:
The
So now we are ready to do our example above, but with correct notation.
x
1 2 3 4 And here it is:
f
15 27 8 5
Example: (continued)
From the previous example, calculate totals:
x
1 2 3 4 TOTALS: And the Mean is then easy:
fx
15 54 24 20 113
Range of a Function
Range can also mean all the output values of a function, seeDomain, Range and Codomain.
Quartiles
Quartiles are the values that divide a list of numbers into quarters. First put the list of numbers in order Then cut the list into four equal parts The Quartiles are at the "cuts"
Like this:
Example: 5, 8, 4, 4, 6, 3, 8
Put them in order: 3, 4, 4, 5, 6, 8, 8 Cut the list into quarters:
And the result is: Quartile 1 (Q1) = 4 Quartile 2 (Q2), which is also the Median, = 5 Quartile 3 (Q3) = 8
Sometimes a "cut" is between two numbers ... the Quartile is the average of the two numbers.
Example: 1, 3, 3, 4, 5, 6, 6, 7, 8, 8
The numbers are already in order Cut the list into quarters:
Q2 = (5+6)/2 = 5.5
And the result is: Quartile 1 (Q1) = 3 Quartile 2 (Q2) = 5.5 Quartile 3 (Q3) = 7
Interquartile Range
The "Interquartile Range" is from Q1 to Q3:
Example:
Example: Box and Whisker Plot and Interquartile Range for 4, 17, 7, 14, 18, 12, 3, 16, 10, 4, 4, 11
Put them in order:
Q3 - Q1 = 15 - 4 = 11
Standard Deviation
The Standard Deviation is a measure of how spread out numbers are. Its symbol is
The formula is easy: it is the square root of the Variance. So now you ask, "What is the Variance?"
Variance
The Variance is defined as: The average of the squared differences from the Mean. To calculate the variance follow these steps: Work out the Mean (the simple average of the numbers)
Then for each number: subtract the Mean and square the result
Example
You and your friends have just measured the heights of your dogs (in millimeters):
The heights (at the shoulders) are: 600mm, 470mm, 170mm, 430mm and 300mm. Find out the Mean, the Variance, and the Standard Deviation. Your first step is to find the Mean:
Answer:
Mean = 600 + 470 + 170 + 430 + 300 5 = 1970 5 = 394
so the mean (average) height is 394 mm. Let's plot this on the chart:
To calculate the Variance, take each difference, square it, and then average the result:
So, the Variance is 21,704. And the Standard Deviation is just the square root of Variance, so:
And the good thing about the Standard Deviation is that it is useful. Now we can show which heights are within one Standard Deviation (147mm) of the Mean:
So, using the Standard Deviation we have a "standard" way of knowing what is normal, and what is extra large or extra small.
Rottweilers are tall dogs. And Dachshunds are a bit short ... but don't tell them! Now try the Standard Deviation Calculator.
Sample Variance = 108,520 / 4 = 27,130 Sample Standard Deviation = 27,130 = 164 (to the nearest mm)
Think of it as a "correction" when your data is only a sample.
Formulas
Here are the two formulas, explained at Standard Deviation Formulas if you want to know more:
Looks complicated, but the important change is to divide by N-1 (instead of N) when calculating a Sample Variance.
4+4-4-4 =0 4
4+4+4+4 =4 4
7+1+6+2 =4 4
Oh No! It also gives a value of 4, Even though the differences are more spread out! So let us try squaring each difference (and taking the square root at the end):
42 + 42 + 42 + 42 4
64 =4 4
72 + 12 + 62 + 22 4
90 = 4.74... 4
That is nice! The Standard Deviation is bigger when the differences are more spread out ... just what we want! In fact this method is a similar idea to distance between points, just applied in a different way. And it is easier to use algebra on squares and square roots than absolute values, which makes the standard deviation easy to use in other areas of mathematics.
Normal Distribution
Data can be "distributed" (spread out) in different ways.
But there are many cases where the data tends to be around a central value with no bias left or right, and it gets close to a "Normal Distribution" like this:
A Normal Distribution
The "Bell Curve" is a Normal Distribution. And the yellow histogram shows some data that follows it closely, but not perfectly (which is usual).
Many things closely follow a Normal Distribution: heights of people size of things produced by machines errors in measurements blood pressure marks on a test
The Normal Distribution has: mean = median = mode symmetry about the center 50% of values less than the mean
Quincunx
You can see a normal distribution being created by random chance! It is called the Quincunx and it is an amazing machine. Have a play with it!
Standard Deviations
The Standard Deviation is a measure of how spread out numbers are (read that page for details on how to calculate it). When you calculate the standard deviation of your data, you will find that (generally):
Example: 95% of students at school are between 1.1m and 1.7m tall.
Assuming this data is normally distributed can you calculate the mean and standard deviation? The mean is halfway between 1.1m and 1.7m:
95% is 2 standard deviations either side of the mean (a total of 4 standard deviations) so:
It is good to know the standard deviation, because we can say that any value is: likely to be within 1 standard deviation (68 out of 100 will be) very likely to be within 2 standard deviations (95 out of 100 will be) almost certainly within 3 standard deviations (997 out of 1000 will be)
Standard Scores
The number of standard deviations from the mean is also called the "Standard Score", "sigma" or "z-score". Get used to those words!
You can see on the bell curve that 1.85m is 3 standard deviations from the mean of 1.4, so:
It is also possible to calculate how many standard deviations 1.85 is from the mean How far is 1.85 from the mean? It is 1.85 - 1.4 = 0.45m from the mean How many standard deviations is that? The standard deviation is 0.15m, so: 0.45m / 0.15m = 3 standard deviations
So to convert a value to a Standard Score ("z-score"): first subtract the mean, then divide by the Standard Deviation
You can take any Normal Distribution and convert it to The Standard Normal Distribution.
first subtract the mean: 26 - 38.8 = -12.8, then divide by the Standard Deviation: -12.8/11.4 = -1.12
So 26 is -1.12 Standard Deviations from the Mean Here are the first three conversions Original Value 26 33 65 ... And here they are graphically: Calculation (26-38.8) / 11.4 = (33-38.8) / 11.4 = (65-38.8) / 11.4 = ... Standard Score (z-score) -1.12 -0.51 +2.30 ...
Here is the formula for z-score that we have been using: z is the "z-score" (Standard Score) x is the value to be standardized is the mean is the standard deviation
In More Detail
Here is the Standard Normal Distribution with percentages for every half of a standard deviation, and cumulative percentages:
Example: Your score in a recent test was 0.5 standard deviations above the average, how many people scored lower than you did? Between 0 and 0.5 is 19.1% Less than 0 is 50% (left half of the curve) So the total less than you is:
Some values are less than 1000g ... can you fix that?
31% of the bags are less than 1000g, which is cheating the customer!
Because it is a random thing we can't stop bags having less than 1000g, but we can reduce it a lot ... if 1000g was at -3 standard deviations there would be only 0.1% (very
small)
at -2.5 standard deviations we can calculate:
below 3 is 0.1% and between 3 and 2.5 standard deviations is 0.5%, together that is 0.1%+0.5% = 0.6% (a good choice I think)
So let us adjust the machine to have 1000g at 2.5 standard deviations from the mean. We could adjust it to: increase the amount of sugar in each bag (this would change the mean), or make it more accurate (this would reduce the standard deviation)
10g / 2.5 = 4g
So the standard deviation should be 4g, like this: (We hope the machine is that accurate!)
Or perhaps we could have some combination of better accuracy and slightly larger average size, I will leave that up to you!
Correlation
When two sets of data are strongly linked together we say they have a High Correlation. The word Correlation is made of Co- (meaning "together"), and Relation Correlation is Positive when the values increase together, and Correlation is Negative when one value decreases as the other increases
Like this:
1 is a perfect positive correlation 0 is no correlation (the values don't seem linked at all) -1 is a perfect negative correlation
The value shows how good the correlation is (not how steep the line is), and if it is positive or negative.
$408
You can easily see that warmer weather leads to more sales, the relationship is good but not perfect.
In fact the correlation is 0.9575 ... see at the end how I calculated it.
Moral of the story: make a Scatter Plot, and look at it! You may see more than the correlation value says.
How To Calculate
How did I calculate the value 0.9575 at the top? I used "Pearson's Correlation". There is software that can calculate it for you, such as the CORREL() function in Excel or OpenOffice Calc ...
Here is how I calculated the first Ice Cream example (values rounded to 1 or 0 decimal places):
As a formula it is:
Where:
You probably won't have to calculate it like that, but at least you know it is not "magic", but simply a routine set of calculations.
Approximate Values
There are also approximate ways to calculate a correlation coefficient, such as "Spearman's rank correlation coefficient", but I prefer using a spreadsheet like above.
Skewed Data
Data can be "skewed", meaning it tends to have a long tail on one side or the other:
Negative Skew
No Skew
Positive Skew
Negative Skew?
Why is it called negative skew? Because the long "tail" is on the negative side of the peak. People sometimes say it is "skewed to the left" (the long tail is on the left hand side) The mean is also on the left of the peak.
Positive Skew
And positive skew is when the long tail is on the positive side of the peak, and some people say it is "skewed to the right". The mean is on the right of the peak value.
Calculating Skewness
"Skewness" (the amount of skew) can be calculated, for example you could use the SKEW() function in Excel or OpenOffice Calc.
The Range
The range is the most obvious measure of dispersion and is the difference between the lowest and highest values in a dataset. In figure 1, the size of the largest semester 1 tutorial group is 6 students and the size of the smallest group is 4 students, resulting in a range of 2 (6-4). In semester 2, the largest tutorial group size is 7 students and the smallest tutorial group contains 3 students, therefore the range is 4 (7-3).
The range is simple to compute and is useful when you wish to evaluate the whole of a dataset. The range is useful for showing the spread within a dataset and for comparing the spread between similar datasets. An example of the use of the range to compare spread within datasets is provided in table 1. The scores of individual students in the examination and coursework component of a module are shown.
To find the range in marks the highest and lowest values need to be found from the table. The highest coursework mark was 48 and the lowest was 27 giving a range of 21. In the examination, the highest mark was 45 and the lowest 12 producing a range of 33. This indicates that there was wider variation in the students performance in the examination than in the coursework for this module. Since the range is based solely on the two most extreme values within the dataset, if one of these is either exceptionally high or low (sometimes referred to as outlier) it will result in a range that is not typical of the variability within the dataset. For example, imagine in the above example that one student failed to hand in any coursework and was awarded a mark of zero, however they sat the exam and scored 40. The range for the coursework marks would now become 48 (48-0), rather than 21, however the new range is not typical of the dataset as a whole and is distorted by the outlier in the coursework marks. In order to reduce the problems caused by outliers in a dataset, the inter-quartile range is often calculated instead of the range.
quarters along the dataset. Therefore, the upper quartile lies half way between the median and the highest value in the dataset whilst the lower quartile lies halfway between the median and the lowest value in the dataset. The inter-quartile range is found by subtracting the lower quartile from the upper quartile. For example, the examination marks for 20 students following a particular module are arranged in order of magnitude.
The median lies at the mid-point between the two central values (10th and 11th) = half-way between 60 and 62 = 61 The lower quartile lies at the mid-point between the 5th and 6th values = half-way between 52 and 53 = 52.5 The upper quartile lies at the mid-point between the 15th and 16th values = half-way between 70 and 71 = 70.5 The inter-quartile range for this dataset is therefore 70.5 - 52.5 = 18 whereas the range is: 80 - 43 = 37. The inter-quartile range provides a clearer picture of the overall dataset by removing/ignoring the outlying values. Like the range however, the inter-quartile range is a measure of dispersion that is based upon only two values from the dataset. Statistically, the standard deviation is a more powerful measure of dispersion because it takes into account every value in the dataset. The standard deviation is explored in the next section of this guide.
The method Excel uses to calculate quartiles is not commonly used and tends to produce unusual results particularly when the dataset contains only a few values. For this reason you may be best to calculate the inter-quartile range by hand.
If the mean of a dataset is 25 and its standard deviation is 1.6, then 68% of the values in the dataset will lie between MEAN-1SD (25-1.6=23.4) andMEAN+1SD (25+1.6=26.6) 99% of the values will lie between MEAN-3SD (25-4.8=20.2) and MEAN+3SD(25+4.8=29.8). If the dataset had the same mean of 25 but a larger standard deviation (for example, 2.3) it would indicate that the values were more dispersed. The frequency distribution for a dispersed dataset would still show a normal distribution but when plotted on a graph the shape of the curve will be flatter as in figure 4.
Where x represents each value in the population, is the mean value of the population, is the summation (or total), and N is the number of values in the population. The standard deviation of a sample is known as S and is calculated using:
Where x represents each value in the population, x is the mean value of the sample, is the summation (or total), and n-1 is the number of values in the sample minus 1.
The same data can be used to demonstrate how to calculate the sample standard deviation in Excel. In this case, imagine that the data in column A represent the kilometres per litre found for a sample of 5 new cars tested by the manufacturer. The population standard deviation is calculated using =STDEV(VALUES) and in this case the command is =STDEV(A2:A6) which produces an answer of 0.55. The sample standard deviation will always be greater than the population standard deviation when they are calculated for the same dataset. This is because the formula for the sample standard deviation has to take into account the possibility of there being more variation in the true population than has been measured in the sample. Based on their sample of 5 cars, and therefore using the sample standard deviation, the manufacturers could state with 99% confidence that similar cars will do between 14.57(MEAN-3SD) and 17.87 (MEAN+3SD) kilometres per litre . These examples show the quick method of calculating standard deviations using a cell range. Each of the commands can also be written out in a longer format with the individual kilometres/litre entered. For example entering: =STDEV(16.13,16.40,15.81,17.07,15.69) produces an identical result to=STDEV(A2:A6). However, if one of the values in column A was found to be incorrect and adjusted, the cell range method would automatically update the calculation of the standard deviation whereas the longer format will require manual adjustment of the command. Further information about using Excel to perform calculations can be found here.
Summary
The range, inter-quartile range and standard deviation are all measures that indicate the amount of variability within a dataset. The range is the simplest measure of variability to calculate but can be misleading if the dataset contains extreme values. The inter-quartile range reduces this problem by considering the variability within the middle 50% of the dataset. The standard deviation is the most robust measure of variability since it takes into account a measure of how every value in the dataset varies from the mean. However, care must be taken when calculating the standard deviation to consider whether the entire population or a sample is being examined and to use the appropriate formula.