Statistics for Data Science with Practice in Python

This is Day 9 of the 100 Days Data Science Bootcamp series, from noob to expert. See the full code on GitHub or the original write-up. Day 8 covered statistics in Python at a high level; this post works through the calculations directly.

Understanding and working with data is an essential skill for businesses, researchers, and professionals of all backgrounds. Python’s NumPy and pandas libraries provide a wide range of statistical and data analysis capabilities, covered in more depth in our NumPy 101 beginner’s guide. This article works through mean, median, mode, variance, and standard deviation, then moves into percentiles, quartiles, z-scores, and filling in missing values, all with runnable Python.

Mean:

The mean is the average value of a set of data. It is calculated by adding all the values in a set of data and then dividing by the number of values in the set. For example, if we have a set of data {1, 2, 3, 4, 5}, the mean would be (1 + 2 + 3 + 4 + 5) / 5 = 3.

# Importing libraries
import numpy as np

# Creating a sample data set
data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

# Mean
mean = np.mean(data)
print("Mean:", mean)

Mean: 5.5

Median:

The median is the middle value of a set of data when it is arranged in numerical order. If the set has an odd number of values, the median is the middle value. If the set has an even number of values, the median is the average of the two middle values. For example, if we have a set of data {1, 2, 3, 4, 5}, the median would be 3.

# Median
median = np.median(data)
print("Median:", median)

Median: 5.5

Mode:

The mode is the value that appears most frequently in a set of data. A set of data can have multiple modes or no mode at all. For example, if we have a set of data {1, 2, 2, 3, 4, 5}, the mode would be 2.

# Mode
import statistics as st
mode = st.mode(data)
print("Mode:", mode)

Mode: 1

Range:

The range is the difference between the highest and lowest values in a set of data. For example, if we have a set of data {1, 2, 3, 4, 5}, the range would be 5 - 1 = 4.

# Range
range = np.ptp(data)
print("Range:", range)

Range: 9

Variance:

The variance is a measure of how much the values in a set of data deviate from the mean. It is calculated by taking the sum of the squares of the differences between each value and the mean, and then dividing by the number of values in the set.

# Variance
variance = np.var(data)
print("Variance:", variance)

Variance: 8.25

Standard deviation:

The standard deviation is a measure of how spread out the values in a set of data are. It is calculated by taking the square root of the variance.

# Standard deviation
std_dev = np.std(data)
print("Standard deviation:", std_dev)

Standard deviation: 2.8722813232690143

Percentiles and quartiles:

Percentiles and quartiles are measures of the distribution of a set of data. A percentile is a value that separates a set of data into 100 equal parts. A quartile is a value that separates a set of data into 4 equal parts.

# Percentiles
percentile = np.percentile(data, [25, 50, 75])
print("25th percentile:", percentile[0])
print("50th percentile (Median):", percentile[1])
print("75th percentile:", percentile[2])

25th percentile: 3.25 50th percentile (Median): 5.5 75th percentile: 7.75

Z-scores:

A z-score is a measure of how far away a value is from the mean in terms of standard deviations. It shows up constantly outside plain descriptive stats too, including in the significance testing covered in our A/B testing and statistical significance guide. It is calculated by taking the difference between a value and the mean, and then dividing by the standard deviation.

# Z-scores
z_scores = (data - mean) / std_dev
print("Z-scores:", z_scores)

Z-scores: [-1.5666989 -1.21854359 -0.87038828 -0.52223297 -0.17407766 0.17407766 0.52223297 0.87038828 1.21854359 1.5666989 ]

Filling missing values and creating new columns

Real datasets rarely arrive complete. pandas gives you fillna() for missing values and plain assignment for new columns, both covered at length in our pandas guide with exercises:

import pandas as pd

df = pd.DataFrame({"score": [88, None, 76, 91, None]})

# Fill missing values with the column mean
df["score"] = df["score"].fillna(df["score"].mean())

# Add a new column derived from an existing one
df["z_score"] = (df["score"] - df["score"].mean()) / df["score"].std()

print(df)

Filling with the mean is the simplest option, and it works fine when missing values are rare and roughly random. If the missingness is systematic (say, higher scores are more likely to be missing because of dropout), filling with the mean biases the statistic instead of just estimating around it, so it is worth checking why a value is missing before deciding how to fill it. Other common fill strategies include the median (for skewed data), a fixed placeholder value, or forward-filling the previous row’s value in time-series data.

Summary:

The key concepts of statistics such as mean, median, mode, variance, standard deviation, percentiles, quartiles, and z-scores are explained in detail, along with examples of how to calculate these values using Python libraries such as NumPy and pandas. Additionally, the article also covers more advanced topics such as filling missing values and creating new columns in a dataset. The article is suitable for both beginners and experienced data analysts, providing them with the knowledge and tools they need to work with data in Python. The article provides sample data in CSV format which can be used to practice the concepts explained.

Exercise Question you will find in the exercise notebook of Day 6 on GitHub.

If you liked it then…[

](https://www.buymeacoffee.com/anurag629)