SciPy - Statistical Tests and Inference

---

---

Statistical tests and inference involve deriving conclusions about a population from sample data. These methodologies are fundamental for validating hypotheses, analyzing data trends, and making informed decisions in research, economics, engineering and many other fields. SciPys scipy.stats module offers a comprehensive set of tools to perform various statistical tests and data inferences.

Important Statistical Tests in SciPy

The scipy.stats library in Python includes a variety of functions to execute tests such as t-tests, chi-square tests and ANOVA, helping you validate assumptions and test hypotheses in different applications.

SciPy provides several statistical tests designed to assess different types of data and determine if observed differences or relationships are statistically significant. These tests play a critical role in hypothesis testing and analysis.

t-Test

A t-test is used to assess whether the means of two groups are different from one another typically applied in situations like comparing the results of two sample groups. The function scipy.stats.ttest_ind() can be used to perform a t-test on two independent samples.

The following example demonstrates how to perform a t-test on two datasets −

from scipy.stats import ttest_ind
import numpy as np

# Generate sample data
group1 = np.random.normal(0, 1, 100)
group2 = np.random.normal(0.5, 1, 100)

# Conduct the t-test
stat, p_value = ttest_ind(group1, group2)

print(f"t-statistic: {stat:.4f}")
print(f"p-value: {p_value:.4f}")

Here is the result of the t-test showing the t-statistic and p-value which help us to determine if the differences between the two groups are statistically significant −

t-statistic: -3.1020
p-value: 0.0022

Chi-Squared Test

The Chi-Squared Test is typically used to analyze categorical data, determining whether there is an association between two categorical variables. It's useful in situations like contingency tables where data is grouped into categories.

To perform Chi-Squared Test, SciPy provides the scipy.stats.chi2_contingency() function −

from scipy.stats import chi2_contingency
import numpy as np

# Example data in a contingency table
data = np.array([[10, 20], [20, 30]])

# Run the chi-squared test
chi2_stat, p_val, dof, expected = chi2_contingency(data)

print(f"Chi-squared statistic: {chi2_stat:.4f}")
print(f"p-value: {p_val:.4f}")
print(f"Degrees of freedom: {dof}")
print(f"Expected values: \n{expected}")

Below is the output of the Chi-squared test showing the statistic, p-value, degrees of freedom, and expected values:

Chi-squared statistic: 0.1280
p-value: 0.7205
Degrees of freedom: 1
Expected values:
[[11.25 18.75]
 [18.75 31.25]]

ANOVA (Analysis of Variance)

ANOVA tests whether there are significant differences among the means of three or more groups. It's useful when comparing multiple datasets to determine if at least one of them is different from the others.

To perform a one-way ANOVA we can use the scipy.stats.f_oneway() function, following is the example which performs the Annova test −

from scipy.stats import f_oneway
import numpy as np

# Example data from three groups
group1 = np.random.normal(0, 1, 100)
group2 = np.random.normal(1, 1, 100)
group3 = np.random.normal(2, 1, 100)

# Run one-way ANOVA
f_stat, p_value = f_oneway(group1, group2, group3)

print(f"F-statistic: {f_stat:.4f}")
print(f"p-value: {p_value:.4f}")

Heres the result of the ANOVA test showing the F-statistic and p-value, which help us assess whether the group means are statistically different:

F-statistic: 75.5012
p-value: 0.0000

Normality Tests

To determine if a dataset follows a normal distribution we can use normality tests like the Shapiro-Wilk Test or D'Agostino and Pearson's Test available in SciPy. The scipy.stats.shapiro() function conducts the Shapiro-Wilk test to check normality −

from scipy.stats import shapiro
import numpy as np

# Example data
data = np.random.normal(0, 1, 100)

# Perform Shapiro-Wilk normality test
stat, p_value = shapiro(data)

print(f"Test statistic: {stat:.4f}")
print(f"p-value: {p_value:.4f}")

Following is the output of the Shapiro-Wilk test helps to evaluate if the sample data is consistent with a normal distribution −

Test statistic: 0.9878
p-value: 0.4939

Using Statistical Inference in SciPy

SciPy provides essential tools for making inferences about a population from sample data, such as −

• p-value: This is used to determine the statistical significance of test results. A p-value below a threshold (commonly 0.05) suggests a significant result.
• Confidence Intervals: Estimate the range in which a population parameter (such as the mean) lies based on sample data.
• Effect Size: Quantifies the magnitude of an observed effect or difference.

Using these methods the researchers can perform thorough statistical analyses and make decisions backed by solid evidence from their data.