NumPy - Identifying Missing Values

---

---

Identifying Missing Values in Arrays

Identifying missing values in arrays means finding where data is missing, often represented as NaN (Not a Number) in NumPy. You can identify missing values in arrays using the NumPy np.isnan() function.

NaN is a special floating-point value defined by the IEEE floating-point standard. It is used to represent undefined or unrepresentable values, such as the result of 0/0 or a mathematical operation involving NaN.

Using the isnan() Function

The np.isnan() function in NumPy is used to identify NaN (Not a Number) values in an array.

This function checks each element in the array and returns a boolean array of the same shape, where each element is True if the corresponding element in the original array is NaN and False otherwise. Following is the syntax −

numpy.isnan(x)

Where, x is the input array in which to check for NaN values.

Example

In the following example, we use np.isnan() function to create a mask that identifies NaN values in the array −

import numpy as np

# Creating an array with NaN values
arr = np.array([1.0, 2.5, np.nan, 4.7, np.nan, 6.2])

# Identifying NaN values using np.isnan()
nan_mask = np.isnan(arr)

print("Original Array:\n", arr)
print("NaN Mask:\n", nan_mask)

Following is the output obtained −

Original Array:
[1.  2.5 nan 4.7 nan 6.2]
NaN Mask:
 [False False  True False  True False]

Identifying Missing Values in Multi-dimensional Arrays

Identifying missing values in multi-dimensional arrays refers to detecting NaN values across various dimensions of the array, such as in 2D matrices or 3D tensors.

This process is similar to working with 1D arrays but requires handling multiple dimensions while maintaining clarity on where the missing values are located.

Example

In this example, we use the np.isnan() function to create a mask that identifies NaN values in a 2D array −

import numpy as np 

# Creating a 2D array with NaN values
arr_2d = np.array([[1.0, np.nan, 3.5],
                   [np.nan, 5.1, 6.3]])

# Identifying NaN values in the 2D array
nan_mask_2d = np.isnan(arr_2d)

print("Original 2D Array:\n", arr_2d)
print("NaN Mask 2D:\n", nan_mask_2d)

This will produce the following result −

Original 2D Array:
[[1.  nan 3.5]
 [nan 5.1 6.3]]
NaN Mask 2D:
[[False  True False]
[ True False False]]

Identifying Missing Values in Structured Arrays

Identifying missing values in structured arrays involves detecting NaN or other placeholders within fields of the array, especially when the array contains mixed data types and multiple fields.

Structured arrays are complex because each field can have its own data type, so handling missing values requires attention to each field individually.

Example

In the example below, we use the np.isnan() function to create a mask that identifies NaN values specifically in the 'age' field of a structured array −

import numpy as np

# Creating a structured array with NaN values
dtype = [('name', 'U10'), ('age', 'f8')]
structured_arr = np.array([('Alice', 25), ('Bob', np.nan), ('Cathy', 23)], dtype=dtype)

# Checking for NaN values in the 'age' field
nan_mask_structured = np.isnan(structured_arr['age'])

print("Structured Array:\n", structured_arr)
print("NaN Mask for 'age' field:\n", nan_mask_structured)

Following is the output of the above code −

Structured Array:
[('Alice', 25.) ('Bob', nan) ('Cathy', 23.)]
NaN Mask for 'age' field:
[False  True False]

Counting Missing Values in an Array

To determine the number of missing values in an array, you can use the np.isnan() function, which returns a boolean array indicating where the NaN values are located.

Each element in this boolean array is "True" if the corresponding element in the original array is NaN, and "False" otherwise. By summing this boolean array, you effectively count the number of True values, which corresponds to the number of missing values.

Example

In the following example, we generate a boolean mask using np.isnan() function to identify NaN values in the array. We then count the number of NaN values by summing the mask, which provides the total count of missing values −

import numpy as np

# Create an array with some NaN values
arr = np.array([1.0, 2.0, np.nan, 4.0, np.nan])

# Generate a boolean array indicating NaN values
nan_mask = np.isnan(arr)

# Count the number of NaN values
nan_count = np.sum(nan_mask)

print("Boolean mask of NaN values:")
print(nan_mask)
print("Number of NaN values:")
print(nan_count)

The output obtained is as shown below −

Boolean mask of NaN values:
[False False  True False  True]
Number of NaN values:
2

Boolean Indexing with np.isnan() Function

Once you have identified the missing values using np.isnan() function, you can combine this with Boolean indexing to perform various operations on those values.

Boolean indexing allows you to create a mask based on the condition (e.g., whether an element is NaN) and then use this mask to filter, replace, or analyze the elements that meet this condition.

Example: Filtering Out Missing Values

You can use Boolean indexing to filter out the missing values from your array, retaining only the non-missing values −

import numpy as np

# Create an array with some NaN values
arr = np.array([1.0, 2.0, np.nan, 4.0, np.nan])

# Generate a boolean array indicating NaN values
nan_mask = np.isnan(arr)

# Filter out NaN values
filtered_arr = arr[~nan_mask]

print("Original array:")
print(arr)
print("Filtered array (without NaN values):")
print(filtered_arr)

After executing the above code, we get the following output −

Original array:
[ 1.  2. nan  4. nan]
Filtered array (without NaN values):
[1. 2. 4.]

Example: Replacing Missing Values

You can replace NaN values with a specific value, such as the mean or median of the non-missing values −

import numpy as np

# Create an array with some NaN values
arr = np.array([1.0, 2.0, np.nan, 4.0, np.nan])

# Calculate the mean of non-NaN values
mean_value = np.nanmean(arr)

# Replace NaN values with the mean value
arr_with_replacement = np.where(np.isnan(arr), mean_value, arr)

print("Original array:")
print(arr)
print("Array with NaN replaced by mean:")
print(arr_with_replacement)

The result produced is as follows −

Original array:
[ 1.  2. nan  4. nan]
Array with NaN replaced by mean:
[1.         2.         2.33333333 4.         2.33333333]

Example: Analyzing Missing Values

You can use Boolean indexing to analyze the distribution or patterns of missing values, for instance, checking which rows or columns have missing data −

import numpy as np

# Create a 2D array with some NaN values
arr_2d = np.array([[1.0, np.nan, 3.0],
                   [4.0, np.nan, 6.0],
                   [np.nan, 8.0, 9.0]])

# Identify NaN values
nan_mask_2d = np.isnan(arr_2d)

# Count NaN values per row
nan_count_per_row = np.sum(nan_mask_2d, axis=1)

print("Original 2D array:")
print(arr_2d)
print("NaN count per row:")
print(nan_count_per_row)

We get the output as shown below −

Original 2D array:
[[ 1. nan  3.]
 [ 4. nan  6.]
 [nan  8.  9.]]
NaN count per row:
[1 1 1]