<jupyter_start><jupyter_text>*This notebook contains an excerpt from the [Python Data Science Handbook](http://shop.oreilly.com/product/0636920034919.do) by <NAME>; the content is available [on GitHub](https://github.com/jakevdp/PythonDataScienceHandbook).*  Handling Missing Data<jupyter_code>import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>NaN and None in Pandas``NaN`` and ``None`` both have their place, and Pandas is built to handle the two of them nearly interchangeably, converting between them where appropriate:<jupyter_code>pd.Series([1, np.nan, 2, None])
1 + np.nan
1 + None<jupyter_output><empty_output><jupyter_text>Operating on Null Values- ``isnull()``: Generate a boolean mask indicating missing values- ``notnull()``: Opposite of ``isnull()``- ``dropna()``: Return a filtered version of the data- ``fillna()``: Return a copy of the data with missing values filled or imputed  Detecting null valuesPandas data structures have two useful methods for detecting null data: ``isnull()`` and ``notnull()``.Either one will return a Boolean mask over the data. For example:<jupyter_code>data = pd.Series([1, np.nan, 'hello', None])
data
data.isnull()
data[data.notnull()]<jupyter_output><empty_output><jupyter_text>Dropping null valuesIn addition to the masking used before, there are the convenience methods, ``dropna()``(which removes NA values) and ``fillna()`` (which fills in NA values). For a ``Series``,the result is straightforward:<jupyter_code>data.dropna()
df = pd.DataFrame([[1,      np.nan, 2],
                   [2,      3,      5],
                   [np.nan, 4,      6]])
df<jupyter_output><empty_output><jupyter_text>We cannot drop single values from a ``DataFrame``; we can only drop full rows or full columns.Depending on the application, you might want one or the other, so ``dropna()`` gives a number of options for a ``DataFrame``.By default, ``dropna()`` will drop all rows in which *any* null value is present:<jupyter_code>df.dropna()<jupyter_output><empty_output><jupyter_text>Alternatively, you can drop NA values along a different axis; ``axis=1`` drops all columns containing a null value:<jupyter_code>df.dropna(axis='columns')<jupyter_output><empty_output><jupyter_text>But this drops some good data as well; you might rather be interested in dropping rows or columns with *all* NA values, or a majority of NA values.This can be specified through the ``how`` or ``thresh`` parameters, which allow fine control of the number of nulls to allow through.The default is ``how='any'``, such that any row or column (depending on the ``axis`` keyword) containing a null value will be dropped.You can also specify ``how='all'``, which will only drop rows/columns that are *all* null values:<jupyter_code>df[3] = np.nan
df
df.dropna(axis='columns', how='any')
df.dropna(axis='columns', how='all')<jupyter_output><empty_output><jupyter_text>For finer-grained control, the ``thresh`` parameter lets you specify a minimum number of non-null values for the row/column to be kept:<jupyter_code>df
df.dropna(axis='rows', thresh=3)<jupyter_output><empty_output><jupyter_text>Filling null valuesSometimes rather than dropping NA values, you'd rather replace them with a valid value.This value might be a single number like zero, or it might be some sort of imputation or interpolation from the good values.You could do this in-place using the ``isnull()`` method as a mask, but because it is such a common operation Pandas provides the ``fillna()`` method, which returns a copy of the array with the null values replaced.Consider the following ``Series``:<jupyter_code>data = pd.Series([1, np.nan, 2, None, 3], index=list('abcde'))
data
data.fillna(0)
data.fillna(data.mean())
data
# forward-fill
data.fillna(method='ffill')
# back-fill
data.fillna(method='bfill')
df
df.fillna(0)
df
df.fillna(method='ffill', axis=1)
df.fillna(method='ffill', axis=0)
df.replace(np.nan, 7)
df.replace(5, 7)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup