<jupyter_start><jupyter_text>Python Pandas Exercise: Course on ML and DS  Why Pandas is so important for data ScientistPandas is built on top of the **NumPy** package, meaning a lot of the structure of NumPy is used or replicated in Pandas. Data in pandas is often used to feed statistical analysis in **SciPy**, plotting functions from **Matplotlib**, and machine learning algorithms in **Scikit-learn**.we are using jupyter notebook in this course, but any other text editor is fine to work with pandas  Exercise Pipeline- Steps Install and import pandas `conda install pandas`OR `pip install pandas`  Read data from csv file- IMDV-Movie.csv ``` python df = pd.read_csv(filename)df``` NOte: CSVs don't have indexes like our DataFrames, so all we need to do is just designate the `index_col` when reading: ``` pythondf = pd.read_csv(filename, index_col=0)df```  Viewing your dataThe first thing to do when opening a new dataset is print out a few rows to keep as a visual reference. We accomplish this with `.head()`:  Getting info about your data`.info()` should be one of the very first commands you run after loading your data:  work with missing values1. Check for Null values 2. Replace nulls with non-null values, a technique known as **imputation**<jupyter_code>#  df.isnull()<jupyter_output><empty_output><jupyter_text>Notice `isnull()` returns a DataFrame where each cell is either True or False depending on that cell's null status.To count the number of nulls in each column we use an aggregate function for summing:<jupyter_code>#df.isnull().sum()<jupyter_output><empty_output><jupyter_text>`.isnull()` just by iteself isn't very useful, and is usually used in conjunction with other methods, like `sum()`.We can see now that our data has **128** missing values for `revenue_millions` and **64** missing values for `metascore`.  Removing null values<jupyter_code>#df.dropna()<jupyter_output><empty_output><jupyter_text>This operation will delete any **row** with at least a single null value, but it will return a new DataFrame without altering the original one. You could specify `inplace=True` in this method as well.So in the case of our dataset, this operation would remove 128 rows where `revenue_millions` is null and 64 rows where `metascore` is null. This obviously seems like a waste since there's perfectly good data in the other columns of those dropped rows. That's why we'll look at imputation next.Other than just dropping rows, you can also drop columns with null values by setting `axis=1`:<jupyter_code>movies_df.dropna(axis=1)
movies_df.shape<jupyter_output><empty_output><jupyter_text>As we learned above, this is a tuple that represents the shape of the DataFrame, i.e. 1000 rows and 11 columns. Note that the *rows* are at index zero of this tuple and *columns* are at **index one** of this tuple. This is why `axis=1` affects columns. This comes from NumPy, and is a great example of why learning NumPy is worth your time.  ImputationImputation is a conventional feature engineering technique used to keep valuable data that have null values. There may be instances where dropping every row with a null value removes too big a chunk from your dataset, so instead we can impute that null with another value, usually the **mean** or the **median** of that column. Let's look at imputing the missing values in the `revenue_millions` column. First we'll extract that column into its own variable:<jupyter_code># revenue = df['revenue_millions']<jupyter_output><empty_output><jupyter_text>Using square brackets is the general way we select columns in a DataFrame. If you remember back to when we created DataFrames from scratch, the keys of the `dict` ended up as column names. Now when we select columns of a DataFrame, we use brackets just like if we were accessing a Python dictionary. `revenue` now contains a Series:<jupyter_code># revenue.head()<jupyter_output><empty_output><jupyter_text>Slightly different formatting than a DataFrame, but we still have our `Title` index. We'll impute the missing values of revenue using the mean. Here's the mean value:<jupyter_code>#revenue_mean = revenue.mean()

#revenue_mean<jupyter_output><empty_output><jupyter_text>With the mean, let's fill the nulls using `fillna()`:<jupyter_code>#revenue.fillna(revenue_mean, inplace=True)<jupyter_output><empty_output><jupyter_text>We have now replaced all nulls in `revenue` with the mean of the column. Notice that by using `inplace=True` we have actually affected the original `movies_df`:<jupyter_code>#df.isnull().sum()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup