<jupyter_start><jupyter_text>LU1 (Working with data) - Example<jupyter_code># imports, feel free to ignore this for now 
import pandas as pd 
from matplotlib import pyplot as plt 
from utils import get_toy_data
% matplotlib inline<jupyter_output><empty_output><jupyter_text>1 - Working with jupyter notebooks (this will be insultingly basic)  Welcome! You are now within a [Jupyter notebook](http://jupyter.org/). We can do a few cool things:<jupyter_code># basic math 
10 + 15
# defining variables 
some_number = 5
some_animal = 'cat'
# variables can be used later 
print('I have %0.0f tins of %s food' % (some_number, some_animal))
# defining functions 
def sum_all(a, b, c): 
    return a + b + c
sum_all(2, 3 , 4)<jupyter_output><empty_output><jupyter_text>Writing in markdown:   some odd type of subtitle Look mom, I can write markdown! `def Mom==impressed`> Do you see any Teletubbies in here? *yay formatting!* ... **be bold** Loading images:  ![title](https://media.giphy.com/media/ljUXHv2x2BpjG/giphy.gif) To learn more: [Lessons by datacamp](https://www.datacamp.com/community/tutorials/tutorial-jupyter-notebook)  2 - Loading data  Great, we're ready to get to work! Let's start by loading our dataset, in our case some mushrooms!  ![title](https://mojohealth.com.au/assets/upload/data/mushrooms.jpg) You already have some mushroom datasets in your `data` folder. We will load them using Pandas, using [read_csv](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html)<jupyter_code># load the csv that is at path data/mushrooms.csv, into a Pandas DataFrame called data
data = pd.read_csv('data/mushrooms.csv')

# print the type 
print('Our dataset is now of the following type: %s' % type(data))<jupyter_output>Our dataset is now of the following type: <class 'pandas.core.frame.DataFrame'><jupyter_text>This dataset will now be a [Pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html). It's ok (at first) to think of DataFrames as tables, a bit like spreadsheets.   3 - Understanding data  The first thing to do with any dataset is... to look at it! Let's use [head](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.head.html) to look at the first 5 lines of our DataFrame<jupyter_code>data.head(5)<jupyter_output><empty_output><jupyter_text>How many rows and columns do we have? We can find out with the [.shape](http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.DataFrame.shape.html) command<jupyter_code>data.shape<jupyter_output><empty_output><jupyter_text>What are the columns?<jupyter_code>data.columns<jupyter_output><empty_output><jupyter_text>We can subset a column, by using brackets notation:<jupyter_code># subset the column is_poisonous, and save it into a Pandas Series called poison 
poison = data['is_poisonous']


# print the type 
print('poison is a variable of the following type: %s' % type(poison))<jupyter_output>poison is a variable of the following type: <class 'pandas.core.series.Series'><jupyter_text>You will notice that `poison` is now a Pandas Series (fancy name for "column"). Don't worry too much about this for now.  We can use [value_counts](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.value_counts.html) to understand how many poisonous and non-poisonous mushrooms we have<jupyter_code># How many poisonous and non poisonous mushrooms do we have? 
data['is_poisonous'].value_counts()<jupyter_output><empty_output><jupyter_text>Let's see another column:<jupyter_code># How many of each gill color do we have? 
data['gill-color'].value_counts()<jupyter_output><empty_output><jupyter_text>Sometimes, we have continuous variables, such as `height`.  If we do `value_counts` on `height` is will be pretty useless, as it has too many unique vaues. In these cases, using [a histogram](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.hist.html) is generally better:<jupyter_code># draw a histogram for the height column
data['height'].hist()
# the following is Matplotlib styling code
plt.xlabel('height')
plt.ylabel('count')
plt.show()<jupyter_output><empty_output><jupyter_text>For a continuous variable, we can get a bunch of descriptive statistics:<jupyter_code># Mean mushroom height: 
data['height'].mean()
# Minimum mushroom height: 
data['height'].min()
# max mushroom height: 
data['height'].max()<jupyter_output><empty_output><jupyter_text>We can also look at a single mushroom, but subsetting the second line:<jupyter_code>data.iloc[2]  # get the second line<jupyter_output><empty_output><jupyter_text>5 - Groupby Ok, now for something a bit trickier. We want to know what is the percentage of mushrooms that are poison, for each "color of spore-print" (whatever a spore-print is). So, we need to `groupby` the color of the spore, and then take the `mean` of `is_poisonous`<jupyter_code># mean poison by color = grouping by sprint color, take the mean of each group's is_poison
mean_poison_by_color = data.groupby('spore-print-color')['is_poisonous'].mean()<jupyter_output><empty_output><jupyter_text>So what does this look like?<jupyter_code>mean_poison_by_color<jupyter_output><empty_output><jupyter_text>As often happens, it is easier to just plot this, by adding a `.plot` at the end. In this case we<jupyter_code># plot the mean_poison_by_color, with a horizontal bar plot
mean_poison_by_color.plot(kind='barh')
# matplotlib styling (axis labels)
plt.xlabel('Percentage poisonous mushrooms')
plt.show()<jupyter_output><empty_output><jupyter_text>Very interesting, we can tell that if the color is `r` (presumably red) the mushroom is always poisonous, but if is `y` (probably yellow) it is never poisonous.   6 - Dummifying  As you might notice, our dataset has multiple types. We features that are categorical (contain strings), such as `gill-color`, a continuous variable (`height`), and a discrete variable (`is_poisonous`).<jupyter_code># just calling head so we can see the data again 
data[['gill-color', 'spore-print-color', 'is_poisonous', 'height']].head(3)<jupyter_output><empty_output><jupyter_text>For some applications, such as making predictions, we need all the data to be numerical. For that we have an excellent tool called [get_dummies](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html), that will _one hot encode_ (make 0 or 1) our categorical data. Easier with an example:<jupyter_code>toy_data = get_toy_data()

# let's look at it 
toy_data<jupyter_output><empty_output><jupyter_text>Now let's get dummies on this dataset<jupyter_code>pd.get_dummies(toy_data)<jupyter_output><empty_output><jupyter_text>So, now let's do this on our mushroom dataset:<jupyter_code>data = pd.get_dummies(data)
data.head()
print('We now have %0.0f rows and %0.0f columns'  % (data.shape[0], data.shape[1]))<jupyter_output>We now have 8124 rows and 29 columns<jupyter_text>7 - Saving data Finally, we'll save our "prepared mushrooms" to a csv, using pandas' to_csv function.<jupyter_code>data.to_csv('data/prepared_mushrooms.csv', # the path where we want it to be saved 
            index=False)  # we don't care about the [0, 1, 2, ...] index. Sometimes the index is relevant, but not here<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup