<jupyter_start><jupyter_text>Analyze A/B Test Results  Table of Contents- [Introduction](intro)- [Part I - Probability](probability)- [Part II - A/B Test](ab_test)- [Part III - Regression](regression) IntroductionA/B tests are very commonly performed by data analysts and data scientists.  It is important that you get some practice working with the difficulties of these In this project, We will be working to understand the results of an A/B test run by an e-commerce website.  The goal is to work through this notebook to help the company understand if they should implement the new page, keep the old page, or perhaps run the experiment longer to make their decision. Part I - ProbabilityTo get started, let's import our libraries.<jupyter_code>import pandas as pd
import numpy as np
import random
import matplotlib.pyplot as plt
%matplotlib inline
random.seed(42)<jupyter_output><empty_output><jupyter_text>`1.` Now, read in the `ab_data.csv` data. Store it in `df`. a. Read in the dataset and take a look at the top few rows here:<jupyter_code>df = pd.read_csv('ab_data.csv')
df.head()<jupyter_output><empty_output><jupyter_text>b. Find the number of rows in the dataset.<jupyter_code>df.shape[0]<jupyter_output><empty_output><jupyter_text>c. The number of unique users in the dataset.<jupyter_code>df.user_id.nunique()<jupyter_output><empty_output><jupyter_text>d. The proportion of users converted.<jupyter_code>df.loc[df.converted == 1].user_id.nunique()/df.user_id.nunique()<jupyter_output><empty_output><jupyter_text>e. The number of times the `new_page` and `treatment` don't match.<jupyter_code>Not_Match1 = df.loc[(df.landing_page == 'new_page') & (df.group != 'treatment')].index
Not_Match2 = df.loc[(df.landing_page == 'old_page') & (df.group != 'control')].index
df.loc[(df.landing_page == 'new_page') & (df.group != 'treatment')].user_id.nunique() + \
df.loc[(df.landing_page == 'old_page') & (df.group != 'control')].user_id.nunique()<jupyter_output><empty_output><jupyter_text>f. Do any of the rows have missing values?<jupyter_code>df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 294478 entries, 0 to 294477
Data columns (total 5 columns):
user_id         294478 non-null int64
timestamp       294478 non-null object
group           294478 non-null object
landing_page    294478 non-null object
converted       294478 non-null int64
dtypes: int64(2), object(3)
memory usage: 11.2+ MB<jupyter_text>`2.` For the rows where **treatment** does not match with **new_page** or **control** does not match with **old_page**, we cannot be sure if this row truly received the new or old page. Thus, we will create a new dataframe to have only the data with these records.a. Store the new dataframe in **df2**.<jupyter_code>df2 = df.drop(Not_Match1)
df2.drop(Not_Match2, inplace = True)
df2.shape
# Double Check all of the correct rows were removed - this should be 0
df2[((df2['group'] == 'treatment') == (df2['landing_page'] == 'new_page')) == False].shape[0]<jupyter_output><empty_output><jupyter_text>`3.` Analysis on df2  a. How many unique **user_id**s are in **df2**?<jupyter_code>df2.user_id.nunique()<jupyter_output><empty_output><jupyter_text>b. Check if any repeated users.<jupyter_code>u1 = df2.loc[df2.user_id.drop_duplicates().index]
#u1.shape, df2.shape
df_diff = pd.concat([df2,u1]).drop_duplicates(keep=False)
df_diff.user_id<jupyter_output><empty_output><jupyter_text>c. What is the row information for the repeat **user_id**?<jupyter_code>df_diff<jupyter_output><empty_output><jupyter_text>d. Remove **one** of the rows with a duplicate **user_id**, but keep your dataframe as **df2**.<jupyter_code>df2.drop_duplicates('user_id', inplace = True)<jupyter_output><empty_output><jupyter_text>`4.` Further investigation on df2a. What is the probability of an individual converting regardless of the page they receive?<jupyter_code>df2[df2.converted == 1].shape[0]/df2.shape[0]<jupyter_output><empty_output><jupyter_text>b. Given that an individual was in the `control` group, what is the probability they converted?<jupyter_code>control_group = df2[df2.group == 'control']
pconverted_control = control_group[control_group.converted == 1].shape[0]/control_group.shape[0]
pconverted_control<jupyter_output><empty_output><jupyter_text>c. Given that an individual was in the `treatment` group, what is the probability they converted?<jupyter_code>treatment_group = df2[df2.group == 'treatment']
pconverted_treatment = treatment_group[treatment_group.converted == 1].shape[0]/treatment_group.shape[0]
pconverted_treatment<jupyter_output><empty_output><jupyter_text>d. What is the probability that an individual received the new page?<jupyter_code>df2.landing_page.value_counts()/df2.shape[0]<jupyter_output><empty_output><jupyter_text>Conclusion **As probaility of converting is not significantly higher in control group we can see that there is no such evidence to conclude that the treatment page will lead to more conversions.**  Part II - A/B TestNotice that because of the time stamp associated with each event, you could technically run a hypothesis test continuously as each observation was observed.  However, then the hard question is do you stop as soon as one page is considered significantly better than another or does it need to happen consistently for a certain amount of time?  How long do you run to render a decision that neither page is better than another?  These questions are the difficult parts associated with A/B tests in general.  `1.` For now, we will consider the need to make the decision just based on all the data provided.  If you want to assume that the old page is better unless the new page proves to be definitely better at a Type I error rate of 5%, we can write the null and alternative as follow: **null :  P_old >= P_new Alternative: P_old < P_new** `2.` Assume under the null hypothesis, $p_{new}$ and $p_{old}$ both have "true" success rates equal to the **converted** success rate regardless of page - that is $p_{new}$ and $p_{old}$ are equal. Furthermore, assume they are equal to the **converted** rate in **ab_data.csv** regardless of the page. Use a sample size for each page equal to the ones in **ab_data.csv**.  Perform the sampling distribution for the difference in **converted** between the two pages over 10,000 iterations of calculating an estimate from the null.   a. What is the **conversion rate** for $p_{new}$ under the null?<jupyter_code>pnew = df2.loc[df2.converted == 1].user_id.nunique()/df2.user_id.nunique()
pnew<jupyter_output><empty_output><jupyter_text>b. What is the **conversion rate** for $p_{old}$ under the null?<jupyter_code>pold = pnew
pold<jupyter_output><empty_output><jupyter_text>c. What is $n_{new}$, the number of individuals in the treatment group?<jupyter_code>nnew_df = df2.query('group == "treatment"')
nnew = nnew_df.shape[0]
nnew<jupyter_output><empty_output><jupyter_text>d. What is $n_{old}$, the number of individuals in the control group?<jupyter_code>nold_df = df2.query('group == "control"')
nold = nold_df.shape[0]
nold<jupyter_output><empty_output><jupyter_text>e. Simulate $n_{new}$ transactions with a conversion rate of $p_{new}$ under the null.  Store these $n_{new}$ 1's and 0's in **new_page_converted**.<jupyter_code>#new_page_converted = np.random.binomial( n = nnew.converted , p = pnew )
new_page_converted = np.random.binomial(n = nnew_df.converted , p = pnew)
new_page_converted<jupyter_output><empty_output><jupyter_text>f. Simulate $n_{old}$ transactions with a conversion rate of $p_{old}$ under the null.  Store these $n_{old}$ 1's and 0's in **old_page_converted**.<jupyter_code>old_page_converted = np.random.binomial( n = nold_df.converted , p = pold)<jupyter_output><empty_output><jupyter_text>g. Find $p_{new}$ - $p_{old}$ for your simulated values from part (e) and (f).<jupyter_code>pnew2 = new_page_converted.mean()
pold2 = old_page_converted.mean()
pnew2 - pold2<jupyter_output><empty_output><jupyter_text>h. Create 10,000 $p_{new}$ - $p_{old}$ values using the same simulation process you used in parts (a) through (g) above. Store all 10,000 values in a NumPy array called **p_diffs**.<jupyter_code>new_page_converted = np.random.binomial(n = nnew , p = pnew, size = 10000) / nnew
old_page_converted = np.random.binomial(n = nold , p = pold, size = 10000) / nold
p_diffs = new_page_converted - old_page_converted<jupyter_output><empty_output><jupyter_text>i. Plot a histogram of the **p_diffs**.  Does this plot look like what you expected?  Use the matching problem in the classroom to assure you fully understand what was computed here.<jupyter_code>plt.hist(p_diffs);<jupyter_output><empty_output><jupyter_text>j. What proportion of the **p_diffs** are greater than the actual difference observed in **ab_data.csv**?<jupyter_code>observed_difference= df2['converted'][df2['group'] == 'treatment'].mean() - df2['converted'][df2['group'] == 'control'].mean()
(p_diffs > observed_difference).mean(),observed_difference<jupyter_output><empty_output><jupyter_text>k. Please explain using the vocabulary you've learned in this course what you just computed in part **j.**  What is this value called in scientific studies?  What does this value mean in terms of whether or not there is a difference between the new and old pages? **If we simulate 10000 time the difference between the conversion rates under the null hypothesis, about 97% of the resulting array will be large than actual difference computed directly from the dataset. This value often called p-value which indicates worst-case error ( Type- I error).If this error is smaller than certain threshold we will have enough evidence to reject the null hypothesis and accept alternative hypothesis. Typically this threshold is about 5%. In our case, we don't have enough evidence to reject null hypothesis which means we don't recommend to change to the new page based on this data.** l. We could also use a built-in to achieve similar results.  Though using the built-in might be easier to code, the above portions are a walkthrough of the ideas that are critical to correctly thinking about statistical significance. Fill in the below to calculate the number of conversions for each page, as well as the number of individuals who received each page. Let `n_old` and `n_new` refer the the number of rows associated with the old page and new pages, respectively.<jupyter_code>import statsmodels.api as sm

convert_old = df2.query('group == "control" and converted == 1').shape[0]
convert_new = df2.query('group == "treatment" and converted == 1').shape[0]
n_old = df2.query('group == "control"').shape[0]
n_new = df2.query('group == "treatment"').shape[0]<jupyter_output>/opt/conda/lib/python3.6/site-packages/statsmodels/compat/pandas.py:56: FutureWarning: The pandas.core.datetools module is deprecated and will be removed in a future version. Please use the pandas.tseries module instead.
  from pandas.core import datetools<jupyter_text>m. We can also use `stats.proportions_ztest` to compute the test statistic and p-value.<jupyter_code>#z_score, p_value = sm.stats.proportions_ztest([convert_new, convert_old], [n_new, n_old], alternative = 'larger')
z_score, p_value = sm.stats.proportions_ztest([convert_old, convert_new], [n_old, n_new],alternative = 'smaller')

z_score, p_value<jupyter_output><empty_output><jupyter_text>Conclusion **We can see that p_value is simular to what we obtained above(about 90 % ). It means that it is not expected that new page will lead to more conversions.**   Part III - A regression approach`1.` In this final part, you will see that the result you achieved in the A/B test in Part II above can also be achieved by performing regression. a. Since each row is either a conversion or no conversion, what type of regression should you be performing in this case? **Logistic Regression** b. The goal is to use **statsmodels** to fit the regression model we specified in part **a.** to see if there is a significant difference in conversion based on which page a customer receives. However, first we need to create in df2 a column for the intercept, and create a dummy variable column for which page each user received.  Add an **intercept** column, as well as an **ab_page** column, which is 1 when an individual receives the **treatment** and 0 if **control**.<jupyter_code>df2['intercept'] = 1 
df2 = df2.join(pd.get_dummies(df.group))
df2.drop(['control'], axis = 1 ,inplace = True)
df2.rename(columns = {'treatment': 'ab_page'}, inplace = True)
df2.head()<jupyter_output><empty_output><jupyter_text>c. Use **statsmodels** to instantiate your regression model on the two columns you created in part b., then fit the model using the two columns you created in part **b.** to predict whether or not an individual converts.<jupyter_code>import statsmodels.api as sm
!pip install --upgrade --use-feature=2020-resolver statsmodels
lg_mod = sm.Logit(df2.converted, df2[['ab_page', 'intercept']])
results = lg_mod.fit()<jupyter_output>Usage:   
  pip install [options] <requirement specifier> [package-index-options] ...
  pip install [options] -r <requirements file> [package-index-options] ...
  pip install [options] [-e] <vcs project url> ...
  pip install [options] [-e] <local project path> ...
  pip install [options] <archive url/path> ...

no such option: --use-feature
Optimization terminated successfully.
         Current function value: 0.366118
         Iterations 6<jupyter_text>d. Provide the summary of your model below, and use it as necessary to answer the following questions.<jupyter_code>results.summary2()<jupyter_output><empty_output><jupyter_text>**The p value given in this summary is the 2 sided test value. This can be related to 1 sided test value with the following formula ( Pside2 = 2(1 - Pside1) ). If we use Pside1 = .9 then Pside2 = .2 which is very close to value obtained above. I am not sure though why it is slightly lower than .2.** f. Now, we are considering other things that might influence whether or not an individual converts.  **We can add higher order terms like square or cubic values. In addition we can multiply two features. While this increase model ability to fit more complex data, it turns out that interpreting results will be more difficult.** g. Now along with testing if the conversion rate changes for different pages, also we will add an effect based on which country a user lives in. We will need to read in the **countries.csv** dataset and merge together your datasets on the appropriate rows.<jupyter_code>countries = pd.read_csv('countries.csv')
df_countries = df2.merge(countries , on = 'user_id',how = 'left')
df_countries = df_countries.join(pd.get_dummies(df_countries.country))
df_countries['intercept'] = 1
lg_mod2 = sm.Logit(df_countries.converted, df_countries[['ab_page', 'intercept','UK','US']])
results2 = lg_mod2.fit()
results2.summary2()<jupyter_output>Optimization terminated successfully.
         Current function value: 0.366113
         Iterations 6<jupyter_text>**It seems like the country columns has no significnce on the model results.**  h. Though we have now looked at the individual factors of country and page on conversion, we would now like to look at an interaction between page and country to see if there significant effects on conversion.  Create the necessary additional columns, and fit the new model.<jupyter_code>df_countries['interaction_CA'] = df_countries['CA'] * df_countries['ab_page']
df_countries['interaction_UK'] = df_countries['UK'] * df_countries['ab_page']
lg_mod3 = sm.Logit(df_countries.converted, df_countries[['ab_page', 'intercept','UK','US','interaction_CA','interaction_UK']])
results3 = lg_mod3.fit()
results3.summary2()<jupyter_output>Optimization terminated successfully.
         Current function value: 0.366109
         Iterations 6
StarcoderdataJupyterStructuredCleanDedup