<jupyter_start><jupyter_text>Importing Historical Data files Historical Datasets can be created from Adaptive models. They contain all thevalues of the predictors at the time of decision plus the associated outcome.To enable this feature please see the Pega documentation.Once enabled, the system will start generating datafiles in the repository. Theseare in multi-line JSON format.In this example we use a dataset that is included in the data folder of the repository (*hds.zip*). As a prerequisite to using this notebook, please first unzip this file into a folder called "hds".<jupyter_code>path_to_json = '../data/hds'
json_files = [pos_json for pos_json in os.listdir(path_to_json) if pos_json.endswith('.json')]
df = pd.DataFrame()
for i in json_files:
    df = pd.concat([pd.read_json(path_to_json + "/" + i, lines=True), df])
df = df.reset_index(drop=True)
df<jupyter_output><empty_output><jupyter_text>Fields    The dataset contains predictors of various sources such as Customer data, IH predictors, other parameterized predictors as well as the context keys for the model. It also contains the outcomes at decision time.There will also be some "meta" fields that describe where the data is coming from aswell as an ID so the individual decision can be tied back to a specific version of themodel.<jupyter_code>cust_col = []
IH_col = []
param_col = []
dec_col = []
context_col = []
other_col = []
for col in df.columns:
    if 'Customer_' in col: cust_col.append(col)
    elif 'IH_' in col: IH_col.append(col)
    elif 'Param_' in col: param_col.append(col)
    elif 'Decision_' in col: dec_col.append(col)
    elif 'Context_' in col: context_col.append(col)
    else: other_col.append(col)
df[cust_col].info()
df[dec_col].info()
df[IH_col].info()
df[param_col].info()
df[context_col].describe()
df[other_col].head()<jupyter_output><empty_output><jupyter_text>Correcting datetime fieldsUsing the CDH utils library it is easily possible to convert the string datetime fields into actual datetime values. From there, the date value (year, month, day) is extracted from Decision Time field.<jupyter_code>df['Decision_DecisionTime'] = [fromPRPCDateTime(i) for i in df['Decision_DecisionTime']]
df['Decision_OutcomeTime'] = [fromPRPCDateTime(i) for i in df['Decision_OutcomeTime']]
df['Day'] = [i.strftime('%Y-%m-%d') for i in df['Decision_DecisionTime']]
df['Customer_Age'] = df['Customer_Age'].astype(float)<jupyter_output><empty_output><jupyter_text>Exploring Data Customer Age over TimeThe boxplot below shows the Customer Age value over time. The line in the middle of each box represents the median value. It is also possible to see the outliers.<jupyter_code>fig, ax = plt.subplots(figsize=(10, 5))
sns.boxplot(x='Day', y='Customer_Age', data=df.sort_values('Day'))
ax.set_ylabel('Customer Age', fontsize=13)
ax.set_xlabel('Day', fontsize=13)
for i in ax.get_xmajorticklabels()+ax.get_ymajorticklabels():
    i.set_fontsize(12)<jupyter_output><empty_output><jupyter_text>Missing ValuesShowing the percentage of missing values per day. Missing valuesare a common source of issues in models.<jupyter_code>fig, ax = plt.subplots(figsize=(10, 5))
df_plot = df[['Customer_Age', 'Day']].groupby('Day').apply(
    lambda x:x['Customer_Age'].isnull().sum()*100/x.shape[0]).reset_index().sort_values('Day')
pal = sns.color_palette("Reds_d", len(df_plot))
rank = df_plot[0].argsort()
sns.barplot(x='Day', y=0, data=df_plot, palette=np.array(pal[::-1])[rank])
ax.set_ylabel('Percentage of Missing Values', fontsize=13)
ax.set_xlabel('Day', fontsize=13)
ax.set_title('Customer Age', fontsize=13)
for i in ax.get_xmajorticklabels()+ax.get_ymajorticklabels():
    i.set_fontsize(12)
ax.yaxis.set_major_formatter(mtick.PercentFormatter())<jupyter_output><empty_output><jupyter_text>Predictor performance (AUC) over time Since the Historical Dataset contains the outcomes, we can also plot the day-over-day predictor performance. We can "bin" the predictor using a simple quantile binning,then use the number of "positives" and "negatives" to calculate the AUC.This is not exactly the same binning as ADM is doing but for numeric fields thiswill be a good approximation. For symbolic fields the binning should take a similar approach as ADM is doing, by grouping labels by z-ratio first.<jupyter_code>df = df.sort_values('Customer_Age')
df['Customer_Age_Binned'] = pd.cut(np.array(df.sort_values('Customer_Age')['Customer_Age']), 
                                   bins=20,right = False, include_lowest =True )
df_AUC = df[['Customer_Age_Binned', 'Decision_Outcome', 'Customer_Age', 'Day']].groupby(
    ['Day', 'Customer_Age_Binned', 'Decision_Outcome']).count().rename(columns={'Customer_Age':'count'}).reset_index()

auc_per_day = {}
for day in df_AUC.sort_values('Day')['Day'].unique():
    _df = df_AUC[df_AUC['Day']==day].pivot(index='Customer_Age_Binned', 
                                           columns='Decision_Outcome', values='count').reset_index()
    _df = _df[~((_df['Accepted'].isnull()) & (_df['Rejected'].isnull()))].sort_values('Customer_Age_Binned')
    _df['Accepted'] = _df['Accepted'].fillna(0)
    _df['Rejected'] = _df['Rejected'].fillna(0)
    if _df['Accepted'].sum()==0:
        auc_per_day[day] = 0.5
    else:
        auc_per_day[day] = auc_from_bincounts(_df['Accepted'], _df['Rejected'])
auc_per_day = pd.DataFrame.from_dict(auc_per_day, orient='index').reset_index().rename(columns={0:'AUC', 'index':'Day'})
auc_per_day['AUC (%)'] = auc_per_day['AUC']*100
auc_per_day
fig, ax = plt.subplots(figsize=(10, 5))
sns.pointplot(x='Day', y='AUC (%)', data=auc_per_day)
ax.set_ylabel('AUC (%)', fontsize=13)
ax.set_xlabel('Day', fontsize=13)
ax.set_title('Predictor: Customer Age', fontsize=13)
for i in ax.get_xmajorticklabels()+ax.get_ymajorticklabels():
    i.set_fontsize(12)
ax.yaxis.set_major_formatter(mtick.PercentFormatter())
ax.set_ylim([50, 86])<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup