<jupyter_start><jupyter_text>Preparing the Data for Structure Learning since id doesn't have significance for the output we will drop it out and also unnamed column is nan<jupyter_code># y includes our labels and x includes our features
df, non_numeric_cols = check_numeric(data)
df = label_encode(df, non_numeric_cols)
y = data.diagnosis                          # M or B 
list = ['Unnamed: 32','id','diagnosis']
x = data.drop(list,axis = 1 )
x.head()<jupyter_output>['diagnosis']<jupyter_text>Splitting data into training and test set for feature selection<jupyter_code># split data train 70 % and test 30 %
logistic =Logistic()
x_train, x_test, y_train, y_test = logistic.splitdata(x, y)<jupyter_output><empty_output><jupyter_text>using univariate feature selection, we will use SelectKBest that removes all but the k highest scoring features..<jupyter_code># find best scored 8 features
select_feature = SelectKBest(chi2, k=8).fit(x_train, y_train)
print('Score list:', select_feature.scores_)
print('Feature list:', x_train.columns)<jupyter_output>Score list: [1.77946492e+02 6.06916433e+01 1.34061092e+03 3.66899557e+04
 1.00015175e-01 3.41839493e+00 1.30547650e+01 7.09766457e+00
 1.95982847e-01 3.42575072e-04 2.45882967e+01 4.07131026e-02
 1.72696840e+02 6.12741067e+03 1.32470372e-03 3.74071521e-01
 6.92896719e-01 2.01587194e-01 1.39557806e-03 2.65927071e-03
 3.25782599e+02 1.16958562e+02 2.40512835e+03 7.50217341e+04
 2.63226314e-01 1.19077581e+01 2.58858117e+01 8.90751003e+00
 1.00635138e+00 1.23087347e-01]
Feature list: Index(['radius_mean', 'texture_mean', 'perimeter_mean', 'area_mean',
       'smoothness_mean', 'compactness_mean', 'concavity_mean',
       'concave points_mean', 'symmetry_mean', 'fractal_dimension_mean',
       'radius_se', 'texture_se', 'perimeter_se', 'area_se', 'smoothness_se',
       'compactness_se', 'concavity_se', 'concave points_se', 'symmetry_se',
       'fractal_dimension_se', 'radius_worst', 'texture_worst',
       'perimeter_worst', 'area_worst', 'smoothness_worst',
       'compactness_worst', 'c[...]<jupyter_text>our best 8 features are ['radius_mean', 'texture_mean', 'perimeter_mean', 'area_mean',       'smoothness_mean', 'compactness_mean', 'concavity_mean',       'concave points_mean']  so we will be using this number of feature for our causal graph  Structure from Domain Knowledge We can manually define a structure model by specifying the relationships between different features. First, we must create an empty structure model.<jupyter_code>sm = StructureModel()<jupyter_output><empty_output><jupyter_text>We can apply now the NOTEARS algorithm to learn the structure<jupyter_code>#filter original dataset to retain the important features
#the dimension of the dataset has reduced from 569,32 to 569,9
stru_data = df [['diagnosis','area_mean', 'concavity_mean', 'concave points_mean', 'radius_worst',
       'perimeter_worst', 'area_worst', 'concavity_worst',
       'concave points_worst']]
filehandler.to_csv(stru_data,'../input/extracted.csv')
stru_data.head()
sm = from_pandas(stru_data)
# drawing the image
structure.drawgraph(sm,path= '../output/graph.png')
#add constraint to diagnosis without lasso regularization
sm_constrained = from_pandas(stru_data, tabu_parent_nodes=['diagnosis'], w_threshold=0.8)
structure.drawgraph(sm_constrained,path = '../output/graph1.png')
#put constraint on diagnosis column with lasso regularization
sm_lasso = from_pandas_lasso(stru_data, tabu_parent_nodes=['diagnosis'], w_threshold=0.8, beta=0.8)
structure.drawgraph(sm_lasso,path ='../output/graph2.png')<jupyter_output>INFO:structure:plots causal graph successfully<jupyter_text>Structure Review concavity affects  and area Radius affects perimeter and area, the graph captures these relationships.  Jaccard's similarity index<jupyter_code>#similarity between a graph plotted without lasso regularization and one plotted with lasso regularization
def jaccard_similarity(g, h):
    i = set(g).intersection(h)
    return round(len(i) / (len(g) + len(h) - len(i)),3)

jaccard_similarity(sm_constrained.edges(), sm_lasso.edges())<jupyter_output><empty_output><jupyter_text>Using Union and intersection<jupyter_code>def jaccar_score(g, h):    
    i = set(g).intersection(set(h))
    u = set(g).union(set(h))
    return len(i) / float(len(u))
jaccar_score(sm_constrained.edges(), sm_lasso.edges())<jupyter_output><empty_output><jupyter_text>The score and similarity are pretty much the same let us if we can increase this number. let us Split the data row wise and test the model as we increase the data<jupyter_code>df_1 = stru_data.iloc[:100,:]
df_2 = stru_data.iloc[:200,:]
df_3 = stru_data.iloc[:300,:]
df_4 = stru_data.iloc[:400,:]
df_5 = stru_data.iloc[:500,:]
df_6 = stru_data.iloc[:,:]<jupyter_output><empty_output><jupyter_text>let us draw using the first version of data<jupyter_code>sm_df1 = from_pandas_lasso(df_1, tabu_parent_nodes=['diagnosis'], w_threshold=0.8, beta=0.8)
structure.drawgraph(sm_df1,path ='../output/graph3.png' )
sm_df2 = from_pandas_lasso(df_2, tabu_parent_nodes=['diagnosis'], w_threshold=0.8, beta=0.8)
structure.drawgraph(sm_df2,path = '../output/graph4.png')

# similarity between df_1 and df_2
jaccard_similarity(sm_df1.edges(), sm_df2.edges())
sm_df3 = from_pandas_lasso(df_3, tabu_parent_nodes=['diagnosis'], w_threshold=0.8, beta=0.8)
structure.drawgraph(sm_df3, path = '../output/graph5.png' )

# similarity between df_2 and df_3
jaccard_similarity(sm_df2.edges(), sm_df3.edges())
sm_df4 = from_pandas_lasso(df_4, tabu_parent_nodes=['diagnosis'], w_threshold=0.8, beta=0.8)
structure.drawgraph(sm_df4, path = '../output/graph6.png' )
# similarity between df_2 and df_3
jaccard_similarity(sm_df3.edges(), sm_df4.edges())
sm_df5 = from_pandas_lasso(df_5, tabu_parent_nodes=['diagnosis'], w_threshold=0.8, beta=0.8)
structure.drawgraph(sm_df5, path = '../output/graph6.png' )
# similarity between df_2 and df_3
jaccard_similarity(sm_df4.edges(), sm_df5.edges())
sm_df6 = from_pandas_lasso(df_6, tabu_parent_nodes=['diagnosis'], w_threshold=0.8, beta=0.8)
structure.drawgraph(sm_df6, path = '../output/graph7.png' )
# similarity between df_2 and df_3
jaccard_similarity(sm_df5.edges(), sm_df6.edges())<jupyter_output><empty_output><jupyter_text>The graph has similarity of 1 between the 5th  and 6th dataframe, the 5th dataframe had 500 observations while the 6th dataframe had all the 569 observations.  FINAL GRAPH  Lets us add some edges to the graph. Radius will not directly affect the outcome but area, perimeter and concavity will affect the outcome<jupyter_code>sm_lasso.add_edge("concavity_mean", "diagnosis")
sm_lasso.add_edge("area_worst", "diagnosis")
sm_lasso.add_edge("area_mean", "diagnosis")
sm_lasso.add_edge("concavity_worst", "diagnosis")
sm_lasso.add_edge("perimeter_worst", "diagnosis")<jupyter_output><empty_output><jupyter_text>Lets us draw the graph with the new edges<jupyter_code>structure.drawgraphs(sm_lasso, path = '../output/graph8.png' )<jupyter_output>INFO:structure:plots causal graph successfully<jupyter_text>now let us train using logistic regression model for the whole dataset<jupyter_code># let us train random forest model
log = Logistic()
log.logistic(df,x_train,y_train)<jupyter_output><empty_output><jupyter_text>let us see the measure of how much all the variablles overfit<jupyter_code>log.accuracy(df,x_train,y_train,x_test,y_test,path = '../output/whole_data_Set')<jupyter_output>Accuracy is:  0.9707602339181286<jupyter_text>now let us see the accuracy of the extracted features<jupyter_code># y includes our labels and x includes our features
df, non_numeric_cols = check_numeric(data)
df = label_encode(df, non_numeric_cols)
y = data.diagnosis                          # M or B 
x = data[['area_mean', 'area_se', 'texture_mean', 'concavity_worst','concavity_mean']]
df = data[['area_mean', 'area_se', 'texture_mean', 'concavity_worst','concavity_mean','diagnosis']]<jupyter_output>['diagnosis']<jupyter_text>Let us split the extracted features<jupyter_code>x_train, x_test, y_train, y_test = log.splitdata(x, y)<jupyter_output><empty_output><jupyter_text>let us train the model using the extracted features<jupyter_code>log.logistic(df,x_train,y_train)<jupyter_output><empty_output><jupyter_text>let us see the measure of how much all the variablles overfit for the extracted features fom the causal graph<jupyter_code>log.accuracy(df,x_train,y_train,x_test,y_test,path = '../output/extracted_data_Set')<jupyter_output>Accuracy is:  0.9590643274853801
StarcoderdataJupyterStructuredCleanDedup