<jupyter_start><jupyter_text>Working with Watson OpenScale - Custom Machine Learning Provider This notebook should be run using with **Python 3.7.x** runtime environment. **If you are viewing this in Watson Studio and do not see Python 3.7.x in the upper right corner of your screen, please update the runtime now.** It requires service credentials for the following services:  * Watson OpenScale  * A Custom ML provider which is hosted in a VM that can be accessible from CPD PODs, specifically OpenScale PODs namely ML Gateway fairness, quality, drift, and explain.  * DB2 - as part of this notebook, we make use of an existing data mart.  The notebook will configure a OpenScale data mart subscription for Custom ML Provider deployment. We configure and execute the fairness, explain, quality and drift monitors.  Custom Machine Learning Provider SetupFollowing code can be used to start a gunicorn/flask application that can be hosted in a VM, such that it can be accessable from CPD system.This code does the following:* It wraps a Watson Machine Learning model that is deployed to a space.* So the hosting application URL should contain the SPACE ID and the DEPLOYMENT ID. Then, the same can be used to talk to the target WML model/deployment.* Having said that, this is only for this tutorial purpose, and you can define your Custom ML provider endpoint in any fashion you want, such that it wraps your own custom ML engine.* The scoring request and response payload should confirm to the schema as described here at: https://dataplatform.cloud.ibm.com/docs/content/wsj/model/wos-frameworks-custom.html* To start the application using the below code, make sure you install following python packages in your VM:python -m pip install gunicornpython -m pip install flaskpython -m pip install numpypython -m pip install pandaspython -m pip install requestspython -m pip install joblib==0.11python -m pip install scipy==0.19.1python -m pip install --user numpy scipy matplotlib ipython jupyter pandas sympy nosepython -m pip install ibm_watson_machine_learning-----------------```from flask import Flask, request, abort, jsonifyimport jsonimport base64import requests, ioimport pandas as pdfrom ibm_watson_machine_learning import APIClientapp = Flask(__name__)WML_CREDENTIALS = {                   "url": "https://namespace1-cpd-namespace1.apps.xxxxx.os.fyre.ibm.com",                   "username": "admin",                   "password" : "<PASSWORD>",                   "instance_id": "wml_local",                   "version" : "3.5"                  }@app.route('/spaces//deployments//predictions', methods=['POST'])def wml_scoring(space_id, deployment_id):	if not request.json:		abort(400)	wml_credentials = WML_CREDENTIALS	payload_scoring = {        "input_data": [            request.json        ]    }	wml_client = APIClient(wml_credentials)	wml_client.set.default_space(space_id)	records_list=[]	scoring_response = wml_client.deployments.score(deployment_id, payload_scoring)	return jsonify(scoring_response["predictions"][0])if __name__ == '__main__':    app.run(host='xxxx.fyre.ibm.com', port=9443, debug=True)```-----------------  Setup   Package installation<jupyter_code>import warnings
warnings.filterwarnings('ignore')
!pip install --upgrade pyspark==2.4 --no-cache | tail -n 1

!pip install --upgrade pandas==0.25.3 --no-cache | tail -n 1
!pip install --upgrade requests==2.23 --no-cache | tail -n 1
!pip install numpy==1.16.4 --no-cache | tail -n 1
!pip install scikit-learn==0.20 --no-cache | tail -n 1
!pip install SciPy --no-cache | tail -n 1
!pip install lime --no-cache | tail -n 1

!pip install --upgrade ibm-watson-machine-learning --user | tail -n 1
!pip install --upgrade ibm-watson-openscale --no-cache | tail -n 1
!pip install --upgrade ibm-wos-utils --no-cache | tail -n 1<jupyter_output><empty_output><jupyter_text>Action: restart the kernel!  Configure credentials - WOS_CREDENTIALS (CP4D)- WML_CREDENTIALS (CP4D)- DATABASE_CREDENTIALS (DB2 on CP4D or Cloud Object Storage (COS))- SCHEMA_NAME<jupyter_code>#masked
WOS_CREDENTIALS = {
    "url": "https://namespace1-cpd-namespace1.apps.xxxxx.os.fyre.ibm.com",
    "username": "admin",
    "password": "<PASSWORD>",
    "version": "3.5"
}
CUSTOM_ML_PROVIDER_SCORING_URL = 'https://xxxxx.fyre.ibm.com:9443/spaces/$SPACE_ID/deployments/$DEPLOYMENT_ID/predictions'
scoring_url = CUSTOM_ML_PROVIDER_SCORING_URL
label_column="Risk"
model_type = "binary"
import os
import base64
import json
import requests
from requests.auth import HTTPBasicAuth<jupyter_output><empty_output><jupyter_text>Save training data to Cloud Object Storage Cloud object storage details¶In next cells, you will need to paste some credentials to Cloud Object Storage. If you haven't worked with COS yet please visit getting started with COS tutorial. You can find COS_API_KEY_ID and COS_RESOURCE_CRN variables in Service Credentials in menu of your COS instance. Used COS Service Credentials must be created with Role parameter set as Writer. Later training data file will be loaded to the bucket of your instance and used as training refecence in subsription. COS_ENDPOINT variable can be found in Endpoint field of the menu.<jupyter_code>IAM_URL="https://iam.ng.bluemix.net/oidc/token"
# masked
COS_API_KEY_ID = "*****"
COS_RESOURCE_CRN = "*****"
COS_ENDPOINT = "https://s3.us.cloud-object-storage.appdomain.cloud" # Current list avaiable at https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints
BUCKET_NAME = "*****"
FILE_NAME = "german_credit_data_biased_training.csv"<jupyter_output><empty_output><jupyter_text>Load and explore data<jupyter_code>!rm german_credit_data_biased_training.csv
!wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/german_credit_data_biased_training.csv<jupyter_output><empty_output><jupyter_text>Explore data<jupyter_code>training_data_references = [
                {
                    "id": "Credit Risk",
                    "type": "s3",
                    "connection": {
                        "access_key_id": COS_API_KEY_ID,
                        "endpoint_url": COS_ENDPOINT,
                        "resource_instance_id":COS_RESOURCE_CRN
                    },
                    "location": {
                        "bucket": BUCKET_NAME,
                        "path": FILE_NAME,
                    }
                }
            ]<jupyter_output><empty_output><jupyter_text>Construct the scoring payload<jupyter_code>import pandas as pd

df = pd.read_csv("german_credit_data_biased_training.csv")
df.head()
cols_to_remove = [label_column]
def get_scoring_payload(no_of_records_to_score = 1):

    for col in cols_to_remove:
        if col in df.columns:
            del df[col] 

    fields = df.columns.tolist()
    values = df[fields].values.tolist()

    payload_scoring ={"fields": fields, "values": values[:no_of_records_to_score]}  
    return payload_scoring
#debug
payload_scoring = get_scoring_payload(1)
payload_scoring<jupyter_output><empty_output><jupyter_text>Method to perform scoring<jupyter_code>def custom_ml_scoring():
    header = {"Content-Type": "application/json", "x":"y"}
    
    print(scoring_url)
    scoring_response = requests.post(scoring_url, json=payload_scoring, headers=header, verify=False)
    jsonify_scoring_response = scoring_response.json()
    return jsonify_scoring_response<jupyter_output><empty_output><jupyter_text>Method to perform payload logging<jupyter_code>import uuid
scoring_id = None
from ibm_watson_openscale.supporting_classes.payload_record import PayloadRecord
def payload_logging(payload_scoring, scoring_response):
    scoring_id = str(uuid.uuid4())
    records_list=[]
    
    #manual PL logging for custom ml provider
    pl_record = PayloadRecord(scoring_id=scoring_id, request=payload_scoring, response=scoring_response, response_time=int(460))
    records_list.append(pl_record)
    wos_client.data_sets.store_records(data_set_id = payload_data_set_id, request_body=records_list)
    
    time.sleep(5)
    pl_records_count = wos_client.data_sets.get_records_count(payload_data_set_id)
    print("Number of records in the payload logging table: {}".format(pl_records_count))
    return scoring_id<jupyter_output><empty_output><jupyter_text>Score the model and print the scoring response Sample Scoring<jupyter_code>custom_ml_scoring()<jupyter_output><empty_output><jupyter_text>Configure OpenScale The notebook will now import the necessary libraries and set up a Python OpenScale client.<jupyter_code>from ibm_watson_openscale import APIClient
from ibm_watson_openscale.utils import *
from ibm_watson_openscale.supporting_classes import *
from ibm_watson_openscale.supporting_classes.enums import *
from ibm_watson_openscale.base_classes.watson_open_scale_v2 import *
from ibm_cloud_sdk_core.authenticators import CloudPakForDataAuthenticator

import json
import requests
import base64
from requests.auth import HTTPBasicAuth
import time<jupyter_output><empty_output><jupyter_text>Get a instance of the OpenScale SDK client<jupyter_code>authenticator = CloudPakForDataAuthenticator(
        url=WOS_CREDENTIALS['url'],
        username=WOS_CREDENTIALS['username'],
        password=WOS_CREDENTIALS['password'],
        disable_ssl_verification=True
    )

wos_client = APIClient(service_url=WOS_CREDENTIALS['url'],authenticator=authenticator)
wos_client.version<jupyter_output><empty_output><jupyter_text>Set up datamartWatson OpenScale uses a database to store payload logs and calculated metrics. If database credentials were not supplied above, the notebook will use the free, internal lite database. If database credentials were supplied, the datamart will be created there unless there is an existing datamart and the KEEP_MY_INTERNAL_POSTGRES variable is set to True. If an OpenScale datamart exists in Db2 or PostgreSQL, the existing datamart will be used and no data will be overwritten.Prior instances of the model will be removed from OpenScale monitoring.<jupyter_code>wos_client.data_marts.show()
data_marts = wos_client.data_marts.list().result.data_marts
if len(data_marts) == 0:
    raise Exception("Missing data mart.")
data_mart_id=data_marts[0].metadata.id
print('Using existing datamart {}'.format(data_mart_id))
data_mart_details = wos_client.data_marts.list().result.data_marts[0]
data_mart_details.to_dict()
wos_client.service_providers.show()<jupyter_output><empty_output><jupyter_text>Remove existing service provider connected with used WML instance.Multiple service providers for the same engine instance are avaiable in Watson OpenScale. To avoid multiple service providers of used WML instance in the tutorial notebook the following code deletes existing service provder(s) and then adds new one.<jupyter_code>SERVICE_PROVIDER_NAME = "Custom ML Provider Demo - All Monitors"
SERVICE_PROVIDER_DESCRIPTION = "Added by tutorial WOS notebook to showcase monitoring Fairness, Quality, Drift and Explainability against a Custom ML provider."
service_providers = wos_client.service_providers.list().result.service_providers
for service_provider in service_providers:
    service_instance_name = service_provider.entity.name
    if service_instance_name == SERVICE_PROVIDER_NAME:
        service_provider_id = service_provider.metadata.id
        wos_client.service_providers.delete(service_provider_id)
        print("Deleted existing service_provider for WML instance: {}".format(service_provider_id))<jupyter_output><empty_output><jupyter_text>Add service providerWatson OpenScale needs to be bound to the Watson Machine Learning instance to capture payload data into and out of the model.Note: You can bind more than one engine instance if needed by calling wos_client.service_providers.add method. Next, you can refer to particular service provider using service_provider_id.<jupyter_code>request_headers = {"Content-Type": "application/json", "Custom_header_X": "Custom_header_X_value_Y"}
MLCredentials = {}
added_service_provider_result = wos_client.service_providers.add(
        name=SERVICE_PROVIDER_NAME,
        description=SERVICE_PROVIDER_DESCRIPTION,
        service_type=ServiceTypes.CUSTOM_MACHINE_LEARNING,
        request_headers=request_headers,
        operational_space_id = "production",
        credentials=MLCredentials,
        background_mode=False
    ).result
service_provider_id = added_service_provider_result.metadata.id
print(wos_client.service_providers.get(service_provider_id).result)
print('Data Mart ID : ' + data_mart_id)
print('Service Provider ID : ' + service_provider_id)<jupyter_output><empty_output><jupyter_text>Subscriptions Remove existing credit risk subscriptionsThis code removes previous subscriptions to the model to refresh the monitors with the new model and new data.<jupyter_code>wos_client.subscriptions.show()<jupyter_output><empty_output><jupyter_text>Remove the existing subscription<jupyter_code>SUBSCRIPTION_NAME = "Custom ML Subscription - All Monitors"
subscriptions = wos_client.subscriptions.list().result.subscriptions
for subscription in subscriptions:
    if subscription.entity.asset.name == "[asset] " + SUBSCRIPTION_NAME:
        sub_model_id = subscription.metadata.id
        wos_client.subscriptions.delete(subscription.metadata.id)
        print('Deleted existing subscription for model', sub_model_id)<jupyter_output><empty_output><jupyter_text>This code creates the model subscription in OpenScale using the Python client API. Note that we need to provide the model unique identifier, and some information about the model itself.<jupyter_code>feature_columns=["CheckingStatus","LoanDuration","CreditHistory","LoanPurpose","LoanAmount","ExistingSavings","EmploymentDuration","InstallmentPercent","Sex","OthersOnLoan","CurrentResidenceDuration","OwnsProperty","Age","InstallmentPlans","Housing","ExistingCreditsCount","Job","Dependents","Telephone","ForeignWorker"]
cat_features=["CheckingStatus","CreditHistory","LoanPurpose","ExistingSavings","EmploymentDuration","Sex","OthersOnLoan","OwnsProperty","InstallmentPlans","Housing","Job","Telephone","ForeignWorker"]
import uuid
asset_id = str(uuid.uuid4())
asset_name = '[asset] ' + SUBSCRIPTION_NAME
url = ''

asset_deployment_id = str(uuid.uuid4())
asset_deployment_name = asset_name
asset_deployment_scoring_url = scoring_url

scoring_endpoint_url = scoring_url
scoring_request_headers = {
        "Content-Type": "application/json",
        "Custom_header_X": "Custom_header_X_value_Y"
    }
subscription_details = wos_client.subscriptions.add(
        data_mart_id=data_mart_id,
        service_provider_id=service_provider_id,
        asset=Asset(
            asset_id=asset_id,
            name=asset_name,
            url=url,
            asset_type=AssetTypes.MODEL,
            input_data_type=InputDataType.STRUCTURED,
            problem_type=ProblemType.BINARY_CLASSIFICATION
        ),
        deployment=AssetDeploymentRequest(
            deployment_id=asset_deployment_id,
            name=asset_deployment_name,
            deployment_type= DeploymentTypes.ONLINE,
            scoring_endpoint=ScoringEndpointRequest(
                url=scoring_endpoint_url,
                request_headers=scoring_request_headers
            )
        ),
        asset_properties=AssetPropertiesRequest(
            label_column=label_column,
            probability_fields=["probability"],
            prediction_field="predictedLabel",
            feature_fields = feature_columns,
            categorical_fields = cat_features,
            training_data_reference=TrainingDataReference(type="cos",
                                                          location=COSTrainingDataReferenceLocation(bucket = BUCKET_NAME,
                                                                                                    file_name = FILE_NAME),
                                                          connection=COSTrainingDataReferenceConnection.from_dict({
                                                                        "resource_instance_id": COS_RESOURCE_CRN,
                                                                        "url": COS_ENDPOINT,
                                                                        "api_key": COS_API_KEY_ID,
                                                                        "iam_url": IAM_URL}))
        )
    ).result
subscription_id = subscription_details.metadata.id
print('Subscription ID: ' + subscription_id)
import time

time.sleep(5)
payload_data_set_id = None
payload_data_set_id = wos_client.data_sets.list(type=DataSetTypes.PAYLOAD_LOGGING, 
                                                target_target_id=subscription_id, 
                                                target_target_type=TargetTypes.SUBSCRIPTION).result.data_sets[0].metadata.id
if payload_data_set_id is None:
    print("Payload data set not found. Please check subscription status.")
else:
    print("Payload data set id:", payload_data_set_id)<jupyter_output><empty_output><jupyter_text>Before the payload logging wos_client.subscriptions.get(subscription_id).result.to_dict()  Score the model so we can configure monitorsNow that the WML service has been bound and the subscription has been created, we need to send a request to the model before we configure OpenScale. This allows OpenScale to create a payload log in the datamart with the correct schema, so it can capture data coming into and out of the model.<jupyter_code>no_of_records_to_score = 100<jupyter_output><empty_output><jupyter_text>Construct the scoring payload<jupyter_code>payload_scoring = get_scoring_payload(no_of_records_to_score)<jupyter_output><empty_output><jupyter_text>Perform the scoring against the Custom ML Provider<jupyter_code>scoring_response = custom_ml_scoring()<jupyter_output><empty_output><jupyter_text>Perform payload logging by passing the scoring payload and scoring response<jupyter_code>scoring_id = payload_logging(payload_scoring, scoring_response)<jupyter_output><empty_output><jupyter_text>The scoring id, which would be later used for explanation of the randomly picked transactions<jupyter_code>print('scoring_id: ' + str(scoring_id))<jupyter_output><empty_output><jupyter_text>Fairness configuration  The code below configures fairness monitoring for our model. It turns on monitoring for two features, sex and age. In each case, we must specify:    Which model feature to monitor One or more majority groups, which are values of that feature that we expect to receive a higher percentage of favorable outcomes One or more minority groups, which are values of that feature that we expect to receive a higher percentage of unfavorable outcomes The threshold at which we would like OpenScale to display an alert if the fairness measurement falls below (in this case, 80%) Additionally, we must specify which outcomes from the model are favourable outcomes, and which are unfavourable. We must also provide the number of records OpenScale will use to calculate the fairness score. In this case, OpenScale's fairness monitor will run hourly, but will not calculate a new fairness rating until at least 100 records have been added. Finally, to calculate fairness, OpenScale must perform some calculations on the training data, so we provide the dataframe containing the data.  Create Fairness Monitor Instance<jupyter_code>target = Target(
    target_type=TargetTypes.SUBSCRIPTION,
    target_id=subscription_id

)
parameters = {
    "features": [
        {"feature": "Sex",
         "majority": ['male'],
         "minority": ['female']
         },
        {"feature": "Age",
         "majority": [[26, 75]],
         "minority": [[18, 25]]
         }
    ],
    "favourable_class": ["No Risk"],
    "unfavourable_class": ["Risk"],
    "min_records": 100
}
thresholds = [{
    "metric_id": "fairness_value",
    "specific_values": [{
            "applies_to": [{
                "key": "feature",
                "type": "tag",
                "value": "Age"
            }],
            "value": 95
        },
        {
            "applies_to": [{
                "key": "feature",
                "type": "tag",
                "value": "Sex"
            }],
            "value": 95
        }
    ],
    "type": "lower_limit",
    "value": 80.0
}]

fairness_monitor_details = wos_client.monitor_instances.create(
    data_mart_id=data_mart_id,
    background_mode=False,
    monitor_definition_id=wos_client.monitor_definitions.MONITORS.FAIRNESS.ID,
    target=target,
    parameters=parameters,
    thresholds=thresholds).result
fairness_monitor_instance_id = fairness_monitor_details.metadata.id<jupyter_output><empty_output><jupyter_text>Get Fairness Monitor Instance<jupyter_code>wos_client.monitor_instances.show()<jupyter_output><empty_output><jupyter_text>Get run detailsIn case of production subscription, initial monitoring run is triggered internally. Checking its status<jupyter_code>runs = wos_client.monitor_instances.list_runs(fairness_monitor_instance_id, limit=1).result.to_dict()
fairness_monitoring_run_id = runs["runs"][0]["metadata"]["id"]
run_status = None
while(run_status not in ["finished", "error"]):
    run_details = wos_client.monitor_instances.get_run_details(fairness_monitor_instance_id, fairness_monitoring_run_id).result.to_dict()
    run_status = run_details["entity"]["status"]["state"]
    print('run_status: ', run_status)
    if run_status in ["finished", "error"]:
        break
    time.sleep(10)<jupyter_output><empty_output><jupyter_text>Fairness run output<jupyter_code>wos_client.monitor_instances.get_run_details(fairness_monitor_instance_id, fairness_monitoring_run_id).result.to_dict()
wos_client.monitor_instances.show_metrics(monitor_instance_id=fairness_monitor_instance_id)<jupyter_output><empty_output><jupyter_text>Configure Explainability We provide OpenScale with the training data to enable and configure the explainability features.<jupyter_code>target = Target(
    target_type=TargetTypes.SUBSCRIPTION,
    target_id=subscription_id
)
parameters = {
    "enabled": True
}
explain_monitor_details = wos_client.monitor_instances.create(
    data_mart_id=data_mart_id,
    background_mode=False,
    monitor_definition_id=wos_client.monitor_definitions.MONITORS.EXPLAINABILITY.ID,
    target=target,
    parameters=parameters
).result

explain_monitor_details.metadata.id
scoring_ids = []
sample_size = 2
import random
for i in range(0, sample_size):
    n = random.randint(1,100)
    scoring_ids.append(scoring_id + '-' + str(n))
print("Running explanations on scoring IDs: {}".format(scoring_ids))
explanation_types = ["lime", "contrastive"]
result = wos_client.monitor_instances.explanation_tasks(scoring_ids=scoring_ids, explanation_types=explanation_types).result
print(result)<jupyter_output><empty_output><jupyter_text>Explanation tasks<jupyter_code>explanation_task_ids=result.metadata.explanation_task_ids
explanation_task_ids<jupyter_output><empty_output><jupyter_text>Wait for the explanation tasks to complete - all of them<jupyter_code>import time
def finish_explanation_tasks():
    finished_explanations = []
    finished_explanation_task_ids = []
    
    # Check for the explanation task status for finished status. 
    # If it is in-progress state, then sleep for some time and check again. 
    # Perform the same for couple of times, so that all tasks get into finished state.
    for i in range(0, 5):
        # for each explanation
        print('iteration ' + str(i))
        
        #check status for all explanation tasks
        for explanation_task_id in explanation_task_ids:
            if explanation_task_id not in finished_explanation_task_ids:
                result = wos_client.monitor_instances.get_explanation_tasks(explanation_task_id=explanation_task_id).result
                print(explanation_task_id + ' : ' + result.entity.status.state)
                if (result.entity.status.state == 'finished' or result.entity.status.state == 'error') and explanation_task_id not in finished_explanation_task_ids:
                    finished_explanation_task_ids.append(explanation_task_id)
                    finished_explanations.append(result)


        # if there is altest one explanation task that is not yet completed, then sleep for sometime, 
        # and check for all those tasks, for which explanation is not yet completeed.
        
        if len(finished_explanation_task_ids) != sample_size:
            print('sleeping for some time..')
            time.sleep(10)
        else:
            break
                    
    return finished_explanations<jupyter_output><empty_output><jupyter_text>You may have to run the below multiple times till all explanation tasks are either finished or error'ed.<jupyter_code>finished_explanations = finish_explanation_tasks()
len(finished_explanations)
def construct_explanation_features_map(feature_name, feature_weight):
    if feature_name in explanation_features_map:
        explanation_features_map[feature_name].append(feature_weight)
    else:
        explanation_features_map[feature_name] = [feature_weight]
explanation_features_map = {}
for result in finished_explanations:
    print('\n>>>>>>>>>>>>>>>>>>>>>>\n')
    print('explanation task: ' + str(result.metadata.explanation_task_id) + ', perturbed:' + str(result.entity.perturbed))
    if result.entity.explanations is not None:
        explanations = result.entity.explanations
        for explanation in explanations:
            if 'predictions' in explanation:
                predictions = explanation['predictions']
                for prediction in predictions:
                    predicted_value = prediction['value']
                    probability = prediction['probability']
                    print('prediction : ' + str(predicted_value) + ', probability : ' + str(probability))
                    if 'explanation_features' in prediction:
                        explanation_features = prediction['explanation_features']
                        for explanation_feature in explanation_features:
                            feature_name = explanation_feature['feature_name']
                            feature_weight = explanation_feature['weight']
                            if (feature_weight >= 0 ):
                                feature_weight_percent = round(feature_weight * 100, 2)
                                print(str(feature_name) + ' : ' + str(feature_weight_percent))
                                task_feature_weight_map = {}
                                task_feature_weight_map[result.metadata.explanation_task_id] = feature_weight_percent
                                construct_explanation_features_map(feature_name, feature_weight_percent)
        print('\n>>>>>>>>>>>>>>>>>>>>>>\n')
explanation_features_map
import matplotlib.pyplot as plt
for key in explanation_features_map.keys():
    #plot_graph(key, explanation_features_map[key])
    values = explanation_features_map[key]
    plt.title(key)
    plt.ylabel('Weight')
    plt.bar(range(len(values)), values)
    plt.show()<jupyter_output><empty_output><jupyter_text>Quality monitoring and feedback logging   Enable quality monitoring The code below waits ten seconds to allow the payload logging table to be set up before it begins enabling monitors. First, it turns on the quality (accuracy) monitor and sets an alert threshold of 70%. OpenScale will show an alert on the dashboard if the model accuracy measurement (area under the curve, in the case of a binary classifier) falls below this threshold.The second paramater supplied, min_records, specifies the minimum number of feedback records OpenScale needs before it calculates a new measurement. The quality monitor runs hourly, but the accuracy reading in the dashboard will not change until an additional 50 feedback records have been added, via the user interface, the Python client, or the supplied feedback endpoint.<jupyter_code>import time

#time.sleep(10)
target = Target(
        target_type=TargetTypes.SUBSCRIPTION,
        target_id=subscription_id
)
parameters = {
    "min_feedback_data_size": 90
}
thresholds = [
                {
                    "metric_id": "area_under_roc",
                    "type": "lower_limit",
                    "value": .80
                }
            ]
quality_monitor_details = wos_client.monitor_instances.create(
    data_mart_id=data_mart_id,
    background_mode=False,
    monitor_definition_id=wos_client.monitor_definitions.MONITORS.QUALITY.ID,
    target=target,
    parameters=parameters,
    thresholds=thresholds
).result
quality_monitor_instance_id = quality_monitor_details.metadata.id
quality_monitor_instance_id<jupyter_output><empty_output><jupyter_text>Feedback logging The code below downloads and stores enough feedback data to meet the minimum threshold so that OpenScale can calculate a new accuracy measurement. It then kicks off the accuracy monitor. The monitors run hourly, or can be initiated via the Python API, the REST API, or the graphical user interface.<jupyter_code>!rm additional_feedback_data_v2.json
!wget https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/credit_risk/additional_feedback_data_v2.json<jupyter_output><empty_output><jupyter_text>Get feedback logging dataset ID<jupyter_code>feedback_dataset_id = None
feedback_dataset = wos_client.data_sets.list(type=DataSetTypes.FEEDBACK, 
                                                target_target_id=subscription_id, 
                                                target_target_type=TargetTypes.SUBSCRIPTION).result
feedback_dataset_id = feedback_dataset.data_sets[0].metadata.id
if feedback_dataset_id is None:
    print("Feedback data set not found. Please check quality monitor status.")
with open('additional_feedback_data_v2.json') as feedback_file:
    additional_feedback_data = json.load(feedback_file)
wos_client.data_sets.store_records(feedback_dataset_id, request_body=additional_feedback_data, background_mode=False)
wos_client.data_sets.get_records_count(data_set_id=feedback_dataset_id)
run_details = wos_client.monitor_instances.run(monitor_instance_id=quality_monitor_instance_id, background_mode=False).result
wos_client.monitor_instances.show_metrics(monitor_instance_id=quality_monitor_instance_id)<jupyter_output><empty_output><jupyter_text>Drift configuration   Drift detection model generationPlease update the score function which will be used forgenerating drift detection model which will used for drift detection . This might take sometime to generate model and time taken depends on the training dataset size. The output of the score function should be a 2 arrays 1. Array of model prediction 2. Array of probabilities - User is expected to make sure that the data type of the "class label" column selected and the prediction column are same . For eg : If class label is numeric , the prediction array should also be numeric- Each entry of a probability array should have all the probabities of the unique class lable .  For eg: If the model_type=multiclass and unique class labels are A, B, C, D . Each entry in the probability array should be a array of size 4 . Eg : [ [50,30,10,10] ,[40,20,30,10]...]  **Note:**- *User is expected to add "score" method , which should output prediction column array and probability column array.*- *The data type of the label column and prediction column should be same . User needs to make sure that label column and prediction column array should have the same unique class labels*- **Please update the score function below with the help of templates documented [here](https://github.com/IBM-Watson/aios-data-distribution/blob/master/Score%20function%20templates%20for%20drift%20detection.md)**<jupyter_code>import pandas as pd

df = pd.read_csv("german_credit_data_biased_training.csv")
df.head()
def score(training_data_frame):
    #The data type of the label column and prediction column should be same .
    #User needs to make sure that label column and prediction column array should have the same unique class labels
    prediction_column_name = "predictedLabel"
    probability_column_name = "probability"
    
    feature_columns = list(training_data_frame.columns)
    training_data_rows = training_data_frame[feature_columns].values.tolist()
    
    payload_scoring_records = {
          "fields": feature_columns,
          "values": [x for x in training_data_rows]
      }
    
    header = {"Content-Type": "application/json", "x":"y"}
    scoring_response_raw = requests.post(scoring_url, json=payload_scoring_records, headers=header, verify=False)
    scoring_response = scoring_response_raw.json()

    probability_array = None
    prediction_vector = None
    
    prob_col_index = list(scoring_response.get('fields')).index(probability_column_name)
    predict_col_index = list(scoring_response.get('fields')).index(prediction_column_name)

    if prob_col_index < 0 or predict_col_index < 0:
      raise Exception("Missing prediction/probability column in the scoring response")

    import numpy as np
    probability_array = np.array([value[prob_col_index] for value in scoring_response.get('values')])
    prediction_vector = np.array([value[predict_col_index] for value in scoring_response.get('values')])

    return probability_array, prediction_vector<jupyter_output><empty_output><jupyter_text>Define the drift detection input<jupyter_code>drift_detection_input = {
    "feature_columns": feature_columns,
    "categorical_columns": cat_features,
    "label_column": label_column,
    "problem_type": model_type
}
print(drift_detection_input)<jupyter_output><empty_output><jupyter_text>Generate drift detection model<jupyter_code>!rm drift_detection_model.tar.gz
from ibm_wos_utils.drift.drift_trainer import DriftTrainer
drift_trainer = DriftTrainer(df,drift_detection_input)
if model_type != "regression":
    #Note: batch_size can be customized by user as per the training data size
    drift_trainer.generate_drift_detection_model(score,batch_size=df.shape[0])

#Note: Two column constraints are not computed beyond two_column_learner_limit(default set to 200)
#User can adjust the value depending on the requirement
drift_trainer.learn_constraints(two_column_learner_limit=200)
drift_trainer.create_archive()
!ls -al
filename = 'drift_detection_model.tar.gz'<jupyter_output><empty_output><jupyter_text>Upload the drift detection model to OpenScale subscription<jupyter_code>wos_client.monitor_instances.upload_drift_model(
        model_path=filename,
        archive_name=filename,
        data_mart_id=data_mart_id,
        subscription_id=subscription_id,
        enable_data_drift=True,
        enable_model_drift=True
     )<jupyter_output><empty_output><jupyter_text>Delete the existing drift monitor instance for the subscription<jupyter_code>monitor_instances = wos_client.monitor_instances.list().result.monitor_instances
for monitor_instance in monitor_instances:
    monitor_def_id=monitor_instance.entity.monitor_definition_id
    if monitor_def_id == "drift" and monitor_instance.entity.target.target_id == subscription_id:
        wos_client.monitor_instances.delete(monitor_instance.metadata.id)
        print('Deleted existing drift monitor instance with id: ', monitor_instance.metadata.id)
target = Target(
    target_type=TargetTypes.SUBSCRIPTION,
    target_id=subscription_id

)
parameters = {
    "min_samples": 100,
    "drift_threshold": 0.1,
    "train_drift_model": False,
    "enable_model_drift": True,
    "enable_data_drift": True
}

drift_monitor_details = wos_client.monitor_instances.create(
    data_mart_id=data_mart_id,
    background_mode=False,
    monitor_definition_id=wos_client.monitor_definitions.MONITORS.DRIFT.ID,
    target=target,
    parameters=parameters
).result

drift_monitor_instance_id = drift_monitor_details.metadata.id
drift_monitor_instance_id<jupyter_output><empty_output><jupyter_text>Drift run<jupyter_code>drift_run_details = wos_client.monitor_instances.run(monitor_instance_id=drift_monitor_instance_id, background_mode=False)
time.sleep(5)
wos_client.monitor_instances.show_metrics(monitor_instance_id=drift_monitor_instance_id)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup