<jupyter_start><jupyter_text>Copyright (c) Microsoft Corporation. All rights reserved.Licensed under the MIT License.  AutoML 08: Remote Execution with DataStoreIn this sample accesses a data file on a remote DSVM through DataStore. Advantagets of using data store1. DataStore secures the access details.2. DataStore supports read, write to blob and file store3. AutoML natively supports copying data from DataStore to DSVMMake sure you have executed the [00.configuration](00.configuration.ipynb) before running this notebook.In this notebook you would see1. Configuring the DSVM to allow files to be access directly by the get_data method.2. get_data returning data from a local file.  Create ExperimentAs part of the setup you have already created a Workspace. For AutoML you would need to create an Experiment. An Experiment is a named object in a Workspace, which is used to run experiments.<jupyter_code>import logging
import os
import random

from matplotlib import pyplot as plt
from matplotlib.pyplot import imshow
import numpy as np
import pandas as pd
from sklearn import datasets

import azureml.core
from azureml.core.experiment import Experiment
from azureml.core.workspace import Workspace
from azureml.train.automl import AutoMLConfig
from azureml.train.automl.run import AutoMLRun
ws = Workspace.from_config()

# choose a name for experiment
experiment_name = 'automl-remote-datastore-file'
# project folder
project_folder = './sample_projects/automl-remote-dsvm-file'

experiment=Experiment(ws, experiment_name)

output = {}
output['SDK version'] = azureml.core.VERSION
output['Subscription ID'] = ws.subscription_id
output['Workspace'] = ws.name
output['Resource Group'] = ws.resource_group
output['Location'] = ws.location
output['Project Directory'] = project_folder
output['Experiment Name'] = experiment.name
pd.set_option('display.max_colwidth', -1)
pd.DataFrame(data=output, index=['']).T<jupyter_output><empty_output><jupyter_text>DiagnosticsOpt-in diagnostics for better experience, quality, and security of future releases<jupyter_code>from azureml.telemetry import set_diagnostics_collection
set_diagnostics_collection(send_diagnostics=True)<jupyter_output><empty_output><jupyter_text>Create a Remote Linux DSVMNote: If creation fails with a message about Marketplace purchase eligibilty, go to portal.azure.com, start creating DSVM there, and select "Want to create programmatically" to enable programmatic creation. Once you've enabled it, you can exit without actually creating VM.**Note**: By default SSH runs on port 22 and you don't need to specify it. But if for security reasons you can switch to a different port (such as 5022), you can append the port number to the address. [Read more](https://render.githubusercontent.com/documentation/sdk/ssh-issue.md) on this.<jupyter_code>from azureml.core.compute import DsvmCompute
from azureml.core.compute_target import ComputeTargetException

compute_target_name = 'mydsvm'

try:
    dsvm_compute = DsvmCompute(workspace=ws, name=compute_target_name)
    print('found existing:', dsvm_compute.name)
except ComputeTargetException:
    dsvm_config = DsvmCompute.provisioning_configuration(vm_size="Standard_D2_v2")
    dsvm_compute = DsvmCompute.create(ws, name=compute_target_name, provisioning_configuration=dsvm_config)
    dsvm_compute.wait_for_completion(show_output=True)<jupyter_output><empty_output><jupyter_text>Copy data file to localDownload the data file.<jupyter_code>mkdir data
df = pd.read_csv("https://automldemods.blob.core.windows.net/datasets/PlayaEvents2016,_1.6MB,_3.4k-rows.cleaned.2.tsv",
                     delimiter="\t", quotechar='"')
df.to_csv("data/data.tsv", sep="\t", quotechar='"', index=False)<jupyter_output><empty_output><jupyter_text>Upload data to the cloud Now make the data accessible remotely by uploading that data from your local machine into Azure so it can be accessed for remote training. The datastore is a convenient construct associated with your workspace for you to upload/download data, and interact with it from your remote compute targets. It is backed by Azure blob storage account.The data.tsv files are uploaded into a directory named data at the root of the datastore.<jupyter_code>from azureml.core import Workspace, Datastore
#blob_datastore = Datastore(ws, blob_datastore_name)
ds = ws.get_default_datastore()
print(ds.datastore_type, ds.account_name, ds.container_name)
# ds.upload_files("data.tsv")
ds.upload(src_dir='./data', target_path='data', overwrite=True, show_progress=True)<jupyter_output><empty_output><jupyter_text>Configure & RunFirst let's create a DataReferenceConfigruation object to inform the system what data folder to download to the copmute target.<jupyter_code>from azureml.core.runconfig import DataReferenceConfiguration
dr = DataReferenceConfiguration(datastore_name=ds.name, 
                   path_on_datastore='data', 
                   mode='download', # download files from datastore to compute target
                   overwrite=True)
from azureml.core.runconfig import RunConfiguration

# create a new RunConfig object
conda_run_config = RunConfiguration(framework="python")

# Set compute target to the Linux DSVM
conda_run_config.target = dsvm_compute.name
# set the data reference of the run coonfiguration
conda_run_config.data_references = {ds.name: dr}<jupyter_output><empty_output><jupyter_text>Create Get Data FileFor remote executions you should author a get_data.py file containing a get_data() function. This file should be in the root directory of the project. You can encapsulate code to read data either from a blob storage or local disk in this file.The *get_data()* function returns a [dictionary](README.mdgetdata).<jupyter_code>if not os.path.exists(project_folder):
    os.makedirs(project_folder)
%%writefile $project_folder/get_data.py

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
import os
from os.path import expanduser, join, dirname

def get_data():
    # Burning man 2016 data
    df = pd.read_csv(join(dirname(os.path.realpath(__file__)),
                          os.environ["AZUREML_DATAREFERENCE_workspacefilestore"],
                          "data.tsv"), delimiter="\t", quotechar='"')
    # get integer labels
    le = LabelEncoder()
    le.fit(df["Label"].values)
    y = le.transform(df["Label"].values)
    df = df.drop(["Label"], axis=1)

    df_train, _, y_train, _ = train_test_split(df, y, test_size=0.1, random_state=42)

    return { "X" : df.values, "y" : y }<jupyter_output><empty_output><jupyter_text>Instantiate AutoML You can specify automl_settings as **kwargs** as well. Also note that you can use the get_data() symantic for local excutions too. Note: For Remote DSVM and Batch AI you cannot pass Numpy arrays directly to the fit method.|Property|Description||-|-||**primary_metric**|This is the metric that you want to optimize. Classification supports the following primary metrics accuracyAUC_weightedbalanced_accuracyaverage_precision_score_weightedprecision_score_weighted||**max_time_sec**|Time limit in seconds for each iteration||**iterations**|Number of iterations. In each iteration Auto ML trains a specific pipeline with the data||**n_cross_validations**|Number of cross validation splits||**concurrent_iterations**|Max number of iterations that would be executed in parallel.  This should be less than the number of cores on the DSVM|**preprocess**| *True/False* Setting this to *True* enables Auto ML to perform preprocessing on the input to handle *missing data*, and perform some common *feature extraction*||**max_cores_per_iteration**| Indicates how many cores on the compute target would be used to train a single pipeline. Default is *1*, you can set it to *-1* to use all cores|<jupyter_code>automl_settings = {
    "max_time_sec": 3600,
    "iterations": 10,
    "n_cross_validations": 5,
    "primary_metric": 'AUC_weighted',
    "preprocess": True,
    "max_cores_per_iteration": 2,
    "verbosity": logging.INFO
}
automl_config = AutoMLConfig(task = 'classification',
                             debug_log = 'automl_errors.log',
                             path=project_folder,
                             run_configuration=conda_run_config,
                             #compute_target = dsvm_compute,
                             data_script = project_folder + "/get_data.py",
                             **automl_settings
                            )<jupyter_output><empty_output><jupyter_text>Training the Model For remote runs the execution is asynchronous, so you will see the iterations get populated as they complete. You can interact with the widgets/models even when the experiment is running to retreive the best model up to that point. Once you are satisfied with the model you can cancel a particular iteration or the whole run.<jupyter_code>remote_run = experiment.submit(automl_config, show_output=False)<jupyter_output><empty_output><jupyter_text>Exploring the Results  Widget for monitoring runsThe widget will sit on "loading" until the first iteration completed, then you will see an auto-updating graph and table show up. It refreshed once per minute, so you should see the graph update as child runs complete.You can click on a pipeline to see run properties and output logs. Logs are also available on the DSVM under /tmp/azureml_run/{iterationid}/azureml-logsNOTE: The widget displays a link at the bottom. This links to a web-ui to explore the individual run details.<jupyter_code>from azureml.train.widgets import RunDetails
RunDetails(remote_run).show()<jupyter_output><empty_output><jupyter_text>Retrieve All Child RunsYou can also use sdk methods to fetch all the child runs and see individual metrics that we log.<jupyter_code>children = list(remote_run.get_children())
metricslist = {}
for run in children:
    properties = run.get_properties()
    metrics = {k: v for k, v in run.get_metrics().items() if isinstance(v, float)}    
    metricslist[int(properties['iteration'])] = metrics

rundata = pd.DataFrame(metricslist).sort_index(1)
rundata<jupyter_output><empty_output><jupyter_text>Canceling runsYou can cancel ongoing remote runs using the *cancel()* and *cancel_iteration()* functions<jupyter_code># Cancel the ongoing experiment and stop scheduling new iterations
remote_run.cancel()

# Cancel iteration 1 and move onto iteration 2
# remote_run.cancel_iteration(1)<jupyter_output><empty_output><jupyter_text>Retrieve the Best ModelBelow we select the best pipeline from our iterations. The *get_output* method on automl_classifier returns the best run and the fitted model for the last *fit* invocation. There are overloads on *get_output* that allow you to retrieve the best run and fitted model for *any* logged metric or a particular *iteration*.<jupyter_code>best_run, fitted_model = remote_run.get_output()<jupyter_output><empty_output><jupyter_text>Best Model based on any other metric<jupyter_code># lookup_metric = "accuracy"
# best_run, fitted_model = remote_run.get_output(metric=lookup_metric)<jupyter_output><empty_output><jupyter_text>Model from a specific iteration<jupyter_code># iteration = 1
# best_run, fitted_model = remote_run.get_output(iteration=iteration)<jupyter_output><empty_output><jupyter_text>Register fitted model for deployment<jupyter_code>#description = 'AutoML Model'
#tags = None
#remote_run.register_model(description=description, tags=tags)
#remote_run.model_id # Use this id to deploy the model as a web service in Azure<jupyter_output><empty_output><jupyter_text>Testing the Fitted Model<jupyter_code>import sklearn
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
from pandas_ml import ConfusionMatrix

df = pd.read_csv("https://automldemods.blob.core.windows.net/datasets/PlayaEvents2016,_1.6MB,_3.4k-rows.cleaned.2.tsv",
                     delimiter="\t", quotechar='"')

# get integer labels
le = LabelEncoder()
le.fit(df["Label"].values)
y = le.transform(df["Label"].values)
df = df.drop(["Label"], axis=1)

_, df_test, _, y_test = train_test_split(df, y, test_size=0.1, random_state=42)

ypred = fitted_model.predict(df_test.values)

ypred_strings = le.inverse_transform(ypred)
ytest_strings = le.inverse_transform(y_test)

cm = ConfusionMatrix(ytest_strings, ypred_strings)

print(cm)

cm.plot()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup