<jupyter_start><jupyter_text>Copyright (c) Microsoft Corporation. All rights reserved.Licensed under the MIT License.  AutoML 08: Remote Execution with Text fileIn this sample accesses a data file on a remote DSVM.  This is more efficient than reading the file from Blob storage in the get_data method.Make sure you have executed the [00.configuration](00.configuration.ipynb) before running this notebook.In this notebook you would see1. Configuring the DSVM to allow files to be access directly by the get_data method.2. get_data returning data from a local file.  Create ExperimentAs part of the setup you have already created a Workspace. For AutoML you would need to create an Experiment. An Experiment is a named object in a Workspace, which is used to run experiments.<jupyter_code>import logging
import os
import random

from matplotlib import pyplot as plt
from matplotlib.pyplot import imshow
import numpy as np
import pandas as pd
from sklearn import datasets

import azureml.core
from azureml.core.experiment import Experiment
from azureml.core.workspace import Workspace
from azureml.train.automl import AutoMLConfig
from azureml.train.automl.run import AutoMLRun
ws = Workspace.from_config()

# choose a name for experiment
experiment_name = 'automl-remote-dsvm-file'
# project folder
project_folder = './sample_projects/automl-remote-dsvm-file'

experiment=Experiment(ws, experiment_name)

output = {}
output['SDK version'] = azureml.core.VERSION
output['Subscription ID'] = ws.subscription_id
output['Workspace'] = ws.name
output['Resource Group'] = ws.resource_group
output['Location'] = ws.location
output['Project Directory'] = project_folder
output['Experiment Name'] = experiment.name
pd.set_option('display.max_colwidth', -1)
pd.DataFrame(data=output, index=['']).T<jupyter_output><empty_output><jupyter_text>DiagnosticsOpt-in diagnostics for better experience, quality, and security of future releases<jupyter_code>from azureml.telemetry import set_diagnostics_collection
set_diagnostics_collection(send_diagnostics=True)<jupyter_output><empty_output><jupyter_text>Create a Remote Linux DSVMNote: If creation fails with a message about Marketplace purchase eligibilty, go to portal.azure.com, start creating DSVM there, and select "Want to create programmatically" to enable programmatic creation. Once you've enabled it, you can exit without actually creating VM.**Note**: By default SSH runs on port 22 and you don't need to specify it. But if for security reasons you can switch to a different port (such as 5022), you can append the port number to the address. [Read more](https://render.githubusercontent.com/documentation/sdk/ssh-issue.md) on this.<jupyter_code>from azureml.core.compute import DsvmCompute

dsvm_name = 'mydsvm'
try:
    dsvm_compute = DsvmCompute(ws, dsvm_name)
    print('found existing dsvm.')
except:
    print('creating new dsvm.')
    dsvm_config = DsvmCompute.provisioning_configuration(vm_size = "Standard_D2_v2")
    dsvm_compute = DsvmCompute.create(ws, name = dsvm_name, provisioning_configuration = dsvm_config)
    dsvm_compute.wait_for_completion(show_output = True)<jupyter_output><empty_output><jupyter_text>Copy data file to the DSVMDownload the data file.Copy the data file to the DSVM under the folder: /tmp/data<jupyter_code>df = pd.read_csv("https://automldemods.blob.core.windows.net/datasets/PlayaEvents2016,_1.6MB,_3.4k-rows.cleaned.2.tsv",
                     delimiter="\t", quotechar='"')
df.to_csv("data.tsv", sep="\t", quotechar='"', index=False)

# Now copy the file data.tsv to the folder /tmp/data on the DSVM<jupyter_output><empty_output><jupyter_text>Create Get Data FileFor remote executions you should author a get_data.py file containing a get_data() function. This file should be in the root directory of the project. You can encapsulate code to read data either from a blob storage or local disk in this file.The *get_data()* function returns a [dictionary](README.mdgetdata).<jupyter_code>if not os.path.exists(project_folder):
    os.makedirs(project_folder)
%%writefile $project_folder/get_data.py

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
import os

def get_data():
    # Burning man 2016 data
    df = pd.read_csv('/tmp/data/data.tsv',
                     delimiter="\t", quotechar='"')
    # get integer labels
    le = LabelEncoder()
    le.fit(df["Label"].values)
    y = le.transform(df["Label"].values)
    df = df.drop(["Label"], axis=1)

    df_train, _, y_train, _ = train_test_split(df, y, test_size=0.1, random_state=42)

    return { "X" : df.values, "y" : y }<jupyter_output><empty_output><jupyter_text>Instantiate AutoML You can specify automl_settings as **kwargs** as well. Also note that you can use the get_data() symantic for local excutions too. Note: For Remote DSVM and Batch AI you cannot pass Numpy arrays directly to the fit method.|Property|Description||-|-||**primary_metric**|This is the metric that you want to optimize. Classification supports the following primary metrics accuracyAUC_weightedbalanced_accuracyaverage_precision_score_weightedprecision_score_weighted||**max_time_sec**|Time limit in seconds for each iteration||**iterations**|Number of iterations. In each iteration Auto ML trains a specific pipeline with the data||**n_cross_validations**|Number of cross validation splits||**concurrent_iterations**|Max number of iterations that would be executed in parallel.  This should be less than the number of cores on the DSVM|**preprocess**| *True/False* Setting this to *True* enables Auto ML to perform preprocessing on the input to handle *missing data*, and perform some common *feature extraction*||**max_cores_per_iteration**| Indicates how many cores on the compute target would be used to train a single pipeline. Default is *1*, you can set it to *-1* to use all cores|<jupyter_code>automl_settings = {
    "max_time_sec": 3600,
    "iterations": 10,
    "n_cross_validations": 5,
    "primary_metric": 'AUC_weighted',
    "preprocess": True,
    "max_cores_per_iteration": 2,
    "verbosity": logging.INFO
}
automl_config = AutoMLConfig(task = 'classification',
                             debug_log = 'automl_errors.log',
                             path=project_folder,
                             compute_target = dsvm_compute,
                             data_script = project_folder + "/get_data.py",
                             **automl_settings
                            )<jupyter_output><empty_output><jupyter_text>Training the Model For remote runs the execution is asynchronous, so you will see the iterations get populated as they complete. You can interact with the widgets/models even when the experiment is running to retreive the best model up to that point. Once you are satisfied with the model you can cancel a particular iteration or the whole run.<jupyter_code>remote_run = experiment.submit(automl_config, show_output=False)<jupyter_output><empty_output><jupyter_text>Exploring the Results  Widget for monitoring runsThe widget will sit on "loading" until the first iteration completed, then you will see an auto-updating graph and table show up. It refreshed once per minute, so you should see the graph update as child runs complete.You can click on a pipeline to see run properties and output logs. Logs are also available on the DSVM under /tmp/azureml_run/{iterationid}/azureml-logsNOTE: The widget displays a link at the bottom. This links to a web-ui to explore the individual run details.<jupyter_code>from azureml.train.widgets import RunDetails
RunDetails(remote_run).show()<jupyter_output><empty_output><jupyter_text>Retrieve All Child RunsYou can also use sdk methods to fetch all the child runs and see individual metrics that we log.<jupyter_code>children = list(remote_run.get_children())
metricslist = {}
for run in children:
    properties = run.get_properties()
    metrics = {k: v for k, v in run.get_metrics().items() if isinstance(v, float)}    
    metricslist[int(properties['iteration'])] = metrics

rundata = pd.DataFrame(metricslist).sort_index(1)
rundata<jupyter_output><empty_output><jupyter_text>Canceling runsYou can cancel ongoing remote runs using the *cancel()* and *cancel_iteration()* functions<jupyter_code># Cancel the ongoing experiment and stop scheduling new iterations
# remote_run.cancel()

# Cancel iteration 1 and move onto iteration 2
# remote_run.cancel_iteration(1)<jupyter_output><empty_output><jupyter_text>Retrieve the Best ModelBelow we select the best pipeline from our iterations. The *get_output* method on automl_classifier returns the best run and the fitted model for the last *fit* invocation. There are overloads on *get_output* that allow you to retrieve the best run and fitted model for *any* logged metric or a particular *iteration*.<jupyter_code>best_run, fitted_model = remote_run.get_output()<jupyter_output><empty_output><jupyter_text>Best Model based on any other metric<jupyter_code># lookup_metric = "accuracy"
# best_run, fitted_model = remote_run.get_output(metric=lookup_metric)<jupyter_output><empty_output><jupyter_text>Model from a specific iteration<jupyter_code># iteration = 1
# best_run, fitted_model = remote_run.get_output(iteration=iteration)<jupyter_output><empty_output><jupyter_text>Register fitted model for deployment<jupyter_code>description = 'AutoML Model'
tags = None
remote_run.register_model(description=description, tags=tags)
remote_run.model_id # Use this id to deploy the model as a web service in Azure<jupyter_output><empty_output><jupyter_text>Testing the Fitted Model<jupyter_code>import sklearn
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
from pandas_ml import ConfusionMatrix

df = pd.read_csv("https://automldemods.blob.core.windows.net/datasets/PlayaEvents2016,_1.6MB,_3.4k-rows.cleaned.2.tsv",
                     delimiter="\t", quotechar='"')

# get integer labels
le = LabelEncoder()
le.fit(df["Label"].values)
y = le.transform(df["Label"].values)
df = df.drop(["Label"], axis=1)

_, df_test, _, y_test = train_test_split(df, y, test_size=0.1, random_state=42)

ypred = fitted_model.predict(df_test.values)

ypred_strings = le.inverse_transform(ypred)
ytest_strings = le.inverse_transform(y_test)

cm = ConfusionMatrix(ytest_strings, ypred_strings)

print(cm)

cm.plot()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup