<jupyter_start><jupyter_text>RAPIDS + Dask examples on AI Notebooks**This example is not an officially supported Google product, does not have a SLA/SLO, and should not be used in production.**  OverviewGPUs can greatly accelerate all stages of an ML pipeline: pre-processing, training, and inference. In this blog, we will be focusing on the pre-processing and training stages, using Python in a Jupyter Notebook environment.First, we will use Dask/RAPIDS to read a dataset into NVIDIA GPU memory and execute some basic functions. Then, we’ll use Dask to scale beyond our GPU memory capacity.Next, we’ll scale XGBoost across multiple NVIDIA A100 Tensor Core GPUs, by submitting an AI Platform Training job with a custom container. Finally, you can deploy your model for online inference, accelerated by GPUs, using AI Platform Predictions. DatasetThe dataset we will be using for this tutorial is simulated particle activity data that was released for the Higgs Boson Machine Learning Challenge. We will be replicating the dataset, and using different subsets of Higgs (some larger, some smaller) to demonstrate the scaling ability of Dask on AI Platform. Executing makedataset.sh will download the dataset, replicate it, and upload it to GCS in the github. You can execute this script in your AI Notebook, or run it in a separate GCE instance, or run it locally if you wish.  Costs This tutorial uses billable components of Google Cloud Platform (GCP):* Cloud AI Platform* Cloud StorageLearn about [Cloud AI Platformpricing](https://cloud.google.com/ml-engine/docs/pricing) and [Cloud Storagepricing](https://cloud.google.com/storage/pricing), and use the [PricingCalculator](https://cloud.google.com/products/calculator/)to generate a cost estimate based on your projected usage.  Set up your local development environment**If you are using Colab or AI Platform Notebooks**, your environment already meetsall the requirements to run this notebook. You can skip this step.**Otherwise**, make sure your environment meets this notebook's requirements.You need the following:* The Google Cloud SDK* Git* Python 3* virtualenv* Jupyter notebook running in a virtual environment with Python 3The Google Cloud guide to [Setting up a Python developmentenvironment](https://cloud.google.com/python/setup) and the [Jupyterinstallation guide](https://jupyter.org/install) provide detailed instructionsfor meeting these requirements. The following steps provide a condensed set ofinstructions:1. [Install and initialize the Cloud SDK.](https://cloud.google.com/sdk/docs/)2. [Install Python 3.](https://cloud.google.com/python/setupinstalling_python)3. [Install   virtualenv](https://cloud.google.com/python/setupinstalling_and_using_virtualenv)   and create a virtual environment that uses Python 3.4. Activate that environment and run `pip install jupyter` in a shell to install   Jupyter.5. Run `jupyter notebook` in a shell to launch Jupyter.6. Open this notebook in the Jupyter Notebook Dashboard.  Before you begin Set up your GCP project**The following steps are required, regardless of your notebook environment.**1. [Select or create a GCP project.](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)3. [Enable the AI Platform APIs and Compute Engine APIs.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component)4. If you are running this notebook locally, you will need to install [Google Cloud SDK](https://cloud.google.com/sdk).5. Enter your project ID in the cell below. Then run the  cell to make sure theCloud SDK uses the right project for all the commands in this notebook.**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands.  Authenticate your GCP account**If you are using AI Platform Notebooks**, your environment is alreadyauthenticated. Skip this step.**If you are using Colab**, run the cell below and follow the instructionswhen prompted to authenticate your account via oAuth.**Otherwise**, follow these steps:1. In the GCP Console, go to the [**Create service account key**   page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).2. From the **Service account** drop-down list, select **New service account**.3. In the **Service account name** field, enter a name.4. From the **Role** drop-down list, select   **Machine Learning Engine > AI Platform Admin** and   **Storage > Storage Object Admin**.5. Click *Create*. A JSON file that contains your key downloads to yourlocal environment.6. Enter the path to your service account key as the`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell.<jupyter_code>## Create conda environment - execute in terminal
'''
conda create -n rapids-0.17 -c rapidsai -c nvidia \
-c conda-forge -c dask -c defaults rapids-blazing=0.17 python=3.7 cudatoolkit=11.0  gcsfs=0.7.1 -y

conda activate rapids-0.17
'''
!git clone https://github.com/remylouisew/rapids_AIP.git
import dask_cudf
import cupy as cp
import time
import gcsfs
import os, json
import subprocess
import pandas as pd
import time


GCS_BUCKET = "bucket_name/optional_folder" #replace with your GCS bucket name or GCS folder path
# Instatiate LocalCUDACluster to assign dask processes to GPUs

from dask_cuda import LocalCUDACluster
from dask.distributed import Client, wait

cluster = LocalCUDACluster()
client = Client(cluster)

# Read dataset into Dask DMatrix from Google Cloud Storage

colnames = ['label'] + ['feature%02d' % i for i in range(1, 29)]
train_dir='gs://' + GCS_BUCKET + '/abcdefghij/*.csv' #GCS public bucket. We're using GCS's wildcard functionality to select the files we need
df = dask_cudf.read_csv(train_dir, header=None, names=colnames, chunksize=None)

print("Number of partitions is", df.npartitions)
%%time
# Some basic functions using Dask

df["key"] = df.feature02.round()
group_means = df.groupby("key").mean().persist()
wait(group_means);

group_means.head()

#group_means.compute() will output the cuDF Dataframe as a Pandas DataFrame. A good workflow would be to summarize your data using Dask, then output to pandas for plotting or other pandas functions.

# This will allow you to see memory usage your GPUs in real-time 
# Better to execute in terminal rather than in Notebook
!nvidia-smi dmon
# Now restart the kernal so that you can instatiate a new LocalCUDAcluster, 
# this time one that will spill to host memory when the GPU memory is exceeded

import dask_cudf
import cupy as cp
import argparse
import time
import gcsfs
import dask_cudf
import os, json
import subprocess
from dask_cuda import LocalCUDACluster
from dask.distributed import Client, wait
from dask.utils import parse_bytes

cluster = LocalCUDACluster(CUDA_VISIBLE_DEVICES="0",
    #Dask will spill to disk by default, but setting device_memory_limit allows you to control when that will happen
    device_memory_limit=parse_bytes("14GB"),)
client = Client(cluster)
client

%%time
# Read in larger dataset (20GB) from GCS

colnames = ['label'] + ['feature%02d' % i for i in range(1, 29)]
train_dir='gs://' + GCS_BUCKET + 'abcdefghi*/*.csv' #GCS public bucket
df = dask_cudf.read_csv(train_dir, header=None, names=colnames, chunksize=None)

print("Number of partitions is", df.npartitions)

# Run the dask functions, which will require nearly double the memory available on the GPU
df["key"] = df.feature02.round()
group_means = df.groupby("key").mean().persist()
wait(group_means);

group_means.head()<jupyter_output>Number of partitions is 30
CPU times: user 5.99 s, sys: 1.11 s, total: 7.1 s
Wall time: 3min 15s
StarcoderdataJupyterStructuredCleanDedup