<jupyter_start><jupyter_text>Scale Seldon Deployments based on Prometheus Metrics.This notebook shows how you can scale Seldon Deployments based on Prometheus metrics via KEDA. [KEDA](https://keda.sh/) is a Kubernetes-based Event Driven Autoscaler. With KEDA, you can drive the scaling of any container in Kubernetes based on the number of events needing to be processed. With the support of KEDA in Seldon, you can scale your seldon deployments with any scalers listed [here](https://keda.sh/docs/2.0/scalers/).In this example we will scale the seldon deployment with Prometheus metrics as an example.  Install Seldon CoreInstall Seldon Core as described in [docs](https://docs.seldon.io/projects/seldon-core/en/latest/workflow/install.html)Make sure add `--set keda.enabled=true`  Install Seldon Core Analyticseldon-core-analytics contains Prometheus and Grafana installation with a basic Grafana dashboard showing the default Prometheus metrics exposed by Seldon for each inference graph deployed. Later we will use the Prometheus service installed to provide metrics in order to scale the Seldon models.Install Seldon Core Analytics as described in [docs](https://docs.seldon.io/projects/seldon-core/en/latest/analytics/analytics.html)<jupyter_code>!helm install seldon-core-analytics ../../helm-charts/seldon-core-analytics -n seldon-system --wait<jupyter_output><empty_output><jupyter_text>Install KEDA<jupyter_code>!kubectl delete -f https://github.com/kedacore/keda/releases/download/v2.0.0/keda-2.0.0.yaml
!kubectl apply -f https://github.com/kedacore/keda/releases/download/v2.0.0/keda-2.0.0.yaml
!kubectl get pod -n keda<jupyter_output><empty_output><jupyter_text>Create model with KEDA To create a model with KEDA autoscaling you just need to add a KEDA spec referring in the Deployment, e.g.:```yamlkedaSpec:  pollingInterval: 15                                 Optional. Default: 30 seconds  minReplicaCount: 1                                  Optional. Default: 0  maxReplicaCount: 5                                  Optional. Default: 100  triggers:  - type: prometheus    metadata:       Required      serverAddress: http://seldon-core-analytics-prometheus-seldon.seldon-system.svc.cluster.local      metricName: access_frequency      threshold: '10'      query: rate(seldon_api_executor_client_requests_seconds_count{seldon_app=~"seldon-model-example"}[10s]```The full SeldonDeployment spec is shown below.<jupyter_code>VERSION = !cat ../../version.txt
VERSION = VERSION[0]
VERSION
%%writefile model_with_keda_prom.yaml
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: seldon-model
spec:
  name: test-deployment
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: seldonio/mock_classifier:1.5.0-dev
          imagePullPolicy: IfNotPresent
          name: classifier
          resources:
            requests:
              cpu: '0.5'
      kedaSpec:
        pollingInterval: 15                                # Optional. Default: 30 seconds
        minReplicaCount: 1                                 # Optional. Default: 0
        maxReplicaCount: 5                                 # Optional. Default: 100
        triggers:
        - type: prometheus
          metadata:
            # Required
            serverAddress: http://seldon-core-analytics-prometheus-seldon.seldon-system.svc.cluster.local
            metricName: access_frequency
            threshold: '10'
            query: rate(seldon_api_executor_client_requests_seconds_count{seldon_app=~"seldon-model-example"}[1m])
    graph:
      children: []
      endpoint:
        type: REST
      name: classifier
      type: MODEL
    name: example

!kubectl create -f model_with_keda_prom.yaml
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=seldon-model -o jsonpath='{.items[0].metadata.name}')<jupyter_output><empty_output><jupyter_text>Create Load We label some nodes for the loadtester. We attempt the first two as for Kind the first node shown will be the master.<jupyter_code>!kubectl label nodes $(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') role=locust
!kubectl label nodes $(kubectl get nodes -o jsonpath='{.items[1].metadata.name}') role=locust<jupyter_output><empty_output><jupyter_text>Before add loads to the model, there is only one replica<jupyter_code>!kubectl get deployment seldon-model-example-0-classifier
!helm install seldon-core-loadtesting seldon-core-loadtesting --repo https://storage.googleapis.com/seldon-charts \
    --set locust.host=http://seldon-model-example:8000 \
    --set oauth.enabled=false \
    --set locust.hatchRate=1 \
    --set locust.clients=1 \
    --set loadtest.sendFeedback=0 \
    --set locust.minWait=0 \
    --set locust.maxWait=0 \
    --set replicaCount=1<jupyter_output><empty_output><jupyter_text>After a few mins you should see the deployment scaled to 5 replicas<jupyter_code>import json
import time


def getNumberPods():
    dp = !kubectl get deployment seldon-model-example-0-classifier -o json
    dp = json.loads("".join(dp))
    return dp["status"]["replicas"]


scaled = False
for i in range(60):
    pods = getNumberPods()
    print(pods)
    if pods > 1:
        scaled = True
        break
    time.sleep(5)
assert scaled
!kubectl get deployment/seldon-model-example-0-classifier scaledobject/seldon-model-example-0-classifier<jupyter_output><empty_output><jupyter_text>Remove Load<jupyter_code>!helm delete seldon-core-loadtesting<jupyter_output><empty_output><jupyter_text>After 5-10 mins you should see the deployment replica number decrease to 1<jupyter_code>!kubectl get pods,deployments,hpa,scaledobject
!kubectl delete -f model_with_keda_prom.yaml<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup