|
|
--- |
|
|
license: apache-2.0 |
|
|
--- |
|
|
|
|
|
### DISTILBERT RUNNING ON [DEEPSPARSE](https://github.com/neuralmagic/deepsparse) GOES BRHMMMMMMMM. 🚀🚀🚀 |
|
|
|
|
|
This model is 👇 |
|
|
|
|
|
███████╗ ██████╗ █████╗ ██████╗ ███████╗ ███████╗ |
|
|
██╔════╝ ██╔══██╗ ██╔══██╗ ██╔══██╗ ██╔════╝ ██╔════╝ |
|
|
███████╗ ██████╔╝ ███████║ ██████╔╝ ███████╗ █████╗ |
|
|
╚════██║ ██╔═══╝ ██╔══██║ ██╔══██╗ ╚════██║█ █╔══╝ |
|
|
███████║ ██║ ██║ ██║ ██║ ██ ║███████║ ███████╗ |
|
|
╚══════╝ ╚═╝ ╚═╝ ╚═╝ ╚═╝ ╚═ ╝╚══════╝ ╚══════╝ |
|
|
|
|
|
 |
|
|
|
|
|
|
|
|
### LOOKS LIKE THIS 👇 |
|
|
|
|
|
 |
|
|
|
|
|
### Inference endpoints, outside of outliers (4ms) is avg. latency on 2 vCPUs: |
|
|
|
|
|
 |
|
|
|
|
|
|
|
|
### Handler for access to inference endpoints |
|
|
|
|
|
```python |
|
|
class EndpointHandler: |
|
|
|
|
|
def __init__(self, path=""): |
|
|
|
|
|
self.pipeline = Pipeline.create(task="text-classification", model_path=path) |
|
|
|
|
|
def __call__(self, data: Dict[str, Any]) -> Dict[str, str]: |
|
|
""" |
|
|
Args: |
|
|
data (:obj:): prediction input text |
|
|
""" |
|
|
inputs = data.pop("inputs", data) |
|
|
|
|
|
start = perf_counter() |
|
|
prediction = self.pipeline(inputs) |
|
|
end = perf_counter() |
|
|
latency = end - start |
|
|
|
|
|
return { |
|
|
"labels": prediction.labels, |
|
|
"scores": prediction.scores, |
|
|
"latency (secs.)": latency |
|
|
} |
|
|
``` |
|
|
|
|
|
̷͈̍ |
|
|
̵̳͒R̶̙̓i̸̟͘c̴̻̆k̸̑͜ÿ̷̳́ |
|
|
̸̪̚ |
|
|
̷͖̀ |
