<jupyter_start><jupyter_text>Copyright (c) Microsoft Corporation. All rights reserved.  Licensed under the MIT License.  Inference PyTorch Bert Model with ONNX Runtime on GPU In this tutorial, you'll learn how to load a Bert model from PyTorch, convert it to ONNX, and inference it for high performance using ONNX Runtime and NVIDIA GPU. In the following sections, we are going to use the Bert model trained with Stanford Question Answering Dataset (SQuAD) dataset as an example. Bert SQuAD model is used in question answering scenarios, where the answer to every question is a segment of text from the corresponding reading passage, or the question might be unanswerable.This notebook is for GPU inference. For CPU inference, please look at another notebook [Inference PyTorch Bert Model with ONNX Runtime on CPU](PyTorch_Bert-Squad_OnnxRuntime_CPU.ipynb).  0. Prerequisites It requires your machine to have a GPU, and a python environment with [PyTorch](https://pytorch.org/) installed before running this notebook. GPU Environment Setup using AnaCondaFirst, we install [AnaConda](https://www.anaconda.com/distribution/) in a target machine and open an AnaConda prompt window when it is done. Then run the following commands to create a conda environment. This notebook is tested with PyTorch 1.5.0 and OnnxRuntime 1.3.0.```consoleconda create -n gpu_env python=3.6conda activate gpu_envconda install -c anaconda ipykernelconda install -c conda-forge ipywidgetspython -m ipykernel install --user --name=gpu_envjupyter notebook```Finally, launch Jupyter Notebook and you can choose gpu_env as kernel to run this notebook.Onnxruntime-gpu need specified version of CUDA and cuDNN. You can find the Requirements [here]( http://www.onnxruntime.ai/docs/how-to/install.html). Remember to add the directories to PATH environment variable (See [CUDA and cuDNN Path](CUDA-and-cuDNN-Path) below).<jupyter_code>import sys

run_install = False # Only need install once
if run_install:
    if sys.platform in ['linux', 'win32']: # Linux or Windows
        !{sys.executable} -m pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio===0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
    else: # Mac
        print("PyTorch 1.9 MacOS Binaries do not support CUDA, install from source instead")

    !{sys.executable} -m pip install onnxruntime-gpu==1.8.1 onnx==1.9.0 onnxconverter_common==1.8.1

    # Install other packages used in this notebook.
    !{sys.executable} -m pip install transformers==4.8.2
    !{sys.executable} -m pip install psutil pytz pandas py-cpuinfo py3nvml coloredlogs wget netron sympy
import torch
import onnx
import onnxruntime
import transformers
print("pytorch:", torch.__version__)
print("onnxruntime:", onnxruntime.__version__)
print("onnx:", onnx.__version__)
print("transformers:", transformers.__version__)<jupyter_output>pytorch: 1.9.0+cu111
onnxruntime: 1.8.1
onnx: 1.9.0
transformers: 4.8.2<jupyter_text>1. Load Pretrained Bert model  We begin by downloading the SQuAD data file and store them in the specified location.<jupyter_code>import os

cache_dir = "./squad"
if not os.path.exists(cache_dir):
    os.makedirs(cache_dir)

predict_file_url = "https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json"
predict_file = os.path.join(cache_dir, "dev-v1.1.json")
if not os.path.exists(predict_file):
    import wget
    print("Start downloading predict file.")
    wget.download(predict_file_url, predict_file)
    print("Predict file downloaded.")<jupyter_output><empty_output><jupyter_text>Let's first define some constant variables.<jupyter_code># Whether allow overwriting existing ONNX model and download the latest script from GitHub
enable_overwrite = True

# Total samples to inference, so that we can get average latency
total_samples = 1000

# ONNX opset version
opset_version=11<jupyter_output><empty_output><jupyter_text>Specify some model configuration variables.<jupyter_code># fine-tuned model from https://huggingface.co/models?search=squad
model_name_or_path = "bert-large-uncased-whole-word-masking-finetuned-squad"
max_seq_length = 128
doc_stride = 128
max_query_length = 64<jupyter_output><empty_output><jupyter_text>Start to load model from pretrained. This step could take a few minutes.<jupyter_code># The following code is adapted from HuggingFace transformers
# https://github.com/huggingface/transformers/blob/master/examples/run_squad.py

from transformers import (BertConfig, BertForQuestionAnswering, BertTokenizer)

# Load pretrained model and tokenizer
config_class, model_class, tokenizer_class = (BertConfig, BertForQuestionAnswering, BertTokenizer)
config = config_class.from_pretrained(model_name_or_path, cache_dir=cache_dir)
tokenizer = tokenizer_class.from_pretrained(model_name_or_path, do_lower_case=True, cache_dir=cache_dir)
model = model_class.from_pretrained(model_name_or_path,
                                    from_tf=False,
                                    config=config,
                                    cache_dir=cache_dir)
# load some examples
from transformers.data.processors.squad import SquadV1Processor

processor = SquadV1Processor()
examples = processor.get_dev_examples(None, filename=predict_file)

from transformers import squad_convert_examples_to_features
features, dataset = squad_convert_examples_to_features( 
            examples=examples[:total_samples], # convert enough examples for this notebook
            tokenizer=tokenizer,
            max_seq_length=max_seq_length,
            doc_stride=doc_stride,
            max_query_length=max_query_length,
            is_training=False,
            return_dataset='pt'
        )<jupyter_output>100%|██████████| 48/48 [00:03<00:00, 14.24it/s]
convert squad examples to features: 100%|██████████| 1000/1000 [00:08<00:00, 112.67it/s]
add example index and unique id: 100%|██████████| 1000/1000 [00:00<00:00, 836518.55it/s]<jupyter_text>2. Export the loaded model Once the model is loaded, we can export the loaded PyTorch model to ONNX.<jupyter_code>output_dir = "./onnx"
if not os.path.exists(output_dir):
    os.makedirs(output_dir)   
export_model_path = os.path.join(output_dir, 'bert-base-cased-squad_opset{}.onnx'.format(opset_version))

import torch
use_gpu = torch.cuda.is_available()
device = torch.device("cuda" if use_gpu else "cpu")

# Get the first example data to run the model and export it to ONNX
data = dataset[0]
inputs = {
    'input_ids':      data[0].to(device).reshape(1, max_seq_length),
    'attention_mask': data[1].to(device).reshape(1, max_seq_length),
    'token_type_ids': data[2].to(device).reshape(1, max_seq_length)
}

# Set model to inference mode, which is required before exporting the model because some operators behave differently in 
# inference and training mode.
model.eval()
model.to(device)

if enable_overwrite or not os.path.exists(export_model_path):
    with torch.no_grad():
        symbolic_names = {0: 'batch_size', 1: 'max_seq_len'}
        torch.onnx.export(model,                                            # model being run
                          args=tuple(inputs.values()),                      # model input (or a tuple for multiple inputs)
                          f=export_model_path,                              # where to save the model (can be a file or file-like object)
                          opset_version=opset_version,                      # the ONNX version to export the model to
                          do_constant_folding=True,                         # whether to execute constant folding for optimization
                          input_names=['input_ids',                         # the model's input names
                                       'input_mask', 
                                       'segment_ids'],
                          output_names=['start', 'end'],                    # the model's output names
                          dynamic_axes={'input_ids': symbolic_names,        # variable length axes
                                        'input_mask' : symbolic_names,
                                        'segment_ids' : symbolic_names,
                                        'start' : symbolic_names,
                                        'end' : symbolic_names})
        print("Model exported at ", export_model_path)<jupyter_output>/disk/conda3/envs/gpu_env/lib/python3.6/site-packages/transformers/modeling_utils.py:1974: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  input_tensor.shape[chunk_dim] == tensor_shape for input_tensor in input_tensors<jupyter_text>3. PyTorch Inference Use PyTorch to evaluate an example input for comparison purpose.<jupyter_code>import time

# Measure the latency. It is not accurate using Jupyter Notebook, it is recommended to use standalone python script.
latency = []
with torch.no_grad():
    for i in range(total_samples):
        data = dataset[i]
        inputs = {
            'input_ids':      data[0].to(device).reshape(1, max_seq_length),
            'attention_mask': data[1].to(device).reshape(1, max_seq_length),
            'token_type_ids': data[2].to(device).reshape(1, max_seq_length)
        }
        start = time.time()
        outputs = model(**inputs)
        latency.append(time.time() - start)
print("PyTorch {} Inference time = {} ms".format(device.type, format(sum(latency) * 1000 / len(latency), '.2f')))<jupyter_output>PyTorch cuda Inference time = 16.56 ms<jupyter_text>4. Inference ONNX Model with ONNX Runtime  CUDA and cuDNN Pathonnxruntime-gpu has dependency on [CUDA](https://developer.nvidia.com/cuda-downloads) and [cuDNN](https://developer.nvidia.com/cudnn). Required CUDA version can be found [here](http://www.onnxruntime.ai/docs/reference/execution-providers/CUDA-ExecutionProvider.htmlrequirements)<jupyter_code># Change to True when onnxruntime (like onnxruntime-gpu 1.0.0 ~ 1.1.2) cannot be imported.
add_cuda_path = False

# For Linux, see https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#environment-setup
# Below is example for Windows
if add_cuda_path:
    cuda_dir = 'D:/NVidia/CUDA/v11.0/bin'
    cudnn_dir = 'D:/NVidia/CUDA/v11.0/bin'
    if not (os.path.exists(cuda_dir) and os.path.exists(cudnn_dir)):
        raise ValueError("Please specify correct path for CUDA and cuDNN. Otherwise onnxruntime cannot be imported.")
    else:
        if cuda_dir == cudnn_dir:
            os.environ["PATH"] = cuda_dir + ';' + os.environ["PATH"]
        else:
            os.environ["PATH"] = cuda_dir + ';' + cudnn_dir + ';' + os.environ["PATH"]<jupyter_output><empty_output><jupyter_text>Now we are ready to inference the model with ONNX Runtime.<jupyter_code>import psutil
import onnxruntime
import numpy

assert 'CUDAExecutionProvider' in onnxruntime.get_available_providers()
device_name = 'gpu'

sess_options = onnxruntime.SessionOptions()

# Optional: store the optimized graph and view it using Netron to verify that model is fully optimized.
# Note that this will increase session creation time so enable it for debugging only.
sess_options.optimized_model_filepath = os.path.join(output_dir, "optimized_model_{}.onnx".format(device_name))

# Please change the value according to best setting in Performance Test Tool result.
sess_options.intra_op_num_threads=psutil.cpu_count(logical=True)

session = onnxruntime.InferenceSession(export_model_path, sess_options)

latency = []
for i in range(total_samples):
    data = dataset[i]
    # TODO: use IO Binding (see https://www.onnxruntime.ai/python/api_summary.html) to improve performance.
    ort_inputs = {
        'input_ids':  data[0].cpu().reshape(1, max_seq_length).numpy(),
        'input_mask': data[1].cpu().reshape(1, max_seq_length).numpy(),
        'segment_ids': data[2].cpu().reshape(1, max_seq_length).numpy()
    }
    start = time.time()
    ort_outputs = session.run(None, ort_inputs)
    latency.append(time.time() - start)
    
print("OnnxRuntime {} Inference time = {} ms".format(device_name, format(sum(latency) * 1000 / len(latency), '.2f')))<jupyter_output>OnnxRuntime gpu Inference time = 25.28 ms<jupyter_text>We can compare the output of PyTorch and ONNX Runtime. We can see some results are not close. It is because ONNX Runtime uses some approximation in CUDA optimization. Based on our evaluation on SQuAD data set, F1 score is on par for models before and after optimization.<jupyter_code>print("***** Verifying correctness *****")
for i in range(2):    
    print('PyTorch and ONNX Runtime output {} are close:'.format(i), numpy.allclose(ort_outputs[i], outputs[i].cpu(), rtol=1e-02, atol=1e-02))
    diff = ort_outputs[i] - outputs[i].cpu().numpy()
    max_diff = numpy.max(numpy.abs(diff))
    avg_diff = numpy.average(numpy.abs(diff))
    print(f'maximum_diff={max_diff} average_diff={avg_diff}')<jupyter_output>***** Verifying correctness *****
PyTorch and ONNX Runtime output 0 are close: True
maximum_diff=5.7220458984375e-06 average_diff=1.3103708624839783e-06
PyTorch and ONNX Runtime output 1 are close: True
maximum_diff=5.7220458984375e-06 average_diff=1.2257369235157967e-06<jupyter_text>Inference with Actual Sequence LengthNote that ONNX model is exported using dynamic length axis. It is recommended to use actual sequence input without padding instead of fixed length input for best performance. Let's see how it can be applied to this model.From an example input below, we can see zero padding at the end of each sequence.<jupyter_code># An example input (we can see padding). From attention_mask, we can deduce the actual length.
inputs<jupyter_output><empty_output><jupyter_text>The original sequence length is 128. After removing paddings, the sequence length is reduced. Input with smaller sequence length need less computation, thus we can see there is improvement on inference latency.<jupyter_code>import statistics

latency = []
lengths = []
for i in range(total_samples):
    data = dataset[i]
    # Instead of using fixed length (128), we can use actual sequence length (less than 128), which helps to get better performance.
    actual_sequence_length = sum(data[1].numpy())
    lengths.append(actual_sequence_length)
    opt_inputs = {
        'input_ids':  data[0].numpy()[:actual_sequence_length].reshape(1, actual_sequence_length),
        'input_mask': data[1].numpy()[:actual_sequence_length].reshape(1, actual_sequence_length),
        'segment_ids': data[2].numpy()[:actual_sequence_length].reshape(1, actual_sequence_length)
    }
    start = time.time()
    opt_outputs = session.run(None, opt_inputs)
    latency.append(time.time() - start)
print("Average length", statistics.mean(lengths))
print("OnnxRuntime {} Inference time with actual sequence length = {} ms".format(device_name, format(sum(latency) * 1000 / len(latency), '.2f')))<jupyter_output>Average length 94
OnnxRuntime gpu Inference time with actual sequence length = 21.93 ms<jupyter_text>Let's compare the output and see whether the results are close.**Note**: Need end-to-end evaluation on performance and accuracy if you use this strategy.<jupyter_code>print("***** Comparing results with/without paddings *****")
for i in range(2):
    print('Output {} are close:'.format(i), numpy.allclose(opt_outputs[i], ort_outputs[i][:,:len(opt_outputs[i][0])], rtol=1e-03, atol=1e-03))<jupyter_output>***** Comparing results with/without paddings *****
Output 0 are close: True
Output 1 are close: True<jupyter_text>5. Offline Optimization and Test ToolsIt is recommended to try [OnnxRuntime Transformer Model Optimization Tool](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/python/tools/transformers) on the exported ONNX models. It could help verify whether the model can be fully optimized, and get performance test results.  Transformer OptimizerAlthough OnnxRuntime could optimize Bert model exported by PyTorch. Sometime, model cannot be fully optimized due to different reasons:* A new subgraph pattern is generated by new version of export tool, and the pattern is not covered by older version of OnnxRuntime. * The exported model uses dynamic axis and this makes it harder for shape inference of the graph. That blocks some optimization to be applied.* Some optimization is better to be done offline. Like change input tensor type from int64 to int32 to avoid extra Cast nodes, or convert model to float16 to achieve better performance in V100 or T4 GPU.We have python script **optimizer.py**, which is more flexible in graph pattern matching and model conversion (like float32 to float16). You can also use it to verify whether a Bert model is fully optimized.In this example, we can see that it introduces optimization that is not provided by onnxruntime: SkipLayerNormalization and bias fusion, which is not fused in OnnxRuntime due to shape inference as mentioned.It will also tell whether the model is fully optimized or not. If not, that means you might need change the script to fuse some new pattern of subgraph.Example Usage:```from onnxruntime.transformers import optimizeroptimized_model = optimizer.optimize_model(export_model_path, model_type='bert', num_heads=12, hidden_size=768)optimized_model.save_model_to_file(optimized_model_path)```You can also use command line like the following:  Float32 ModelLet us optimize the ONNX model using the script. The first example will output model with float32 to store weights. This is the choice for most GPUs without Tensor Core.If your GPU (like V100 or T4) has Tensor Core, jump to [Float16 Model](6.-Model-Optimization-with-Float16) section since that will give you better performance than Float32 model.<jupyter_code>optimized_fp32_model_path = './onnx/bert-base-cased-squad_opt_{}_fp32.onnx'.format('gpu' if use_gpu else 'cpu')

!{sys.executable} -m onnxruntime.transformers.optimizer --input $export_model_path --output $optimized_fp32_model_path<jupyter_output>apply: Fused LayerNormalization count: 49
               apply: Fused Gelu count: 24
adjust_reshape_and_expand: Removed Reshape and Expand count: 0
               apply: Fused SkipLayerNormalization count: 48
               apply: Fused Attention count: 24
         prune_graph: Graph pruned: 0 inputs, 0 outputs and 5 nodes are removed
               apply: Fused EmbedLayerNormalization(with mask) count: 1
         prune_graph: Graph pruned: 0 inputs, 0 outputs and 3 nodes are removed
         prune_graph: Graph pruned: 0 inputs, 0 outputs and 0 nodes are removed
               apply: Fused BiasGelu count: 24
               apply: Fused SkipLayerNormalization(add bias) count: 48
            optimize: opset verion: 11
  save_model_to_file: Sort graphs in topological order
  save_model_to_file: Output model to ./onnx/bert-base-cased-squad_opt_gpu_fp32.onnx
get_fused_operator_statistics: Optimized operators:{'EmbedLayerNormalization': 1, 'Attention': 24, 'Gelu': 0, 'FastGelu[...]<jupyter_text>Optimized GraphWe can open the optimized model using [Netron](https://github.com/lutzroeder/netron) to visualize.The graph is like the following:Sometime, optimized graph is slightly different. For example, FastGelu is replaced by BiasGelu for CPU inference; When the option --input_int32 is used, Cast nodes for inputs are removed.<jupyter_code>import netron

# change it to True if want to view the optimized model in browser
enable_netron = False
if enable_netron:
    # If you encounter error "access a socket in a way forbidden by its access permissions", install Netron as standalone application instead.
    netron.start(optimized_fp32_model_path)<jupyter_output><empty_output><jupyter_text>Performance Test ToolThe following will create 1000 random inputs of batch_size 1 and sequence length 128, then measure the average latency and throughput numbers.Note that the test uses fixed sequence length. If you use [dynamic sequence length](Inference-with-Actual-Sequence-Length), actual performance depends on the distribution of sequence length.**Attention**: Latency numbers from Jupyter Notebook are not accurate. See [Attional Info](7.-Additional-Info) for more info.<jupyter_code>GPU_OPTION = '--use_gpu' if use_gpu else ''

!{sys.executable} -m onnxruntime.transformers.bert_perf_test --model $optimized_fp32_model_path --batch_size 1 --sequence_length 128 --samples 1000 --test_times 1 $GPU_OPTION<jupyter_output>test setting TestSetting(batch_size=1, sequence_length=128, test_cases=1000, test_times=1, use_gpu=True, intra_op_num_threads=None, seed=3, verbose=False)
Generating 1000 samples for batch_size=1 sequence_length=128
Running test: model=bert-base-cased-squad_opt_gpu_fp32.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=4,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,use_gpu=True
Average latency = 23.72 ms, Throughput = 42.15 QPS
Running test: model=bert-base-cased-squad_opt_gpu_fp32.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=3,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,use_gpu=True
Average latency = 24.24 ms, Throughput = 41.25 QPS
Running test: model=bert-base-cased-squad_opt_gpu_fp32.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=2,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,use_gpu=True
Average latency = 24.36 ms, Throughput = 41.05 QPS
Running test: model=bert-base-cased-squad_opt_gpu[...]<jupyter_text>Let's load the summary file and take a look. Note that blank value in OMP_NUM_THREADS or OMP_WAIT_POLICY means the environment variable does not exist.<jupyter_code>import os
import glob     
import pandas
latest_result_file = max(glob.glob("./onnx/perf_results_GPU_B1_S128_*.txt"), key=os.path.getmtime)
result_data = pandas.read_table(latest_result_file)
print("Float32 model perf results from", latest_result_file)
# Remove some columns that have same values for all rows.
columns_to_remove = ['model', 'graph_optimization_level', 'batch_size', 'sequence_length', 'test_cases', 'test_times', 'use_gpu']
result_data.drop(columns_to_remove, axis=1, inplace=True)
result_data<jupyter_output>Float32 model perf results from ./onnx/perf_results_GPU_B1_S128_20210714-001817.txt<jupyter_text>From above result, we can see that latency is very close for different settings. The default setting (intra_op_num_threads=0, OMP_NUM_THREADS and OMP_WAIT_POLICY does not exist) performs the best.  Model Results Comparison ToolWhen a BERT model is optimized, some approximation is used in calculation. If your BERT model has three inputs, a script compare_bert_results.py can be used to do a quick verification. The tool will generate some fake input data, and compare the inference outputs of the original and optimized models. If outputs are all close, it is safe to use the optimized model.For GPU inference, the absolute or relative difference is larger than those numbers of CPU inference. Note that slight difference in output will not impact final result. We did end-to-end evaluation using SQuAD data set using a fine-tuned squad model, and F1 score is almost the same before/after optimization.<jupyter_code>!{sys.executable} -m onnxruntime.transformers.compare_bert_results --baseline_model $export_model_path --optimized_model $optimized_fp32_model_path --batch_size 1 --sequence_length 128 --samples 100 --rtol 0.01 --atol 0.01 $GPU_OPTION<jupyter_output>100% passed for 100 random inputs given thresholds (rtol=0.01, atol=0.01).
maximum absolute difference=5.316734313964844e-05
maximum relative difference=0.00012461667938623577<jupyter_text>6. Model Optimization with Float16The optimizer.py script have an option **--float16** to convert model to use float16 to store weights. After the conversion, it could be faster to run in GPU with tensor cores like V100 or T4.Let's run tools to measure the performance on V100. The results show significant performance improvement: latency is about 3.4 ms for float32 model, and 1.8 ms for float16 model.<jupyter_code>optimized_fp16_model_path = './onnx/bert-base-cased-squad_opt_{}_fp16.onnx'.format('gpu' if use_gpu else 'cpu')
!{sys.executable} -m onnxruntime.transformers.optimizer --input $export_model_path --output $optimized_fp16_model_path --float16
GPU_OPTION = '--use_gpu' if use_gpu else ''
!python -m onnxruntime.transformers.bert_perf_test --model $optimized_fp16_model_path --batch_size 1 --sequence_length 128 --samples 1000 --test_times 1 $GPU_OPTION
import os
import glob     
import pandas
latest_result_file = max(glob.glob("./onnx/perf_results_GPU_B1_S128_*.txt"), key=os.path.getmtime)
result_data = pandas.read_table(latest_result_file)
print("Float32 model perf results from", latest_result_file)
# Remove some columns that have same values for all rows.
columns_to_remove = ['model', 'graph_optimization_level', 'batch_size', 'sequence_length', 'test_cases', 'test_times', 'use_gpu']
result_data.drop(columns_to_remove, axis=1, inplace=True)
result_data<jupyter_output>Float32 model perf results from ./onnx/perf_results_GPU_B1_S128_20210714-002224.txt<jupyter_text>Throughput TuningSome application need best throughput under some constraint on latency. This can be done by testing performance of different batch sizes. The tool could help on this.Here is an example that check the performance of multiple batch sizes (1, 2, 4, 8, 16, 32 and 64) using default settings.<jupyter_code>GPU_OPTION = '--use_gpu' if use_gpu else ''
THREAD_SETTING = '--intra_op_num_threads 3'
!{sys.executable} -m onnxruntime.transformers.bert_perf_test --model $optimized_fp16_model_path --batch_size 1 2 4 8 16 32 --sequence_length 128 --samples 1000 --test_times 1 $THREAD_SETTING $GPU_OPTION
import os
import glob     
import pandas
latest_result_file = max(glob.glob("./onnx/perf_results_*.txt"), key=os.path.getmtime)
result_data = pandas.read_table(latest_result_file)
print("Float16 model summary from", latest_result_file)
columns_to_remove = ['model', 'graph_optimization_level', 'test_cases', 'test_times', 'use_gpu', 'sequence_length']
columns_to_remove.extend(['intra_op_num_threads'])
result_data.drop(columns_to_remove, axis=1, inplace=True)
result_data<jupyter_output>Float16 model summary from ./onnx/perf_results_GPU_B1-2-4-8-16-32_S128_20210714-002816.txt<jupyter_text>7. Additional InfoNote that running Jupyter Notebook has significant impact on performance result. You can close Jupyter Notebook and other applications, then run the performance test in a console to get more accurate performance numbers.We have a [benchmark script](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/transformers/run_benchmark.sh). It is recommended to use it measure inference speed of OnnxRuntime.[OnnxRuntime C API](https://github.com/microsoft/onnxruntime/blob/master/docs/C_API.md) could get slightly better performance than python API. If you use C API in inference, you can use OnnxRuntime_Perf_Test.exe built from source to measure performance instead.Here is the machine configuration that generated the above results. You might get slower or faster result according to your hardware.<jupyter_code>!{sys.executable} -m onnxruntime.transformers.machine_info --silent<jupyter_output>{
  "gpu": {
    "driver_version": "450.51.05",
    "devices": [
      {
        "memory_total": 15843721216,
        "memory_available": 9313189888,
        "name": "<NAME>"
      }
    ]
  },
  "cpu": {
    "brand": "AMD EPYC 7V12 64-Core Processor",
    "cores": 4,
    "logical_cores": 4,
    "hz": [
      2445417000,
      0
    ],
    "l2_cache": 524288,
    "flags": [
      "3dnowext",
      "3dnowprefetch",
      "abm",
      "adx",
      "aes",
      "apic",
      "arat",
      "avx",
      "avx2",
      "bmi1",
      "bmi2",
      "clflush",
      "clflushopt",
      "clwb",
      "cmov",
      "cmp_legacy",
      "cpuid",
      "cr8_legacy",
      "cx16",
      "cx8",
      "de",
      "extd_apicid",
      "f16c",
      "fma",
      "fpu",
      "fsgsbase",
      "fxsr",
      "fxsr_opt",
      "ht",
      "hypervisor",
      "lahf_lm",
      "lm",
      "mca",
      "mce",
      "misalignsse",
      "mmx",
      "mmx[...]
StarcoderdataJupyterStructuredCleanDedup