<jupyter_start><jupyter_text>100 iterations<jupyter_code>profile('cuda')
profile('cuda', size='(64,10,256,256)')<jupyter_output>Profiling over 100 runs after 10 warmup runs.
Testing on torch.float16:
 relu_fwd:        345.1µs ± 5.984µs, 337.7µs, 367.6µs
 relu_bwd:        1.257ms ± 201.6µs, 1.205ms, 2.986ms
 leaky_relu_fwd:  343.0µs ± 30.66µs, 334.0µs, 643.7µs
 leaky_relu_bwd:  1.213ms ± 59.66µs, 1.204ms, 1.806ms
 softplus_fwd:    791.0µs ± 4.405µs, 787.4µs, 808.7µs
 softplus_bwd:    1.638ms ± 127.4µs, 1.624ms, 2.906ms
 silu_jit_fwd:    609.4µs ± 6.577µs, 603.5µs, 637.8µs
 silu_jit_bwd:    1.430ms ± 12.35µs, 1.422ms, 1.550ms
 silu_native_fwd: 434.2µs ± 23.30µs, 426.4µs, 659.0µs
 silu_native_bwd: 1.239ms ± 300.0µs, 1.206ms, 4.223ms
 mish_naive_fwd:  1.623ms ± 9.061µs, 1.616ms, 1.690ms
 mish_naive_bwd:  3.512ms ± 620.0µs, 3.439ms, 9.614ms
 mish_jit_fwd:    1.362ms ± 8.539µs, 1.347ms, 1.408ms
 mish_jit_bwd:    2.329ms ± 4.493µs, 2.319ms, 2.335ms
 mish_cuda_fwd:   1.011ms ± 11.01µs, 1.004ms, 1.110ms
 mish_cuda_bwd:   1.883ms ± 1.018µs, 1.882ms, 1.891ms
 mish_native_fwd: 731.8µs ± 45.41µs, 720.1µs, 1.135ms
 mish_nati[...]<jupyter_text>50 iterations<jupyter_code>profile('cuda', n_repeat=50)
profile('cuda', size='(64,10,256,256)', n_repeat=50)<jupyter_output>Profiling over 50 runs after 10 warmup runs.
Testing on torch.float16:
 relu_fwd:        345.7µs ± 22.92µs, 337.4µs, 500.6µs
 relu_bwd:        1.206ms ± 801.7ns, 1.205ms, 1.208ms
 leaky_relu_fwd:  339.2µs ± 6.653µs, 333.2µs, 367.8µs
 leaky_relu_bwd:  1.252ms ± 165.8µs, 1.205ms, 2.157ms
 softplus_fwd:    793.0µs ± 5.494µs, 786.9µs, 819.3µs
 softplus_bwd:    1.763ms ± 565.6µs, 1.624ms, 4.381ms
 silu_jit_fwd:    928.1µs ± 57.32µs, 911.5µs, 1.326ms
 silu_jit_bwd:    3.481ms ± 79.44µs, 3.465ms, 4.035ms
 silu_native_fwd: 435.2µs ± 18.88µs, 426.5µs, 563.2µs
 silu_native_bwd: 1.210ms ± 1.364µs, 1.207ms, 1.213ms
 mish_naive_fwd:  1.621ms ± 5.045µs, 1.614ms, 1.636ms
 mish_naive_bwd:  3.443ms ± 1.689µs, 3.440ms, 3.448ms
 mish_jit_fwd:    1.635ms ± 6.902µs, 1.628ms, 1.655ms
 mish_jit_bwd:    5.092ms ± 222.7µs, 5.057ms, 6.651ms
 mish_cuda_fwd:   1.007ms ± 4.318µs, 1.003ms, 1.023ms
 mish_cuda_bwd:   1.886ms ± 25.02µs, 1.882ms, 2.062ms
 mish_native_fwd: 725.3µs ± 5.760µs, 719.0µs, 745.0µs
 mish_nativ[...]<jupyter_text>25 iterations<jupyter_code>profile('cuda', n_repeat=25, warmup=5)
profile('cuda', size='(64,10,256,256)', n_repeat=25, warmup=5)<jupyter_output>Profiling over 25 runs after 5 warmup runs.
Testing on torch.float16:
 relu_fwd:        343.4µs ± 5.465µs, 337.7µs, 362.5µs
 relu_bwd:        1.237ms ± 128.3µs, 1.209ms, 1.865ms
 leaky_relu_fwd:  352.0µs ± 55.62µs, 336.8µs, 623.7µs
 leaky_relu_bwd:  1.284ms ± 351.7µs, 1.210ms, 3.007ms
 softplus_fwd:    793.2µs ± 6.308µs, 786.1µs, 813.8µs
 softplus_bwd:    1.631ms ± 1.120µs, 1.629ms, 1.634ms
 silu_jit_fwd:    923.6µs ± 7.414µs, 912.9µs, 936.7µs
 silu_jit_bwd:    3.513ms ± 187.5µs, 3.470ms, 4.431ms
 silu_native_fwd: 432.8µs ± 5.688µs, 425.4µs, 450.3µs
 silu_native_bwd: 1.214ms ± 2.596µs, 1.211ms, 1.225ms
 mish_naive_fwd:  1.628ms ± 10.02µs, 1.617ms, 1.654ms
 mish_naive_bwd:  3.459ms ± 68.51µs, 3.443ms, 3.795ms
 mish_jit_fwd:    1.637ms ± 7.579µs, 1.629ms, 1.659ms
 mish_jit_bwd:    5.069ms ± 22.44µs, 5.062ms, 5.178ms
 mish_cuda_fwd:   1.011ms ± 6.176µs, 1.005ms, 1.037ms
 mish_cuda_bwd:   1.887ms ± 937.0ns, 1.885ms, 1.889ms
 mish_native_fwd: 725.0µs ± 3.944µs, 722.3µs, 738.5µs
 mish_native[...]
StarcoderdataJupyterStructuredCleanDedup