Skip to content

validated_model_list.md

Latest commit

 

History

History
1841 lines (1810 loc) · 33.5 KB

validated_model_list.md

File metadata and controls

1841 lines (1810 loc) · 33.5 KB
Raw

Validated Models

Validated MLPerf Models

Model Framework Support Example
ResNet50 v1.5 TensorFlow Yes Link
PyTorch Yes Link
DLRM PyTorch Yes Link
BERT-large TensorFlow Yes Link
PyTorch Yes Link
SSD-ResNet34 TensorFlow Yes Link
PyTorch Yes Link
RNN-T PyTorch Yes Link
3D-UNet TensorFlow WIP
PyTorch Yes Link

Full Validated Models on Intel Xeon Platinum 8380 Scalable processor

The below tables are models enabled by the Intel® Neural Compressor.

Performance varies by use, configuration and other factors. See backup for configuration details. For more complete information about performance and benchmark results, visit www.intel.com/benchmarks

Performance results are based on testing as of ​​02/23/2022 and may not reflect all publicly available ​updates. No product or component can be absolutely secure.

Intel optimizations, for Intel compilers or other products, may not optimize to the same degree for non-Intel products.

Your costs may vary.

Intel technologies may require enabled hardware, software or service activation.

© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others​.​​

TensorFlow 2.x models

Framework version model Accuracy Performance
1s4c10ins1bs/throughput
(samples/sec)
INT8 FP32 Acc Ratio[(INT8-FP32)/FP32] INT8 FP32 Performance Ratio[INT8/FP32]
intel-tensorflow 2.7.0 densenet121 73.57% 72.89% 0.93% 374.30 296.40 1.26x
intel-tensorflow 2.7.0 densenet161 76.24% 76.29% -0.07% 216.60 163.90 1.32x
intel-tensorflow 2.7.0 densenet169 74.40% 74.65% -0.33% 294.75 242.15 1.22x
intel-tensorflow 2.7.0 efficientnet_b0 77.81% 76.76% 1.37% 561.50 434.01 1.29x
intel-tensorflow 2.7.0 faster_rcnn_inception_resnet_v2 37.99% 38.33% -0.89% 3.88 2.17 1.78x
intel-tensorflow 2.7.0 faster_rcnn_inception_resnet_v2_saved 37.90% 38.33% -1.12% 3.88 2.17 1.79x
intel-tensorflow 2.7.0 faster_rcnn_resnet101 30.28% 30.39% -0.36% 67.16 18.54 3.62x
intel-tensorflow 2.7.0 faster_rcnn_resnet101_saved 30.37% 30.39% -0.07% 66.95 16.49 4.06x
intel-tensorflow 2.7.0 inception_resnet_v2 80.44% 80.40% 0.05% 265.70 133.92 1.98x
intel-tensorflow 2.7.0 inception_v1 70.48% 69.74% 1.06% 2192.55 1052.98 2.08x
intel-tensorflow 2.7.0 inception_v2 74.36% 73.97% 0.53% 1799.20 816.74 2.20x
intel-tensorflow 2.7.0 inception_v3 77.28% 76.75% 0.69% 923.76 386.05 2.39x
intel-tensorflow 2.7.0 inception_v4 80.40% 80.27% 0.16% 572.49 191.00 3.00x
intel-tensorflow 2.7.0 mask_rcnn_inception_v2 28.54% 28.72% -0.63% 134.68 51.15 2.63x
intel-tensorflow 2.7.0 mask_rcnn_inception_v2_ckpt 28.54% 28.72% -0.63% 134.74 50.79 2.65x
intel-tensorflow 2.7.0 mobilenetv1 71.79% 70.96% 1.17% 3633.27 1382.64 2.63x
intel-tensorflow 2.7.0 mobilenetv2 71.89% 71.76% 0.18% 2504.63 1418.82 1.77x
intel-tensorflow 2.7.0 resnet101 77.50% 76.45% 1.37% 853.41 342.71 2.49x
intel-tensorflow 2.7.0 resnet50_fashion 77.80% 78.12% -0.41% 3742.45 1957.16 1.91x
intel-tensorflow 2.7.0 resnet50v1.0 74.11% 74.27% -0.22% 1474.03 486.21 3.03x
intel-tensorflow 2.7.0 resnet50v1.5 76.82% 76.46% 0.47% 1224.52 414.92 2.95x
intel-tensorflow 2.7.0 resnetv2_101 72.67% 71.87% 1.11% 440.30 315.23 1.40x
intel-tensorflow 2.7.0 resnetv2_152 73.03% 72.37% 0.91% 305.00 216.73 1.41x
intel-tensorflow 2.7.0 resnetv2_50 70.33% 69.64% 0.99% 746.79 555.71 1.34x
intel-tensorflow 2.7.0 ssd_mobilenet_v1 22.97% 23.13% -0.69% 866.75 450.34 1.92x
intel-tensorflow 2.7.0 ssd_mobilenet_v1_ckpt 22.99% 23.13% -0.61% 867.26 386.89 2.24x
intel-tensorflow 2.7.0 ssd_resnet34 21.69% 22.09% -1.81% 41.17 10.76 3.83x
intel-tensorflow 2.7.0 ssd_resnet50_v1 37.86% 38.00% -0.37% 68.03 24.68 2.76x
intel-tensorflow 2.7.0 ssd_resnet50_v1_ckpt 37.81% 38.00% -0.50% 68.75 20.70 3.32x
intel-tensorflow 2.7.0 vgg16 72.66% 70.89% 2.50% 624.03 172.04 3.63x
intel-tensorflow 2.7.0 vgg19 72.72% 71.01% 2.41% 542.61 141.44 3.84x

TensorFlow 1.x models

Framework version model Accuracy Performance
1s4c10ins1bs/throughput
(samples/sec)
INT8 FP32 Acc Ratio[(INT8-FP32)/FP32] INT8 FP32 Performance Ratio[INT8/FP32]
intel-tensorflow 1.15UP3 bert_base_mrpc 86.52% 86.52% 0.00% 255.77 136.04 1.88x
intel-tensorflow 1.15UP3 bert_large_squad 92.41 92.98 -0.61% 23.53 11.62 2.02x
intel-tensorflow 1.15UP3 inception_v1_slim 70.49% 69.77% 1.03% 1941.78 781.35 2.49x
intel-tensorflow 1.15UP3 inception_v2_slim 74.35% 73.98% 0.50% 1559.10 646.87 2.41x
intel-tensorflow 1.15UP3 inception_v3_slim 78.32% 77.99% 0.42% 918.83 282.69 3.25x
intel-tensorflow 1.15UP3 inception_v4_slim 80.30% 80.19% 0.14% 507.39 141.36 3.59x
intel-tensorflow 1.15UP3 resnet_v1_101_slim 77.52% 76.40% 1.47% 805.67 218.12 3.69x
intel-tensorflow 1.15UP3 resnet_v1_152_slim 77.08% 76.81% 0.35% 572.84 145.86 3.93x
intel-tensorflow 1.15UP3 resnet_v1_50_slim 76.38% 75.18% 1.60% 1446.45 403.51 3.58x
intel-tensorflow 1.15UP3 resnetv2_101_slim 72.62% 71.91% 0.99% 459.04 249.91 1.84x
intel-tensorflow 1.15UP3 resnetv2_152_slim 72.95% 72.40% 0.76% 326.76 169.80 1.92x
intel-tensorflow 1.15UP3 resnetv2_50_slim 70.47% 69.72% 1.08% 775.54 469.69 1.65x
intel-tensorflow 1.15UP3 vgg16_slim 72.78% 70.89% 2.67% 591.48 146.10 4.05x
intel-tensorflow 1.15UP3 vgg19_slim 72.60% 71.01% 2.24% 494.30 118.19 4.18x

PyTorch models

Framework version model Accuracy Performance
1s4c10ins1bs/throughput
(samples/sec)
INT8 FP32 Acc Ratio[(INT8-FP32)/FP32] INT8 FP32 Performance Ratio[INT8/FP32]
pytorch 1.9.0+cpu albert_base_mrpc 88.77% 88.50% 0.31% 31.28 27.45 1.14x
pytorch 1.9.0+cpu barthez_mrpc 83.12% 83.81% -0.82% 119.18 74.13 1.61x
pytorch 1.9.0+cpu blendcnn 68.40% 68.40% 0.00% 5138.45 4672.94 1.10x
pytorch 1.9.0+cpu camembert_base_mrpc 87.39% 86.82% 0.66% 173.50 99.62 1.74x
pytorch 1.9.0+cpu ctrl_mrpc 82.00% 82.00% 0.00% 21.31 8.50 2.51x
pytorch 1.9.0+cpu deberta_mrpc 91.48% 90.91% 0.63% 104.94 63.68 1.65x
pytorch 1.9.0+cpu distilbert_base_mrpc 88.27% 89.16% -1.00% 311.52 196.45 1.59x
pytorch 1.9.0+cpu flaubert_mrpc 80.82% 80.19% 0.78% 356.18 300.38 1.19x
pytorch 1.9.0+cpu inception_v3 69.43% 69.52% -0.13% 472.93 216.80 2.18x
pytorch 1.9.0+cpu longformer_mrpc 91.82% 91.46% 0.39% 20.72 16.65 1.24x
pytorch 1.9.0+cpu mbart_wnli 56.34% 56.34% 0.00% 57.97 28.33 2.05x
pytorch 1.9.0+cpu mobilenet_v2 70.54% 71.84% -1.81% 782.34 532.65 1.47x
pytorch 1.9.0+cpu peleenet 71.66% 72.10% -0.62% 513.07 388.10 1.32x
pytorch 1.9.0+cpu resnet18 69.57% 69.76% -0.27% 828.53 402.89 2.06x
pytorch 1.9.0+cpu resnet18_fx 69.57% 69.76% -0.27% 833.23 383.38 2.17x
pytorch 1.9.0+cpu resnet18_qat 69.83% 69.76% 0.09% 826.88 392.36 2.11x
pytorch 1.9.0+cpu resnet18_qat_fx 69.85% 69.76% 0.12% 833.85 383.65 2.17x
pytorch 1.9.0+cpu resnet50 75.98% 76.15% -0.21% 515.56 194.38 2.65x
pytorch 1.9.0+cpu resnet50_ipex 75.57% 76.15% -0.75% 672.90 307.53 2.19x
pytorch 1.9.0+cpu resnet50_qat 76.02% 76.15% -0.17% 509.97 187.82 2.72x
pytorch 1.9.0+cpu resnext101_32x8d 79.15% 79.31% -0.20% 203.85 70.25 2.90x
pytorch 1.9.0+cpu rnnt 92.47 92.54 -0.08% 78.90 18.81 4.19x
pytorch 1.9.0+cpu roberta_base_mrpc 87.76% 88.18% -0.48% 175.81 100.59 1.75x
pytorch 1.9.0+cpu se_resnext50_32x4d 79.05% 79.08% -0.04% 373.85 165.22 2.26x
pytorch 1.9.0+cpu squeezebert_mrpc 87.25% 87.65% -0.46% 171.52 145.03 1.18x
pytorch 1.9.0+cpu ssd_resnet34_fx 19.51 19.63 -0.61% 28.74 7.28 3.95x
pytorch 1.9.0+cpu ssd_resnet34_qat_fx 17.80% 17.30% 2.89% 291.32 99.99 2.91x
pytorch 1.9.0+cpu transfo_xl_mrpc 82.09% 81.20% 1.09% 9.61 7.28 1.32x
pytorch 1.9.0+cpu yolo_v3 24.46% 24.54% -0.36% 99.827 37.337 2.67x

PyTorch models

Framework version model Accuracy Performance
1s4c10ins1bs/throughput
(samples/sec)
INT8 FP32 Acc Ratio[(INT8-FP32)/FP32] INT8 FP32 Performance Ratio[INT8/FP32]
pytorch 1.8.0+cpu bert_base_cola 58.92% 59.07% -0.26% 176.71 101.25 1.75x
pytorch 1.8.0+cpu bert_base_mrpc 90.35% 90.41% -0.07% 166.14 101.59 1.64x
pytorch 1.8.0+cpu bert_base_mrpc_qat 89.30% 89.50% -0.22% 181.32 102.55 1.77x
pytorch 1.8.0+cpu bert_base_rte 69.53% 69.14% 0.56% 176.97 101.55 1.74x
pytorch 1.8.0+cpu bert_base_sst-2 91.35% 91.83% -0.52% 179.68 101.56 1.77x
pytorch 1.8.0+cpu bert_base_sts-b 89.07% 89.76% -0.77% 179.08 103.04 1.74x
pytorch 1.8.0+cpu bert_large_cola 62.07% 62.83% -1.21% 87.10 33.96 2.56x
pytorch 1.8.0+cpu bert_large_mrpc 88.97% 89.91% -1.05% 86.95 33.84 2.57x
pytorch 1.8.0+cpu bert_large_qnli 91.54% 91.84% -0.32% 89.92 33.84 2.66x
pytorch 1.8.0+cpu bert_large_rte 72.27% 71.88% 0.54% 37.55 33.78 1.11x

MXNet models

Framework version model Accuracy Performance
1s4c10ins1bs/throughput
(samples/sec)
INT8 FP32 Acc Ratio[(INT8-FP32)/FP32] INT8 FP32 Performance Ratio[INT8/FP32]
mxnet 1.7.0 inceptionv3 77.73% 77.64% 0.11% 974.39 273.37 3.56x
mxnet 1.7.0 mobilenet1.0 71.69% 72.22% -0.74% 7289.97 2584.76 2.82x
mxnet 1.7.0 mobilenetv2_1.0 70.78% 70.87% -0.12% 5981.16 2145.92 2.79x
mxnet 1.7.0 resnet152_v1 78.21% 78.54% -0.42% 561.44 149.04 3.77x
mxnet 1.7.0 resnet18_v1 70.02% 70.14% -0.17% 3541.71 812.57 4.36x
mxnet 1.7.0 resnet50v1 76.08% 76.33% -0.32% 1573.97 414.04 3.80x
mxnet 1.7.0 squeezenet1.0 56.74% 56.96% -0.38% 4920.07 1481.78 3.32x
mxnet 1.7.0 ssd-mobilenet1.0 74.94% 75.54% -0.79% 737.96 176.71 4.18x

ONNX Models

Framework version model Accuracy Performance
1s4c10ins1bs/throughput
(samples/sec)
INT8 FP32 Acc Ratio[(INT8-FP32)/FP32] INT8 FP32 Performance Ratio[INT8/FP32]
onnxrt-runtime 1.8.0 alexnet 54.74% 54.79% -0.09% 1182.08 507.93 2.33x
onnxrt-runtime 1.8.0 bert_base_mrpc_dynamic 85.54% 86.03% -0.57% 355.71 144.14 2.47x
onnxrt-runtime 1.8.0 bert_base_mrpc_static 85.29% 86.03% -0.86% 681.90 293.39 2.32x
onnxrt-runtime 1.8.0 bert_squad_model_zoo 80.43 80.67 -0.29% 106.55 59.56 1.79x
onnxrt-runtime 1.8.0 caffenet 56.19% 56.30% -0.20% 1754.53 561.43 3.13x
onnxrt-runtime 1.8.0 distilbert_base_mrpc 84.56% 84.56% 0.00% 1568.66 550.81 2.85x
onnxrt-runtime 1.8.0 efficientnet 77.58% 77.70% -0.15% 1091.30 935.77 1.17x
onnxrt-runtime 1.8.0 fcn 64.66% 64.98% -0.49% 18.24 12.63 1.44x
onnxrt-runtime 1.8.0 googlenet-12 67.61% 67.79% -0.27% 930.16 720.33 1.29x
onnxrt-runtime 1.8.0 inception_v1 67.23% 67.24% -0.01% 960.98 763.02 1.26x
onnxrt-runtime 1.8.0 mobilebert_mrpc 85.54% 86.27% -0.85% 766.13 651.07 1.18x
onnxrt-runtime 1.8.0 mobilebert_squad_mlperf 89.84 90.02 -0.20% 92.29 81.50 1.13x
onnxrt-runtime 1.8.0 mobilenet_v3_mlperf 75.59% 75.74% -0.20% 2962.13 1880.56 1.58x
onnxrt-runtime 1.8.0 resnet_v1_5_mlperf 76.13% 76.46% -0.43% 845.61 496.12 1.70x
onnxrt-runtime 1.8.0 resnet50_v1_5 72.28% 72.29% -0.01% 859.07 494.28 1.74x
onnxrt-runtime 1.8.0 resnet50-v1-12 74.76% 74.99% -0.31% 1011.74 521.56 1.94x
onnxrt-runtime 1.8.0 roberta_base_mrpc 88.73% 89.46% -0.82% 696.59 282.72 2.46x
onnxrt-runtime 1.8.0 shufflenet-v2-12 66.13% 66.36% -0.35% 4575.88 2699.85 1.69x
onnxrt-runtime 1.8.0 squeezenet 56.55% 56.87% -0.56% 5119.43 3695.21 1.39x
onnxrt-runtime 1.8.0 ssd_mobilenet_v1 22.21% 23.10% -3.85% 731.38 624.29 1.17x
onnxrt-runtime 1.8.0 ssd_mobilenet_v2 23.84% 24.68% -3.40% 552.41 444.97 1.24x
onnxrt-runtime 1.8.0 ssd-12 18.84% 18.98% -0.74% 15.18 10.30 1.47x
onnxrt-runtime 1.8.0 vgg16 66.53% 66.69% -0.24% 141.56 121.22 1.17x
onnxrt-runtime 1.8.0 vgg16_model_zoo 72.28% 72.40% -0.17% 252.78 120.95 2.09x
onnxrt-runtime 1.8.0 zfnet 55.89% 55.96% -0.13% 565.28 307.55 1.84x

INC-ENGINE Models

Backend model Accuracy Performance
1s4c10ins1bs/throughput
(samples/sec)
INT8 FP32 Acc Ratio[(INT8-FP32)/FP32] INT8 FP32 Performance Ratio[INT8/FP32]
INC-ENGINE bert_base_mrpc 82.35% 83.09% -0.89% 487.41 140.978 3.46x
INC-ENGINE bert_base_nli_mean_tokens_stsb 89.26% 89.55% -0.32% 504.147 141.504 3.56x
INC-ENGINE bert_base_sparse_mrpc 70.34% 70.59% -0.35% 507.585 142.876 3.55x
INC-ENGINE bert_large_squad 90.70 90.87 -0.19% 45.32 12.531 3.62x
INC-ENGINE distilbert_base_uncased_emotion 93.85% 94.20% -0.37% 999.973 283.975 3.52x
INC-ENGINE distilbert_base_uncased_mrpc 84.07% 84.07% 0.00% 996.79 280.88 3.55x
INC-ENGINE distilbert_base_uncased_sst2 90.14% 90.25% -0.12% 999.98 283.96 3.52x
INC-ENGINE distilroberta_base_wnli 56.34% 56.34% 0.00% 1032.043 291.782 3.54x
INC-ENGINE dlrm 78.07% 78.10% -0.04% 54898.34 48331.14 1.14x
INC-ENGINE finbert_financial_phrasebank 82.68% 82.80% -0.14% 922.877 272.751 3.38x
INC-ENGINE minilm_l6_h384_uncased_sst2 89.33% 90.14% -0.90% 2690.501 1002.695 2.68x
INC-ENGINE paraphrase_xlm_r_multilingual_v1_stsb 86.71% 87.23% -0.60% 511.919 142.851 3.58x
INC-ENGINE roberta_base_mrpc 89.71% 88.97% 0.83% 508.184 142.483 3.57x

BACKUP

System ConfigurationIntel Xeon Platinum 8380 Scalable processor
Test Date March 01 01:00:00 UTC 2022
Manufacturer Intel Corporation
Product Name WilsonCity
BIOS Version WLYDCRB1.SYS.0025.D65.2106110056
OS Ubuntu 20.04.1 LTS
Kernel 5.4.0-77-generic
Microcode 0x8d9522d4
CPU Model Intel(R) Xeon(R) Platinum 8380 CPU @ 2.30GHz
Base Frequency 2.3GHZ
Thread(s) per Core 2
Core(s) per Socket 40
Socket(s) 2
Turbo Enabled
Power & Perf Policy Performance
Installed 512GB (16x32GB DDR4 3200MT/s [3200MT/s])
NIC Summary I210 Gigabit Network Connection
Drive Summary 1x ST1000NX0423 931.5G, 1x INTEL_SSDSC2BB48 447.1G 1x INTEL_SSDSC2KG96 894.3G

Validated Pruning Models

Tasks FWK Model fp32 baseline gradient sensitivity with 20% sparsity +onnx dynamic quantization on pruned model
accuracy% drop% perf gain (sample/s) accuracy% drop% perf gain (sample/s)
SST-2 pytorch bert-base accuracy = 92.32 accuracy = 91.97 -0.38 1.30x accuracy = 92.20 -0.13 1.86x
QQP pytorch bert-base [accuracy, f1] = [91.10, 88.05] [accuracy, f1] = [89.97, 86.54] [-1.24, -1.71] 1.32x [accuracy, f1] = [89.75, 86.60] [-1.48, -1.65] 1.81x
Tasks FWK Model fp32 baseline Pattern Lock on 70% Unstructured Sparsity Pattern Lock on 50% 1:2 Structured Sparsity
accuracy% drop% accuracy% drop%
MNLI pytorch bert-base [m, mm] = [84.57, 84.79] [m, mm] = [82.45, 83.27] [-2.51, -1.80] [m, mm] = [83.20, 84.11] [-1.62, -0.80]
SST-2 pytorch bert-base accuracy = 92.32 accuracy = 91.51 -0.88 accuracy = 92.20 -0.13
QQP pytorch bert-base [accuracy, f1] = [91.10, 88.05] [accuracy, f1] = [90.48, 87.06] [-0.68, -1.12] [accuracy, f1] = [90.92, 87.78] [-0.20, -0.31]
QNLI pytorch bert-base accuracy = 91.54 accuracy = 90.39 -1.26 accuracy = 90.87 -0.73
QnA pytorch bert-base [em, f1] = [79.34, 87.10] [em, f1] = [77.27, 85.75] [-2.61, -1.54] [em, f1] = [78.03, 86.50] [-1.65, -0.69]
Framework Model fp32 baseline Compression dataset acc(drop)%
Pytorch resnet18 69.76 30% sparsity on magnitude ImageNet 69.47(-0.42)
Pytorch resnet18 69.76 30% sparsity on gradient sensitivity ImageNet 68.85(-1.30)
Pytorch resnet50 76.13 30% sparsity on magnitude ImageNet 76.11(-0.03)
Pytorch resnet50 76.13 30% sparsity on magnitude and post training quantization ImageNet 76.01(-0.16)
Pytorch resnet50 76.13 30% sparsity on magnitude and quantization aware training ImageNet 75.90(-0.30)

Validated Knowledge Distillation Examples

Example Name Dataset Student
(Accuracy)		 Teacher
(Accuracy)		 Student With Distillation
(Accuracy Improvement)
ResNet example ImageNet ResNet18
(0.6739)		 ResNet50
(0.7399)		 0.6845
(0.0106)
BlendCnn example MRPC BlendCnn
(0.7034)		 BERT-Base
(0.8382)		 0.7034
(0)
BiLSTM example SST-2 BiLSTM
(0.7913)		 RoBERTa-Base
(0.9404)		 0.8085
(0.0172)