Benchmarks on Built-in Knowledage Graphs

DGL-KE provides five built-in knowledge graphs:

Dataset #nodes #edges #relations
FB15k 14951 592213 1345
FB15k-237 14541 310116 237
wn18 40943 151442 18
wn18rr 40943 93003 11
Freebase 86054151 338586276 14824

Users can specify one of the datasets with --dataset option in their tasks.

DGL-KE provides benchmark results on FB15k, wn18, as well as Freebase. Users can go to the corresponded folder to check out the scripts and results. All the benchmark results are done by AWS EC2. For multi-cpu and distributed training, the target instance is r5dn.24xlarge, which has 48 CPU cores and 768 GB memory. Also, r5dn.xlarge has 100Gbit network throughput, which is powerful for distributed training. For GPU training, our target instance is p3.16xlarge, which has 64 CPU cores and 8 Nvidia v100 GPUs. For users, you can choose your own instance by your demand and tune the hyper-parameters for the best performance.

All the scripts can be found on this page.

FB15k

One-GPU training

Models MR MRR HITS-1 HITS-3 HITS-10 TIME
TransE_l1 47.34 0.672 0.557 0.763 0.849 201
TransE_l2 47.04 0.649 0.525 0.746 0.844 167
DistMult 61.43 0.696 0.586 0.782 0.873 150
ComplEx 64.73 0.757 0.672 0.826 0.886 171
RESCAL 124.5 0.661 0.589 0.704 0.787 1252
TransR 59.99 0.670 0.585 0.728 0.808 530
RotatE 43.85 0.726 0.632 0.799 0.873 1405

8-GPU training

Models MR MRR HITS-1 HITS-3 HITS-10 TIME
TransE_l1 48.59 0.662 0.542 0.756 0.846 53
TransE_l2 47.52 0.627 0.492 0.733 0.838 49
DistMult 59.44 0.679 0.566 0.764 0.864 47
ComplEx 64.98 0.750 0.668 0.814 0.883 49
RESCAL 133.3 0.643 0.570 0.685 0.773 179
TransR 66.51 0.666 0.581 0.724 0.803 90
RotatE 50.04 0.685 0.581 0.763 0.851 120

Multi-CPU training

Models MR MRR HITS-1 HITS-3 HITS-10 TIME
TransE_l1 48.32 0.645 0.521 0.741 0.838 140
TransE_l2 45.28 0.633 0.501 0.735 0.840 58
DistMult 62.63 0.647 0.529 0.733 0.846 58
ComplEx 67.83 0.694 0.590 0.772 0.863 69

Distributed training

Models MR MRR HITS-1 HITS-3 HITS-10 TIME
TransE_l1 38.26 0.691 0.591 0.765 0.853 104
TransE_l2 34.84 0.645 0.510 0.754 0.854 31
DistMult 51.85 0.661 0.532 0.762 0.864 57
ComplEx 62.52 0.667 0.567 0.737 0.836 65

wn18

One-GPU training

Models MR MRR HITS-1 HITS-3 HITS-10 TIME
TransE_l1 355.4 0.764 0.602 0.928 0.949 327
TransE_l2 209.4 0.560 0.306 0.797 0.943 223
DistMult 419.0 0.813 0.702 0.921 0.948 133
ComplEx 318.2 0.932 0.914 0.948 0.959 144
RESCAL 563.6 0.848 0.792 0.898 0.928 308
TransR 432.8 0.609 0.452 0.736 0.850 906
RotatE 451.6 0.944 0.940 0.945 0.950 671

8-GPU training

Models MR MRR HITS-1 HITS-3 HITS-10 TIME
TransE_l1 348.8 0.739 0.553 0.927 0.948 111
TransE_l2 198.9 0.559 0.305 0.798 0.942 71
DistMult 798.8 0.806 0.705 0.903 0.932 66
ComplEx 535.0 0.938 0.931 0.944 0.949 53
RotatE 487.7 0.943 0.939 0.945 0.951 127

Multi-CPU training

Models MR MRR HITS-1 HITS-3 HITS-10 TIME
TransE_l1 376.3 0.593 0.264 0.926 0.949 925
TransE_l2 218.3 0.528 0.259 0.777 0.939 210
DistMult 837.4 0.791 0.675 0.904 0.933 362
ComplEx 806.3 0.904 0.881 0.926 0.937 281

Distributed training

Models MR MRR HITS-1 HITS-3 HITS-10 TIME
TransE_l1 136.0 0.848 0.768 0.927 0.950 759
TransE_l2 85.04 0.797 0.672 0.921 0.958 144
DistMult 278.5 0.872 0.816 0.926 0.939 275
ComplEx 333.8 0.838 0.796 0.870 0.906 273

Freebase

8-GPU training

Models MR MRR HITS-1 HITS-3 HITS-10 TIME
TransE_l2 23.56 0.736 0.663 0.782 0.873 4767
DistMult 46.19 0.833 0.813 0.842 0.869 4281
ComplEx 46.70 0.834 0.815 0.843 0.869 8356
TransR 49.68 0.696 0.653 0.716 0.773 14235
RotatE 93.20 0.769 0.748 0.779 0.804 9060

Multi-CPU training

Models MR MRR HITS-1 HITS-3 HITS-10 TIME
TransE_l2 30.82 0.815 0.766 0.848 0.902 6993
DistMult 44.16 0.834 0.815 0.843 0.869 7146
ComplEx 45.62 0.835 0.817 0.843 0.870 8732

Distributed training

Models MR MRR HITS-1 HITS-3 HITS-10 TIME
TransE_l2 34.25 0.764 0.705 0.802 0.869 1633
DistMult 75.15 0.769 0.751 0.779 0.801 1679
ComplEx 77.83 0.771 0.754 0.779 0.802 2293