Benchmarks on Built-in Knowledage Graphs ------------------------------------------ DGL-KE provides five built-in knowledge graphs: +------------+------------+-----------+------------+ | Dataset | #nodes | #edges | #relations | +============+============+===========+============+ | FB15k | 14951 | 592213 | 1345 | +------------+------------+-----------+------------+ | FB15k-237 | 14541 | 310116 | 237 | +------------+------------+-----------+------------+ | wn18 | 40943 | 151442 | 18 | +------------+------------+-----------+------------+ | wn18rr | 40943 | 93003 | 11 | +------------+------------+-----------+------------+ | Freebase | 86054151 | 338586276 | 14824 | +------------+------------+-----------+------------+ Users can specify one of the datasets with ``--dataset`` option in their tasks. DGL-KE provides benchmark results on ``FB15k``, ``wn18``, as well as ``Freebase``. Users can go to the corresponded folder to check out the scripts and results. All the benchmark results are done by AWS EC2. For multi-cpu and distributed training, the target instance is ``r5dn.24xlarge``, which has 48 CPU cores and 768 GB memory. Also, ``r5dn.xlarge`` has 100Gbit network throughput, which is powerful for distributed training. For GPU training, our target instance is ``p3.16xlarge``, which has 64 CPU cores and 8 Nvidia v100 GPUs. For users, you can choose your own instance by your demand and tune the hyper-parameters for the best performance. All the scripts can be found on this `page.`__ .. __: https://github.com/awslabs/dgl-ke/tree/master/examples FB15k ^^^^^^ One-GPU training +-----------+-------+-------+--------+--------+---------+------+ | Models | MR | MRR | HITS-1 | HITS-3 | HITS-10 | TIME | +===========+=======+=======+========+========+=========+======+ | TransE_l1 | 47.34 | 0.672 | 0.557 | 0.763 | 0.849 | 201 | +-----------+-------+-------+--------+--------+---------+------+ | TransE_l2 | 47.04 | 0.649 | 0.525 | 0.746 | 0.844 | 167 | +-----------+-------+-------+--------+--------+---------+------+ | DistMult | 61.43 | 0.696 | 0.586 | 0.782 | 0.873 | 150 | +-----------+-------+-------+--------+--------+---------+------+ | ComplEx | 64.73 | 0.757 | 0.672 | 0.826 | 0.886 | 171 | +-----------+-------+-------+--------+--------+---------+------+ | RESCAL | 124.5 | 0.661 | 0.589 | 0.704 | 0.787 | 1252 | +-----------+-------+-------+--------+--------+---------+------+ | TransR | 59.99 | 0.670 | 0.585 | 0.728 | 0.808 | 530 | +-----------+-------+-------+--------+--------+---------+------+ | RotatE | 43.85 | 0.726 | 0.632 | 0.799 | 0.873 | 1405 | +-----------+-------+-------+--------+--------+---------+------+ 8-GPU training +-----------+-------+-------+--------+--------+---------+------+ | Models | MR | MRR | HITS-1 | HITS-3 | HITS-10 | TIME | +===========+=======+=======+========+========+=========+======+ | TransE_l1 | 48.59 | 0.662 | 0.542 | 0.756 | 0.846 | 53 | +-----------+-------+-------+--------+--------+---------+------+ | TransE_l2 | 47.52 | 0.627 | 0.492 | 0.733 | 0.838 | 49 | +-----------+-------+-------+--------+--------+---------+------+ | DistMult | 59.44 | 0.679 | 0.566 | 0.764 | 0.864 | 47 | +-----------+-------+-------+--------+--------+---------+------+ | ComplEx | 64.98 | 0.750 | 0.668 | 0.814 | 0.883 | 49 | +-----------+-------+-------+--------+--------+---------+------+ | RESCAL | 133.3 | 0.643 | 0.570 | 0.685 | 0.773 | 179 | +-----------+-------+-------+--------+--------+---------+------+ | TransR | 66.51 | 0.666 | 0.581 | 0.724 | 0.803 | 90 | +-----------+-------+-------+--------+--------+---------+------+ | RotatE | 50.04 | 0.685 | 0.581 | 0.763 | 0.851 | 120 | +-----------+-------+-------+--------+--------+---------+------+ Multi-CPU training +-----------+-------+-------+--------+--------+---------+------+ | Models | MR | MRR | HITS-1 | HITS-3 | HITS-10 | TIME | +===========+=======+=======+========+========+=========+======+ | TransE_l1 | 48.32 | 0.645 | 0.521 | 0.741 | 0.838 | 140 | +-----------+-------+-------+--------+--------+---------+------+ | TransE_l2 | 45.28 | 0.633 | 0.501 | 0.735 | 0.840 | 58 | +-----------+-------+-------+--------+--------+---------+------+ | DistMult | 62.63 | 0.647 | 0.529 | 0.733 | 0.846 | 58 | +-----------+-------+-------+--------+--------+---------+------+ | ComplEx | 67.83 | 0.694 | 0.590 | 0.772 | 0.863 | 69 | +-----------+-------+-------+--------+--------+---------+------+ Distributed training +-----------+-------+-------+--------+--------+---------+------+ | Models | MR | MRR | HITS-1 | HITS-3 | HITS-10 | TIME | +===========+=======+=======+========+========+=========+======+ | TransE_l1 | 38.26 | 0.691 | 0.591 | 0.765 | 0.853 | 104 | +-----------+-------+-------+--------+--------+---------+------+ | TransE_l2 | 34.84 | 0.645 | 0.510 | 0.754 | 0.854 | 31 | +-----------+-------+-------+--------+--------+---------+------+ | DistMult | 51.85 | 0.661 | 0.532 | 0.762 | 0.864 | 57 | +-----------+-------+-------+--------+--------+---------+------+ | ComplEx | 62.52 | 0.667 | 0.567 | 0.737 | 0.836 | 65 | +-----------+-------+-------+--------+--------+---------+------+ wn18 ^^^^ One-GPU training +-----------+-------+-------+--------+--------+---------+------+ | Models | MR | MRR | HITS-1 | HITS-3 | HITS-10 | TIME | +===========+=======+=======+========+========+=========+======+ | TransE_l1 | 355.4 | 0.764 | 0.602 | 0.928 | 0.949 | 327 | +-----------+-------+-------+--------+--------+---------+------+ | TransE_l2 | 209.4 | 0.560 | 0.306 | 0.797 | 0.943 | 223 | +-----------+-------+-------+--------+--------+---------+------+ | DistMult | 419.0 | 0.813 | 0.702 | 0.921 | 0.948 | 133 | +-----------+-------+-------+--------+--------+---------+------+ | ComplEx | 318.2 | 0.932 | 0.914 | 0.948 | 0.959 | 144 | +-----------+-------+-------+--------+--------+---------+------+ | RESCAL | 563.6 | 0.848 | 0.792 | 0.898 | 0.928 | 308 | +-----------+-------+-------+--------+--------+---------+------+ | TransR | 432.8 | 0.609 | 0.452 | 0.736 | 0.850 | 906 | +-----------+-------+-------+--------+--------+---------+------+ | RotatE | 451.6 | 0.944 | 0.940 | 0.945 | 0.950 | 671 | +-----------+-------+-------+--------+--------+---------+------+ 8-GPU training +-----------+-------+-------+--------+--------+---------+------+ | Models | MR | MRR | HITS-1 | HITS-3 | HITS-10 | TIME | +===========+=======+=======+========+========+=========+======+ | TransE_l1 | 348.8 | 0.739 | 0.553 | 0.927 | 0.948 | 111 | +-----------+-------+-------+--------+--------+---------+------+ | TransE_l2 | 198.9 | 0.559 | 0.305 | 0.798 | 0.942 | 71 | +-----------+-------+-------+--------+--------+---------+------+ | DistMult | 798.8 | 0.806 | 0.705 | 0.903 | 0.932 | 66 | +-----------+-------+-------+--------+--------+---------+------+ | ComplEx | 535.0 | 0.938 | 0.931 | 0.944 | 0.949 | 53 | +-----------+-------+-------+--------+--------+---------+------+ | RotatE | 487.7 | 0.943 | 0.939 | 0.945 | 0.951 | 127 | +-----------+-------+-------+--------+--------+---------+------+ Multi-CPU training +-----------+-------+-------+--------+--------+---------+------+ | Models | MR | MRR | HITS-1 | HITS-3 | HITS-10 | TIME | +===========+=======+=======+========+========+=========+======+ | TransE_l1 | 376.3 | 0.593 | 0.264 | 0.926 | 0.949 | 925 | +-----------+-------+-------+--------+--------+---------+------+ | TransE_l2 | 218.3 | 0.528 | 0.259 | 0.777 | 0.939 | 210 | +-----------+-------+-------+--------+--------+---------+------+ | DistMult | 837.4 | 0.791 | 0.675 | 0.904 | 0.933 | 362 | +-----------+-------+-------+--------+--------+---------+------+ | ComplEx | 806.3 | 0.904 | 0.881 | 0.926 | 0.937 | 281 | +-----------+-------+-------+--------+--------+---------+------+ Distributed training +-----------+-------+-------+--------+--------+---------+------+ | Models | MR | MRR | HITS-1 | HITS-3 | HITS-10 | TIME | +===========+=======+=======+========+========+=========+======+ | TransE_l1 | 136.0 | 0.848 | 0.768 | 0.927 | 0.950 | 759 | +-----------+-------+-------+--------+--------+---------+------+ | TransE_l2 | 85.04 | 0.797 | 0.672 | 0.921 | 0.958 | 144 | +-----------+-------+-------+--------+--------+---------+------+ | DistMult | 278.5 | 0.872 | 0.816 | 0.926 | 0.939 | 275 | +-----------+-------+-------+--------+--------+---------+------+ | ComplEx | 333.8 | 0.838 | 0.796 | 0.870 | 0.906 | 273 | +-----------+-------+-------+--------+--------+---------+------+ Freebase ^^^^^^^^ 8-GPU training +-----------+-------+-------+--------+--------+---------+------+ | Models | MR | MRR | HITS-1 | HITS-3 | HITS-10 | TIME | +===========+=======+=======+========+========+=========+======+ | TransE_l2 | 23.56 | 0.736 | 0.663 | 0.782 | 0.873 | 4767 | +-----------+-------+-------+--------+--------+---------+------+ | DistMult | 46.19 | 0.833 | 0.813 | 0.842 | 0.869 | 4281 | +-----------+-------+-------+--------+--------+---------+------+ | ComplEx | 46.70 | 0.834 | 0.815 | 0.843 | 0.869 | 8356 | +-----------+-------+-------+--------+--------+---------+------+ | TransR | 49.68 | 0.696 | 0.653 | 0.716 | 0.773 |14235 | +-----------+-------+-------+--------+--------+---------+------+ | RotatE | 93.20 | 0.769 | 0.748 | 0.779 | 0.804 | 9060 | +-----------+-------+-------+--------+--------+---------+------+ Multi-CPU training +-----------+-------+-------+--------+--------+---------+------+ | Models | MR | MRR | HITS-1 | HITS-3 | HITS-10 | TIME | +===========+=======+=======+========+========+=========+======+ | TransE_l2 | 30.82 | 0.815 | 0.766 | 0.848 | 0.902 | 6993 | +-----------+-------+-------+--------+--------+---------+------+ | DistMult | 44.16 | 0.834 | 0.815 | 0.843 | 0.869 | 7146 | +-----------+-------+-------+--------+--------+---------+------+ | ComplEx | 45.62 | 0.835 | 0.817 | 0.843 | 0.870 | 8732 | +-----------+-------+-------+--------+--------+---------+------+ Distributed training +-----------+-------+-------+--------+--------+---------+------+ | Models | MR | MRR | HITS-1 | HITS-3 | HITS-10 | TIME | +===========+=======+=======+========+========+=========+======+ | TransE_l2 | 34.25 | 0.764 | 0.705 | 0.802 | 0.869 | 1633 | +-----------+-------+-------+--------+--------+---------+------+ | DistMult | 75.15 | 0.769 | 0.751 | 0.779 | 0.801 | 1679 | +-----------+-------+-------+--------+--------+---------+------+ | ComplEx | 77.83 | 0.771 | 0.754 | 0.779 | 0.802 | 2293 | +-----------+-------+-------+--------+--------+---------+------+