Venue
Journal of machine learning research
In the last few years, graph neural networks (GNNs) have become the standard toolkit for analyzing and learning from data on graphs. This emerging field has witnessed an extensive growth of promising techniques that have been applied with success to computer science, mathematics, biology, physics and chemistry. But for any successful field to become mainstream and reliable, benchmarks must be developed to quantify progress. This led us in March 2020 to release a benchmark framework that i) comprises of a diverse collection of mathematical and real-world graphs, ii) enables fair model comparison with the same parameter budget to identify key architectures, iii) has an open-source, easy-to-use and reproducible code infrastructure, and iv) is flexible for researchers to experiment with new theoretical ideas. As of December 2022, the GitHub repository 1 has reached 2,000 stars and 380 forks, which demonstrates the utility of the proposed open-source framework through the wide usage by the GNN community. In this paper, we present an updated version of our benchmark with a concise presentation of the aforementioned framework characteristics, an additional medium-sized molecular dataset AQSOL, similar to the popular ZINC, but with a real-world measured chemical target, and discuss how this framework can be leveraged to explore new GNN designs and insights. As a proof of value of our benchmark, we study the case of graph positional encoding (PE) in GNNs, which was introduced with this benchmark and has since spurred interest of exploring more powerful PE for Transformers and GNNs in a robust experimental setting.
The paper presents an open-source benchmarking framework for Graph Neural Networks (GNNs). It outlines the necessity for standardized benchmarks to assess progress in GNN research across various fields like computer science, biology, and physics. The updated framework now includes additional datasets, notably AQSOL, alongside a diverse collection of mathematical and real-world graphs allowing for fair model comparison. It highlights the effective use of positional encodings in enhancing the performance of GNNs, particularly through the use of Laplacian eigenvectors for graph positional encoding. The framework encourages exploration of GNN designs and aims to establish a modular and reproducible codebase for future research.
This paper employs the following methods:
- Graph Neural Networks (GNNs)
- Message Passing Graph Convolutional Networks (MP-GCNs)
- Weisfeiler-Lehman (WL) GNNs
- GatedGCN
- GraphSage
- GAT
- MoNet
- vanilla GCN
- 3WLGNN
- RingGNN
- GIN
The following datasets were used in this research:
- AQSOL
- ZINC
- OGBL-COLLAB
- WikiCS
- MNIST
- CIFAR10
- CSL
- CYCLES
- GraphTheoryProp
- SBM (PATTERN/CLUSTER)
- TSP
- Mean Absolute Error (MAE)
- Classification Accuracy
- Hits@K
- F1 Score
- Proposed benchmark framework facilitates better comparison and insights on GNNs
- Laplacian Positional Encoding improves MP-GCNs performance
- Anisotropic GNNs show enhanced results compared to isotropic models
The authors identified the following limitations:
- Challenges in universally defining datasets and their representative quality
- Absence of consensus on experimental settings in the GNN literature
- Number of GPUs: 4
- GPU Type: None specified
Graph Neural Networks
Benchmarking
Datasets
Graph Representation Learning
Positional Encoding