Partial Graph Reasoning for Neural Network Regularization

  • 1University of Sydney
  • 2University of New South Wales
  • 3Paige AI
  • 4Tencent AI Lab

Overview

overview

Regularizers helped deep neural networks prevent feature co-adaptations. Dropout, as a commonly used regularization technique, stochastically disables neuron activations during network optimization. However, such complete feature disposal can affect the feature representation and network understanding. Toward better descriptions of latent representations, we present DropGraph that learns regularization function by constructing a stand-alone graph from the backbone features. DropGraph first samples stochastic spatial feature vectors and then incorporates graph reasoning methods to generate feature map distortions. This add-on graph regularizes the network during training and can be completely skipped during inference. We provide intuitions on the linkage between graph reasoning and Dropout with further discussions on how partial graph reasoning method reduces feature correlations. To this end, we extensively study the modeling of graph vertex dependencies and the utilization of the graph for distorting backbone feature maps. DropGraph was validated on four tasks with a total of 7 different datasets. The experimental results show that our method outperforms other state-of-the-art regularizers while leaving the base model structure unmodified during inference.

Highlights

  • An input-sensitive learnbale regularizer that distorts network feature maps.
  • Re-formulation of graph reasoning as network regularization.
  • Evaluations have been made on 4 tasks across 7 different datasets with state-of-the-art results.
    • Image classification: CIFAR (on ResNet and RegNet) and ImageNet (on ResNet and RegNet).
    • Image semantic segmentation: Pascal VOC 2012 (on FCN and DeepLabV3) and MoNuSeg (on U-Net and Attention U-Net) .
    • Point cloud classification: ModelNet40 (on DGCNN).
    • Graph representation learning: node classification on Cora & graph recognition on Protein.

Video

(Coming soon)

BibTeX

If you find our data or project useful in your research, please cite:

(Coming soon)

Acknowledgments

The website template was borrowed from Chaoyi Zhang.