Neural Rule Ensembles

Artificial Neural Networks form the basis of very powerful learning methods. It has been observed that a naive application of fully connected neural networks often leads to overfitting. In an attempt to circumvent this issue, a prior knowledge pertaining to feature interactions can be encoded into these networks. This defines a task-specific structure on an underlying representation and helps in reducing the number of learnable parameters. Convolutional Neural Network is such an adaptation of artificial neural networks for image datasets which exploits the spatial relationship among the features and explicitly encodes the translational equivariance. Similarly, Recurrent Neural Networks are designed to exploit the temporal relationship inherent in sequential data. However, for tabular datasets, any prior structure on feature relationships is not apparent. In this work, we use decision trees to capture such feature interactions for this kind of datasets and define a mapping to encode extracted relationships into a neural network. This addresses the initialization related concerns of fully connected neural networks and enables learning of compact representations compared to state of the art tree-based approaches. Empirical evaluations and simulation studies show the superiority of such an approach over fully connected neural networks and tree-based approaches.