Applications of Tropical Mathematics to Neural Network Architecture

Abstract

Using the methods provided by tropical mathematics we can simplify the structure of a neural network, which increases its explainability, without decreasing its accuracy. This paper aims to explore the use of tropical functions in neural networks and compare their efficiency with classical ones. Theoretical framework of tropical mathematics is a semiring with idempotent addition, which is a natural approach to piecewise-linear neural networks, e.g. networks with ReLU activation. Within this approach, piecewise-linear convex function is a tropical polynomial, and general piecewise-linear functions are tropical rational functions. Thus a layer of a neural network with linear preactivation and ReLU activation can be viewed as a vector-valued tropical rational function, which in turn can be represented by two tropical layers. Two tropical layers were implemented, and five tropical architectures were constructed. The models were trained on a heart disease dataset, aiming to determine the presence of heart disease. All models had the same hyperparameters. Each of the models was trained for 100 epochs using Adam and SGD optimizers. The results of the comparison showed that the best accuracy was achieved by a mixed-architecture model using two linear layers. The comparison results showed that the best accuracy was achieved by a mixed-architecture model with two linear layers with a min-layer and a max-layer in between. This accuracy was achieved by using an Adam optimizer. The classical model scored 77.3% and the tropical 77.7%.