Influence of the mantissa finiteness on the accuracy of gradient-free optimization methods

Vostrikov D.D., Konin G.O., Lobanov A.V., Matyukhin V.V.
 pdf (6086K)

Gradient-free optimization methods or zeroth-order methods are widely used in training neural networks, reinforcement learning, as well as in industrial tasks where only the values of a function at a point are available (working with non-analytical functions). In particular, the method of error back propagation in PyTorch works exactly on this principle. There is a well-known fact that computer calculations use heuristics of floating-point numbers, and because of this, the problem of finiteness of the mantissa arises.

In this paper, firstly, we reviewed the most popular methods of gradient approximation: Finite forward/central difference (FFD/FCD), Forward/Central wise component (FWC/CWC), Forward/Central randomization on $l_2$ sphere (FSSG2/CFFG2); secondly, we described current theoretical representations of the noise introduced by the inaccuracy of calculating the function at a point: adversarial noise, random noise; thirdly, we conducted a series of experiments on frequently encountered classes of problems, such as quadratic problem, logistic regression, SVM, to try to determine whether the real nature of machine noise corresponds to the existing theory. It turned out that in reality (at least for those classes of problems that were considered in this paper), machine noise turned out to be something between adversarial noise and random, and therefore the current theory about the influence of the mantissa limb on the search for the optimum in gradient-free optimization problems requires some adjustment.

Keywords: mantissa finiteness, gradient-free optimization, gradient approximation, gradient descent, quadratic problem, logistic regression
Citation in English: Vostrikov D.D., Konin G.O., Lobanov A.V., Matyukhin V.V. Influence of the mantissa finiteness on the accuracy of gradient-free optimization methods // Computer Research and Modeling, 2023, vol. 15, no. 2, pp. 259-280
Citation in English: Vostrikov D.D., Konin G.O., Lobanov A.V., Matyukhin V.V. Influence of the mantissa finiteness on the accuracy of gradient-free optimization methods // Computer Research and Modeling, 2023, vol. 15, no. 2, pp. 259-280
DOI: 10.20537/2076-7633-2023-15-2-259-280
URL: http://crm-en.ics.org.ru/journal/article/3315/
Creative Commons License This work is licensed under a Creative Commons Attribution-NoDerivs 3.0 Unported License.

Indexed in Scopus

Full-text version of the journal is also available on the web site of the scientific electronic library eLIBRARY.RU

The journal is included in the Russian Science Citation Index

The journal is included in the RSCI

International Interdisciplinary Conference "Mathematics. Computing. Education"

Copyright © 2009–2024 Institute of Computer Science