On the relations of stochastic convex optimization problems with empirical risk minimization problems on $p$-norm balls

Dvinskikh D.M., Pirau V.V., Gasnikov A.V.
 pdf (192K)

In this paper, we consider convex stochastic optimization problems arising in machine learning applications (e. g., risk minimization) and mathematical statistics (e. g., maximum likelihood estimation). There are two main approaches to solve such kinds of problems, namely the Stochastic Approximation approach (online approach) and the Sample Average Approximation approach, also known as the Monte Carlo approach, (offline approach). In the offline approach, the problem is replaced by its empirical counterpart (the empirical risk minimization problem). The natural question is how to define the problem sample size, i. e., how many realizations should be sampled so that the quite accurate solution of the empirical problem be the solution of the original problem with the desired precision. This issue is one of the main issues in modern machine learning and optimization. In the last decade, a lot of significant advances were made in these areas to solve convex stochastic optimization problems on the Euclidean balls (or the whole space). In this work, we are based on these advances and study the case of arbitrary balls in the $p$-norms. We also explore the question of how the parameter $p$ affects the estimates of the required number of terms as a function of empirical risk.

In this paper, both convex and saddle point optimization problems are considered. For strongly convex problems, the existing results on the same sample sizes in both approaches (online and offline) were generalized to arbitrary norms. Moreover, it was shown that the strong convexity condition can be weakened: the obtained results are valid for functions satisfying the quadratic growth condition. In the case when this condition is not met, it is proposed to use the regularization of the original problem in an arbitrary norm. In contradistinction to convex problems, saddle point problems are much less studied. For saddle point problems, the sample size was obtained under the condition of $\gamma$-growth of the objective function. When $\gamma = 1$, this condition is the condition of sharp minimum in convex problems. In this article, it was shown that the sample size in the case of a sharp minimum is almost independent of the desired accuracy of the solution of the original problem.

Keywords: convex optimization, stochastic optimization, regularization, empirical risk minimization, stochastic approximation, sample average approximation, quadratic growth condition, sharp minimum
Citation in English: Dvinskikh D.M., Pirau V.V., Gasnikov A.V. On the relations of stochastic convex optimization problems with empirical risk minimization problems on $p$-norm balls // Computer Research and Modeling, 2022, vol. 14, no. 2, pp. 309-319
Citation in English: Dvinskikh D.M., Pirau V.V., Gasnikov A.V. On the relations of stochastic convex optimization problems with empirical risk minimization problems on $p$-norm balls // Computer Research and Modeling, 2022, vol. 14, no. 2, pp. 309-319
DOI: 10.20537/2076-7633-2022-14-2-309-319
URL: http://crm-en.ics.org.ru/journal/article/3185/
Creative Commons License This work is licensed under a Creative Commons Attribution-NoDerivs 3.0 Unported License.

Indexed in Scopus

Full-text version of the journal is also available on the web site of the scientific electronic library eLIBRARY.RU

The journal is included in the Russian Science Citation Index

The journal is included in the RSCI

International Interdisciplinary Conference "Mathematics. Computing. Education"

Copyright © 2009–2024 Institute of Computer Science