All issues

This article reviews both historical achievements and modern results in the field of Markov Decision Process (MDP) and convex optimization. This review is the first attempt to cover the field of reinforcement learning in Russian in the context of convex optimization. The fundamental Bellman equation and the criteria of optimality of policy — strategies based on it, which make decisions based on the known state of the environment at the moment, are considered. The main iterative algorithms of policy optimization based on the solution of the Bellman equations are also considered. An important section of this article was the consideration of an alternative to the $Q$-learning approach — the method of direct maximization of the agent’s average reward for the chosen strategy from interaction with the environment. Thus, the solution of this convex optimization problem can be represented as a linear programming problem. The paper demonstrates how the convex optimization apparatus is used to solve the problem of Reinforcement Learning (RL). In particular, it is shown how the concept of strong duality allows us to naturally modify the formulation of the RL problem, showing the equivalence between maximizing the agent’s reward and finding his optimal strategy. The paper also discusses the complexity of MDP optimization with respect to the number of state–action–reward triples obtained as a result of interaction with the environment. The optimal limits of the MDP solution complexity are presented in the case of an ergodic process with an infinite horizon, as well as in the case of a non-stationary process with a finite horizon, which can be restarted several times in a row or immediately run in parallel in several threads. The review also reviews the latest results on reducing the gap between the lower and upper estimates of the complexity of MDP optimization with average remuneration (Averaged MDP, AMDP). In conclusion, the real-valued parametrization of agent policy and a class of gradient optimization methods through maximizing the $Q$-function of value are considered. In particular, a special class of MDPs with restrictions on the value of policy (Constrained Markov Decision Process, CMDP) is presented, for which a general direct-dual approach to optimization with strong duality is proposed.

The main aim, formulated in the first part of article, is to carry out detailed numerical studies of the chemical, ionization, optical, and temperature characteristics of the lower ionosphere perturbed by powerful radio emission. The brief review of the main experimental and theoretical researches of physical phenomena occurring in the ionosphere when it is heated by high-power high-frequency radio waves from heating facilities is given. The decisive role of the $D$-region of the ionosphere in the absorption of radio beam energy is shown. A detailed analysis of kinetic processes in the disturbed $D$-region, which is the most complex in kinetic terms, has been performed. It is shown that for a complete description of the ionization-chemical and optical characteristics of the disturbed region, it is necessary to take into account more than 70 components, which, according to their main physical content, can be conveniently divided into five groups. A kinetic model is presented to describe changes in the concentrations of components interacting (the total number of reactions is 259). The system of kinetic equations was solved using a semi-implicit numerical method specially adapted to such problems. Based on the proposed structure, a software package was developed in which the algorithm scheme allowed changing both the content of individual program blocks and their number, which made it possible to conduct detailed numerical studies of individual processes in the behavior of the parameters of the perturbed region. The complete numerical algorithm is based on the two-temperature approximation, in which the main attention was paid to the calculation of the electron temperature, since its behavior is determined by inelastic kinetic processes involving electrons. The formulation of the problem is of a rather general nature and makes it possible to calculate the parameters of the disturbed ionosphere in a wide range of powers and frequencies of radio emission. Based on the developed numerical technique, it is possible to study a wide range of phenomena both in the natural and disturbed ionosphere.

Automatic recognition of personal identity by biometric features is based on unique peculiarities or characteristics of people. Biometric identification process consist in making of reference templates and comparison with new input data. Iris pattern recognition algorithms presents high accuracy and low identification errors percent on practice. Iris pattern advantages over other biometric features are determined by its high degree of freedom (nearly 249), excessive density of unique features and constancy. High recognition reliability level is very important because it provides search in big databases. Unlike one-to-one check mode that is applicable only to small calculation count it allows to work in one-to-many identification mode. Every biometric identification system appears to be probabilistic and qualitative characteristics description utilizes such parameters as: recognition accuracy, false acceptance rate and false rejection rate. These characteristics allows to compare identity recognition methods and asses the system performance under any circumstances. This article explains the mathematical model of iris pattern biometric identification and its characteristics. Besides, there are analyzed results of comparison of model and real recognition process. To make such analysis there was carried out the review of existing iris pattern recognition methods based on different unique features vector. The Python-based software package is described below. It builds-up probabilistic distributions and generates large test data sets. Such data sets can be also used to educate the identification decision making neural network. Furthermore, synergy algorithm of several iris pattern identification methods was suggested to increase qualitative characteristics of system in comparison with the use of each method separately.

The model of market pricing in duopoly describing the prices dynamics as a two-dimensional map is presented. It is shown that the fixed point of the map coincides with the local Nash-equilibrium price in duopoly game. There have been numerically identified a bifurcation of the fixed point, shown the scheme of transition from periodic to chaotic mode through a doubling period. To ensure the sustainability of local Nashequilibrium price the controlling chaos mechanism has been proposed. This mechanism allows to harmonize the economic interests of the firms and to form the balanced pricing policy.

The traditional approach to computing optimal game strategies of firms on oligopolistic markets and of indicators of such markets consists in studying linear dynamical games with quadratic criteria and solving generalized matrix Riccati equations.

The other approach proposed by the author is based on methods of operational calculus (in particular, Z-transform). This approach makes it possible to achieve economic meaningful decisions under wider field of parameter values. It characterizes by simplicity of computations and by necessary for economic analysis visibility. One of its advantages is that in many cases important for economic practice, it, in contrast to the traditional approach, provides the ability to make calculations using widespread spreadsheets, which allows to study the prospects for the development of oligopolistic markets to a wide range of professionals and consumers.

The article deals with the practical aspects of determining the optimal Nash–Cournot strategies of participants in oligopolistic markets on the basis of operational calculus, in particular the technique of computing the optimal Nash–Cournot strategies in Excel. As an illustration of the opportinities of the proposed methods of calculation, examples close to the practical problems of forecasting indicators of the markets of high-tech products are studied.

The results of calculations obtained by the author for numerous examples and real economic systems, both using the obtained relations on the basis of spreadsheets and using extended Riccati equations, are very close. In most of the considered practical problems, the deviation of the indicators calculated in accordance with the two approaches, as a rule, does not exceed 1.5–2%. The highest value of relative deviations (up to 3–5%) is observed at the beginning of the forecasting period. In typical cases, the period of relatively noticeable deviations is 3–5 moments of time. After the transition period, there is almost complete agreement of the values of the required indicators using both approaches.

The dynamics of public attention to COVID-19 epidemic is studied. The level of public attention is described by the daily number of search requests in Google made by users from a given country. In the empirical part of the work, data on the number of requests and the number of infected cases for a number of countries are considered. It is shown that in all cases the maximum of public attention occurs earlier than the maximum daily number of newly infected individuals. Thus, for a certain period of time, the growth of the epidemics occurs in parallel with the decline in public attention to it. It is also shown that the decline in the number of requests is described by an exponential function of time. In order to describe the revealed empirical pattern, a mathematical model is proposed, which is a modification of the model of the decline in attention after a one-time political event. The model develops the approach that considers decision-making by an individual as a member of the society in which the information process takes place. This approach assumes that an individual’s decision about whether or not to make a request on a given day about COVID is based on two factors. One of them is an attitude that reflects the individual’s long-term interest in a given topic and accumulates the individual’s previous experience, cultural preferences, social and economic status. The second is the dynamic factor of public attention to the epidemic, which changes during the process under consideration under the influence of informational stimuli. With regard to the subject under consideration, information stimuli are related to epidemic dynamics. The behavioral hypothesis is that if on some day the sum of the attitude and the dynamic factor exceeds a certain threshold value, then on that day the individual in question makes a search request on the topic of COVID. The general logic is that the higher the rate of infection growth, the higher the information stimulus, the slower decreases public attention to the pandemic. Thus, the constructed model made it possible to correlate the rate of exponential decrease in the number of requests with the rate of growth in the number of cases. The regularity found with the help of the model was tested on empirical data. It was found that the Student’s statistic is 4.56, which allows us to reject the hypothesis of the absence of a correlation with a significance level of 0.01.

Modeling combat operations is an urgent scientific and practical task aimed at providing commanders and staffs with quantitative grounds for making decisions. The authors proposed the function of victory in combat and military operations, based on the function of the conflict by G. Tullock and taking into account the scale of combat (military) operations. On a sufficient volume of military statistics, the scale parameter was assessed and its values were found for the tactical, operational and strategic levels. The game-theoretic models «offensive – defense», in which the sides solve the immediate and subsequent tasks, having the formation of troops in one or several echelons, have been investigated. At the first stage of modeling, the solution of the immediate task is found — the breakthrough (holding) of defense points, at the second — the solution of the subsequent task — the defeat of the enemy in the depth of the defense (counterattack and restoration of defense). For the tactical level, using the Nash equilibrium, solutions were found for the closest problem (distribution of the forces of the sides by points of defense) in an antagonistic game according to three criteria: a) breakthrough of the weakest point, b) breakthrough of at least one point, and c) weighted average probability. It is shown that it is advisable for the attacking side to use the criterion of «breaking through at least one point», in which, all other things being equal, the maximum probability of breaking through the points of defense is ensured. At the second stage of modeling for a particular case (the sides are guided by the criterion of breaking through the weakest point when breaking through and holding defense points), the problem of distributing forces and facilities between tactical tasks (echelons) was solved according to two criteria: a) maximizing the probability of breaking through the defense point and the probability of defeating the enemy in depth defense, b) maximizing the minimum value of the named probabilities (the criterion of the guaranteed result). Awareness is an important aspect of combat operations. Several examples of reflexive games (games characterized by complex mutual awareness) and information management are considered. It is shown under what conditions information control increases the player’s payoff, and the optimal information control is found.

A crucial component of distributed computing systems is network infrastructure. While networking forms the backbone of such systems, it is often the invisible partner to storage and computing resources. We propose to integrate Network Elements directly into distributed systems through the workload management layer. There are many reasons for this approach. As the complexity and demand for distributed systems grow, it is important to use existing infrastructure efficiently. For example, one could use network performance measurements in the decision making mechanisms of workload management systems. New advanced technologies allow one to programmatically define network configuration, for example SDN — Software Defined Networks. We will describe how these methods are being used within the PanDA workload management system of the ATLAS collaboration.

In general, solving the information explosion problem can be delegated to systems for automatic processing of digital data. These systems are intended for recognizing, sorting, meaningfully processing and presenting data in formats readable and interpretable by humans. The creation of intelligent knowledge extraction systems that handle unstructured data would be a natural solution in this area. At the same time, the evident progress in these tasks for structured data contrasts with the limited success of unstructured data processing, and, in particular, document processing. Currently, this research area is undergoing active development and investigation. The present paper is a systematic survey on both Russian and international publications that are dedicated to the leading trend in automatic text data processing: Text Mining (TM). We cover the main tasks and notions of TM, as well as its place in the current AI landscape. Furthermore, we analyze the complications that arise during the processing of texts written in natural language (NLP) which are weakly structured and often provide ambiguous linguistic information. We describe the stages of text data preparation, cleaning, and selecting features which, alongside the data obtained via morphological, syntactic, and semantic analysis, constitute the input for the TM process. This process can be represented as mapping a set of text documents to «knowledge». Using the case of stock trading, we demonstrate the formalization of the problem of making a trade decision based on a set of analytical recommendations. Examples of such mappings are methods of Information Retrieval (IR), text summarization, sentiment analysis, document classification and clustering, etc. The common point of all tasks and techniques of TM is the selection of word forms and their derivatives used to recognize content in NL symbol sequences. Considering IR as an example, we examine classic types of search, such as searching for word forms, phrases, patterns and concepts. Additionally, we consider the augmentation of patterns with syntactic and semantic information. Next, we provide a general description of all NLP instruments: morphological, syntactic, semantic and pragmatic analysis. Finally, we end the paper with a comparative analysis of modern TM tools which can be helpful for selecting a suitable TM platform based on the user’s needs and skills.

Deposit benchmark interest rates are a policy implemented by banking regulators to calculate the interest rates offered to depositors, maintaining equitable and competitive rates within the financial industry. It functions as a benchmark for determining the pricing of different banking products, expenses, and financial choices. The benchmark rate will have a direct impact on the amount of money deposited, which in turn will determine the amount of money available for lending.We are motivated to analyze the influence of deposit benchmark interest rates on the dynamics of banking loans. This study examines the issue using a difference equation of banking loans. In this process, the decision on the loan amount in the next period is influenced by both the present loan volume and the information on its marginal profit. An analysis is made of the loan equilibrium point and its stability. We also analyze the bifurcations that arise in the model. To ensure a stable banking loan, it is necessary to set the benchmark rate higher than the flip value and lower than the transcritical bifurcation values. The confirmation of this result is supported by the bifurcation diagram and its associated Lyapunov exponent. Insufficient deposit benchmark interest rates might lead to chaotic dynamics in banking lending. Additionally, a bifurcation diagram with two parameters is also shown. We do numerical sensitivity analysis by examining contour plots of the stability requirements, which vary with the deposit benchmark interest rate and other parameters. In addition, we examine a nonstandard difference approach for the previous model, assess its stability, and make a comparison with the standard model. The outcome of our study can provide valuable insights to the banking regulator in making informed decisions regarding deposit benchmark interest rates, taking into account several other banking factors.