All issues

This article reviews both historical achievements and modern results in the field of Markov Decision Process (MDP) and convex optimization. This review is the first attempt to cover the field of reinforcement learning in Russian in the context of convex optimization. The fundamental Bellman equation and the criteria of optimality of policy — strategies based on it, which make decisions based on the known state of the environment at the moment, are considered. The main iterative algorithms of policy optimization based on the solution of the Bellman equations are also considered. An important section of this article was the consideration of an alternative to the $Q$-learning approach — the method of direct maximization of the agent’s average reward for the chosen strategy from interaction with the environment. Thus, the solution of this convex optimization problem can be represented as a linear programming problem. The paper demonstrates how the convex optimization apparatus is used to solve the problem of Reinforcement Learning (RL). In particular, it is shown how the concept of strong duality allows us to naturally modify the formulation of the RL problem, showing the equivalence between maximizing the agent’s reward and finding his optimal strategy. The paper also discusses the complexity of MDP optimization with respect to the number of state–action–reward triples obtained as a result of interaction with the environment. The optimal limits of the MDP solution complexity are presented in the case of an ergodic process with an infinite horizon, as well as in the case of a non-stationary process with a finite horizon, which can be restarted several times in a row or immediately run in parallel in several threads. The review also reviews the latest results on reducing the gap between the lower and upper estimates of the complexity of MDP optimization with average remuneration (Averaged MDP, AMDP). In conclusion, the real-valued parametrization of agent policy and a class of gradient optimization methods through maximizing the $Q$-function of value are considered. In particular, a special class of MDPs with restrictions on the value of policy (Constrained Markov Decision Process, CMDP) is presented, for which a general direct-dual approach to optimization with strong duality is proposed.

The course and outcome of the battle is largely dependent on the morale of the troops, characterized by the percentage of loss in killed and wounded, in which the troops still continue to fight. Every fight is a psychological act of ending his rejection of one of the parties. Typically, models of battle psychological factor taken into account in the decision of Lanchester equations (the condition of equality of forces, when the number of one of the parties becomes zero). It is emphasized that the model Lanchester type satisfactorily describe the dynamics of the battle only in the initial stages. To resolve this contradiction is proposed to use a modification of Lanchester's equations, taking into account the fact that at any moment of the battle on the enemy firing not affected and did not abandon the battle fighters. The obtained differential equations are solved by numerical method and allow the dynamics to take into account the influence of psychological factor and evaluate the completion time of the conflict. Computational experiments confirm the known military theory is the fact that the fight usually ends in refusal of soldiers of one of the parties from its continuation (avoidance of combat in various forms). Along with models of temporal and spatial dynamics proposed to use a modification of the technology features of the conflict of S. Skaperdas, based on the principles of combat. To estimate the probability of victory of one side in the battle takes into account the interest of the maturing sides of the bloody casualties and increased military superiority.

The paper develops a new mathematical method of the joint signal and noise calculation at the Rice statistical distribution based on combing the maximum likelihood method and the method of moments. The calculation of the sough-for values of signal and noise is implemented by processing the sampled measurements of the analyzed Rician signal’s amplitude. The explicit equations’ system has been obtained for required signal and noise parameters and the results of its numerical solution are provided confirming the efficiency of the proposed technique. It has been shown that solving the two-parameter task by means of the proposed technique does not lead to the increase of the volume of demanded calculative resources if compared with solving the task in one-parameter approximation. An analytical solution of the task has been obtained for the particular case of small value of the signal-to-noise ratio. The paper presents the investigation of the dependence of the sought for parameters estimation accuracy and dispersion on the quantity of measurements in experimental sample. According to the results of numerical experiments, the dispersion values of the estimated sought-for signal and noise parameters calculated by means of the proposed technique change in inverse proportion to the quantity of measurements in a sample. There has been implemented a comparison of the accuracy of the soughtfor Rician parameters’ estimation by means of the proposed technique and by earlier developed version of the method of moments. The problem having been considered in the paper is meaningful for the purposes of Rician data processing, in particular, at the systems of magnetic-resonance visualization, in devices of ultrasonic visualization, at optical signals’ analysis in range-measuring systems, at radar signals’ analysis, as well as at solving many other scientific and applied tasks that are adequately described by the Rice statistical model.

The article deals with the process of creating an effective algorithm for determining the amount of emitted quanta from an X-ray tube in computer tomography (CT) studies. An analysis of domestic and foreign literature showed that most of the work in the field of radiometry and radiography takes the tabulated values of X-ray absorption coefficients into account, while individual dose factors are not taken into account at all since many studies are lacking the Dose Report. Instead, an average value is used to simplify the calculation of statistics. In this regard, it was decided to develop a method to detect the amount of ionizing quanta by analyzing the noise of CT data. As the basis of the algorithm, we used Poisson and Gauss distribution mathematical model of owns’ design of logarithmic value. The resulting mathematical model was tested on the CT data of a calibration phantom consisting of three plastic cylinders filled with water, the X-ray absorption coefficient of which is known from the table values. The data were obtained from several CT devices from different manufacturers (Siemens, Toshiba, GE, Phillips). The developed algorithm made it possible to calculate the number of emitted X-ray quanta per unit time. These data, taking into account the noise level and the radiuses of the cylinders, were converted to X-ray absorption values, after which a comparison was made with tabulated values. As a result of this operation, the algorithm used with CT data of various configurations, experimental data were obtained, consistent with the theoretical part and the mathematical model. The results showed good accuracy of the algorithm and mathematical apparatus, which shows reliability of the obtained data. This mathematical model is already used in the noise reduction program of the CT of own design, where it participates as a method of creating a dynamic threshold of noise reduction. At the moment, the algorithm is being processed to work with real data from computer tomography of patients.

The traditional approach to computing optimal game strategies of firms on oligopolistic markets and of indicators of such markets consists in studying linear dynamical games with quadratic criteria and solving generalized matrix Riccati equations.

The other approach proposed by the author is based on methods of operational calculus (in particular, Z-transform). This approach makes it possible to achieve economic meaningful decisions under wider field of parameter values. It characterizes by simplicity of computations and by necessary for economic analysis visibility. One of its advantages is that in many cases important for economic practice, it, in contrast to the traditional approach, provides the ability to make calculations using widespread spreadsheets, which allows to study the prospects for the development of oligopolistic markets to a wide range of professionals and consumers.

The article deals with the practical aspects of determining the optimal Nash–Cournot strategies of participants in oligopolistic markets on the basis of operational calculus, in particular the technique of computing the optimal Nash–Cournot strategies in Excel. As an illustration of the opportinities of the proposed methods of calculation, examples close to the practical problems of forecasting indicators of the markets of high-tech products are studied.

The results of calculations obtained by the author for numerous examples and real economic systems, both using the obtained relations on the basis of spreadsheets and using extended Riccati equations, are very close. In most of the considered practical problems, the deviation of the indicators calculated in accordance with the two approaches, as a rule, does not exceed 1.5–2%. The highest value of relative deviations (up to 3–5%) is observed at the beginning of the forecasting period. In typical cases, the period of relatively noticeable deviations is 3–5 moments of time. After the transition period, there is almost complete agreement of the values of the required indicators using both approaches.

In this work, the problem of constructing an electronic analogue of heterogeneous DNA is solved with the help of the methods of mathematical modeling. Electronic analogs of that type, along with other physical models of living systems, are widely used as a tool for studying the dynamic and functional properties of these systems. The solution to the problem is based on an algorithm previously developed for homogeneous (synthetic) DNA and modified in such a way that it can be used for the case of inhomogeneous (native) DNA. The algorithm includes the following steps: selection of a model that simulates the internal mobility of DNA; construction of a transformation that allows you to move from the DNA model to its electronic analogue; search for conditions that provide an analogy of DNA equations and electronic analogue equations; calculation of the parameters of the equivalent electrical circuit. To describe inhomogeneous DNA, the model was chosen that is a system of discrete nonlinear differential equations simulating the angular deviations of nitrogenous bases, and Hamiltonian corresponding to these equations. The values of the coefficients in the model equations are completely determined by the dynamic parameters of the DNA molecule, including the moments of inertia of nitrous bases, the rigidity of the sugar-phosphate chain, and the constants characterizing the interactions between complementary bases in pairs. The inhomogeneous Josephson line was used as a basis for constructing an electronic model, the equivalent circuit of which contains four types of cells: A-, T-, G-, and C-cells. Each cell, in turn, consists of three elements: capacitance, inductance, and Josephson junction. It is important that the A-, T-, G- and C-cells of the Josephson line are arranged in a specific order, which is similar to the order of the nitrogenous bases (A, T, G and C) in the DNA sequence. The transition from DNA to an electronic analog was carried out with the help of the A-transformation which made it possible to calculate the values of the capacitance, inductance, and Josephson junction in the A-cells. The parameter values for the T-, G-, and C-cells of the equivalent electrical circuit were obtained from the conditions imposed on the coefficients of the model equations and providing an analogy between DNA and the electronic model.

This article discusses the influence of the shadow, informal and household sectors on the dynamics of a stochastic model with heterogeneous (heterogeneous) agents. The study uses the integration of the general equilibrium approach to explain the behavior of demand, supply and prices in an economy with several interacting markets, and a multi-agent approach. The analyzed model describes an economy with aggregated uncertainty and with an infinite number of heterogeneous agents (households). The source of heterogeneity is the idiosyncratic income shocks of agents in the legal and shadow sectors of the economy. In the analysis, an algorithm is used to approximate the dynamics of the distribution function of the capital stocks of individual agents — the dynamics of its first and second moments. The synthesis of the agent approach and the general equilibrium approach is carried out using computer implementation of the recursive feedback between microagents and macroenvironment. The behavior of the impulse response functions of the main variables of the model confirms the positive influence of the shadow economy (below a certain limit) on minimizing the rate of decline in economic indicators during recessions, especially for developing economies. The scientific novelty of the study is the combination of a multi-agent approach and a general equilibrium approach for modeling macroeconomic processes at the regional and national levels. Further research prospects may be associated with the use of more detailed general equilibrium models, which allow, in particular, to describe the behavior of heterogeneous groups of agents in the entrepreneurial sector of the economy.

Numerical simulation of moving sportsman external flow is presented. The unique method is developed for obtaining integral aerodynamic characteristics, which were the function of the flow regime (i.e. angle of attack, flow speed) and body position. Individual anthropometric characteristics and moving boundaries of sportsman (or sports equipment) during the race are taken into consideration.

Numerical simulation is realized using FlowVision CFD. The software is based on the finite volume method, high-performance numerical methods and reliable mathematical models of physical processes. A Cartesian computational grid is used by FlowVision, the grid generation is a completely automated process. Local grid adaptation is used for solving high-pressure gradient and object complex shape. Flow simulation process performed by solutions systems of equations describing movement of fluid and/or gas in the computational domain, including: mass, moment and energy conservation equations; state equations; turbulence model equations. FlowVision permits flow simulation near moving bodies by means of computational domain transformation according to the athlete shape changes in the motion. Ski jumper aerodynamic characteristics are studied during all phases: take-off performance in motion, in-run and flight. Projected investigation defined simulation method, which includes: inverted statement of sportsman external flow development (velocity of the motion is equal to air flow velocity, object is immobile); changes boundary of the body technology defining; multiple calculations with the national team member data projecting. The research results are identification of the main factors affected to jumping performance: aerodynamic forces, rotating moments etc. Developed method was tested with active sportsmen. Ski jumpers used this method during preparations for Sochi Olympic Games 2014. A comparison of the predicted characteristics and experimental data shows a good agreement. Method versatility is underlined by performing swimmer and skater flow simulation. Designed technology is applicable for sorts of natural and technical objects.

The paper is devoted to the study of relationships between discrete and continuous models financial processes and their probabilistic characteristics. First, a connection is established between the price processes of stocks, hedging portfolio and options in the models conditioned by binomial perturbations and their limit perturbations of the Brownian motion type. Secondly, analogues in the coefficients of stochastic equations with various random processes, continuous and jumpwise, and in the coefficients corresponding deterministic equations for their probabilistic characteristics. Statement of the results on the connections and finding analogies, obtained in this paper, led to the need for an adequate presentation of preliminary information and results from financial mathematics, as well as descriptions of related objects of stochastic analysis. In this paper, partially new and known results are presented in an accessible form for those who are not specialists in financial mathematics and stochastic analysis, and for whom these results are important from the point of view of applications. Specifically, the following sections are presented.

• In one- and n-period binomial models, it is proposed a unified approach to determining on the probability space a risk-neutral measure with which the discounted option price becomes a martingale. The resulting martingale formula for the option price is suitable for numerical simulation. In the following sections, the risk-neutral measures approach is applied to study financial processes in continuous-time models.

• In continuous time, models of the price of shares, hedging portfolios and options are considered in the form of stochastic equations with the Ito integral over Brownian motion and over a compensated Poisson process. The study of the properties of these processes in this section is based on one of the central objects of stochastic analysis — the Ito formula. Special attention is given to the methods of its application.

• The famous Black – Scholes formula is presented, which gives a solution to the partial differential equation for the function $v(t, x)$, which, when $x = S (t)$ is substituted, where $S(t)$ is the stock price at the moment time $t$, gives the price of the option in the model with continuous perturbation by Brownian motion.

• The analogue of the Black – Scholes formula for the case of the model with a jump-like perturbation by the Poisson process is suggested. The derivation of this formula is based on the technique of risk-neutral measures and the independence lemma.