All issues

This article reviews both historical achievements and modern results in the field of Markov Decision Process (MDP) and convex optimization. This review is the first attempt to cover the field of reinforcement learning in Russian in the context of convex optimization. The fundamental Bellman equation and the criteria of optimality of policy — strategies based on it, which make decisions based on the known state of the environment at the moment, are considered. The main iterative algorithms of policy optimization based on the solution of the Bellman equations are also considered. An important section of this article was the consideration of an alternative to the $Q$-learning approach — the method of direct maximization of the agent’s average reward for the chosen strategy from interaction with the environment. Thus, the solution of this convex optimization problem can be represented as a linear programming problem. The paper demonstrates how the convex optimization apparatus is used to solve the problem of Reinforcement Learning (RL). In particular, it is shown how the concept of strong duality allows us to naturally modify the formulation of the RL problem, showing the equivalence between maximizing the agent’s reward and finding his optimal strategy. The paper also discusses the complexity of MDP optimization with respect to the number of state–action–reward triples obtained as a result of interaction with the environment. The optimal limits of the MDP solution complexity are presented in the case of an ergodic process with an infinite horizon, as well as in the case of a non-stationary process with a finite horizon, which can be restarted several times in a row or immediately run in parallel in several threads. The review also reviews the latest results on reducing the gap between the lower and upper estimates of the complexity of MDP optimization with average remuneration (Averaged MDP, AMDP). In conclusion, the real-valued parametrization of agent policy and a class of gradient optimization methods through maximizing the $Q$-function of value are considered. In particular, a special class of MDPs with restrictions on the value of policy (Constrained Markov Decision Process, CMDP) is presented, for which a general direct-dual approach to optimization with strong duality is proposed.

The course and outcome of the battle is largely dependent on the morale of the troops, characterized by the percentage of loss in killed and wounded, in which the troops still continue to fight. Every fight is a psychological act of ending his rejection of one of the parties. Typically, models of battle psychological factor taken into account in the decision of Lanchester equations (the condition of equality of forces, when the number of one of the parties becomes zero). It is emphasized that the model Lanchester type satisfactorily describe the dynamics of the battle only in the initial stages. To resolve this contradiction is proposed to use a modification of Lanchester's equations, taking into account the fact that at any moment of the battle on the enemy firing not affected and did not abandon the battle fighters. The obtained differential equations are solved by numerical method and allow the dynamics to take into account the influence of psychological factor and evaluate the completion time of the conflict. Computational experiments confirm the known military theory is the fact that the fight usually ends in refusal of soldiers of one of the parties from its continuation (avoidance of combat in various forms). Along with models of temporal and spatial dynamics proposed to use a modification of the technology features of the conflict of S. Skaperdas, based on the principles of combat. To estimate the probability of victory of one side in the battle takes into account the interest of the maturing sides of the bloody casualties and increased military superiority.

The paper develops a new mathematical method of the joint signal and noise calculation at the Rice statistical distribution based on combing the maximum likelihood method and the method of moments. The calculation of the sough-for values of signal and noise is implemented by processing the sampled measurements of the analyzed Rician signal’s amplitude. The explicit equations’ system has been obtained for required signal and noise parameters and the results of its numerical solution are provided confirming the efficiency of the proposed technique. It has been shown that solving the two-parameter task by means of the proposed technique does not lead to the increase of the volume of demanded calculative resources if compared with solving the task in one-parameter approximation. An analytical solution of the task has been obtained for the particular case of small value of the signal-to-noise ratio. The paper presents the investigation of the dependence of the sought for parameters estimation accuracy and dispersion on the quantity of measurements in experimental sample. According to the results of numerical experiments, the dispersion values of the estimated sought-for signal and noise parameters calculated by means of the proposed technique change in inverse proportion to the quantity of measurements in a sample. There has been implemented a comparison of the accuracy of the soughtfor Rician parameters’ estimation by means of the proposed technique and by earlier developed version of the method of moments. The problem having been considered in the paper is meaningful for the purposes of Rician data processing, in particular, at the systems of magnetic-resonance visualization, in devices of ultrasonic visualization, at optical signals’ analysis in range-measuring systems, at radar signals’ analysis, as well as at solving many other scientific and applied tasks that are adequately described by the Rice statistical model.

The article deals with the process of creating an effective algorithm for determining the amount of emitted quanta from an X-ray tube in computer tomography (CT) studies. An analysis of domestic and foreign literature showed that most of the work in the field of radiometry and radiography takes the tabulated values of X-ray absorption coefficients into account, while individual dose factors are not taken into account at all since many studies are lacking the Dose Report. Instead, an average value is used to simplify the calculation of statistics. In this regard, it was decided to develop a method to detect the amount of ionizing quanta by analyzing the noise of CT data. As the basis of the algorithm, we used Poisson and Gauss distribution mathematical model of owns’ design of logarithmic value. The resulting mathematical model was tested on the CT data of a calibration phantom consisting of three plastic cylinders filled with water, the X-ray absorption coefficient of which is known from the table values. The data were obtained from several CT devices from different manufacturers (Siemens, Toshiba, GE, Phillips). The developed algorithm made it possible to calculate the number of emitted X-ray quanta per unit time. These data, taking into account the noise level and the radiuses of the cylinders, were converted to X-ray absorption values, after which a comparison was made with tabulated values. As a result of this operation, the algorithm used with CT data of various configurations, experimental data were obtained, consistent with the theoretical part and the mathematical model. The results showed good accuracy of the algorithm and mathematical apparatus, which shows reliability of the obtained data. This mathematical model is already used in the noise reduction program of the CT of own design, where it participates as a method of creating a dynamic threshold of noise reduction. At the moment, the algorithm is being processed to work with real data from computer tomography of patients.

The traditional approach to computing optimal game strategies of firms on oligopolistic markets and of indicators of such markets consists in studying linear dynamical games with quadratic criteria and solving generalized matrix Riccati equations.

The other approach proposed by the author is based on methods of operational calculus (in particular, Z-transform). This approach makes it possible to achieve economic meaningful decisions under wider field of parameter values. It characterizes by simplicity of computations and by necessary for economic analysis visibility. One of its advantages is that in many cases important for economic practice, it, in contrast to the traditional approach, provides the ability to make calculations using widespread spreadsheets, which allows to study the prospects for the development of oligopolistic markets to a wide range of professionals and consumers.

The article deals with the practical aspects of determining the optimal Nash–Cournot strategies of participants in oligopolistic markets on the basis of operational calculus, in particular the technique of computing the optimal Nash–Cournot strategies in Excel. As an illustration of the opportinities of the proposed methods of calculation, examples close to the practical problems of forecasting indicators of the markets of high-tech products are studied.

The results of calculations obtained by the author for numerous examples and real economic systems, both using the obtained relations on the basis of spreadsheets and using extended Riccati equations, are very close. In most of the considered practical problems, the deviation of the indicators calculated in accordance with the two approaches, as a rule, does not exceed 1.5–2%. The highest value of relative deviations (up to 3–5%) is observed at the beginning of the forecasting period. In typical cases, the period of relatively noticeable deviations is 3–5 moments of time. After the transition period, there is almost complete agreement of the values of the required indicators using both approaches.

In this work, the problem of constructing an electronic analogue of heterogeneous DNA is solved with the help of the methods of mathematical modeling. Electronic analogs of that type, along with other physical models of living systems, are widely used as a tool for studying the dynamic and functional properties of these systems. The solution to the problem is based on an algorithm previously developed for homogeneous (synthetic) DNA and modified in such a way that it can be used for the case of inhomogeneous (native) DNA. The algorithm includes the following steps: selection of a model that simulates the internal mobility of DNA; construction of a transformation that allows you to move from the DNA model to its electronic analogue; search for conditions that provide an analogy of DNA equations and electronic analogue equations; calculation of the parameters of the equivalent electrical circuit. To describe inhomogeneous DNA, the model was chosen that is a system of discrete nonlinear differential equations simulating the angular deviations of nitrogenous bases, and Hamiltonian corresponding to these equations. The values of the coefficients in the model equations are completely determined by the dynamic parameters of the DNA molecule, including the moments of inertia of nitrous bases, the rigidity of the sugar-phosphate chain, and the constants characterizing the interactions between complementary bases in pairs. The inhomogeneous Josephson line was used as a basis for constructing an electronic model, the equivalent circuit of which contains four types of cells: A-, T-, G-, and C-cells. Each cell, in turn, consists of three elements: capacitance, inductance, and Josephson junction. It is important that the A-, T-, G- and C-cells of the Josephson line are arranged in a specific order, which is similar to the order of the nitrogenous bases (A, T, G and C) in the DNA sequence. The transition from DNA to an electronic analog was carried out with the help of the A-transformation which made it possible to calculate the values of the capacitance, inductance, and Josephson junction in the A-cells. The parameter values for the T-, G-, and C-cells of the equivalent electrical circuit were obtained from the conditions imposed on the coefficients of the model equations and providing an analogy between DNA and the electronic model.

Instrument of numerical-analytical modeling of characteristics of propagation of electromagnetic waves in chaotic space plasma with taking into account effects of gravitation is developed for interpretation of data of measurements of astrophysical precision instruments of new education. The task of propagation of waves in curved (Riemann’s) space is solved in Euclid’s space by introducing of the effective index of refraction of vacuum. The gravitational potential can be calculated for various model of distribution of mass of astrophysical objects and at solution of Poisson’s equation. As a result the effective index of refraction of vacuum can be evaluated. Approximate model of the effective index of refraction is suggested with condition that various objects additively contribute in total gravitational field. Calculation of the characteristics of electromagnetic waves in the gravitational field of astrophysical objects is performed by the approximation of geometrical optics with condition that spatial scales of index of refraction a lot more wavelength. Light differential equations in Euler’s form are formed the basis of numerical-analytical instrument of modeling of trajectory characteristic of waves. Chaotic inhomogeneities of space plasma are introduced by model of spatial correlation function of index of refraction. Calculations of refraction scattering of waves are performed by the approximation of geometrical optics. Integral equations for statistic moments of lateral deviations of beams in picture plane of observer are obtained. Integrals for moments are reduced to system of ordinary differential equations the firsts order with using analytical transformations for cooperative numerical calculation of arrange and meansquare deviations of light. Results of numerical-analytical modeling of trajectory picture of propagation of electromagnetic waves in interstellar space with taking into account impact of gravitational fields of space objects and refractive scattering of waves on inhomogeneities of index of refraction of surrounding plasma are shown. Based on the results of modeling quantitative estimation of conditions of stochastic blurring of the effect of gravitational lensing of electromagnetic waves at various frequency ranges is performed. It’s shown that operating frequencies of meter range of wavelengths represent conditional low-frequency limit for observational of the effect of gravitational lensing in stochastic space plasma. The offered instrument of numerical-analytical modeling can be used for analyze of structure of electromagnetic radiation of quasar propagating through group of galactic.

This article discusses the influence of the shadow, informal and household sectors on the dynamics of a stochastic model with heterogeneous (heterogeneous) agents. The study uses the integration of the general equilibrium approach to explain the behavior of demand, supply and prices in an economy with several interacting markets, and a multi-agent approach. The analyzed model describes an economy with aggregated uncertainty and with an infinite number of heterogeneous agents (households). The source of heterogeneity is the idiosyncratic income shocks of agents in the legal and shadow sectors of the economy. In the analysis, an algorithm is used to approximate the dynamics of the distribution function of the capital stocks of individual agents — the dynamics of its first and second moments. The synthesis of the agent approach and the general equilibrium approach is carried out using computer implementation of the recursive feedback between microagents and macroenvironment. The behavior of the impulse response functions of the main variables of the model confirms the positive influence of the shadow economy (below a certain limit) on minimizing the rate of decline in economic indicators during recessions, especially for developing economies. The scientific novelty of the study is the combination of a multi-agent approach and a general equilibrium approach for modeling macroeconomic processes at the regional and national levels. Further research prospects may be associated with the use of more detailed general equilibrium models, which allow, in particular, to describe the behavior of heterogeneous groups of agents in the entrepreneurial sector of the economy.

Numerical simulation of moving sportsman external flow is presented. The unique method is developed for obtaining integral aerodynamic characteristics, which were the function of the flow regime (i.e. angle of attack, flow speed) and body position. Individual anthropometric characteristics and moving boundaries of sportsman (or sports equipment) during the race are taken into consideration.

Numerical simulation is realized using FlowVision CFD. The software is based on the finite volume method, high-performance numerical methods and reliable mathematical models of physical processes. A Cartesian computational grid is used by FlowVision, the grid generation is a completely automated process. Local grid adaptation is used for solving high-pressure gradient and object complex shape. Flow simulation process performed by solutions systems of equations describing movement of fluid and/or gas in the computational domain, including: mass, moment and energy conservation equations; state equations; turbulence model equations. FlowVision permits flow simulation near moving bodies by means of computational domain transformation according to the athlete shape changes in the motion. Ski jumper aerodynamic characteristics are studied during all phases: take-off performance in motion, in-run and flight. Projected investigation defined simulation method, which includes: inverted statement of sportsman external flow development (velocity of the motion is equal to air flow velocity, object is immobile); changes boundary of the body technology defining; multiple calculations with the national team member data projecting. The research results are identification of the main factors affected to jumping performance: aerodynamic forces, rotating moments etc. Developed method was tested with active sportsmen. Ski jumpers used this method during preparations for Sochi Olympic Games 2014. A comparison of the predicted characteristics and experimental data shows a good agreement. Method versatility is underlined by performing swimmer and skater flow simulation. Designed technology is applicable for sorts of natural and technical objects.

Based on the Holstein Hamiltonian, the dynamics of the charge introduced into the molecular chain of sites was modeled at different temperatures. In the calculation, the temperature of the chain is set by the initial data ¡ª random Gaussian distributions of velocities and site displacements. Various options for the initial charge density distribution are considered. Long-term calculations show that the system moves to fluctuations near a new equilibrium state. For the same initial velocities and displacements, the average kinetic energy, and, accordingly, the temperature of the T chain, varies depending on the initial distribution of the charge density: it decreases when a polaron is introduced into the chain, or increases if at the initial moment the electronic part of the energy is maximum. A comparison is made with the results obtained previously in the model with a Langevin thermostat. In both cases, the existence of a polaron is determined by the thermal energy of the entire chain.

According to the simulation results, the transition from the polaron mode to the delocalized state occurs in the same range of thermal energy values of a chain of $N$ sites ~ $NT$ for both thermostat options, with an additional adjustment: for the Hamiltonian system the temperature does not correspond to the initially set one, but is determined after long-term calculations from the average kinetic energy of the chain.

In the polaron region, the use of different methods for simulating temperature leads to a number of significant differences in the dynamics of the system. In the region of the delocalized state of charge, for high temperatures, the results averaged over a set of trajectories in a system with a random force and the results averaged over time for a Hamiltonian system are close, which does not contradict the ergodic hypothesis. From a practical point of view, for large temperatures T ≈ 300 K, when simulating charge transfer in homogeneous chains, any of these options for setting the thermostat can be used.