All issues

This article reviews both historical achievements and modern results in the field of Markov Decision Process (MDP) and convex optimization. This review is the first attempt to cover the field of reinforcement learning in Russian in the context of convex optimization. The fundamental Bellman equation and the criteria of optimality of policy — strategies based on it, which make decisions based on the known state of the environment at the moment, are considered. The main iterative algorithms of policy optimization based on the solution of the Bellman equations are also considered. An important section of this article was the consideration of an alternative to the $Q$-learning approach — the method of direct maximization of the agent’s average reward for the chosen strategy from interaction with the environment. Thus, the solution of this convex optimization problem can be represented as a linear programming problem. The paper demonstrates how the convex optimization apparatus is used to solve the problem of Reinforcement Learning (RL). In particular, it is shown how the concept of strong duality allows us to naturally modify the formulation of the RL problem, showing the equivalence between maximizing the agent’s reward and finding his optimal strategy. The paper also discusses the complexity of MDP optimization with respect to the number of state–action–reward triples obtained as a result of interaction with the environment. The optimal limits of the MDP solution complexity are presented in the case of an ergodic process with an infinite horizon, as well as in the case of a non-stationary process with a finite horizon, which can be restarted several times in a row or immediately run in parallel in several threads. The review also reviews the latest results on reducing the gap between the lower and upper estimates of the complexity of MDP optimization with average remuneration (Averaged MDP, AMDP). In conclusion, the real-valued parametrization of agent policy and a class of gradient optimization methods through maximizing the $Q$-function of value are considered. In particular, a special class of MDPs with restrictions on the value of policy (Constrained Markov Decision Process, CMDP) is presented, for which a general direct-dual approach to optimization with strong duality is proposed.

The paper constructs and studies a generalized model describing two-component systems of reaction – diffusion type with power nonlinearity, considering the influence of external noise. A methodology has been developed for analyzing the generalized model, which includes linear stability analysis, nonlinear stability analysis, and numerical simulation of the system’s evolution. The linear analysis technique uses basic approaches, in which the characteristic equation is obtained using a linearization matrix. Nonlinear stability analysis realized up to third-order moments inclusively. For this, the functions describing the dynamics of the components are expanded in Taylor series up to third-order terms. Then, using the Novikov theorem, the averaging procedure is carried out. As a result, the obtained equations form an infinite hierarchically subordinate structure, which must be truncated at some point. To achieve this, contributions from terms higher than the third order are neglected in both the equations themselves and during the construction of the moment equations. The resulting equations form a set of linear equations, from which the stability matrix is constructed. This matrix has a rather complex structure, making it solvable only numerically. For the numerical study of the system’s evolution, the method of variable directions was chosen. Due to the presence of a stochastic component in the analyzed system, the method was modified such that random fields with a specified distribution and correlation function, responsible for the noise contribution to the overall nonlinearity, are generated across entire layers. The developed methodology was tested on the reaction – diffusion model proposed by Barrio et al., according to the results of the study, they showed the similarity of the obtained structures with the pigmentation of fish. This paper focuses on the system behavior analysis in the neighborhood of a non-zero stationary point. The dependence of the real part of the eigenvalues on the wavenumber has been examined. In the linear analysis, a range of wavenumber values is identified in which Turing instability occurs. Nonlinear analysis and numerical simulation of the system’s evolution are conducted for model parameters that, in contrast, lie outside the Turing instability region. Nonlinear analysis found noise intensities of additive noise for which, despite the absence of conditions for the emergence of diffusion instability, the system transitions to an unstable state. The results of the numerical simulation of the evolution of the tested model demonstrate the process of forming spatial structures of Turing type under the influence of additive noise.

The article describes the migration process of a certain population, taking into account the spatial heterogeneity of the local capacity of the ecological niche. It is assumed that this spatial heterogeneity is caused by various natural or artificial factors. The mathematical model of the migration process under consideration is a Cauchy problem on a straight line for some quasi-linear partial differential equation of the first order, which is satisfied by the linear population density under consideration. In this paper, a general solution to this Cauchy problem is found for an arbitrary dependence of the local capacity of an ecological niche on the spatial coordinate. This general solution was applied to describe the migration of the population in question in two different cases: in the case of a dependence of the local capacity of the ecological niche on the spatial coordinate in the form of a smooth step and in the case of a hill-like dependence of the local capacity of the ecological niche on the spatial coordinate. In both cases, the solution to the Cauchy problem is expressed in terms of higher transcendental functions. By applying special relations to the model parameters, these higher transcendental functions are reduced to elementary functions, which makes it possible to obtain exact model solutions explicitly expressed in terms of elementary functions. With the help of these precise solutions, an extensive program of computational experiments has been implemented, showing how the initial population density of the Gaussian form is dispersed by the considered two types of spatial heterogeneity of the local capacity of the ecological niche. These computational experiments have shown that when passing through both step-like and hill-like spatial inhomogeneities of the local capacity of an ecological niche with a narrow Gaussian width of its initial density compared to the characteristic spatial scale of these inhomogeneities, the system forgets its initial state. In particular, if we interpret the system under study as a population living in an extended calm rectilinear river along its bed, then it can be argued that under this initial condition, after the current of this river carries the population under consideration through the area of spatial heterogeneity of the local capacity of the ecological niche, the population density becomes a quasi-rectangular function.

Investigation of logical deterministic cellular automata models of population dynamics allows to reveal detailed individual-based mechanisms. The search for such mechanisms is important in connection with ecological problems caused by overexploitation of natural resources, environmental pollution and climate change. Classical models of population dynamics have the phenomenological nature, as they are “black boxes”. Phenomenological models fundamentally complicate research of detailed mechanisms of ecosystem functioning. We have investigated the role of fecundity and duration of resources regeneration in mechanisms of population growth using four models of ecosystem with one species. These models are logical deterministic cellular automata and are based on physical axiomatics of excitable medium with regeneration. We have modeled catastrophic death of population arising from increasing of resources regeneration duration. It has been shown that greater fecundity accelerates population extinction. The investigated mechanisms are important for understanding mechanisms of sustainability of ecosystems and biodiversity conservation. Prospects of the presented modeling approach as a method of transparent multilevel modeling of complex systems are discussed.

In case of vessel wall damage or contact of blood plasma with a foreign surface, the chain of chemical reactions called coagulation cascade is launched that leading to the formation of a fibrin clot. A key enzyme of the coagulation cascade is thrombin, which catalyzes formation of fibrin from fibrinogen. The distribution of thrombin concentration in blood plasma determines spatio-temporal dynamics of clot formation. Contact pathway of blood coagulation triggers the production of thrombin in response to the contact with a negatively charged surface. If the concentration of thrombin generated at this stage is large enough, further production of thrombin takes place due to positive feedback loops of the coagulation cascade. As a result, thrombin propagates in plasma cleaving fibrinogen that results in the clot formation. The concentration profile and the speed of propagation of thrombin are constant and do not depend on the type of the initial activator.

Such behavior of the coagulation system is well described by the traveling wave solutions in a system of “reaction – diffusion” equations on the concentration of blood factors involved in the coagulation cascade. In this study, we carried out detailed analysis of the mathematical model describing the main reaction of the intrinsic pathway of coagulation cascade.We formulate necessary and sufficient conditions of the existence of the traveling wave solutions. For the considered model the existence of such solutions is equivalent to the existence of the wave solutions in the simplified one-equation model describing the dynamics of thrombin concentration derived under the quasi-stationary approximation.

Simplified model also allows us to obtain analytical estimate of the thrombin propagation rate in the considered model. The speed of the traveling wave for one equation is estimated using the narrow reaction zone method and piecewise linear approximation. The resulting formulas give a good approximation of the velocity of propagation of thrombin in the simplified, as well as in the original model.

By numerical simulation methods the interaction processes of oscillating soliton (breather) with a 180-degree Neel domain wall in the framework of a (2 + 1)-dimensional supersymmetric O(3) nonlinear sigma model is studied. The purpose of this paper is to investigate nonlinear evolution and stability of a system of interacting localized dynamic and topological solutions. To construct the interaction models, were used a stationary breather and domain wall solutions, where obtained in the framework of the two-dimensional sine-Gordon equation by adding specially selected perturbations to the A3-field vector in the isotopic space of the Bloch sphere. In the absence of an external magnetic field, nonlinear sigma models have formal Lorentz invariance, which allows constructing, in particular, moving solutions and analyses the experimental data of the nonlinear dynamics of an interacting solitons system. In this paper, based on the obtained moving localized solutions, models for incident and head-on collisions of breathers with a domain wall are constructed, where, depending on the dynamic parameters of the system, are observed the collisions and reflections of solitons from each other, a long-range interactions and also the decay of an oscillating soliton into linear perturbation waves. In contrast to the breather solution that has the dynamics of the internal degree of freedom, the energy integral of a topologically stable soliton in the all experiments the preserved with high accuracy. For each type of interaction, the range of values of the velocity of the colliding dynamic and topological solitons is determined as a function of the rotation frequency of the A3-field vector in the isotopic space. Numerical models are constructed on the basis of methods of the theory of finite difference schemes, using the properties of stereographic projection, taking into account the group-theoretical features of constructions of the O(N) class of nonlinear sigma models of field theory. On the perimeter of the two-dimensional modeling area, specially developed boundary conditions are established that absorb linear perturbation waves radiated by interacting soliton fields. Thus, the simulation of the interaction processes of localized solutions in an infinite two-dimensional phase space is carried out. A software module has been developed that allows to carry out a complex analysis of the evolution of interacting solutions of nonlinear sigma models of field theory, taking into account it’s group properties in a two-dimensional pseudo-Euclidean space. The analysis of isospin dynamics, as well the energy density and energy integral of a system of interacting dynamic and topological solitons is carried out.

The paper contains numerical simulation of nonisothermal nonlinear flow in a porous medium. Twodimensional unsteady problem of heavy oil, water and steam flow is considered. Oil phase consists of two pseudocomponents: light and heavy fractions, which like the water component, can vaporize. Oil exhibits viscoplastic rheology, its filtration does not obey Darcy's classical linear law. Simulation considers not only the dependence of fluids density and viscosity on temperature, but also improvement of oil rheological properties with temperature increasing.

To solve this problem numerically we use streamline method with splitting by physical processes, which consists in separating the convective heat transfer directed along filtration from thermal conductivity and gravitation. The article proposes a new approach to streamline methods application, which allows correctly simulate nonlinear flow problems with temperature-dependent rheology. The core of this algorithm is to consider the integration process as a set of quasi-equilibrium states that are results of solving system on a global grid. Between these states system solved on a streamline grid. Usage of the streamline method allows not only to accelerate calculations, but also to obtain a physically reliable solution, since integration takes place on a grid that coincides with the fluid flow direction.

In addition to the streamline method, the paper presents an algorithm for nonsmooth coefficients accounting, which arise during simulation of viscoplastic oil flow. Applying this algorithm allows keeping sufficiently large time steps and does not change the physical structure of the solution.

Obtained results are compared with known analytical solutions, as well as with the results of commercial package simulation. The analysis of convergence tests on the number of streamlines, as well as on different streamlines grids, justifies the applicability of the proposed algorithm. In addition, the reduction of calculation time in comparison with traditional methods demonstrates practical significance of the approach.

The article is devoted to the numerical study of shock-wave flows in inhomogeneous media–gas mixtures. In this work, a two-speed two-temperature model is used, in which the dispersed component of the mixture has its own speed and temperature. To describe the change in the concentration of the dispersed component, the equation of conservation of “average density” is solved. This study took into account interphase thermal interaction and interphase pulse exchange. The mathematical model allows the carrier component of the mixture to be described as a viscous, compressible and heat-conducting medium. The system of equations was solved using the explicit Mac-Cormack second-order finite-difference method. To obtain a monotone numerical solution, a nonlinear correction scheme was applied to the grid function. In the problem of shock-wave flow, the Dirichlet boundary conditions were specified for the velocity components, and the Neumann boundary conditions were specified for the other unknown functions. In numerical calculations, in order to reveal the dependence of the dynamics of the entire mixture on the properties of the solid component, various parameters of the dispersed phase were considered — the volume content as well as the linear size of the dispersed inclusions. The goal of the research was to determine how the properties of solid inclusions affect the parameters of the dynamics of the carrier medium — gas. The motion of an inhomogeneous medium in a shock duct divided into two parts was studied, the gas pressure in one of the channel compartments is more important than in the other. The article simulated the movement of a direct shock wave from a high-pressure chamber to a low–pressure chamber filled with a dusty medium and the subsequent reflection of a shock wave from a solid surface. An analysis of numerical calculations showed that a decrease in the linear particle size of the gas suspension and an increase in the physical density of the material from which the particles are composed leads to the formation of a more intense reflected shock wave with a higher temperature and gas density, as well as a lower speed of movement of the reflected disturbance reflected wave.

The reconstruction of charged particle trajectories in tracking detectors is a key problem in the analysis of experimental data for high energy and nuclear physics.

The amount of data in modern experiments is so large that classical tracking methods such as Kalman filter can not process them fast enough. To solve this problem, we have developed two neural network algorithms of track recognition, based on deep learning architectures, for local (track by track) and global (all tracks in an event) tracking in the GEM tracker of the BM@N experiment at JINR (Dubna). The advantage of deep neural networks is the ability to detect hidden nonlinear dependencies in data and the capability of parallel execution of underlying linear algebra operations.

In this work we generalize these algorithms to the cylindrical GEM inner tracker of BESIII experiment. The neural network model RDGraphNet for global track finding, based on the reverse directed graph, has been successfully adapted. After training on Monte Carlo data, testing showed encouraging results: recall of 98% and precision of 86% for track finding.

The local neural network model TrackNETv2 was also adapted to BESIII CGEM successfully. Since the tracker has only three detecting layers, an additional neuro-classifier to filter out false tracks have been introduced. Preliminary tests demonstrated the recall value at the first stage of 99%. After applying the neuro-classifier, the precision was 77% with a slight decrease of the recall to 94%. This result can be improved after the further model optimization.

In this paper, the turbulent Taylor – Couette flow is investigated using two-dimensional modeling based on the averaged Navier – Stokes (RANS) equations and a new two-fluid approach to turbulence at Reynolds numbers in the range from 1000 to 8000. The flow due to a rotating internal and stationary external cylinders. The case of ratio of cylinder diameters 1:2 is considered. It is known that the emerging circular flow is characterized by anisotropic turbulence and mathematical modeling of such flows is a difficult task. To describe such flows, either direct modeling methods are used, which require large computational costs, or rather laborious Reynolds stress methods, or linear RANS models with special corrections for rotation, which are able to describe anisotropic turbulence. In order to compare different approaches to turbulence modeling, the paper presents the numerical results of linear RANS models SARC, SST-RC, Reynolds stress method SSG/LRR-RSM-w2012, DNS direct turbulence modeling, as well as a new two-fluid model. It is shown that the recently developed twofluid model adequately describes the considered flow. In addition, the two-fluid model is easy to implement numerically and has good convergence.