All issues

This article reviews both historical achievements and modern results in the field of Markov Decision Process (MDP) and convex optimization. This review is the first attempt to cover the field of reinforcement learning in Russian in the context of convex optimization. The fundamental Bellman equation and the criteria of optimality of policy — strategies based on it, which make decisions based on the known state of the environment at the moment, are considered. The main iterative algorithms of policy optimization based on the solution of the Bellman equations are also considered. An important section of this article was the consideration of an alternative to the $Q$-learning approach — the method of direct maximization of the agent’s average reward for the chosen strategy from interaction with the environment. Thus, the solution of this convex optimization problem can be represented as a linear programming problem. The paper demonstrates how the convex optimization apparatus is used to solve the problem of Reinforcement Learning (RL). In particular, it is shown how the concept of strong duality allows us to naturally modify the formulation of the RL problem, showing the equivalence between maximizing the agent’s reward and finding his optimal strategy. The paper also discusses the complexity of MDP optimization with respect to the number of state–action–reward triples obtained as a result of interaction with the environment. The optimal limits of the MDP solution complexity are presented in the case of an ergodic process with an infinite horizon, as well as in the case of a non-stationary process with a finite horizon, which can be restarted several times in a row or immediately run in parallel in several threads. The review also reviews the latest results on reducing the gap between the lower and upper estimates of the complexity of MDP optimization with average remuneration (Averaged MDP, AMDP). In conclusion, the real-valued parametrization of agent policy and a class of gradient optimization methods through maximizing the $Q$-function of value are considered. In particular, a special class of MDPs with restrictions on the value of policy (Constrained Markov Decision Process, CMDP) is presented, for which a general direct-dual approach to optimization with strong duality is proposed.

The approach to optimization of integral estimation of bio-systems state is presented. The approach is included the procedures of decreasing of variability of integral estimation based on statistical modeling of experimental data set and optimization the quantity of a state characteristics on a base of their relative contribution to the integral estimation using parallel calculation.

We consider two possible computational methods of the interpretation of experimental data obtained by means of the magnetic force microscopy. These methods of macrospin distribution simulation and reconstruction can be used for research of magnetization reversal processes of nanodots in ordered 2D arrays of nanodots. New approaches to the development of high-performance superscale algorithms for parallel executing on a supercomputer clusters for solving direct and inverse task of the modeling of magnetic states, types of ordering, reversal processes of nanosystems with a collective behavior are proposed. The simulation results are consistent with experimental results.

The work continues research on the ability of a person to improve the productivity of information processing, using parallel work or improving the performance of analyzers. A person receives a series of tasks, the solution of which requires the processing of a certain amount of information. The time and the validity of the decision are recorded. The dependence of the average solution time on the amount of information in the problem is determined by correctly solved problems. In accordance with the proposed method, the problems contain calculations of expressions in two algebras, one of which is associative and the other is nonassociative. To facilitate the work of the subjects in the experiment were used figurative graphic images of elements of algebra. Non-associative calculations were implemented in the form of the game “rock-paper-scissors”. It was necessary to determine the winning symbol in the long line of these figures, considering that they appear sequentially from left to right and play with the previous winner symbol. Associative calculations were based on the recognition of drawings from a finite set of simple images. It was necessary to determine which figure from this set in the line is not enough, or to state that all the pictures are present. In each problem there was no more than one picture. Computation in associative algebra allows the parallel counting, and in the absence of associativity only sequential computations are possible. Therefore, the analysis of the time for solving a series of problems reveals a consistent uniform, sequential accelerated and parallel computing strategy. In the experiments it was found that all subjects used a uniform sequential strategy to solve non-associative problems. For the associative task, all subjects used parallel computing, and some have used parallel computing acceleration of the growth of complexity of the task. A small part of the subjects with a high complexity, judging by the evolution of the solution time, supplemented the parallel account with a sequential stage of calculations (possibly to control the solution). We develop a special method for assessing the rate of processing of input information by a person. It allowed us to estimate the level of parallelism of the calculation in the associative task. Parallelism of level from two to three was registered. The characteristic speed of information processing in the sequential case (about one and a half characters per second) is twice less than the typical speed of human image recognition. Apparently the difference in processing time actually spent on the calculation process. For an associative problem in the case of a minimum amount of information, the solution time is near to the non-associativity case or less than twice. This is probably due to the fact that for a small number of characters recognition almost exhausts the calculations for the used non-associative problem.

The reconstruction of charged particle trajectories in tracking detectors is a key problem in the analysis of experimental data for high energy and nuclear physics.

The amount of data in modern experiments is so large that classical tracking methods such as Kalman filter can not process them fast enough. To solve this problem, we have developed two neural network algorithms of track recognition, based on deep learning architectures, for local (track by track) and global (all tracks in an event) tracking in the GEM tracker of the BM@N experiment at JINR (Dubna). The advantage of deep neural networks is the ability to detect hidden nonlinear dependencies in data and the capability of parallel execution of underlying linear algebra operations.

In this work we generalize these algorithms to the cylindrical GEM inner tracker of BESIII experiment. The neural network model RDGraphNet for global track finding, based on the reverse directed graph, has been successfully adapted. After training on Monte Carlo data, testing showed encouraging results: recall of 98% and precision of 86% for track finding.

The local neural network model TrackNETv2 was also adapted to BESIII CGEM successfully. Since the tracker has only three detecting layers, an additional neuro-classifier to filter out false tracks have been introduced. Preliminary tests demonstrated the recall value at the first stage of 99%. After applying the neuro-classifier, the precision was 77% with a slight decrease of the recall to 94%. This result can be improved after the further model optimization.

We suggested some approaches for solving image processing tasks in the decision support system (DSS) of the Center for Automated Recording of Administrative Offenses of the Road Traffic (CARAO). The main task of this system is to assist the operator in obtaining accurate information about the vehicle registration plate and the vehicle brand/model based on images obtained from the photo and video recording systems. We suggested the approach for vehicle registration plate recognition and brand/model classification on the images based on modern neural network models. LPRNet neural network model supplemented by Spatial Transformer Layer was used to recognize the vehicle registration plate. The ResNeXt-101-32x8d neural network model was used to classify for vehicle brand/model. We suggested the approach to construct the training set for the neural network of vehicle registration plate recognition. The approach is based on computer vision methods and machine learning algorithms. The SIFT algorithm was used to detect and describe local features on images with the vehicle registration plate. DBSCAN clustering was used to detect and delete outliers in such local features. The accuracy of vehicle registration plate recognition was 96% on the testing set. We suggested the approach to improve the efficiency of using the ResNeXt-101-32x8d model at additional training and classification stages. The approach is based on the new architecture of convolutional neural networks with “freezing” weight coefficients of convolutional layers, an additional convolutional layer for parallelizing the classification process, and a set of binary classifiers at the output. This approach significantly reduced the time of additional training of neural network when new vehicle brand/model classification was needed. The final accuracy of vehicle brand/model classification was 99% on the testing set. The proposed approaches were tested and implemented in the DSS of the CARAO of the Republic of Tatarstan.

There are conditions of uneven combustion for tubular powder elements of large elongation used in artillery propelling charges. Here it is necessary to consider in parallel the processes of combustion and movement of powder gases inside and outside the channels of the powder tubes. Without this, it is impossible to adequately formulate and solve the problems of ignition, erosive combustion and stress-strain state of tubular powder elements in the shot process. The paper presents a physical and mathematical formulation of the main problem of the internal ballistics of an artillery shot for a charge consisting of a set of powder tubes. Combustion and movement of a bundle of powder tubes along the barrel channel is modeled by an equivalent tubular charge of all-round combustion. The end and cross-sectional areas of the channel of such a charge (equivalent tube) are equal to the sum of the areas of the ends and cross-sections of the channels of the powder tubes, respectively. The combustion surface of the channel is equal to the sum of the inner surfaces of the tubes in the bundle. The outer combustion surface of the equivalent tube is equal to the sum of the outer surfaces of the tubes in the bundle. It is assumed that the equivalent tube moves along the axis of the bore. The speed of motion of an equivalent tubular charge and its current position are determined from Newton’s second law. To calculate the flow parameters, we used two-dimensional axisymmetric equations of gas dynamics, for the solution of which an axisymmetric orthogonalized difference mesh is constructed, which adapts to the flow conditions. When the tube moves and burns, the difference grid is rearranged taking into account the changing regions of integration. The control volume method is used for the numerical solution of the system of gas-dynamic equations. The gas parameters at the boundaries of the control volumes are determined using a self-similar solution to the Godunov problem of decay for an arbitrary discontinuity. The developed technique was used to calculate the internal ballistics parameters of an artillery shot. This approach is considered for the first time and allows a new approach to the design of tubular artillery charges, since it allows obtaining the necessary information in the form of fields of velocity and pressure of powder gases for calculating the process of gradual ignition, unsteady erosive combustion, stress-strain state and strength of powder elements during the shot. The time dependences of the parameters of the internal ballistics process and the distribution of the main parameters of the flow of combustion products at different times are presented.

The paper is a study of the mathematical model and kinematics of a parallel spherical manipulator. This type of manipulator was proposed back in the 80s of the last century and has since found application in exoskeletons and rehabilitation robots due to its structure, which allows imitating natural joint movements of the human body.

The Parallel Spherical Manipulator is a robot with three legs and two platforms, a base platform and a mobile platform. Its legs consist of two support links that are arc-shaped. Mathematically, the manipulator can be described using two virtual pyramids that are placed on top of each other.

The paper considers two types of manipulator configurations: classical and asymmetric, and solves basic kinematic problems for each. The study shows that the asymmetric design of the manipulator has the maximum workspace, especially when the motors are mounted at the joints of the manipulator’s links inside legs.

To optimize the parameters of the parallel spherical manipulator, we introduced a metric of usable workspace volume. This metric represents the volume of the sector of the sphere in which the robot does not experience internal collisions or singular states. There are three types of singular states possible within a parallel spherical manipulator — serial, parallel, and mixed singularity. We used all three types of singularities to calculate the useful volume. In our research work, we solved the problem related to maximizing the usable volume of the workspace.

Through our research work, we found that the asymmetric configuration of the spherical manipulator maximizes the workspace when the motors are located at the articulation point of the robot leg support arms. At the same time, the parameter $\beta_1$ must be zero degrees to maximize the workspace. This allowed us to create a prototype robot in which we eliminated the use of lower links in legs in favor of a radiused rail along which the motors run. This allowed us to reduce the linear dimensions of the robot itself and gain on the stiffness of the structure.

The results obtained can be used to optimize the parameters of the parallel spherical manipulator in various industrial and scientific applications, as well as for further research of other types of parallel robots and manipulators.

This paper discusses the modeling of the deformation of polymer nanocomposites containing "hard" and "soft" inclusions, using cellular automata and parallel computing. The paper describes an algorithm based on the model, a comparison with experimental data is shown, software for the numerical experiment is described.

The paper proposes a new mathematical model to study the interaction dynamics of the longitudinal wall of a narrow channel with its end seal. The end seal was considered as the edge wall on a spring, i.e. spring-mass system. These walls interaction occurs via a viscous liquid filling the narrow channel; thus required the formulation and solution of the hydroelasticity problem. However, this problem has not been previously studied. The problem consists of the Navier–Stokes equations, the continuity equation, the edge wall dynamics equation, and the corresponding boundary conditions. Two cases of fluid motion in a narrow channel with parallel walls were studied. In the first case, we assumed the liquid motion as the creeping one, and in the second case as the laminar, taking into account the motion inertia. The hydroelasticty problem solution made it possible to determine the distribution laws of velocities and pressure in the liquid layer, as well as the motion law of the edge wall. It is shown that during creeping flow, the liquid physical properties and the channel geometric dimensions completely determine the damping in the considered oscillatory system. Both the end wall velocity and the longitudinal wall velocity affect the damping properties of the liquid layer. If the fluid motion inertia forces were taken into account, their influence on the edge wall vibrations was revealed, which manifested itself in the form of two added masses in the equation of its motion. The added masses and damping coefficients of the liquid layer due to the joint consideration of the liquid layer inertia and its viscosity were determined. The frequency and phase responses of the edge wall were constructed for the regime of steady-state harmonic oscillations. The simulation showed that taking into account the fluid layer inertia and its damping properties leads to a shift in the resonant frequencies to the low-frequency region and an increase in the oscillation amplitudes of the edge wall.