All issues

This article reviews both historical achievements and modern results in the field of Markov Decision Process (MDP) and convex optimization. This review is the first attempt to cover the field of reinforcement learning in Russian in the context of convex optimization. The fundamental Bellman equation and the criteria of optimality of policy — strategies based on it, which make decisions based on the known state of the environment at the moment, are considered. The main iterative algorithms of policy optimization based on the solution of the Bellman equations are also considered. An important section of this article was the consideration of an alternative to the $Q$-learning approach — the method of direct maximization of the agent’s average reward for the chosen strategy from interaction with the environment. Thus, the solution of this convex optimization problem can be represented as a linear programming problem. The paper demonstrates how the convex optimization apparatus is used to solve the problem of Reinforcement Learning (RL). In particular, it is shown how the concept of strong duality allows us to naturally modify the formulation of the RL problem, showing the equivalence between maximizing the agent’s reward and finding his optimal strategy. The paper also discusses the complexity of MDP optimization with respect to the number of state–action–reward triples obtained as a result of interaction with the environment. The optimal limits of the MDP solution complexity are presented in the case of an ergodic process with an infinite horizon, as well as in the case of a non-stationary process with a finite horizon, which can be restarted several times in a row or immediately run in parallel in several threads. The review also reviews the latest results on reducing the gap between the lower and upper estimates of the complexity of MDP optimization with average remuneration (Averaged MDP, AMDP). In conclusion, the real-valued parametrization of agent policy and a class of gradient optimization methods through maximizing the $Q$-function of value are considered. In particular, a special class of MDPs with restrictions on the value of policy (Constrained Markov Decision Process, CMDP) is presented, for which a general direct-dual approach to optimization with strong duality is proposed.

The second part presents numerical studies of the parameters of the lower ionosphere at altitudes of 40–90 km when heated by powerful high-frequency radio waves of various frequencies and powers. The problem statement is considered in the first part of the article. The main attention is paid to the interrelation between the energy and kinetic parameters of the disturbed $D$-region of the ionosphere in the processes that determine the absorption and transformation of the radio beam energy flux in space and time. The possibility of a significant difference in the behavior of the parameters of the disturbed region in the daytime and at nighttime, both in magnitude and in space-time distribution, is shown. In the absence of sufficiently reliable values of the rate constants for a number of important kinetic processes, numerical studies were carried out in stages with the gradual addition of individual processes and kinetic blocks corresponding at the same time to a certain physical content. It is shown that the energy thresholds for inelastic collisions of electrons with air molecules are the main ones. This approach made it possible to detect the effect of the emergence of a self-oscillating mode of changing parameters if the main channel for energy losses in inelastic processes is the most energy-intensive process — ionization. This effect may play a role in plasma studies using high-frequency inductive and capacitive discharges. The results of calculations of the ionization and optical parameters of the disturbed $D$-region for daytime conditions are presented. The electron temperature, density, emission coefficients in the visible and infrared ranges of the spectrum are obtained for various values of the power of the radio beam and its frequency in the lower ionosphere. The height-time distribution of the absorbed radiation power is calculated, which is necessary in studies of higher layers of the ionosphere. The influence on the electron temperature and on the general behavior of the parameters of energy losses by electrons on the excitation of vibrational and metastable states of molecules has been studied in detail. It is shown that under nighttime conditions, when the electron concentration begins at altitudes of about 80 km, and the concentration of heavy particles decreases by two orders of magnitude compared to the average $D$-region, large-scale gas-dynamic motion can develop with sufficient radio emission power The algorithm was developed based on the McCormack method and two-dimensional gas-dynamic calculations of the behavior of the parameters of the perturbed region were performed with some simplifications of the kinetics.

Modeling of channel processes in the study of coastal channel deformations requires the calculation of hydrodynamic flow parameters that take into account the existence of secondary transverse currents formed at channel curvature. Three-dimensional modeling of such processes is currently possible only for small model channels; for real river flows, reduced-dimensional models are needed. At the same time, the reduction of the problem from a three-dimensional model of the river flow movement to a two-dimensional flow model in the cross-section assumes that the hydrodynamic flow under consideration is quasi-stationary and the hypotheses about the asymptotic behavior of the flow along the flow coordinate of the cross-section are fulfilled for it. Taking into account these restrictions, a mathematical model of the problem of the a stationary turbulent calm river flow movement in a channel cross-section is formulated. The problem is formulated in a mixed formulation of velocity — “vortex – stream function”. As additional conditions for problem reducing, it is necessary to specify boundary conditions on the flow free surface for the velocity field, determined in the normal and tangential direction to the cross-section axis. It is assumed that the values of these velocities should be determined from the solution of auxiliary problems or obtained from field or experimental measurement data.

To solve the formulated problem, the finite element method in the Petrov – Galerkin formulation is used. Discrete analogue of the problem is obtained and an algorithm for solving it is proposed. Numerical studies have shown that, in general, the results obtained are in good agreement with known experimental data. The authors associate the obtained errors with the need to more accurately determine the circulation velocities field at crosssection of the flow by selecting and calibrating a more appropriate model for calculating turbulent viscosity and boundary conditions at the free boundary of the cross-section.

Considered the application of physics-informed neural networks using multi layer perceptrons to solve Cauchy initial value problems in which the right-hand sides of the equation are continuous monotonically increasing, decreasing or oscillating functions. With the use of the computational experiments the influence of the construction of the approximate neural network solution, neural network structure, optimization algorithm and software implementation means on the learning process and the accuracy of the obtained solution is studied. The analysis of the efficiency of the most frequently used machine learning frameworks in software development with the programming languages Python and C# is carried out. It is shown that the use of C# language allows to reduce the time of neural networks training by 20–40%. The choice of different activation functions affects the learning process and the accuracy of the approximate solution. The most effective functions in the considered problems are sigmoid and hyperbolic tangent. The minimum of the loss function is achieved at the certain number of neurons of the hidden layer of a single-layer neural network for a fixed training time of the neural network model. It’s also mentioned that the complication of the network structure increasing the number of neurons does not improve the training results. At the same time, the size of the grid step between the points of the training sample, providing a minimum of the loss function, is almost the same for the considered Cauchy problems. Training single-layer neural networks, the Adam method and its modifications are the most effective to solve the optimization problems. Additionally, the application of twoand three-layer neural networks is considered. It is shown that in these cases it is reasonable to use the LBFGS algorithm, which, in comparison with the Adam method, in some cases requires much shorter training time achieving the same solution accuracy. The specificity of neural network training for Cauchy problems in which the solution is an oscillating function with monotonically decreasing amplitude is also investigated. For these problems, it is necessary to construct a neural network solution with variable weight coefficient rather than with constant one, which improves the solution in the grid cells located near by the end point of the solution interval.

For cognitive radio systems, it is important to use efficient algorithms that search for free channels that can be provided to secondary users. Therefore, this paper is devoted to improving the accuracy of prediction frequency resource occupancy of a cellular communication system using spatiotemporal radio environment maps. The formation of a radio environment map is implemented for the fourthgeneration cellular communication system Long-Term Evolution. Taking this into account, a model structure has been developed that includes data generation and allows training and testing of an artificial neural network to predict the occupancy of frequency resources presented as the contents of radio environment map cells. A method for assessing prediction accuracy is described. The simulation model of the cellular communication system is implemented in the MatLab. The developed frequency resource occupancy prediction model is implemented in the Python. The complete file structure of the model is presented. The experiments were performed using artificial neural networks based on the Long Short-Term Memory and Kolmogorov – Arnold neural network architectures, taking into account its modification. It was found that with an equal number of parameters, the Kolmogorov –Arnold neural network learns faster for a given task. The obtained research results indicate an increase in the accuracy of prediction the occupancy of the frequency resource of the cellular communication system when using the Kolmogorov – Arnold neural network.

This paper considers the inverse problem of seismic exploration — determining the structure of the media based on the recorded wave response from it. Large cracks are considered as target objects, whose size and position are to be determined.

he direct problem is solved using the grid-characteristic method. The method allows using physically based algorithms for calculating outer boundaries of the region and contact boundaries inside the region. The crack is assumed to be thin, a special condition on the crack borders is used to describe the crack.

The inverse problem is solved using convolutional neural networks. The input data of the neural network are seismograms interpreted as images. The output data are masks describing the medium on a structured grid. Each element of such a grid belongs to one of two classes — either an element of a continuous geological massif, or an element through which a crack passes. This approach allows us to consider a medium with an unknown number of cracks.

The neural network is trained using only samples with one crack. The final testing of the trained network is performed using additional samples with several cracks. These samples are not involved in the training process. The purpose of testing under such conditions is to verify that the trained network has sufficient generality, recognizes signs of a crack in the signal, and does not suffer from overtraining on samples with a single crack in the media.

The paper shows that a convolutional network trained on samples with a single crack can be used to process data with multiple cracks. The networks detects fairly small cracks at great depths if they are sufficiently spatially separated from each other. In this case their wave responses are clearly distinguishable on the seismogram and can be interpreted by the neural network. If the cracks are close to each other, artifacts and interpretation errors may occur. This is due to the fact that on the seismogram the wave responses of close cracks merge. This cause the network to interpret several cracks located nearby as one. It should be noted that a similar error would most likely be made by a human during manual interpretation of the data. The paper provides examples of some such artifacts, distortions and recognition errors.

This paper examines the impact of the spatial resolution of a discretized (lattice) representation of the environment on the efficiency and correctness of optimal pathfinding in complex environments. Scenarios are considered that may include bottlenecks, non-uniform obstacle distributions, and areas of increased safety requirements in the immediate vicinity of obstacles. Despite the widespread use of lattice representations of the environment in robotics due to their compatibility with sensor data and support for classical trajectory planning algorithms, the resolution of these lattices has a significant impact on both goal reachability and optimal path performance. An algorithm is proposed that combines environmental connectivity analysis, trajectory optimization, and geometric safety refinement. In the first stage, the Leath algorithm is used to estimate the reachability of the target point by identifying a connected component containing the starting position. Upon confirmation of the target point’s reachability, the A* algorithm is applied to the nodes of this component in the second stage to construct a path that simultaneously minimizes both the path length and the risk of collision. In the third stage, a refined obstacle distance estimate is performed for nodes located in safety zones using a combination of the Gilbert – Johnson –Keerthi (GJK) and expanding polyhedron (EPA) algorithms. Experimental analysis revealed a nonlinear relationship between the probability of the existence and effectiveness of an optimal path and the lattice parameters. Specifically, reducing the spatial resolution of the lattice increases the likelihood of connectivity loss and target unreachability, while increasing its spatial resolution increases computational complexity without a proportional improvement in the optimal path’s performance.

The SAS program is the primary data processing tool for the YuMO small-angle neutron scattering spectrometer. The paper presents a retrospective analysis of its two-decade evolution, from a Fortran prototype to a modern software system. The analysis focuses on the architectural decisions that have ensured the program’s long-term viability and its ability to adapt to instrument upgrades.

The core solution was a modular architecture that abstracts the detector system. This enabled the seamless integration of data from two scattering detectors and, later, from a position-sensitive detector. A strict processing pipeline and a unified internal data representation formed the basis for physically grounded algorithms, including weighted merging of spectra, resolution-aware smoothing, and built-in statistical quality control. The program’s interfaces—a command line for batch processing and a graphical user interface for interactive work—are built upon a single computational core, ensuring result consistency and flexibility in use.

Long-term operation has confirmed that the underlying architectural principles naturally align with the key characteristics of international software quality standards, particularly those critical for long-term sustainability. Therefore, the development and evolution of SAS demonstrates a universal set of architectural principles that can serve as a foundation for building sustainable scientific software in related fields of experimental physics.

We consider two possible computational methods of the interpretation of experimental data obtained by means of the magnetic force microscopy. These methods of macrospin distribution simulation and reconstruction can be used for research of magnetization reversal processes of nanodots in ordered 2D arrays of nanodots. New approaches to the development of high-performance superscale algorithms for parallel executing on a supercomputer clusters for solving direct and inverse task of the modeling of magnetic states, types of ordering, reversal processes of nanosystems with a collective behavior are proposed. The simulation results are consistent with experimental results.

Investigation of influence of non-equilibrium initial states and structural disorder on characteristics of anomalous slow non-equilibrium critical behavior of three-dimensional Ising model is carried out. The unique ageing properties and violations of the equilibrium fluctuation-dissipation theorem are observed for considered pure and disordered systems which were prepared in high-temperature initial state and then quenched in their critical points. The heat-bath algorithm description of ageing properties in non-equilibrium critical behavior of three-dimensional Ising model with spin concentrations p = 1.0, p = 0.8, and 0.6 is realized. On the base of analysis of such two-time quantities as autocorrelation function and dynamical susceptibility were demonstrated the ageing effects and were calculated asymptotic values of universal fluctuation-dissipation ratio in these systems. It was shown that the presence of defects leads to aging gain.