All issues

This article reviews both historical achievements and modern results in the field of Markov Decision Process (MDP) and convex optimization. This review is the first attempt to cover the field of reinforcement learning in Russian in the context of convex optimization. The fundamental Bellman equation and the criteria of optimality of policy — strategies based on it, which make decisions based on the known state of the environment at the moment, are considered. The main iterative algorithms of policy optimization based on the solution of the Bellman equations are also considered. An important section of this article was the consideration of an alternative to the $Q$-learning approach — the method of direct maximization of the agent’s average reward for the chosen strategy from interaction with the environment. Thus, the solution of this convex optimization problem can be represented as a linear programming problem. The paper demonstrates how the convex optimization apparatus is used to solve the problem of Reinforcement Learning (RL). In particular, it is shown how the concept of strong duality allows us to naturally modify the formulation of the RL problem, showing the equivalence between maximizing the agent’s reward and finding his optimal strategy. The paper also discusses the complexity of MDP optimization with respect to the number of state–action–reward triples obtained as a result of interaction with the environment. The optimal limits of the MDP solution complexity are presented in the case of an ergodic process with an infinite horizon, as well as in the case of a non-stationary process with a finite horizon, which can be restarted several times in a row or immediately run in parallel in several threads. The review also reviews the latest results on reducing the gap between the lower and upper estimates of the complexity of MDP optimization with average remuneration (Averaged MDP, AMDP). In conclusion, the real-valued parametrization of agent policy and a class of gradient optimization methods through maximizing the $Q$-function of value are considered. In particular, a special class of MDPs with restrictions on the value of policy (Constrained Markov Decision Process, CMDP) is presented, for which a general direct-dual approach to optimization with strong duality is proposed.

This paper provides an overview of studies on nonlinear supratransmission and related phenomena. This effect consists in the transfer of energy at frequencies not supported by the systems under consideration. The supratransmission does not depend on the integrability of the system, it is resistant to damping and various classes of boundary conditions. In addition, a nonlinear discrete medium, under certain general conditions imposed on the structure, can create instability due to external periodic influence. This instability is the generative process underlying the nonlinear supratransmission. This is possible when the system supports nonlinear modes of various nature, in particular, discrete breathers. Then the energy penetrates into the system as soon as the amplitude of the external harmonic excitation exceeds the maximum amplitude of the static breather of the same frequency.

The effect of nonlinear supratransmission is an important property of many discrete structures. A necessary condition for its existence is the discreteness and nonlinearity of the medium. Its manifestation in systems of various nature speaks of its fundamentality and significance. This review considers the main works that touch upon the issue of nonlinear supratransmission in various systems, mainly model ones.

Many teams of authors are studying this effect. First of all, these are models described by discrete equations, including sin-Gordon and the discrete Schr¨odinger equation. At the same time, the effect is not exclusively model and manifests itself in full-scale experiments in electrical circuits, in nonlinear chains of oscillators, as well as in metastable modular metastructures. There is a gradual complication of models, which leads to a deeper understanding of the phenomenon of supratransmission, and the transition to disordered structures and those with elements of chaos structures allows us to talk about a more subtle manifestation of this effect. Numerical asymptotic approaches make it possible to study nonlinear supratransmission in complex nonintegrable systems. The complication of all kinds of oscillators, both physical and electrical, is relevant for various real devices based on such systems, in particular, in the field of nano-objects and energy transport in them through the considered effect. Such systems include molecular and crystalline clusters and nanodevices. In the conclusion of the paper, the main trends in the research of nonlinear supratransmission are given.

The study of the properties of seismic noise in Kamchatka is based on the idea that noise is an important source of information about the processes preceding strong earthquakes. The hypothesis is considered that an increase in seismic hazard is accompanied by a simplification of the statistical structure of seismic noise and an increase in spatial correlations of its properties. The entropy of the distribution of squared wavelet coefficients, the width of the carrier of the multifractal singularity spectrum, and the Donoho – Johnstone index were used as statistics characterizing noise. The values of these parameters reflect the complexity: if a random signal is close in its properties to white noise, then the entropy is maximum, and the other two parameters are minimum. The statistics used are calculated for 6 station clusters. For each station cluster, daily median noise properties are calculated in successive 1-day time windows, resulting in an 18-dimensional (3 properties and 6 station clusters) time series of properties. To highlight the general properties of changes in noise parameters, a principal component method is used, which is applied for each cluster of stations, as a result of which the information is compressed into a 6-dimensional daily time series of principal components. Spatial noise coherences are estimated as a set of maximum pairwise quadratic coherence spectra between the principal components of station clusters in a sliding time window of 365 days. By calculating histograms of the distribution of cluster numbers in which the minimum and maximum values of noise statistics are achieved in a sliding time window of 365 days in length, the migration of seismic hazard areas was assessed in comparison with strong earthquakes with a magnitude of at least 7.

Deep learning’s power stems from complex architectures; however, these can lead to overfitting, where models memorize training data and fail to generalize to unseen examples. This paper proposes a novel probabilistic approach to mitigate this issue. We introduce two key elements: Truncated Log-Uniform Prior and Truncated Log-Normal Variational Approximation, and Automatic Relevance Determination (ARD) with Bayesian Deep Neural Networks (BDNNs). Within the probabilistic framework, we employ a specially designed truncated log-uniform prior for noise. This prior acts as a regularizer, guiding the learning process towards simpler solutions and reducing overfitting. Additionally, a truncated log-normal variational approximation is used for efficient handling of the complex probability distributions inherent in deep learning models. ARD automatically identifies and removes irrelevant features or weights within a model. By integrating ARD with BDNNs, where weights have a probability distribution, we achieve a variational bound similar to the popular variational dropout technique. Dropout randomly drops neurons during training, encouraging the model not to rely heavily on any single feature. Our approach with ARD achieves similar benefits without the randomness of dropout, potentially leading to more stable training.

To evaluate our approach, we have tested the model on two datasets: the Canadian Institute For Advanced Research (CIFAR-10) for image classification and a dataset of Macroscopic Images of Wood, which is compiled from multiple macroscopic images of wood datasets. Our method is applied to established architectures like Visual Geometry Group (VGG) and Residual Network (ResNet). The results demonstrate significant improvements. The model reduced overfitting while maintaining, or even improving, the accuracy of the network’s predictions on classification tasks. This validates the effectiveness of our approach in enhancing the performance and generalization capabilities of deep learning models.

The paper constructs and studies a generalized model describing two-component systems of reaction – diffusion type with power nonlinearity, considering the influence of external noise. A methodology has been developed for analyzing the generalized model, which includes linear stability analysis, nonlinear stability analysis, and numerical simulation of the system’s evolution. The linear analysis technique uses basic approaches, in which the characteristic equation is obtained using a linearization matrix. Nonlinear stability analysis realized up to third-order moments inclusively. For this, the functions describing the dynamics of the components are expanded in Taylor series up to third-order terms. Then, using the Novikov theorem, the averaging procedure is carried out. As a result, the obtained equations form an infinite hierarchically subordinate structure, which must be truncated at some point. To achieve this, contributions from terms higher than the third order are neglected in both the equations themselves and during the construction of the moment equations. The resulting equations form a set of linear equations, from which the stability matrix is constructed. This matrix has a rather complex structure, making it solvable only numerically. For the numerical study of the system’s evolution, the method of variable directions was chosen. Due to the presence of a stochastic component in the analyzed system, the method was modified such that random fields with a specified distribution and correlation function, responsible for the noise contribution to the overall nonlinearity, are generated across entire layers. The developed methodology was tested on the reaction – diffusion model proposed by Barrio et al., according to the results of the study, they showed the similarity of the obtained structures with the pigmentation of fish. This paper focuses on the system behavior analysis in the neighborhood of a non-zero stationary point. The dependence of the real part of the eigenvalues on the wavenumber has been examined. In the linear analysis, a range of wavenumber values is identified in which Turing instability occurs. Nonlinear analysis and numerical simulation of the system’s evolution are conducted for model parameters that, in contrast, lie outside the Turing instability region. Nonlinear analysis found noise intensities of additive noise for which, despite the absence of conditions for the emergence of diffusion instability, the system transitions to an unstable state. The results of the numerical simulation of the evolution of the tested model demonstrate the process of forming spatial structures of Turing type under the influence of additive noise.

This paper examines the impact of the spatial resolution of a discretized (lattice) representation of the environment on the efficiency and correctness of optimal pathfinding in complex environments. Scenarios are considered that may include bottlenecks, non-uniform obstacle distributions, and areas of increased safety requirements in the immediate vicinity of obstacles. Despite the widespread use of lattice representations of the environment in robotics due to their compatibility with sensor data and support for classical trajectory planning algorithms, the resolution of these lattices has a significant impact on both goal reachability and optimal path performance. An algorithm is proposed that combines environmental connectivity analysis, trajectory optimization, and geometric safety refinement. In the first stage, the Leath algorithm is used to estimate the reachability of the target point by identifying a connected component containing the starting position. Upon confirmation of the target point’s reachability, the A* algorithm is applied to the nodes of this component in the second stage to construct a path that simultaneously minimizes both the path length and the risk of collision. In the third stage, a refined obstacle distance estimate is performed for nodes located in safety zones using a combination of the Gilbert – Johnson –Keerthi (GJK) and expanding polyhedron (EPA) algorithms. Experimental analysis revealed a nonlinear relationship between the probability of the existence and effectiveness of an optimal path and the lattice parameters. Specifically, reducing the spatial resolution of the lattice increases the likelihood of connectivity loss and target unreachability, while increasing its spatial resolution increases computational complexity without a proportional improvement in the optimal path’s performance.

The paper presents an optimization approach to microstructure simulation. Porosity function was optimized by numerical method, grain-size model was optimized by complex method based on criteria of model quality. Methods have been validated on examples. Presented new regression model of model quality. Actual application of proposed method is 3D reconstruction of core sample microstructure. Presented results suggest to prolongation of investigations. 

The article deals with the nonlinear model of population dynamics of different ages workers in the regional economy. The model is built on the principles underlying modeling in econophysics. The authors demonstrate the complex dynamics of the model regimes that impose fundamental limits on medium- and long-term forecast of employment in a region. By analogy with the biophysical approach the authors propose a classification of social interactions of the different age-group workers. The model analysis is given for the level of employment among age groups. The verification of the model performs on the statistical data of the Jewish Autonomous Region.

A molecular model of phicobilisome complex with a quenching protein OCP which regulates the energy transfer from phicobilisome to photosystem in photosynthetic apparatus of cyanobacteria has been developed. In the model obtained a well known spatial structure of interacting proteins remains intact and also the energy transfer from phycobilisome to OCP with reasonable rates is possible. Free energy of complex formation was calculated using MM–PBSA approach. By the order of magnitude this energy is about tens of kJ/mole. This value correlates well with experimental observed low stability of this complex. The specific surface energy of interaction between hydrophylic phicobilisome and OCP is twice larger than specific surface energy of their interaction with water. This reflects a high molecular complementary of interacting protein surfaces and is a strong pro argument for proposed model.

Investigation of logical deterministic cellular automata models of population dynamics allows to reveal detailed individual-based mechanisms. The search for such mechanisms is important in connection with ecological problems caused by overexploitation of natural resources, environmental pollution and climate change. Classical models of population dynamics have the phenomenological nature, as they are “black boxes”. Phenomenological models fundamentally complicate research of detailed mechanisms of ecosystem functioning. We have investigated the role of fecundity and duration of resources regeneration in mechanisms of population growth using four models of ecosystem with one species. These models are logical deterministic cellular automata and are based on physical axiomatics of excitable medium with regeneration. We have modeled catastrophic death of population arising from increasing of resources regeneration duration. It has been shown that greater fecundity accelerates population extinction. The investigated mechanisms are important for understanding mechanisms of sustainability of ecosystems and biodiversity conservation. Prospects of the presented modeling approach as a method of transparent multilevel modeling of complex systems are discussed.