Generalization of backpropagation with application to a recurrent gas market model

doi:10.1016/0893-6080(88)90007-X

Neural Networks

Volume 1, Issue 4, 1988, Pages 339-356

https://doi.org/10.1016/0893-6080(88)90007-X Get rights and content

Abstract

Backpropagation is often viewed as a method for adapting artificial neural networks to classify patterns. Based on parts of the book by Rumelhart and colleagues, many authors equate backpropagation with the generalized delta rule applied to fully-connected feedforward networks. This paper will summarize a more general formulation of backpropagation, developed in 1974, which does more justice to the roots of the method in numerical analysis and statistics, and also does more justice to creative approaches expressed by neural modelers in the past year or two. It will discuss applications of backpropagation to forecasting over time (where errors have been halved by using methods other than least squares), to optimization, to sensitivity analysis, and to brain research.

This paper will go on to derive a generalization of backpropagation to recurrent systems (which input their own output), such as hybrids of perceptron-style networks and Grossberg/Hopfield networks. Unlike the proposal of Rumelhart, Hinton, and Williams, this generalization does not require the storage of intermediate iterations to deal with continuous recurrence. This generalization was applied in 1981 to a model of natural gas markets, where it located sources of forecast uncertainty related to the use of least squares to estimate the model parameters in the first place.

References (39)

L.B. Almeida
A learning rule for asynchronous perceptrons with feedback in a combinatorial environment
R.G. Alsmiller
Adjoint sensitivity analysis and its application to LEAP Model 22c
M. Athans et al.
Optimal control theory
(1966)
A. Barto et al.
Neuron-like adaptive elements that can solve difficult learning control problems
IEEE Transactions on Systems, Man and Cybernetics
(1983)
J. Brode et al.
TSP in the Datatran language
(1975)
J. Dennis et al.
Numerical methods for unconstrained optimization and nonlinear equations
(1983)
Energy Information Administration
S. Foote et al.
Extrathalamic modulation of cortical function
Annual Review of Neuroscience
(1987)
G. Forsythe et al.
Computer solution of linear algebraic systems
(1967)
S. Grossberg
Adaptive pattern classification and universal recording, II: Feedback, expectation, olfaction and illusions
Biological Cybernetics
(1976)

S. Grossberg et al.

Predictive regulation of associative learning in neural networks by reinforcement and attentive feedback

D.O. Hebb

The organization of behavior

(1949)

G. Hinton

Connectionist learning procedures

J. Hopfield et al.

Computing with neural circuits: A model

Science

(1986)

MIT Center for Computational Research In Economics and Management Science

Troll Users' Guide

(1980)

D. McNicol et al.

Analysis of the economic effects of accelerated deregulation of natural gas prices

R. O'Neill et al.

An analysis of the natural gas policy act and several alternatives: Part I

D.B. Parker

Learning-logic

D.B. Parker

Second-order backpropagation: Implementing an optimal O(n) approximation to Newton's method in an artificial neural network

(1987)

Cited by (761)

Length of stay prediction for hospital management using domain adaptation
2024, Engineering Applications of Artificial Intelligence
Anticipating inpatient length of stay (LoS) can be of crucial aid for efficient hospital management, admissions planning, resource allocation and care quality improvement. Anticipating LoS can be achieved by applying machine learning techniques on historical patient data for developing predictive models. Ethically, these models cannot serve as substitutes for medical unit heads, solely responsible for authorizing patients’ discharge, but they can be exploited by management systems for effective hospital planning. Therefore, the prediction system should be designed and adapted to work in a true hospital setting. This study focuses on early hospital LoS prediction for different admission units and employs domain adaptation to leverage information learned from a potential source domain. Time series data from 110,079 admissions into 8 intensive care units (ICUs) and 60,492 admissions into 9 ICUs were respectively extracted from eICU-CRD and MIMIC-IV and fed into a Long-Short Term Memory network and a Fully connected network to train a source domain model. Learned weights were then transferred either partially or fully from a source domain to target domains. SHapley Additive exPlanations (SHAP) algorithms were used to study the effect of weight transfer on feature importance. Compared against the benchmark, the proposed weight transfer model showed statistically significant gains in prediction accuracy (between 1% and 5%) as well as computation time (up to 2 h saved) for some target domains. The proposed method provides an adaptable clinical decision support system for hospital management that can facilitate processes of data access via the ethical committee, computation infrastructures and time.
Memory based neural network for cumin price forecasting in Gujarat, India
2024, Journal of Agriculture and Food Research
Agricultural price forecasting, with its distinctive characteristics, remains a captivating field of study. In countries like India, grappling with food security challenges, reliable and efficient price forecasting models are of utmost importance. This research focuses on accurate prediction of cumin prices by emphasizing the importance of time series forecasting and the adoption of deep learning models to overcome the limitations of traditional statistical approaches. Deep learning (DL) approaches including Deep Neural Network (DNN), Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) were employed to forecast cumin prices for the entire year (365 days) of 2022. The models were trained and assessed using daily price data from 2002 to 2021 from Unjha market, Gujarat, India, with accuracy metrics including RMSE, MAPE, SMAPE, MASE, and MDA. The superior accuracy of the Stacked LSTM model, particularly its low RMSE, MAPE, and SMAPE scores, along with the highest MDA, marks it as a promising tool for future agricultural price forecasting. Its precision in predicting cumin prices, with a 5 % error pre-sowing and 18 % pre-harvesting, is particularly noteworthy during critical farming periods. These findings can guide farmers in aligning their production schedules with periods of high prices, aiding in economically more beneficial farming practices. Additionally, the model's predictive reliability can assist policymakers and traders in making data-driven decisions, thus playing a significant role in stabilizing market dynamics.
Modeling of wavy water film by application of artificial neural network - a state of art study
2024, Nuclear Engineering and Design
Nowadays, artificial intelligence (AI) tools have attained a state of maturity and have advanced at a quick rate of achievement in various fields of study. Thus, the predictive and classification of artificial intelligence AI capabilities in manufacturing, commercial, and scientific research applications have resulted in remarkable success. It is widely known that two-phase flow is such a complex problem, and it has a vital state of study due to its importance and wide range of applications. Such as water film phenomena are critical for various industries, including nuclear protection, energy generation, absorption processes, internal combustion engines, chemical reactors, and spray cooling. The present study presented a realistic model for a particular two-phase flow problem of water film thickness. Also, we used neurotic networks for prediction and classification, such as the conventional neural network (CNN) and Long Short-Term Memory (LSTM). The proposed model was evaluated using various raw data, including the input liquid Reynolds number, the Reynolds gas number, the angle of inclination, and the thickness of the water film. The water film thickness was determined using a planar laser-induced fluorescence (PLIF) technique on the various inclination platforms. The current experimental work's inclination angle varied between 0 and 50 degrees. However, when the Reynolds number of the input water changed from 175 to 435, the thickness of the water film was determined; Reynolds gas numbers range from 6950 to 34800.
The current study findings indicate an effective alternative method for detecting and forecasting water films on a sloping plate. Nash-Sutcliffe Efficiency (NSE) affected 99 %, 97 %, and 94 % of forecasts, respectively. The findings show tremendous improvement when deep learning uses both LSTM and CNN. The gravitational pressure and air share tension push the water film.
Revolutionizing concrete analysis: An in-depth survey of AI-powered insights with image-centric approaches on comprehensive quality control, advanced crack detection and concrete property exploration
2024, Construction and Building Materials
Over the last two decades, the integration of big data and deep learning technologies has demonstrated remarkable effectiveness across various domains of civil engineering, leading to significant advancements in machine learning methodologies. Regardless of significant progress in various aspects of civil engineering, there appears to be a striking absence of thorough and all-encompassing evaluations that adequately cover critical domains such as structural damage identification (SDI), accurate prediction of concrete properties, precise crack detection, detailed analysis of crack properties and thorough non-destructive testing (NDT) methods applied specifically to concrete structures. This gap in the existing literatures hinders a comprehensive understanding and application of advanced techniques in these crucial areas. To bridge this knowledge gap, this paper offers an all-encompassing exploration of the recent developments and utilizations of artificial intelligence in civil engineering. Emphasizing concrete-centric structures, this study prioritizes two crucial facets: the automatize detection of concrete specific mechanical attributes and crack detection. To achieve this numerous recent surveys have utilized intelligent algorithms primarily grounded in image statistics, coupled with extensive data analytics and deep learning methodologies. By delving into this research, the paper sheds light on the trajectory and challenges encountered by artificial intelligence in civil engineering over the past few years. It highlights studies that emphasize structural maintenance and management, quality control, design optimization, and more, accomplished through image processing and computer vision techniques. In essence, this review seeks to unveil the profound impact of artificial intelligence in transforming civil engineering practices, with a strong emphasis on exploiting image processing and computer vision methods to tackle concrete-related issues.
Nonlinear autoregressive models for high accuracy early prediction of Li-ion battery end-of-life
2023, Journal of Energy Storage
Predictions of the state-of-health (SOH) of Li-ion batteries is an important goal in the monitoring and management of electric vehicles. In recent years, a number of pure machine-learning methods have been proposed for such predictions. In this paper, we instead consider autoregression methods and embedding strategies, which are specifically tailored to time-series problems. For the first time, we comprehensively compare both linear and nonlinear approaches, including six deep learning architectures, autoregressive integrated moving average (ARIMA) models and seasonal ARIMA models. In particular, for the first time we introduce Gaussian process nonlinear autoregression (GPNAR) for SOH prediction and show that it is superior in terms of accuracy and computational costs to the other autoregressive approaches. On the basis of two different datasets, we also demonstrate that accurate early predictions of the end-of-life (based on 50% of the data) is achievable with GPNAR without the use of features, which keeps data acquisition and processing to a minimum. Finally, we show that GPNAR is capable of capturing seasonal trends such as regeneration without additional time-consuming data analyses. Comparisons to other state-of-the-art methods in the recent literature confirm the superior performance of GPNAR.
Learning structural stress virtual sensors from on-board instrumentation of a commercial aircraft
2023, Computers and Structures
This work aims to predict the mechanical stress on the structure of a business jet in service phase from flight instrument only. A significant database obtained from test flights using aircraft instrumented with strain gauges has been provided as part of an Artificial Intelligence challenge organized by the French Ile-de-France region and the aircraft manufacturer Dassault Aviation. Learning techniques are considered to train a prediction model of the aircraft structural stress. The proposed baseline includes a clustering step for phase identification in time series and an ensemble model with two stacked regressors. The model is trained on a dataset of 117 flights and its overall performance is evaluated on a validation set of flight sequences from 186 flights. The main advantages of the proposed learning approach are its prediction accuracy, its training frugality and its interpretability. This paper presents a global data science workflow applied to a problem of structural stress prediction. Despite a development in a constrained time period with no direct access to the data, the proposed approach demonstrates the feasibility of the concept of learned virtual sensor of aircraft structural stress and paves the way for applications to other structures.

View all citing articles on Scopus

View full text

Original contributionGeneralization of backpropagation with application to a recurrent gas market model

Abstract

A learning rule for asynchronous perceptrons with feedback in a combinatorial environment

Adjoint sensitivity analysis and its application to LEAP Model 22c

Optimal control theory

Neuron-like adaptive elements that can solve difficult learning control problems

IEEE Transactions on Systems, Man and Cybernetics

TSP in the Datatran language

Numerical methods for unconstrained optimization and nonlinear equations

Extrathalamic modulation of cortical function

Annual Review of Neuroscience

Computer solution of linear algebraic systems

Adaptive pattern classification and universal recording, II: Feedback, expectation, olfaction and illusions

Biological Cybernetics