Combining domain knowledge models with neural models has been challenging.   End-to-end trained neural models often perform better (lower Mean Square Error) than domain knowledge models or domain/neural combinations, and the combination is inefficient to train.   In this paper, we demonstrate that by composing domain models with machine learning models, by using extrapolative testing sets, and invoking decorrelation objective functions, we create models which can predict more complex systems. The models are interpretable, extrapolative, data-efficient, and capture predictable but complex non-stochastic behavior such as unmodeled degrees of freedom and systemic measurement noise.   We apply this improved modeling paradigm to several simulated systems and an actual physical system in the context of system identification.    Several ways of composing domain models with neural models are examined for time series, boosting, bagging, and auto-encoding on various systems of varying complexity and non-linearity.   Although this work is preliminary, we show that the ability to combine models is a very promising direction for neural modeling.
Improved modeling of complex systems uses hybrid neural/domain model composition, new decorrelation loss functions and extrapolative test sets  This paper conducts experiments to compare the extrapolative predictions of various hybrid models which compose physical models, neural networks and stochastic models, and tackles the challenge of unmodeled dynamics being a bottleneck. This paper presents approaches for combining neural network with non-NN models to predict behavior of complex physical systems.