$4$ Neural Networks as Universal Function Approximators

Neural networks are a flexible class of parametric models which form the basis for a wide range of algorithms in

machine learning. Their widespread success is due both to the ease and efficiency of "training" them, or optimizing

over their parameters, using the backpropagation algorithm, and their ability to approximate any smooth function.

Recall that a feed$-$forward neural network is simply a combination of multiple 'neurons' such that the output of one

neuron is fed as the input to another neuron. More precisely, a neuron