In fields like signal processing (and imaging as well), one designs so-called filters. These filters are defined by the convolutions and are often hand-crafted. One may specify filters for smoothing, edge detection, frequency reshaping, and similar operations. However with neural networks the idea is to automatically learn the filters and use many of them in conjunction with non-linear operations (activation functions).
As an example consider a neural network operating on sound sequence data. Assume that we an input vector x of length d=106. We construct then a neural network with onle hidden layer only with 104 nodes. This means that we will have a weight matrix with 104×106=1010 weights to be determined, together with 104 biases.
Assume furthermore that we have an output layer which is meant to train whether the sound sequence represents a human voice (true) or something else (false). It means that we have only one output node. But since this output node connects to 104 nodes in the hidden layer, there are in total 104 weights to be determined for the output layer, plus one bias. In total we have
NumberParameters=1010+104+104+1≈1010,that is ten billion parameters to determine.