In practice, what is the most accurate description for activation functions $($such as Sigmoid, Sum, Tanh, ReLU$)$ used in neural networks?

Group of answer choices

They must be differentiable

They can be non$-$differentiable, but only for a small amount of points.

They must be non$-$linear to be learnable

They can be any continuous functions