Swish function

The swish function is a mathematical function defined as follows: $\operatorname {swish} (x):=x\times \operatorname {sigmoid} (x)={\frac {x}{1+e^{-x}}}$.

Applications

In 2017, after performing analysis on ImageNet data, researchers from Google alleged that using the function as an activation function in artificial neural networks improves the performance, compared to ReLU and sigmoid functions. It is believed that one reason for the improvement is that the swish function helps alleviate the vanishing gradient problem during backpropagation.