Funciones de activacion

Avatar de Usuario
Site Admin
Mensajes: 8413
Registrado: 09 May 2012 01:38

Funciones de activacion

Mensajepor Dalamar » 04 Abr 2017 16:44

***** EN PROGRESO ***************

logictic sigmoid


Dying Relu problem: Relu neurons can sometimes be pushed into states in which they become inactive for essentially all inputs. In this state, no gradients flow backward through the neuron, and so the neuron becomes stuck in a perpetually inactive state and "dies." In some cases, large numbers of neurons in a network can become stuck in dead states, effectively decreasing the model capacity. This problem typically arises when the learning rate is set too high.



There are plenty more activation functions available and different researchers keep on looking for new ones (e.g. Exponential Linear Units (ELUs), Gaussian Error Linear Units (GELUs), ...) with different/better properties

Regression: linear (because values are unbounded)

Classification: softmax (simple sigmoid works too but softmax works better)

Use simple sigmoid only if your output admits multiple "true" answers, for instance, a network that checks for the presence of various objects in an image. In other words, the output is not a probability distribution (does not need to sum to 1).

Sigmoid and tanh should not be used as activation function for the hidden layer. This is because of the vanishing gradient problem, i.e., if your input is on a higher side (where sigmoid goes flat) then the gradient will be near zero. This will cause very slow or no learning during backpropagation as weights will be updated with really small values.

Detailed explanation here:

The best function for hidden layers is thus ReLu.
¿Te ha gustado este hilo? Compartelo en las redes socialespara que se sume mas gente a la conversacion!

Volver a “Inteligencia Artificial”