3. Hyperbolic Tangent (tanh)

Formula:

$$\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$$

Range: -1 to +1

Meaning:

Similar to sigmoid but centered at 0.
Positive values $\rightarrow$ positive output
Negative values $\rightarrow$ negative output

Use:

Better than Sigmoid in many cases because of zero-centered output.

Problem:

Still suffers from vanishing gradients.

4. Softmax Activation Function

Formula:

$$\text{Softmax}(x_i) = \frac{e^{x_i}}{\sum_{j} e^{x_j}}$$

Meaning:

Converts a list of numbers into probabilities that sum to 1.
Example: output like $[2.3, 1.0, 0.2] \rightarrow \text{softmax} \rightarrow [0.7, 0.2, 0.1]$

Use:

Used in multi-class classification (3 or more classes).

Important:

Always used in output layer, not in hidden layers.

What is Gradient Descent?

Gradient Descent is an iterative optimization algorithm used to minimize a cost function by adjusting model parameters in the direction of the steepest descent of the function's gradient. In simple terms, it finds the optimal values of weights and biases by gradually reducing the error between predicted and actual outputs.

Imagine you're at the top of a hill and your goal is to find the lowest point in the valley. You can't see the entire valley from the top, but you can feel the slope under your feet.

Start at the Top: You begin at the top of the hill (this is like starting with random guesses for the model's parameters).
Feel the Slope: You look around to find out which direction the ground is sloping down. This is like calculating the gradient, which tells you the steepest way downhill.
Take a Step Down: Move in the direction where the slope is steepest (this is adjusting the model's parameters). The bigger the slope, the bigger the step you take.
Repeat: You keep repeating the process — feeling the slope and moving downhill — until you reach the bottom of the valley (this is when the model has learned and minimized the error).

The key idea is that, just like walking down a hill, Gradient Descent moves towards the "bottom" or minimum of the loss function, which represents the error in predictions.

Moving in the opposite direction of the gradient allows the algorithm to gradually descend towards lower values of the function and eventually reach the minimum of the function. These gradients guide the updates ensuring convergence towards the optimal parameter values. Gradual steps used in descent are defined by the learning rate.

What is Learning Rate?

Learning rate is an important hyperparameter in gradient descent that controls how big or small the steps should be when going downwards in gradient for updating model parameters. It is essential to determine how quickly or slowly the algorithm converges toward the minimum of the cost function.

If Learning rate is too small: The algorithm will take tiny steps during iteration and converge very slowly. This can significantly increase training time and computational cost, especially for large datasets.
If Learning rate is too big: The algorithm may take huge steps, overshooting the minimum of the cost function without settling. It fails to converge, causing the algorithm to oscillate. This process is termed as the exploding gradient problem.

To address these problems, we have some techniques that can be used:

Weights Regularization: The initialization of we