Reset Gate ($r_t$): Determines how much of the previous hidden state ($h_{t-1}$) to "forget" when calculating the new candidate hidden state ($\tilde{h}_t$). A small $r_t$ means $h_{t-1}$ is mostly ignored.

$r_t = \sigma(W_r \cdot [h_{t-1}, x_t] + b_r)$

Candidate Hidden State ($\tilde{h}_t$): The potential new hidden state, calculated based on the current input and a "reset" version of the previous hidden state.

$\tilde{h}_t = \tanh(W_h \cdot [r_t \odot h_{t-1}, x_t] + b_h)$

Hidden State Update: The final hidden state $h_t$ is a linear interpolation between the previous hidden state $h_{t-1}$ and the candidate hidden state $\tilde{h}_t$, controlled by the update gate $z_t$.

$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$

Advantages: Often faster to train and requires less data due to fewer parameters compared to LSTMs. Performance is frequently comparable, making GRUs a popular choice for many tasks.

Practical Example: Time Series Forecasting (GRU Many-to-One)

Predicting the next value in a sequence (e.g., stock price, temperature). Input: window of past values. Output: single future value.

import numpy as np
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import GRU, Dense

# 1. Prepare Data (simplified sinusoidal wave)
time_steps = np.linspace(0, 100, 1000)
data = np.sin(time_steps / 10) + np.random.normal(0, 0.1, 1000)
data = data.reshape(-1, 1) # (timesteps, features)

# Create sequences for training
def create_sequences(data, seq_length):
    X, y = [], []
    for i in range(len(data) - seq_length):
        X.append(data[i:(i + seq_length)])
        y.append(data[i + seq_length])
    return np.array(X), np.array(y)

seq_length = 10
X, y = create_sequences(data, seq_length)

# Split into train/test (simplified)
train_size = int(len(X) * 0.8)
X_train, y_train = X[:train_size], y[:train_size]
X_test, y_test = X[train_size:], y[train_size:]

# 2. Build Model
model = Sequential([
    GRU(units=50, activation='tanh', input_shape=(seq_length, 1)),
    Dense(units=1) # Predict a single future value
])

# 3. Compile and Train (simplified)
model.compile(optimizer='adam', loss='mse')
# model.fit(X_train, y_train, epochs=20, batch_size=32, validation_data=(X_test, y_test))
model.summary()

7. Bidirectional RNNs (BiRNNs)

Concept: Enhance RNNs by allowing them to process the input sequence in both forward and backward directions independently. This provides the model with context from both the past and the future for each point in the sequence.
Architecture: Consists of two separate hidden layers:
- A forward layer processes the sequence from left-to-right (e.g., $x_1 \to x_2 \to ... \to x_T$), producing hidden states $\vec{h}_t$.
- A backward layer processes the sequence from right-to-left (e.g., $x_T \to x_{T-1} \to ... \to x_1$), producing hidden states $\overleftarrow{h}_t$.
Combined Output: The final output at time $t$ is typically based on the concatenation or summation of both hidden states: $h_t = [\vec{h}_t; \overleftarrow{h}_t]$ (concatenation is common). This combined representation is then passed to the output layer.
Benefit: Allows the network to utilize context from both past and future time steps, which is crucial for tasks where understanding the full context of a word or event is necessary (e.g., "apple" as a company vs. a fruit). Useful for tasks like speech recognition, machine translation, and named entity recognition.
Limitation: