Building Your First Neural Network

From Perceptrons to Deep Networks

Neural networks mimic the structure of the human brain — interconnected nodes (neurons) that pass signals to one another. Despite the biological metaphor, they are essentially mathematical functions that learn to approximate complex relationships in data.

A Minimal Neural Network with Python

Let's build a simple neural network from scratch using NumPy. This will help demystify what happens beneath the abstraction layers of deep learning frameworks.

import numpy as np

class SimpleNeuralNetwork:
    def __init__(self, input_size, hidden_size, output_size):
        # Initialize weights randomly
        self.W1 = np.random.randn(input_size, hidden_size)
        self.b1 = np.zeros((1, hidden_size))
        self.W2 = np.random.randn(hidden_size, output_size)
        self.b2 = np.zeros((1, output_size))

    def relu(self, x):
        return np.maximum(0, x)

    def softmax(self, x):
        exp_x = np.exp(x - np.max(x, axis=1, keepdims=True))
        return exp_x / exp_x.sum(axis=1, keepdims=True)

    def forward(self, X):
        self.z1 = X @ self.W1 + self.b1
        self.a1 = self.relu(self.z1)
        self.z2 = self.a1 @ self.W2 + self.b2
        self.output = self.softmax(self.z2)
        return self.output

    def backward(self, X, y, output, lr=0.01):
        m = X.shape[0]
        dz2 = output - y  # Gradient of softmax + cross-entropy
        dW2 = (self.a1.T @ dz2) / m
        db2 = dz2.sum(axis=0, keepdims=True) / m
        dz1 = dz2 @ self.W2.T * (self.z1 > 0)
        dW1 = (X.T @ dz1) / m
        db1 = dz1.sum(axis=0, keepdims=True) / m

        self.W2 -= lr * dW2
        self.b2 -= lr * db2
        self.W1 -= lr * dW1
        self.b1 -= lr * db1

Understanding the Flow

Forward Pass

Data flows through the network layer by layer. Each neuron computes a weighted sum of its inputs, adds a bias, and applies an activation function. Popular choices include ReLU, sigmoid, and tanh.