Handwritten Digit Softmax Classifier in C: From 784 Pixels to submission.csv

Once the dataset layout is clear, the most useful part of this handwritten digit project is the C implementation itself. It does not rely on a deep learning framework. Instead, it uses a direct multi-class softmax model that maps a 784-dimensional input vector to ten digit classes.

This is a good kind of project for learning how model formulas become code. You can inspect the weight matrix, the softmax probability calculation, the cross-entropy loss accumulation, and the gradient-based parameter updates without a large abstraction layer getting in the way.

1. The model structure is deliberately small

The main parameters are only:

• W[10][784]: one weight vector of length 784 for each class
• b[10]: one bias term for each class

For a single input sample x, the classifier first computes one raw score per class:

z[k] = b[k];
for (int j = 0; j < FEATURES; j++) {
    z[k] += W[k][j] * x[j];
}

Those ten values are the logits for the current sample.

2. Softmax turns raw scores into probabilities

Raw linear scores are not directly interpretable as probabilities, so the implementation normalizes them with softmax:

p[i] = exp(z[i] - max_z);
sum += p[i];
...
p[i] /= sum;

The subtraction by max_z is a stability trick. It keeps the exponentials from blowing up numerically. After softmax, the probabilities over the ten classes add up to one, and the predicted label is just the class with the largest probability.

3. What the training loop is actually doing

The current project runs 20 epochs with a learning rate of 0.01. In each epoch, it loops through every training sample and repeats the same sequence:

1. Compute ten logits
2. Apply softmax to get a probability distribution
3. Compare that distribution to the true label
4. Update the weights and biases with the resulting error

The update rule is written in a very transparent way:

double error = p[k] - (k == y_train[i] ? 1.0 : 0.0);
for (int j = 0; j < FEATURES; j++) {
    W[k][j] -= LEARNING_RATE * error * X_train[i][j];
}
b[k] -= LEARNING_RATE * error;

If you already know logistic regression or linear multi-class classification, this will look familiar. It is essentially softmax regression trained with stochastic gradient descent.

4. Which metrics are worth checking

During training, the program prints epoch loss and training accuracy. After training, it prints the final training accuracy and a confusion matrix. Those are the most useful outputs to read first:

• Loss: whether optimization is moving in the right direction
• Accuracy: whether the classification result is improving
• Confusion matrix: which digits are most often mixed up

If a few classes remain confused with each other, that is usually a sign that the digit shapes are visually close or that the linear model has reached its representational limit.

5. How submission.csv is generated

After training, the program reads test.csv, calls predict_one for each sample, and writes the result back into the required CSV structure:

ImageId,Label
1,7
2,2
3,1
...

That is the final submission.csv. From an engineering perspective, this step matters because it turns the training code into a complete pipeline that can process unseen inputs and export predictions in a reusable format.

6. How to run it locally

The downloads section now includes the source file plus compressed copies of the training and test data. The current implementation expects train.csv and test.csv in the same working directory:

unzip train.csv.zip
unzip test.csv.zip
gcc digit_softmax_classifier.c -lm -O2 -o digit_classifier
./digit_classifier

A normal run should print:

• the number of training and test samples
• loss and accuracy for each epoch
• final training accuracy and the confusion matrix
• a message confirming that submission.csv was written

7. What this C version does not try to do

The current implementation is already enough for a complete multi-class practice project, but its boundaries are also clear:

• the model is still linear, not convolutional
• a strong training accuracy does not automatically mean the best generalization
• there is no dedicated validation split for tuning
• there is no mini-batch schedule, regularization, or more advanced optimization

Those are not flaws so much as the next layer of work. A clean, understandable, end-to-end baseline is already valuable.

8. Softmax Training Audit Table

The value of this C project is that it can be inspected and reproduced. The table below puts data loading, numerical stability, training signals, and output files into one review framework so a reader can tell whether a change improved the model or merely kept the program running.

Writing softmax straight from the formula will overflow

The softmax formula in section two is the mathematical definition, but transcribing it directly into C produces nan as soon as scores grow even moderately large. It is the classic trap in projects like this and deserves its own treatment.

The culprit is exp(). A float tops out around 3.4e38, and exp(89) already exceeds that; a double makes it to roughly exp(709) before overflowing to inf. Once numerator and denominator are both inf, the quotient is nan, and nan propagates back through every weight — destroying the model in a single step.

With 784-dimensional input and no constraint on early weights, scores in this task reach the hundreds easily. This is not a rare edge case; it is close to inevitable.

The fix rests on an identity: softmax is invariant to subtracting any constant from all scores.

softmax(z_i) = exp(z_i) / Σ exp(z_j)
             = exp(z_i - C) / Σ exp(z_j - C)      for any constant C

Take C as the maximum score in the group and every exponent argument becomes ≤ 0, so exp() returns values in (0, 1] and can never overflow. At least one denominator term equals 1, so division by zero is impossible too.

void softmax(double *z, int n) {
    double max = z[0];
    for (int i = 1; i < n; i++)
        if (z[i] > max) max = z[i];      /* find the maximum first */

    double sum = 0.0;
    for (int i = 0; i < n; i++) {
        z[i] = exp(z[i] - max);          /* shift, then exponentiate */
        sum += z[i];
    }
    for (int i = 0; i < n; i++)
        z[i] /= sum;
}

The cost is one extra pass over ten elements, which is negligible.

The same class of problem reappears in cross-entropy loss, where log(p) returns -inf as p approaches zero. The safe form clamps to a small floor:

loss -= log(p[label] < 1e-15 ? 1e-15 : p[label]);

A practical debugging heuristic: if loss suddenly becomes nan on some iteration, it is almost certainly overflow rather than an excessive learning rate. Too high a learning rate presents as violent oscillation or gradual divergence — there is a progression. Overflow arrives in one step: normal values on one iteration, nan on the next, and nan forever after. Recognise that pattern and go straight to exp() and log() without first suspecting hyperparameters.

9. Where to go next

If you want an interactive version before reading more source code, open the handwritten digit tab in the playground. The browser version does not retrain on the full dataset. Instead, it loads a compact pre-trained softmax demo so you can draw digits, inspect probability scores, and try labeled samples directly in the page.

The source code, zipped datasets, sample submission file, generated submission, and browser model bundle are all available on the downloads page. If you have not read the previous post yet, start with the dataset structure article so the arrays and loops in this C file are easier to place in context.