Out of Sample
Chapter 1

Origin story

Every engineer starts somewhere. Mine has a lot of trains in it.

Chennai, 2006.

Rajesh says: Electronics and instrumentation at MIT, Anna University. Circuits, sensors, things that measure.

Dortmund, 2010.

Rajesh says: A master's in process automation at TU Dortmund. Plants that run themselves, if you set them up right.

Munich, 2012. German Aerospace Center.

Robot says: Beep. Which way am I tipping?

Rajesh says: Some of your joints have no motor. I estimate their state, in MATLAB and Simulink.

Ruhr region, 2014. Blue Triangle.

Rajesh says: MATLAB simulations to help prototype a cancer diagnostic device.

Bangalore, 2015. Alstom.

Rajesh says: Four years writing simulators and test software for railway signalling. I learn how trains talk to the track.

2020.

Rajesh says: Hang on. These trains have been writing logs the whole time. Who is reading them?

Me, it turns out. That year I became a data scientist.

Chapter 2

The black box talks too much

When nobody can read every line, find the bursts and rank them.

Alstom, a project called Incident Detection. A fleet of trains rolls into the depot for maintenance.

Rajesh says: A train is tens to hundreds of subsystems. Every one of them writes its own log.

All of those logs go into one common black box.

subsystem logsblack box

The maintenance operators download the black box. It runs to millions and millions of lines.

Operator says: I barely have time to download this, let alone find out what needs fixing.

Kleinberg's burst detection. In each time window, count one event type against all the events in that window.

Rajesh says: The stream is in one of two states: baseline, or bursty, where the event turns up far more often. The model picks the cheapest run of states that fits the counts.

Rajesh says: Moving up to bursty has a cost, so a short blip is not worth the climb. A real burst pays for it, and each burst gets a weight.

The method is from Jon Kleinberg's paper Bursty and Hierarchical Structure in Streams

blipburstburstybaseline

Each burst points at the train that produced it. The incidents and the trains are ranked by severity.

Operator says: Worst at the top. So I start at the top.

most severe

It shipped as an executable inside Alstom's fleet management system. It cut the time spent on the logs and improved maintenance efficiency by 30%.

Rajesh says: The black box still talks too much.

Operator says: Fine. Now I only listen when it shouts.

Chapter 3

The rule that never fired

When failures are rare, learn what normal looks like.

Every train logs its odometry and ATC signals. ATC means automatic train control.

Rajesh says: The old way to watch them is a rule: trip when a signal crosses a fixed limit.

Rule box says: Inside the limits. All quiet. Zzz.

Rajesh thinks: That wiggle looks wrong to me. It just never crosses the line.

Why not train a classifier on past faults?

Rajesh says: Railway failures are few and far between. There are too few labelled faults to learn from.

Normal running fills the logs. So I teach a model what normal looks like.

Rajesh says: An autoencoder squeezes a window of signal through a narrow middle, then rebuilds it.

Rajesh says: Trained on normal running, it rebuilds normal well. A window it rebuilds badly is unlike anything it trained on. That gap is the anomaly score.

Over 99% recall. It caught failures the rules missed.

Rajesh says: No fault labels needed. Just a lot of very boring Tuesdays.

Rule box says: Zzz.

Chapter 4

Which cause, not which correlation

Naming a root cause needs the top rung of the ladder.

Two alarms go off together before most service failures.

Rajesh thinks: Correlation ranks them equally. So the engineer checks both. Every time.

Pearl's causal hierarchy is a ladder with three rungs.

Rajesh says: Correlation lives on the bottom rung. A root cause lives on the top one.

1. See: what goes with what?2. Do: what if I set this fault?3. Imagine: would it fail without it?

From the logs, causal discovery learns a directed graph: which component states drive which.

Rajesh says: A Causal Bayesian Network gives each box a probability, given the boxes that point into it.

failure

Rajesh says: Set a fault and cut the arrows into it. That is P(failure | do(fault)).

Rajesh says: Then the top rung: given this failure, would it have happened without that fault?

In production on an urban transit fleet. Troubleshooting time fell 86%.

Rajesh says: It became a UAI 2024 paper I co-authored. We found no earlier use of the hierarchy for root cause analysis in an industrial setting. Read the paper

Puzzle ?

Puzzle break

No numbers. Just think about it for ten seconds.

Model A aced every week it trained on. Model B was only fine on those weeks.

Rajesh says: Next quarter starts Monday. Which one do we ship?

AB
Ship A, it aced the test

Bold. But A memorised those weeks. A score on weeks a model has seen rewards remembering them, not forecasting. The next chapter shows what happened when I checked. Turn the page.

Ship B, A looks too good

Closer, but you are still judging from the same weeks A memorised. Suspicion is not a test. The next chapter has the test. Turn the page.

I need to see something first

That is the one. Show me how each does on weeks it has never seen, then decide. The next chapter is what happened when I did exactly that. Turn the page.

Chapter 5

Pick on the holdout

A model earns its place on weeks it has not seen.

A retail forecast: 76 weekly series.

Rajesh says: Each series picked its model at fit time, by its score on the weeks it trained on.

Rajesh thinks: That is grading a student on the exam they wrote the answer key for.

So I race them again on the holdout: weeks held back from fitting.

Rajesh says: On 69 of 76 series, the fit-time winner was not the holdout winner.

Rajesh says: So I retired the in-sample pick. Each series now picks on holdout WAPE: total absolute error over total demand, so big weeks count more.

Rajesh says: Plus a 1.5x cap on how much worse any one series may get.

RIP

Median holdout WAPE, three reruns. Lower is better.

Rajesh says: Same menu of models. A different judge.

0.5740.5290.451
Chapter 6

I found a leak in my own benchmark

Feature importance is step zero.

I publish a benchmark. LightGBM wins by 23 WMAPE points.

Rajesh says: Nice.

Then I open the feature importance export.

Rajesh thinks: One feature takes 68.8% of the gain. The latest lag gets 0.08%. That is not forecasting.

total_qty 68.8%lag_1 0.08%

total_qty was summed over a window that included the holdout.

Rajesh says: The model could see the answer. It was reading each series' scale, not modelling its demand.

holdouttotal_qty

So I retract the claim, in writing.

Rajesh says: Not a quiet edit. A retraction. Then I compute every covariate only from data before each forecast origin.

RETRACTED

Leak-free rerun, all 16 configurations. WMAPE, lower is better.

SeriesLightGBMCroston-Optimized
Intermittent51.962.6
Lumpy69.070.7

Rajesh says: LightGBM still wins. By less. The ranking held.

The leak was worth 9.9 WMAPE points on intermittent demand.

Rajesh says: The lesson I wrote down: feature importance is step zero. I look before I celebrate.

Chapter 7

Fill rate is the verdict

A forecast is judged by the shelf it produces.

The forecast gives a band: P10, P50, P90. The band sets the reorder point.

Rajesh says: When stock falls below that line, we order more. Fill rate is the share of demand the shelf serves.

reorder point

On one material, a Naive model frozen inside a composite collapsed the band.

Rajesh says: The reorder point fell from 3,618 to 489.

3,618489

Fill rate would have dropped about 54 percentage points.

Rajesh says: A forecast can look fine on error and still empty the shelf. A guardrail caught this one before rollout.

So each policy is scored by the fill rate and cost it produces.

Rajesh says: Train to the end of 2024. Validate on 2025. Test on the first half of 2026, which nothing has seen.

trainvalidatetest

The first verdict: on 9 smooth series, test fill rate 97.7% against the simulated incumbent policy's 74.5%, at lower inventory cost.

Rajesh says: On the intermittent and lumpy series, the old policy stays. Sometimes the right answer is: don't ship mine.

9 smooth series: new policyintermittent, lumpy: incumbent

Then the real test: my policy against what the incumbent actually did, on 27 materials over 152 days.

Rajesh says: Cost: 33.9% lower. Demand-weighted fill: 2.36 points lower. Cheaper, and worse at the one job a shelf has.

mine 61.50%incumbent 63.85%

So fill rate stopped being a score to trade for cost. It became a hard floor: each material must match the incumbent's own measured fill, at the lowest cost.

Rajesh says: A simulated rival is easy to beat. The real one gets to set the bar.

Chapter 8

Ten digits

Give the model a tool, not the phone book.

A customs brokerage needs a ten-digit tariff code for each part it handles.

Rajesh says: Finding it is a walk down a hierarchy, one floor at a time.

chapterheadingsubheadingall ten digits

Each step follows the General Rules of Interpretation and thousands of binding rulings.

Rajesh says: Both change every year. Help.

Version one: a single ReAct agent with a 3,000-line prompt.

Rajesh says: When it went wrong, I could not see which step.

I rebuild it as a LangGraph StateGraph: 7 nodes, 9 edges. Each step can be inspected.

Hybrid retrieval finds candidate headings. The graph walks the subheadings against a knowledge graph of rulings and writes a justification with every code.

A Langfuse trace: Gemini 2.5 Pro spends its whole 4k token allowance on thinking.

Rajesh says: Because I pasted the full subheading tree into the prompt. Like handing someone the phone book to find one number.

thinking ... all 4k tokens

A lean prompt, plus a tool that fetches the tree on demand.

Rajesh says: Cost per classification: $0.13 down to $0.09. In production.

$0.13$0.09
Back @

Back page

Letters, links and what comes next.

Rajesh says: Thanks for reading the story so far. If this made you smile, we should probably talk.

Wanted

I am open to staff and principal level applied science roles, research collaboration on causal and probabilistic forecasting, and consulting on supply chain AI.

The boring version

The CV has the career chart and every result. The PDF is the plain edition: two pages, no pixels.

Read the CVDownload the PDF

Now reading

Sutton and Barto, through the University of Alberta reinforcement learning specialization.

On my desk

Now, September 2026

Multi-echelon inventory optimisation for a mining and industrial operator, a graph against relational database benchmark on a US electric vehicle manufacturer's bill of materials, hiring, and Avathon's new AI Center of Excellence, as one of its three seeding members.

Writing

Longer posts, with fewer pixels.

Read the blog

I also run weekend AI workshops for colleagues who do not work in AI.

Colophon. Drawn in the 16 colours of the DawnBringer palette, three or four screen pixels to every art pixel depending on your screen, and six on the cover. Every word in the bubbles is real text, so you can select it, search it and read it with a screen reader. Next chapter: maybe reinforcement learning, once I finish the book.