Beyond IS/OOS Splits: Designing Walk-Forward Windows

Bottom Line: A Single OOS Segment Is Still Just One Test

Splitting data into an optimization set (IS) and a validation set (OOS) is a sound first step when you suspect overfitting. But if you split only once, the OOS result tells you only how the logic did over that one segment.

Our conclusion is this: validate across multiple OOS windows while moving forward in time, and judge each window on whether it was profitable, not just the total. Even when the total is positive, if the profit is concentrated in a few windows, we hold off on the verdict.

Weaknesses of a Fixed Split

The Split Point Drives the Result

With a fixed split, the developer decides where to cut. If the OOS happens to fall in a clearly trending period, the logic looks better than it really is. If it falls in the opposite kind of period, you may throw away logic that was perfectly usable. Either way, the result depends on one choice: the split point.

Look at the OOS Too Often and It Stops Being OOS

Another problem comes from the developer's own behavior. The OOS results are poor, so you adjust the parameters and look at the OOS again. Repeat that and you are effectively selecting for whatever fits the OOS. On paper it is unused data; in practice it has been used for optimization. This is a mistake we could easily make ourselves, and it can't be prevented without constraining the procedure.

How to Build Walk-Forward Windows

Walk-forward treats an "optimization period" and the "validation period immediately after it" as one window, then repeats while shifting forward in time. The validation segment always lies in the future relative to the optimization segment, and you end up with multiple OOS results.

Criteria for Window Length

It is safer to size windows by number of trades than by calendar years. If too few trades fall inside a validation segment, whether it ends up profitable or not is decided by chance. The longer the holding time and the lower the trade frequency, the longer the validation segment needs to be.

Checking Window by Window

1. Lay out the windows so the validation segments do not overlap
2. Optimize in each window and apply the resulting parameters only to the validation segment right after it
3. Record profit/loss, PF, number of trades, and maximum drawdown for each validation segment
4. Check how many windows have PF of 1.0 or higher, and which periods the losing windows fall in
5. Check whether the remaining total is still profitable after excluding the window with the largest profit
6. Look at whether the parameters chosen for each window jump around wildly

There are three key points to check:

• Whether the profitable windows are concentrated in a small part of the sample
• Whether losses in the losing windows are too large relative to profits in the winning windows
• Whether the optimal parameters change drastically between adjacent windows

We don't think every window needs to be profitable. Trend-following logic normally loses in windows with no directional movement. What should be a concern is when you can't explain why a window lost, or when the depth of the loss exceeds what you expected.

Limitations and Caveats

Too Few Trades and the Windows Don't Hold Up

The ATLAS backtest has 1193 trades over a 7.5-year validation, so even when split into multiple windows, each window retains a reasonable number of trades. By contrast, the ETHEREUM EA backtest has 154 trades over a 7.5-year validation. Slicing low-frequency logic like this into fine windows leaves very few trades per window, and a window's profit or loss tells you almost nothing. Your options are to make the windows coarser or to treat per-window evaluation as a rough reference only.

The Published Figures Are Full-Period Values

The PF and win rate shown in the Measured Ranking are values across the entire validation period. They are not per-window results, so they can't establish on their own whether overfitting is present. Please read them with that discount in mind.

The Window Design Itself Can Be Optimized

If you vary the window length and count and pick the combination that makes the results look good, that is also overfitting. The premise is that the window design is fixed before running the validation and never adjusted afterward. From the outside, there is almost no way to confirm that this premise was honored.

The Future Is Still Unknown

Passing walk-forward does not mean the logic can handle markets that have never occurred before. In our own forward tests on demo accounts, several EAs are currently in negative territory. These are demo accounts, not live-account results, and the time since launch is short, so neither good nor bad results are at a stage where a conclusion can be drawn. Validation methods are tools for reducing misses; they don't promise hits.

Summary

• A fixed IS/OOS split is swayed by the split point and by how many times the OOS gets revisited
• Walk-forward creates multiple validation segments and checks them while moving forward
• Look at profit and loss per window, not the total, and confirm profit concentration and parameter stability
• The fewer trades the logic has, the less stable the judgment becomes as you split into more windows
• Fix the window design in advance; even passing tells you nothing certain about future performance

We have also collected our validation approach in the Knowledge Base.