Reduce backtest overfitting: organize out-of-sample and walk-forward research with FiClaw
Separate parameter selection from validation, define chronological windows and use FiClaw to organize strategy specifications, backtests and review evidence.
Published by FiClaw 产品团队 · Published · Updated
阅读中文版Quick answer
Separate parameter selection from evaluation in time: choose rules on earlier data and test them on later data that did not guide the choice. Walk-forward research repeats this sequence and retains each result. FiClaw can assist with specifications, Python, backtests and parameter comparisons; researchers must define the windows, selection rule and review decision.
For researchers moving beyond an initial backtest to parameter selection and a reviewable validation process.
Example validation request: agree on the process before reading results
Create a validation plan for this strategy. List fixed rules, candidate parameters and the selection criterion. Use earlier data for selection and the subsequent window for evaluation. Record dates, data version, chosen settings, costs, trade count and drawdown. Check whether signals or holding periods overlap the boundary. Do not revise this round's parameters using its validation result.
Why the best historical setting may be fragile
Searching many rules, factors and settings on one history creates more opportunities to fit that particular sample. Keep discarded experiments as well as the winning run so a reviewer can see how the choice was made.
Agree on the selection criterion first, including costs, drawdown, trade count and neighboring settings where relevant. Changing the criterion after inspecting validation results makes that data part of the selection process.
Plan chronological validation windows
An out-of-sample test uses observations that did not guide selection. Walk-forward research repeats selection on an earlier window and evaluation on the next one. Specify whether the training window expands or keeps a fixed length.
Preserve time order. Check signals, labels and holding periods that span the split; exclude overlapping observations or leave a gap when needed. Its length depends on the information used by the experiment, rather than a universal number of days.
Keep evidence within the FiClaw research workflow
Put dates, candidates and the selection rule in the specification before inspecting code, running historical backtests and comparing parameters. This is an experiment design, not a claim that FiClaw automates every walk-forward window. Confirm execution options against the current workstation and interfaces.
Inspect each window before combining non-overlapping validation segments. Explain differences in costs, trade counts, drawdown and parameter changes. One successful validation does not establish readiness for deployment or guarantee future returns.
Keep selection, evaluation and conclusions distinct
| Evidence | What to record | Review question |
|---|---|---|
| Selection log | Candidates, attempts, objective and constraints | Were unsuccessful trials retained? |
| Window result | Training/test dates, frozen parameters, costs and metrics | Was the choice made before evaluation? |
| Window summary | Test segments, trade counts, drawdowns and parameter changes | Are overlapping periods counted more than once? |
| Review decision | Failure conditions, missing evidence and next experiment | Are failed validations explained? |
Frequently asked questions
Can I tune the strategy after a weak out-of-sample result?
Yes, but that period then informs selection. Record the change and seek new independent data or subsequent observations rather than calling the reused period an independent test.
Does FiClaw automate a complete walk-forward analysis?
This guide explains validation design and does not promise one-click walk-forward automation. FiClaw assists with specifications, code, backtests and parameter comparisons; confirm scheduling and data requirements against the current version and interfaces.
Does passing validation mean a strategy is ready for live trading?
No. Data timing, costs, execution assumptions, capacity and permissions still need independent review and an authorized decision.
Method references
- scikit-learn: TimeSeriesSplit and its gap parameter
Technical background on chronological splitting; not documentation of FiClaw's internal implementation.