How to review an AI backtest report: a FiClaw research checklist
Review data timing, execution assumptions, costs, risk, parameter selection and reproducibility before relying on a FiClaw AI backtest report.
FiClaw content team · Published · Updated
阅读中文版Quick answer
Review an AI backtest report in this order: strategy rules, data and execution assumptions, return and risk metrics, then parameter selection and reproducibility. FiClaw assists reporting and diagnosis; an attractive equity curve still needs data checks, independent validation and human judgment.
For researchers, team leads and risk reviewers checking FiClaw research output.
Check that the experiment is fully specified
Look for the universe, dates, benchmark, rebalance rules, capital and position constraints, fees and slippage. A strategy name and a final return are not enough to compare experiments or let another researcher reproduce one.
Inspect adjusted prices, suspensions, price limits and unfilled-order handling. For financial factors, check disclosure availability before signal use. For index universes, verify historical membership. Record missing conditions as limitations.
Explain metrics using the underlying trades
Read return, drawdown, Sharpe, turnover, trade count and costs together, with their calculation windows and conventions. Avoid ranking reports generated with different dates, universes or cost assumptions as though they were equivalent.
Inspect holdings and trades to see whether a small number of dates or securities dominate the result. FiClaw diagnosis helps raise questions, but explanations of anomalies still need to be checked against source data and implementation code.
Review parameter selection and reproducibility
Record which parameters and periods were tried, why this version was retained and which alternatives were rejected. If a validation period repeatedly informs changes, it has participated in model selection and is no longer a fully independent check.
Keep the prompt, spec, code version, data conventions, run configuration and review notes together. Public FiClaw example reports illustrate report structure; their figures are not client performance or predictions of future returns.
Trace each report claim back to evidence
| Claim | Evidence to inspect | Review action |
|---|---|---|
| High return | Benchmark, period, costs and concentration | Rerun under consistent assumptions and explain the source |
| Small drawdown | Equity series, exposures and execution rules | Inspect stressed periods and unfilled orders |
| Best parameters | Search space, trial log and validation periods | Compare nearby settings and preserve an untuned period |
| Reusable result | Specification, code, configuration and data conventions | Ask a second researcher to reproduce the run |
Frequently asked questions
Are public example reports investment track records?
No. They illustrate inputs, settings, metrics, diagnosis and review structure. They are not client investment performance and do not promise future returns.
Can I select a strategy using Sharpe or annualized return alone?
One metric is insufficient. Review data and execution assumptions, costs, drawdown, sample periods, parameter selection and independent validation together.
Does a FiClaw report authorize strategy deployment?
A report supports research evaluation. Independent validation, risk review, permissions and operational checks still follow the team's approval process and remain the responsibility of authorized people.