Plumbline · Service sheet

Independent Backtest Audit

How far can a backtest be trusted? We answer with a fixed set of checks, decided in advance and each one verifiable, and give a clear verdict: ready, conditional, or not ready.

Who it is for

Subscribers to stock-pick and signal servicesYou place trades on a stock-pick or signal service's recommendations. Send your account's daily returns and we check what is left once the market is removed, whether one or two years carry the return, and how it compares with what the service publishes. The report goes to you, for your own decisions only, and may not be published or used for marketing.
Independent quant tradersBefore you put your own money into a strategy, check your own backtest: overfitting, costs, look-ahead. The report goes to you, for your own decisions only, and may not be used for marketing, selling signals or raising capital.
AllocatorsFamily offices, funds of funds, seed investors and angel investors assessing the backtests a quant manager shows them. We work for you, and the complete report goes to you.
Quant teams asked by an investor to be auditedThe investor commissions the audit. You supply the data using our templates, and the complete report goes to the investor, not only the parts that pass.

What we check

Look-aheadWhether the backtest used information it could not yet have had; from weights and returns, without the code
Trial countHow many variants were tried before this one was picked, and how much confidence is left once the true count is applied
Performance by eraWhether the result holds in a period that played no part in the selection
Rebalance phaseHow much the result moves with a different rebalance start date, and whether the reported one is a lucky draw
Concentration in timeWhether one or two years, or a few days, carry the return
Cost stressWhat is left with costs doubled
Market exposureThe realized beta, and what is left once the market is removed
Forward recordWhether performance since going live is consistent with the backtest, and whether there is enough evidence to decide
Pre-registrationWhether the strategy spec was frozen before the out-of-sample period began
Data integrityDuplicates, gaps, missing values and outliers

What you receive

Ready: no problems foundConditional: more evidence or a restated number neededNot ready: a key check fails

See a sample report (three of our own strategies)

What you provide

Download the data templates (a zip of CSV files you can fill in with Excel, with instructions). No source code is needed.

Process

  1. Sign a confidentiality agreement
  2. You submit the data and the request form using the templates
  3. We run the checks, and the auditor reviews each one
  4. We deliver the report online

Why us

The lines are fixed before we see your data and are never adjusted afterwards. The checks use published, verifiable methods, and every number in the report can be re-checked, so you are not asked to trust a black box. Anything the data does not allow is reported as not done, never glossed over.

We stay independent: we take no share of an audited strategy's returns, accept no rebates, and do not raise capital for the party being audited.

We applied this discipline to ourselves first. Our research registry holds 63 registered hypotheses, and 49 of them have been closed by rules fixed in advance.

What we do not do

Terms

Confidentiality
Data is used only for this audit, stored apart from our own research data, and returned or deleted at the end as agreed
Data rights
The client warrants that it is entitled to provide the data it submits and is responsible for its source and licensing; we do not review where the data comes from
Price
Pilot pricing: the first audit report for each new client is free; after that, US$200 per report (one report covers one strategy)
Turnaround
The report is delivered within 5 business days of receiving the complete data and request form
Contact
About & contact