Plumbline · Service sheet

Independent Backtest Audit

How far can a backtest be trusted? We answer with a fixed set of checks, decided in advance and each one verifiable, and give a clear verdict: ready, conditional, or not ready.

Draft: pricing and legal terms to be confirmed

中文版

Who it is for

AllocatorsFamily offices, funds of funds, seed investors and angel investors assessing the backtests a quant manager shows them. We work for you, and the complete report goes to you.
Quant teams asked by an investor to be auditedThe investor commissions the audit. You supply the data using our templates, and the complete report goes to the investor, not only the parts that pass.

What we check

Look-aheadWhether the backtest used information it could not yet have had; from weights and returns, without the code
Trial countHow many variants were tried before this one was picked, and how much confidence is left once the true count is applied
Performance by eraWhether the result holds in a period that played no part in the selection
Rebalance phaseHow much the result moves with a different rebalance start date, and whether the reported one is a lucky draw
Concentration in timeWhether one or two years, or a few days, carry the return
Cost stressWhat is left with costs doubled
Market exposureThe realized beta, and what is left once the market is removed
Forward recordWhether performance since going live is consistent with the backtest, and whether there is enough evidence to decide
Pre-registrationWhether the strategy spec was frozen before the out-of-sample period began
Data integrityDuplicates, gaps, missing values and outliers

What you receive

Ready: no problems foundConditional: more evidence or a restated number neededNot ready: a key check fails

See a sample report (three of our own strategies)

What you provide

We send CSV templates you can fill in with Excel. No source code is needed.

Process

  1. Sign a confidentiality agreement
  2. You submit the data and the request form using the templates
  3. We run the checks, and the auditor reviews each one
  4. We deliver the report and walk you through the verdict, in person or online

Why us

These methods are all public. What we do differently: the lines are fixed in advance and never adjusted after seeing the results; every step can be re-checked; and anything the data does not allow is reported as not done, never glossed over.

We applied this discipline to ourselves first. Our research registry holds 63 registered hypotheses, and 49 of them have been closed by rules fixed in advance.

What we do not do

Terms

Confidentiality
Data is used only for this audit, stored apart from our own research data, and returned or deleted at the end as agreed
Data rights
The client warrants that it is entitled to provide the data it submits and is responsible for its source and licensing; we do not review where the data comes from
Price
To be confirmed (open to discussion during the pilot)
Turnaround
To be confirmed
Contact
About & contact