Tutorial 09: Strategy Parameter Optimization & Auditing¶
Overview
| Item | Description |
|---|---|
| Goal | Manage tunable parameters with PARAMS / PARAM_RANGES; search with optimizer.py, validate on rolling out-of-sample windows with walk_forward / walk_forward_analysis, and audit overfitting and bias with validate_backtest |
| Est. Time | ~60 minutes |
| Prerequisites | Tutorial 00, Tutorial 02 (backtesting & metrics), Tutorial 03 (manual optimization); for metric definitions see Report Format Reference |
This chapter explains how to parameterize strategies using PARAMS / PARAM_RANGES, combined with the in-repo
agent/optimizer.py(rule-based search reference),agent/audit_log.py(audit logging), and eqlib's built-in rolling-validation and scientific-validation APIs to build a reproducible parameter tuning and auditing workflow. You can also write your own Python scripts or orchestrate the same steps in any IDE; EasyQuant is not tied to any specific editor or commercial AI product.
Table of Contents¶
- Why Parameterization and Auditing
- Core Conventions
- Command Line: optimizer.py
- Rolling Out-of-Sample Validation: walk_forward
- Overfitting Detection: walk_forward_analysis
- One-Stop Scientific Validation: validate_backtest & ValidationConfig
- ML Hyperparameter Search: optimize_hyperparams
- Building Your Own Optimization Loop (Recommended Approach)
- Audit Logs
- Code Review Checklist (Recommended)
- Further Reading
1. Why Parameterization and Auditing¶
| Pain Point | How Parameterization + Auditing Helps |
|---|---|
| Tuning by gut feel | PARAM_RANGES defines clear search boundaries; results are comparable |
| Decisions not traceable | audit_log/ records metrics and change rationale for every iteration |
| Great in-sample, poor out-of-sample | walk_forward / walk_forward_analysis quantify decay over rolling OOS windows |
| Rule-based scripts too rigid | Run optimizer.py for a baseline first, then write custom diagnostic logic as needed |
2. Core Conventions¶
2.1 Relevant Repository Files¶
| Path | Purpose |
|---|---|
agent/strategy_template.py |
Copy-ready parameterized strategy skeleton |
agent/optimizer.py |
Rule-based parameter search (reference implementation, independently runnable) |
agent/audit_log.py |
Writes JSONL + Markdown audit entries |
2.2 PARAMS and PARAM_RANGES¶
Strategies should define two module-level dictionaries: current values and search space. initialize must read from PARAMS; hardcoded magic numbers are prohibited.
PARAMS = {
"fast_period": 5,
"slow_period": 20,
"stop_loss_pct": 0.08,
"position_pct": 1.0,
"vol_confirm_mul": 1.5,
}
PARAM_RANGES = {
"fast_period": (2, 15, 1),
"slow_period": (10, 60, 5),
"stop_loss_pct": (0.03, 0.15, 0.01),
"position_pct": (0.3, 1.0, 0.1),
"vol_confirm_mul": (1.0, 3.0, 0.25),
}
def initialize(context):
g.fast_period = PARAMS["fast_period"]
g.slow_period = PARAMS["slow_period"]
# ...
2.3 Common Acceptance Criteria (Examples)¶
- Multi-metric thresholds: Sharpe, max drawdown, annual return, win rate
- Cover at least 2 calendar years or an equivalent trading-day span
- Out-of-sample Sharpe stays above a chosen fraction of the in-sample Sharpe (see §4-5)
Set concrete thresholds according to your strategy's risk profile.
3. Command Line: optimizer.py¶
From the repository root (with pip install -e . done), run the reference optimizer, for example:
python agent/optimizer.py \
--strategy agent/strategy_template.py \
--min-sharpe 1.0 \
--max-drawdown 0.20 \
--periods "2022-01-01:2022-12-31" "2023-01-01:2023-12-31"
It demonstrates programmatic search; for more complex diagnostics (e.g., structural strategy changes), call run_backtest, analyze_returns, and other APIs directly in your own scripts or notebooks.
4. Rolling Out-of-Sample Validation: walk_forward¶
A single "tune on the first 70%, validate on the last 30%" split gives you exactly one out-of-sample verdict. eqlib.wfa.walk_forward() slices the full range into alternating, rolling in-sample (IS) / out-of-sample (OOS) windows: each window selects parameters on its IS segment, runs an independent OOS backtest with those parameters, and finally stitches all OOS segments into one equity curve.
from eqlib import walk_forward, analyze_returns
def make_initialize(**params):
"""Parameter factory: params chosen by optimize_fn arrive as keywords."""
merged = {**PARAMS, **params}
def initialize(context):
g.fast_period = merged["fast_period"]
g.slow_period = merged["slow_period"]
g.stop_loss_pct = merged["stop_loss_pct"]
# ... rest of initialization ...
return initialize
def optimize_fn(train_result):
"""IS-segment parameter selection: the returned dict goes to make_initialize. Return {} to keep defaults."""
metrics = analyze_returns(train_result)
if metrics["sharpe_ratio"] < 1.0:
return {"slow_period": 30} # example: slower MA in weak windows
return {}
result = walk_forward(
make_initialize,
optimize_fn=optimize_fn, # pass None to use default params everywhere
start_date="2020-01-01",
end_date="2024-12-31",
train_months=12, # each IS window is 12 months
test_months=3, # each OOS window is 3 months
step_months=3, # step == test window -> non-overlapping OOS
starting_cash=100_000,
)
print(result.summary) # aggregated statistics
result.oos_equity.plot() # stitched OOS equity curve
The three WFAResult attributes:
| Attribute | Type | Description |
|---|---|---|
windows |
list[dict] |
Per-window train_start/end, test_start/end, params, oos_result, oos_metrics |
oos_equity |
pd.Series |
Stitched OOS equity curve (per-window normalized, then chained) |
summary |
dict |
n_windows, profitable_pct (share of profitable OOS windows), avg_oos_sharpe, total_oos_return, oos_max_drawdown, etc. |
How to read it: if profitable_pct is far below what the full-sample win rate suggested, or oos_equity drifts sideways for years, the strategy is regime-sensitive. summary["avg_oos_sharpe"] is the Sharpe you may quote — the full-sample Sharpe embeds data you tuned on.
5. Overfitting Detection: walk_forward_analysis¶
The eqlib.scientific layer's walk_forward_analysis() is lighter-weight: it does not select parameters on the IS segment. It runs fixed-parameter IS and OOS backtests per window and quantifies out-of-sample decay.
from eqlib.scientific import walk_forward_analysis
wfa = walk_forward_analysis(
initialize, # initialize callable (recommended); a run_backtest result dict also works
param_ranges=None, # reserved; currently only recorded in window metadata
train_window="2Y",
test_window="6M",
step="6M",
start_date="2020-01-01",
end_date="2024-12-31",
)
print("OOS/IS Sharpe ratio:", round(wfa.oos_is_ratio, 2))
print("OOS decay:", wfa.is_sharpe_decay)
for w in wfa.windows:
print(w["train_start"], "->", w["test_end"],
"IS sharpe:", round(w["is_metrics"]["sharpe"], 2),
"OOS sharpe:", round(w["oos_metrics"]["sharpe"], 2))
| Return value | Type | Description |
|---|---|---|
windows |
list[dict] |
Per-window train_start/end, test_start/end, is_metrics, oos_metrics (each with sharpe / annual_return / max_drawdown) |
oos_is_ratio |
float |
Mean OOS Sharpe / mean IS Sharpe; well below 1 means out-of-sample decay |
is_sharpe_decay |
bool |
Whether mean OOS Sharpe is below mean IS Sharpe |
Callable or dict
Passing an initialize callable runs every IS / OOS segment as an independent backtest from the initial cash — a true out-of-sample measurement. Passing an existing backtest result dict only slices the equity curve by date, so OOS segments inherit the IS segment's cumulative equity — fast, but not strictly OOS.
Parameter perturbation
walk_forward_analysis currently does not perturb strategy parameters (param_ranges is metadata-only). For parameter sensitivity, combine it with parameter_sensitivity / out_of_sample_test from eqlib.scientific.
6. One-Stop Scientific Validation: validate_backtest & ValidationConfig¶
validate_backtest() chains bias detection (look-ahead / survivorship / selection / data), statistical confidence (bootstrap, Monte Carlo, significance), overfitting detection, and extended risk metrics into one pipeline over a single backtest result:
from eqlib import run_backtest, ValidationConfig
from eqlib.scientific import validate_backtest
result = run_backtest(initialize, "2022-01-01", "2024-12-31")
config = ValidationConfig(
n_bootstrap=1000, # bootstrap resamples
n_monte_carlo=500, # Monte-Carlo simulations
significance_level=0.05, # p-value threshold for significance tests
perturbation_pct=0.10, # parameter perturbation size
random_state=42, # seed for reproducibility
)
validation = validate_backtest(result, config=config)
print(validation.summary()) # overall trust rating + per-module summary
print(validation.trust_rating)
To skip a module, switch it off in ValidationConfig: ValidationConfig(overfitting=False, statistics=False) runs only bias and risk checks. The full parameter list is in the Scientific Validation API.
7. ML Hyperparameter Search: optimize_hyperparams¶
If your strategy uses MLSelector (see Tutorial 11), do not tune model hyperparameters with random-split cross-validation — time series leak. optimize_hyperparams() runs a grid search with TimeSeriesSplit:
from eqlib.ml.tuning import optimize_hyperparams
best = optimize_hyperparams(
pipeline, # FeaturePipeline instance
model_type="random_forest",
X=X_train, # feature matrix
y=y_train, # labels
param_grid={"n_estimators": [50, 100, 200], "max_depth": [3, 5, 8]},
cv_method="time_series_split",
n_splits=5,
scoring="roc_auc",
)
With fewer than n_splits * 2 rows it returns an empty dict and falls back to default parameters. For one-line tuning inside a strategy, use auto_tune_selector (pulls data straight from the selector's pipeline); see the ML API reference.
8. Building Your Own Optimization Loop (Recommended Approach)¶
Baseline backtest (run_backtest / run_strategy)
|
v
analyze_returns(result) and other metrics
|
v
If below target: adjust PARAMS by rule or judgment (stay within PARAM_RANGES)
|
v
Out-of-sample review: walk_forward or walk_forward_analysis (Sections 4-5)
|
v
(Optional) Full validate_backtest audit (Section 6)
|
v
Re-run backtest -> write summary into audit_log (if using audit_log.py)
|
v
Stop condition or max iterations reached
The implementation is entirely up to you: Shell + Python, Makefile, CI jobs all work. One discipline matters: any parameter informed by full-sample data must prove itself again on OOS windows.
9. Audit Logs¶
If you use agent/audit_log.py, each session produces:
audit_log/
├── session_<timestamp>.jsonl # machine-readable
└── session_<timestamp>.md # Markdown summary
jq examples:
jq 'select(.type=="adjustment")' audit_log/session_*.jsonl
jq 'select(.type=="final")' audit_log/session_*.jsonl
Log the key numbers from Sections 4-6 (oos_is_ratio, profitable_pct, trust_rating) into the audit entries as well — they say more about strategy quality than any single full-sample metric.
10. Code Review Checklist (Recommended)¶
After each change to PARAMS or strategy logic, check at least:
- New values fall within
PARAM_RANGESand satisfy cross-constraints (e.g.,fast_period < slow_period). - Tuned parameters are read via
PARAMS["..."], not hardcoded. - No future data, no misuse of same-day closing prices in signals (avoid look-ahead); when in doubt, run
validate_backtestand read the bias report. - Backtest range and data source are consistent; anomalous trading (suspensions, etc.) is handled.
- New parameters pass at least one rolling OOS check (
profitable_pctfromwalk_forward,oos_is_ratiofromwalk_forward_analysis) — not just the full-sample Sharpe.
11. Further Reading¶
- Tutorial 03: Strategy Optimization & Improvement — manual analysis and improvement
- Tutorial 08: All-Weather Alpha Combined Strategy — multi-module combined example
- Tutorial 11: Machine Learning Stock Selection — MLSelector and feature pipelines
- Run a Backtest · Read Reports
- Scientific & Walk-Forward Validation API · ML API Reference
- API Reference — Lifecycle & Attribution