CHANGELOG.md

Changelog

All notable changes to AgentQuant will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Added

• Shadow idea-generation sidecar (src/agent/idea_sidecar.py, issue #43), behind AGENTQUANT_IDEA_SIDECAR=1. Non-learning: it replays a fixed heuristic or seeded random policy over the canonical parameter grid and logs a versioned observation/action/reward trajectory. No new dependencies; the default demo never imports it.
• Unified memory layer (src/memory/, design).
  • Every backtested proposal is recorded as a trial, and holdout results are attached as out-of-sample evidence.
  • Reads use a point-in-time as_of cutoff enforced in SQL. Beliefs are evidence-weighted: OOS first, in-sample deflated for the number of configs tried, weighted by regime similarity.
  • Recall returns a token-budgeted MemoryPack used by agent_graph, the swarm and ProposalGenerator. Its snapshot id is written to RunManifest.memory_snapshot_id.
• Dreaming mode: agentquant dream [--watch]. It backfills legacy memory, replays in-sample-only trials on forward data, writes consolidation notes when verdicts change, and prunes the read log. Heartbeat healthcheck: python -m src.memory.healthcheck.
• docker-compose.yml runs the app with the dreamer sidecar on shared volumes.
• memory: section in config.yaml (mode: off | read | read_write, budgets, seeding, dream settings).
• agentquant memory --beliefs | --trials | --prompt | --stats [--as-of DATE].

Changed

• Memory seeds (generation_method="memory_seed") now come only from configs with out-of-sample "works" beliefs. Previously they were re-proposed in-sample winners from AlphaStore.
• reflect_node no longer writes one failure_records row per sub-threshold result per iteration. Failures live on the trial rows.

[0.3.0] — 2026-09-20

First tagged, installable release (issue #30). Ships as a built wheel/sdist rather than only being usable from an editable source checkout.

Install & demo

Not published to PyPI yet -- install the tagged wheel from the GitHub release, or build it yourself from the tag:

git checkout v0.3.0
python -m build                                 # writes dist/agentquant-0.3.0-py3-none-any.whl
pip install dist/agentquant-0.3.0-py3-none-any.whl          # core: research/backtest CLI, no UI/LLM extras
agentquant run --help

# optional extras, installed only if you need them:
pip install "dist/agentquant-0.3.0-py3-none-any.whl[ui]"    # streamlit dashboard + plotting
pip install "dist/agentquant-0.3.0-py3-none-any.whl[llm]"   # Anthropic/OpenAI/Gemini/Tavily providers
pip install "dist/agentquant-0.3.0-py3-none-any.whl[dev]"   # pytest, for running the test suite

# offline demo, no API keys or network required beyond the initial install:
python run_app.py

Installing without [ui]/[llm] and importing src.cli / src.agent.agent_graph / src.agent.runner works with no missing-import errors; requesting --app without the ui extra fails with an actionable message naming the extra to install, not a traceback.

Measured cold-install cost (this environment, scripts/measure_cold_install.py)

Extras	Install time	site-packages size
(none) — core	11.0s	474 MB
[dev]	10.9s	550 MB
[dev,llm]	21.4s	789 MB
[dev,llm,ui]	24.0s	978 MB

These are one machine's snapshot (network-dependent pip resolution time), not a general performance claim — rerun scripts/measure_cold_install.py to reproduce on your own environment.

Verified for this release

• python -m build produces an installable wheel/sdist; a clean venv installing that wheel (no editable install, no source checkout on sys.path) runs agentquant --help, agentquant run --help, and agentquant memory --help successfully from a directory outside the repo.
• 183 tests pass (pytest tests/ -v --cov=src), 0 skipped/xfailed, 69% src coverage (see #21).
• CI's base-install, wheel-install, and harness-verification jobs (.github/workflows/ci.yml) cover this: base-only install with runtime network blocked, wheel install run from outside the checkout, and the 6-epoch harness evolution script re-run + drift-checked (#19, #20) on every push/PR.

Known limitations

• No calibrated numerical Sharpe-forecast-accuracy metric; falsifiable claims are recorded as text only (see README Evidence Table).
• GA/DE harness-parameter optimizers benchmark against a mock fitness function, not real backtests.
• Without ANTHROPIC_API_KEY/TAVILY_API_KEY set, all LLM-tool-calling code paths (proposal generation, harness mutation arms, the new fetch_and_extract_content literature tool) fail closed to offline fallbacks rather than silently degrading — expect flat/zero numbers for those paths without a real key, not a bug.
• use_ensemble and grid_adaptation_strategy harness knobs are declared but not wired through execution yet; a config requesting them is rejected with UnsupportedHarnessKnobError rather than silently ignored.

Agentic Research Loop — 2026-09-12

This release documents the current agentic design and the features added while iterating from a parameter-tuning backtester toward a self-improving research agent.

Agentic design

AgentQuant runs a bounded, evidence-producing loop:

ANALYZE → HYPOTHESIZE → BACKTEST → REFLECT → STORE
              ↑                         │
              └──── retry on failure ──┘

• Analyze computes market features and classifies the current regime from volatility, momentum, trend, and drawdown signals.
• Hypothesize combines LLM reasoning, stored alpha memory, regime-aware grid search, and random fallback proposals.
• Backtest evaluates proposals with warmup enforcement, transaction costs, and performance metrics.
• Reflect applies quality gates, records falsifiable proposal claims, and decides whether another bounded iteration is justified.
• Store persists accepted evidence and reusable negative evidence for later runs.

The loop is intentionally bounded: every proposal has parameters, a generation method, reasoning, confidence, and measurable outcomes. This keeps agentic behavior comparable to deterministic baselines instead of treating free-form LLM output as evidence by itself.

Iteration history

The implementation has evolved through these agentic stages:

1. Core research platform — multi-strategy OHLCV ingestion, feature engineering, realistic-cost backtesting, metrics, and dashboard views.
2. Bounded agent loop — explicit analyze/hypothesize/backtest/reflect/store nodes with retry limits and a single proposal-generation entrypoint.
3. Regime-aware proposals — VIX percentile and momentum regimes feed the prompt and parameter-grid priors.
4. Persistent learning — StrategyMemory, AlphaStore, and NLA memory retain cross-run context and successful candidates.
5. Tool-using orchestration — Claude-compatible tool schemas, optional Tavily research/sentiment search, proposal parsing, and graceful fallback to local generation.
6. Harness evaluation — falsifiable claim recording, benchmark tooling, generalization-gap measurements, and experimental harness evolution.
7. Failure-aware self-improvement — structured failure records now capture regime, strategy, parameters, failure mode, metric gap, and a counterfactual hypothesis; matched failures are injected as proposal constraints.
8. Robustness evaluation — anchored walk-forward utilities report median and worst-window metrics; counterfactual stress tests perturb regimes, volatility, outlier days, and trends.
9. Agent observability and extension points — trace diagnostics expose node counts, proposal methods, improvements, and acceptances; regime transitions are detectable; generated strategy source is AST-validated before registry registration.

Added

• failure_records SQLite table and FailureRecord dataclass in src/research/alpha_store.py.
• Automatic failure persistence from reflect_node.
• Structured failure-memory prompt context in src/agent/proposal_generator.py.
• TraceRecorder.diagnostics() and .diagnostics_json() for harness reports.
• src/backtest/walk_forward.py for anchored walk-forward evaluation.
• src/backtest/stress_test.py for counterfactual strategy stress tests.
• RegimeChangeDetector for recording regime-label transitions.
• src/strategies/codegen.py for AST validation and controlled strategy registration.
• stress_test_strategy in the agent tool registry.

Validation

• 70 repository tests pass after integrating the latest main changes.
• Unsafe generated imports are rejected before registration.
• No force-push or destructive history rewrite was used for this release.

Added — 2026-08-28

Tool Registry & Orchestration System

• src/agent/tools/registry.py — Tool schema definitions and execution engine
  • 5 core tools: regime context, market sentiment, strategy research, parameter recommendations, quality assessment
  • JSON schema generation for Claude API compatibility
  • Tavily API integration for web search (market sentiment, strategy research)
• src/agent/tools/orchestrator.py — Claude tool-use loop orchestrator
  • Multi-turn tool reasoning with Claude
  • Graceful fallback if Claude/Tavily unavailable
  • Proposal parsing from JSON responses
• src/agent/tools/evals.py — Harness quality evaluation suite
  • run_benchmark_to_assess_quality() tool for self-assessment
  • Metrics: OOS Sharpe, stability, generalization gap, tool accuracy
  • Actionable recommendations for harness improvement
• Tool system documentation and integration guide (docs/TOOL_INTEGRATION_GUIDE.md)
• Example integrations showing gradual migration paths

Agent Graph Enhancements

• Enhanced hypothesize_node with tool orchestration
  • Tries Claude tool-use first, falls back to ProposalGenerator
  • No breaking changes; tools activate automatically
  • Records tool calls in execution trace
• Enhanced reflect_node with falsifiable claim scoring
  • Tracks proposal accuracy (predicted vs. realized)
  • Enables harness to learn which proposals work
  • Foundation for harness self-improvement

Proposal Parsing

• JSON proposal parsing from Claude responses
• Integration with existing ProposalValidator
• Support for falsifiable claims in proposal reasoning
• Graceful degradation if parsing fails

Research Agent Design

• Comprehensive design specification for autonomous research agent
• Three-part architecture: Literature → Hypothesis → Publication
• Research loop: Plan → Search → Hypothesize → Validate → Publish
• Data structures for research tasks, findings, hypotheses, reports
• Quality metrics and research rigor scoring
• Integration points with main agent loop
• 4-phase implementation roadmap

API Key Security

• Updated .env.example with TAVILY_API_KEY, ANTHROPIC_API_KEY placeholders
• .env remains in .gitignore; secrets never committed to repo
• All tools degrade gracefully if API keys missing

Changed

• pyproject.toml — Added anthropic>=0.30, tavily-python>=0.3 to LLM extras
• src/agent/agent_graph.py — Integrated tool orchestration and claim scoring
• .env.example — Added web search and Claude API key templates

Git Commits (2026-08-28)

• ac68642 — Proposal parsing and deep research agent design
• 383deec — Integrate tool orchestrator into hypothesize and reflect nodes
• 62344e8 — Tool registry and orchestration for agentic harness evolution

[0.2.0] — 2026-04-15

Added

• Harness Engineering Foundation — Components for self-improving agent architecture
  • src/agent/strategy_memory.py — Cross-session learning via SQLite
  • src/agent/memory_layer.py — Pattern extraction and historical context retrieval
  • src/research/alpha_store.py — Alpha candidate persistence and recall
  • Look-ahead bias guards in backtest engine
  • Regime-aware parameter grid sampling
• Scientific Baselines — Experimental harness for rigorous evaluation
  • experiments/walk_forward_context.py — Walk-forward backtesting with market regime context
  • experiments/walk_forward_context_with_costs.py — Realistic trading costs
  • experiments/ablation_study.py — Parameter sensitivity analysis
  • experiments/rigorous_baselines.py — Budget-matched baseline comparisons
• Trace & Observability — Execution tracing for agent debugging
  • src/agent/trace.py — TraceRecorder and hierarchical trace storage

Changed

• Agent Graph Refactor — Bounded ReAct loop with 5 explicit nodes
  • analyze → hypothesize → backtest → reflect → store
  • Max iterations with reflect-retry loop on Sharpe threshold
• Proposal Generator Consolidation — Single entrypoint replacing 4 planner files
  • Fallback chain: LLM → Grid Search → Random
  • Proposals constrained to canonical ParameterGrid for valid A/B comparison
  • Regime-aware prior sampling
• Regime Detection — VIX percentile-based instead of absolute levels
  • src/features/regime.py — Statistical regime classification
  • Cross-regime performance tracking for harness learning

Fixed

• Look-ahead bias in backtest metric computation
• Parameter grid constraints validation before backtest execution

[0.1.0] — 2026-01-01

Added

• Core Quantitative Research Platform
  • Multi-strategy support (momentum, mean reversion, volatility, trend following, breakout)
  • OHLCV data ingestion from yfinance
  • Feature engineering (technical indicators, regime signals)
  • Strategy backtesting with realistic costs (slippage, commission, market impact)
  • Metrics computation (Sharpe ratio, returns, drawdown, win rate)
• LLM Integration
  • Multi-planner support (Gemini, LangChain, OpenAI, fallback to grid search)
  • Temperature-controlled proposal generation
  • Configurable retry logic
• Market Data & Features
  • VIX, Treasury yields (FRED API integration)
  • Multi-asset universe support
  • Caching layer for data (24h TTL)
• Streamlit Dashboard
  • Live data sidebar with price, VIX, yields
  • Agent execution lab
  • NLA memory browser
  • Research workspace
• Testing & CI/CD
  • 42 unit tests covering core systems
  • GitHub Actions CI gate
  • Config validation, backtest rigor, memory consistency checks

---

Project Milestones

v2.0 Refactor: From Parameter-Tuning Script to Self-Improving Agent (Q2 2026)

• Introduced bounded ReAct loop with formal state machine
• Cross-session memory persistence (StrategyMemory, AlphaStore)
• Regime-aware sampling and statistical regime detection
• Walk-forward validation harness
• Foundation for harness evolution and tool-based orchestration

v1.0 Foundation: Core Research Platform (Q1 2026)

• Multi-strategy backtesting engine
• LLM-guided parameter selection
• Dashboard and monitoring
• Comprehensive test coverage

---

Roadmap

Q3 2026 — Harness Evolution & Self-Improvement

• Tool-calling orchestrator (in progress)
• Weakness mining from execution traces
• Falsifiable claims tracking and scoring
• Grid evolution (learn parameter space from results)
• Prompt template variation and ranking
• Held-out test window evaluation protocol
• Generalization gap measurement

Q4 2026+ — Agentic Research Loops

• Multi-agent swarm with specialist roles (critic, regime analyst, etc.)
• Iterative hypothesis refinement
• Strategy discovery from web research
• Production deployment harness

---

Breaking Changes

None yet. All changes maintain backward compatibility within v0.x → v1.x → v2.x progression.

---

Contributing

See DESIGN.md for architecture overview and implementation guidelines.

---

References

• DESIGN.md — Technical design document (v2.0)
• EXPERIMENTAL_DETAILS.md — Original experimental methodology
• docs/PAPER_DRAFT.md — Research paper draft