Focal Harvest is a lightweight Python CLI tool with two primary use-cases, featuring a zero-config local statistical fallback for offline operation:
- Automated Research Pipeline: Query search engines, scrape and clean target pages, and synthesize findings into structured reports using Gemini, Claude, or GPT-4o-mini.
- OSINT Collection Framework: Monitor topics across Reddit, Hacker News, Stack Overflow, arXiv, and 17+ platforms with daemon scheduling, Wayback Machine fallbacks, and Discord/Telegram alert dispatch.
Python • AI Agent • Web Research • OSINT • Multi-Engine • 3-Tier Archive Cascade • Zero-Config Offline Fallback
- 📱 Bidirectional Remote Mobile Bot Listener (Telegram & Discord): Message your Focal Harvest instance directly from your mobile device via Telegram or Discord slash commands (
/research,/deep,/url,/status,/help). Supports single-word and quoted multi-word topics (e.g./research "robotics engineering"), multi-URL deep dives (e.g./url 'https://site1.com', 'https://site2.com'), custom focus areas, User ID authorization whitelisting, compact mobile summaries, and native.pdfand.mdreport uploads. See BOT_LISTENER_GUIDE.md for full setup instructions.
Ensure you have Python 3.12+ installed. Run the platform-independent installer:
python install.py(Or run pip install -r requirements.txt for standard manual installation).
Launch the interactive Terminal User Interface (TUI):
python main.pyFor custom automation scripts, you can run the scraping and synthesis pipeline programmatically in Python:
from scraper import search_duckduckgo, scrape_urls_concurrently
from analyzer import synthesize_topics
# 1. Search for a query
results = search_duckduckgo("Gemini 1.5 Flash vs Pro", max_results=3)
urls = [r["url"] for r in results]
# 2. Scrape target pages concurrently in parallel threads
scraped_data = scrape_urls_concurrently(urls)
# 3. Synthesize the findings into a structured Markdown report
report = synthesize_topics(
scraped_data,
query="Gemini 1.5 Flash vs Pro",
spec_topic="pricing comparisons"
)
print(report)Every developer, researcher, and tech blogger repeats the same tedious manual workflow:
Search Query ➔ Open 20 Browser Tabs ➔ Ignore Cookie Walls & Ads ➔ Copy-Paste Snippets ➔ Synthesize with LLM ➔ Save Report
Focal Harvest automates this entire process inside a lightweight, terminal-based pipeline.
| Metric / Operation | Manual Browser Sweep | Heavy Local LLM (Ollama/Llama3) |  Focal Harvest |
|---|---|---|---|
| Setup Package Size | ~150MB (Chrome) | ~4.7GB (Llama3-8B) | <5.2MB (Zero-Dependency core) |
| Cold Startup Time | ~1.5s | ~12.5s (model load) | <0.15s (TUI instant-boot) |
| Scrape & Clean Speed | ~45s per page | ~1.2s per page (raw HTML) | <0.25s per page (parallel readability) |
| Context Density | High fluff (ads/nav) | Medium (all HTML tags) | High Density (Top 15 key sentences) |
| Resource Usage | ~1.2GB RAM | ~6.5GB RAM + dedicated GPU | <85MB RAM (runs on low-end hardware) |
graph LR
A["Focus Query"] --> B["Concurrent Scraper (utils.py)"]
B --> C["Readability Engine (scraper.py)"]
C --> D["Synthesis Engine (analyzer.py)"]
D --> E["TUI Reports / Webhooks (notifier.py)"]
style A fill:#e1f5fe,stroke:#03a9f4,stroke-width:2px,color:#000000
style E fill:#e8f5e9,stroke:#4caf50,stroke-width:2px,color:#000000
- Orchestrated Search: Resolves search parameters via Tavily or DuckDuckGo.
- Concurrent Fetching: Scrapes pages in parallel threads with Cloudflare bypasses.
- HTML Sanitization: Stubs out ads, widgets, and cookie overlays, keeping only clean prose.
- Adaptive Replenishment: Drops blocked/empty pages and automatically fetches replacements to ensure high-density inputs.
- AI/Local Synthesis: Summarizes content using Gemini, Claude, or OpenAI with automatic key failover. Automatically falls back to an offline PositionRank statistical ranker if no API keys are provided.
- Instant Dispatch: Saves structured Markdown/JSON reports locally and pushes alerts to Discord/Telegram.
Here is a real example of the structured Markdown report generated when researching "Gemini 1.5 Flash vs Gemini 1.5 Pro":
- Remote Mobile Slash Commands: Control your Focal Harvest instance from your mobile device via Telegram or Discord slash commands (
/research,/deep,/url,/status,/help). See BOT_LISTENER_GUIDE.md for full setup. - Strict Whitelist Protection: Enforces user ID authorization whitelisting (
discord_allowed_user_ids&telegram_allowed_user_ids) to block unauthorized access. - Dual Document Delivery: Receives a compact mobile summary preview along with both
.pdfand.mdreport document file uploads.
- Intent-Aware Categorization: AI synthesis automatically classifies queries into specialized 3-section frameworks tailored to the domain:
- Software & Tech: Architecture & System Design ➔ Performance & Benchmarks ➔ Tradeoffs & Best Practices
- Entertainment & Media: Overview & Plot Synopsis ➔ Production & Creative Execution ➔ Critical Reception & Impact
- Business & Finance: Industry Overview & Market ➔ Business Model & Financial Metrics ➔ Risk Factors & Strategic Outlook
- Science & Healthcare: Background & Hypotheses ➔ Methodology & Data Analysis ➔ Practical Applications & Future Scope
- News & History: Context & Timeline ➔ Key Developments & Main Findings ➔ Broader Implications & Impact
- General / Informational: Overview & Core Concepts ➔ Detailed Breakdown ➔ Key Takeaways
- Multi-Engine Cascading: Automatically detects search result shortfalls and cascades across Tavily → DuckDuckGo → Bing HTML → Brave Search → Google Aggregated with URL deduplication.
- 3-Tier Web Archive Cascade: Recovers blocked or deleted pages by cascading through Internet Archive Wayback Machine → Memento Web Federated API (Library of Congress & global archives) → Archive.today.
- HTTP/SOCKS5 Proxy Support: Full proxy configuration with password masking (
***) and Option 16 live diagnostic tester queryingapi.ipify.org. - Optional 403 Bypass (
curl_cffi): Impersonates standard Chrome TLS/JA3 signatures and client hints to bypass Akamai and Cloudflare anti-bot blocks. - Wayback Machine Fallback: Automatically redirects protected SPAs (Reddit, Stack Overflow, Quora) to the Internet Archive to bypass login gates.
- Conditional User-Agent Routing: Automatically swaps to
Discordbot/2.0when scraping guest pages, and shifts to a desktop Chrome UA when user cookies are configured to avoid bot-signature blocks. - Unified Cookie Ingestion: Supports Netscape
config/cookies.txtfiles and universal domain-to-cookie maps to scrape pages behind login walls.
Pre-configured parser modules designed to clean and extract structured layouts:
- HN & Reddit: Captures nested comments threads and recursive tree hierarchies.
- Stack Overflow: Cleans code blocks and identifies accepted answers.
- arXiv & Google Scholar: Extracts academic metadata, abstracts, and authors metrics.
- Finance (Yahoo/SEC EDGAR): Scrapes financial streams and converts data tables to Markdown tables.
- ReadTheDocs & Dev.to: Strips navigation sidebars and headers.
- Multi-Provider Failover: Cascades automatically across API keys (Gemini ➡️ OpenAI ➡️ Claude) before dropping to the offline keyword PositionRanker.
- Live Search Grounding: Integrates Gemini's live Google search grounding to retrieve cited real-time facts directly in the model.
- Local-First Cache: Saves MD5-hashed results in
reports/cache/to skip network requests on recurring topics.
- Styled ReportLab PDF: Exports beautiful PDF documents with custom headers, callout boxes, and cell-normalized tables.
- Microsoft Word (.docx): Generates structured DOCX reports formatted with headers and source tables.
- Markdown & JSON: Generates clean raw Markdown reports and JSON data payloads for programmatic pipelines.
sequenceDiagram
participant User
participant CLI as main.py
participant Scraper as scraper.py
participant AI as analyzer.py
participant Notifier as notifier.py
User->>CLI: Selects Option 1 (Single Scrape)
CLI->>Scraper: Search Query & Crawl Targets
Scraper->>Scraper: Run concurrent thread scrapes
Scraper->>CLI: Return clean list of page contents
CLI->>AI: Synthesize contents (AI or Local)
AI->>CLI: Return structured report
CLI->>Notifier: Export Markdown/JSON & Send Webhooks
Notifier->>User: Renders Report in Terminal + sends Alerts
├── install.py # Platform-independent setup script
├── main.py # Interactive TUI Controller (Version v1.6.0)
├── config_manager.py # Reads and writes config.json
├── scraper.py # Multi-engine search replenishment, 3-tier archive cascade, crawler
├── analyzer.py # Multi-LLM provider wrappers, dynamic category synthesis, and PositionRank
├── notifier.py # Markdown/JSON/PDF/DOCX exporter and Discord/Telegram webhook dispatch
├── utils.py # Proxy manager, URL slug topic extractor, safe_request retries
├── std_plugins/ # Built-in plugins (Hacker News, Reddit, Stack Overflow, YouTube, etc.)
└── tests/ # Isolated unit tests suite (123 unit tests, 100% green)
Focal Harvest adheres to the following core maintainer guardrails:
- Lightweight Footprint: Application remains under 5MB and installs in seconds.
- Strict TUI Only: Runs entirely in the console—no local React dashboards or local REST servers.
- Low Hardware Requirements: Runs smoothly on low-spec student laptops.
- No Local AI Downloads: Uses remote APIs (Gemini/OpenAI/Claude) to bypass multi-gigabyte Ollama/Llama downloads.
- No SQL Databases: Stores reports in plain JSON and Markdown files for absolute filesystem transparency.
Focal Harvest is a command-line utility for personal and research purposes.
- Public Scraping: Programmatic scraping of public data is legally protected under established U.S. case law. Respect target servers by running crawls with polite rates.
- Authenticated Scraping: Do not use active personal accounts with session cookies. Use burner/dedicated accounts exclusively to prevent account suspensions.
Distributed under the MIT License. See LICENSE for details.