L2 Assess · github trending · Aug 8, 2026

firecrawl/firecrawl

defermedium confidence

01 · Fit to Stack

strong fit

Firecrawl ships as a Docker Compose stack (API + worker + Redis + headless browser), which drops cleanly onto the existing Proxmox homelab alongside CT 102/103/104, and it can sit behind the same Cloudflare tunnel pattern already used to expose homelab services. The self-hosted, AGPL path matches the operator's stated preference for free-tier and self-hosted tools over paid APIs.

02 · Fit to Vision

partial fit

It extends two active projects (field-notes enrichment, bud attachment/URL extraction) rather than starting something new, which fits the 'improve what exists' pattern. But it is a resource-heavy service (headless Chromium, Redis, worker process) for a solo, budget-conscious operator building systems meant to run mostly unattended, and today field-notes' only source is GitHub trending, where repo metadata already covers most of what CCR needs.

Implementation Options

01Self-hosted Firecrawl as a new CT

field-notes6-10 hoursmedium risk

Stand up Firecrawl via Docker Compose in a new Proxmox CT, expose it internally, and add a fetch-content step in field-notes' enrichment pipeline that pulls Markdown for a scan item's external_url before the CCR call runs.

+No per-scrape API cost

+Full control over rate limits and retention

+Reusable across field-notes and bud

-Adds a fourth homelab service to patch and monitor (API, worker, Redis, browser)

-Chromium-based rendering is memory-hungry for homelab hardware

-AGPL license needs checking against any future redistribution plans

02Firecrawl Cloud API, opt-in per item

field-notes2-4 hourslow risk

Call the managed Firecrawl API from the field-notes enrichment CCR only when a scan item's source is non-GitHub, gated behind a feature flag so cost stays bounded.

+Fast to integrate, no new infrastructure

+Zero maintenance burden

-Recurring per-scrape cost conflicts with budget-conscious, self-hosted preference

-External dependency and API key to manage

-Usage cost scales with scan volume, which the operator does not yet control

03Skip Firecrawl, use a lightweight fetch+readability step

field-notes3-5 hourslow risk

Add a simple httpx + trafilatura (or readability-lxml) extraction step for the same enrichment goal, without a headless browser or new service.

+Minimal resource footprint, no new CT

+No AGPL or licensing question

+A few hours to bolt onto the existing FastAPI enrichment service

-Fails on JS-rendered pages and behind bot detection, which Firecrawl is specifically built to handle

-Lower-quality Markdown output than Firecrawl's LLM-tuned conversion

01 · Value

Would let field-notes' enrichment CCR read full page content from a scan item's linked URL instead of relying on repo metadata alone, and would let bud extract structured content from URLs referenced in emails (receipts, product pages) rather than parsing email text only.

02 · Why It Matters

Content-fetch quality is a real ceiling on enrichment depth across both field-notes and bud, but right now field-notes' only active source is GitHub trending, where the repo README and metadata already give CCR enough signal - so the marginal value of full-page scraping is speculative until a non-GitHub source or a document-heavy bud case actually exists.

03 · What Implementation Looks Like

In field-notes, a new fetch_content() call in the enrichment service (Python/FastAPI) run before the CCR prompt is built, feeding extracted Markdown into the existing BriefingBody generation; the operator would see richer 'Technical Overview' sections for non-GitHub items. In bud, a similar step ahead of the Claude classifier for URLs found in email bodies. A minimal v1 touches only the enrichment request builder in each project - no schema changes.