The problem Firecrawl actually solves
Most AI tools list their features on SPA pages built with React or Next.js. A normal HTTP request returns an empty div and a JavaScript bundle. Building a Puppeteer scraper for 300 different sites means handling 300 different DOM structures, dealing with rate limiting, managing proxies, and parsing inconsistent HTML. Firecrawl does all of this in one API call: you pass a URL and get back clean markdown with the rendered page content, metadata, and even screenshots if you want them.
Real cost: cheaper than the server to run Puppeteer
I scraped 300 AI tool pages for saas.pet. Total cost: $8.40. That is roughly 350 API calls (some retries) at $0.002 per page. A cloud server capable of running headless Chrome costs $20/month minimum, plus you spend 2-3 days writing handlers for each site's DOM structure. Firecrawl's free tier gives 500 credits/month which is enough for small projects. The paid tier starts at $19/month for 3,000 credits. For comparison, just running a Hetzner instance with Chrome installed costs $5/month and burns 2GB RAM.
The structured extraction feature is underrated
Beyond markdown conversion, Firecrawl has an endpoint that extracts structured JSON using a schema you define. I sent it 'extract the tool name, pricing model, and key features as a list' and it returned valid JSON for 280 out of 300 pages. The 20 failures were sites with aggressive Cloudflare protection or pages that loaded content via infinite scroll. For consistent site structures, the extraction is 95% accurate. For varied sites like random AI tool landing pages, expect 85-90% accuracy.
Where it fails and what to do about it
Firecrawl times out on sites that take longer than 30 seconds to load. It cannot handle login-gated content without cookie injection. Infinite scroll pages only capture the first viewport worth of content. For these edge cases, I built a fallback: if Firecrawl returns a timeout, my pipeline falls back to a simple HTTP fetch with axios and parses whatever HTML it gets. For the 20 failures out of 300, the fallback captured enough content to at least get the tool name and description.
Firecrawl vs building your own scraper
Building a Puppeteer scraper for 300 sites: 3 days of development, $20/month server, ongoing maintenance when sites change their DOM, proxy costs if you get IP-banned. Firecrawl: 30 lines of code, $8 total, zero maintenance. The tradeoff is control. Firecrawl's markdown conversion sometimes loses formatting (tables become plain text, code blocks lose syntax highlighting). If you need pixel-perfect HTML preservation, roll your own. If you need usable text for AI ingestion, Firecrawl is the right abstraction level.
