Firecrawl, web pages as markdown for LLMs

Firecrawl, web pages as markdown for LLMs

Firecrawl fetches a web page, renders it in a headless browser, and returns clean markdown instead of raw HTML. The firecrawl/firecrawl repository holds roughly 170 thousand stars, the server code sits under AGPL-3.0, and the official SDKs under MIT. That licence split is the first thing to check before a commercial rollout, because it applies to entirely different layers of the system.

What Firecrawl actually does

The task sounds trivial: take an address, return text. In practice several layers sit between the two, each has to be handled separately, and together they decide whether a homegrown solution takes a week or half a year.

The first layer is fetching. A large share of modern sites returns an almost empty HTML document at the address, with a script building the content in the browser afterwards. A plain HTTP request then receives a skeleton without a single sentence. Firecrawl runs the page in a browser driven by Playwright by default, waits for rendering, and only then reads the document tree.

The second layer is cleaning. A rendered document contains navigation menus, a footer, a cookie consent banner, a related articles section, and a dozen other blocks that are pure noise to a language model. The onlyMainContent option, enabled by default, tries to keep the body of the content alone. For manual tuning there are includeTags and excludeTags, both accepting CSS selectors.

The third layer is conversion. The cleaned document tree becomes markdown in which headings, lists, tables, and code blocks keep their structure. A language model handles that format noticeably better than HTML, and the same text costs fewer tokens along the way, since class attributes and identifiers disappear.

What Firecrawl does not do matters just as much. It does not split text into chunks, does not compute vectors, and stores nothing long term. The result has to be cut up and placed into a vector store yourself, for instance Chroma or Pinecone. It is also not a library running inside your process. It is an HTTP service, and the JavaScript and Python SDKs are thin clients that assemble a request and unpack a response.

The AGPL-3.0 licence and what follows from it

This is the most important section here, because it is where most rollouts that skipped the check end up stuck.

The LICENSE file at the repository root carries the full text of the GNU Affero General Public License version 3, with a copyright notice for Sideguide Technologies Inc. There is no additional clause, no Commons Clause, and no competition restriction. It is plain, unmodified AGPL-3.0, and the GitHub interface recognises it correctly, reporting agpl-3.0.

The README states it more precisely: the project is primarily licensed under AGPL-3.0, while the SDKs and some interface components are under MIT. That split can be verified in the code. The apps/js-sdk and apps/python-sdk directories carry their own LICENSE files with MIT text. The apps/api directory, which is the server, has no file of its own and therefore falls under the root licence.

Three independent sources agree for the SDKs alone, which with projects like this is not the rule. The firecrawl package at version 4.34.1 declares a license field of MIT in the npm registry, and after downloading and unpacking the archive, package/LICENSE genuinely contains MIT text with the Sideguide Technologies Inc. notice. The Python counterpart, firecrawl-py at version 4.37.1, also declares MIT in its PyPI metadata. The firecrawl-mcp package ships under MIT, and the firecrawl-cli command line tool under ISC.

The practical conclusion runs as follows. If you use the cloud service and call it through an SDK, the code you write touches the MIT layer only and there is no issue here. If instead you stand the server up yourself, you enter AGPL-3.0 territory, and its section thirteen covers something ordinary GPL does not: a user interacting with the program over a network has the right to receive the source of the version they connect to, if it has been modified.

There is no exception for self-hosting. Neither the licence file nor the documentation grants an additional permission that would lift that obligation. Nor is there a publicly announced dual licence you can buy for your own deployment. The commercial path free of copyleft exposure remains the cloud service or an individually negotiated Enterprise agreement.

A separate and unsettled question is where the boundary of a derivative work lies when your application talks to a Firecrawl server purely over HTTP and runs in a separate container. Some legal teams treat that separation as sufficient, others do not. This is not legal advice, only a description of the state of affairs: if you plan to self-host inside a commercial product, that call belongs to a lawyer rather than to the engineering team.

The endpoints and how they differ

The version two interface exposes several operations whose names get confused, and each carries a different cost.

The scrape operation fetches one address and returns a document in the requested formats. The crawl operation discovers subpages and fetches all of them, working asynchronously: you receive a job identifier and poll for status. The map operation returns a list of addresses only, with no content, which is the cheapest way to see the size of a site before launching the real fetch.

The search operation combines web search with fetching the results, accepting the sources web, news, and images plus categories such as github or pdf. The parse operation processes an uploaded file instead of an address. On top of those come batch scrape for a list of addresses, extract and agent for pulling data described in a sentence, monitor for checking pages on a schedule, and interact plus browser for browser sessions.

The distinction between map and crawl is worth committing to memory, because the mistake is expensive. Mapping counts as one credit per call, while fetching counts as one credit per page. Launching crawl on a large site purely to learn how many subpages it has can eat an entire monthly allowance in minutes.

Installation and the first call

The JavaScript client requires Node 22 or newer, which is stated outright in the package engines field. The Python client declares compatibility from 3.8 upward and pulls in httpx, requests, websockets, and pydantic version 2 or later.

# the Python client
pip install firecrawl-py

# the JavaScript client, short name or historical name
npm install firecrawl
npm install @mendable/firecrawl-js

# the command line tool, handy for a quick check
npm install -g firecrawl-cli
firecrawl scrape https://example.com

# key read automatically by both SDKs
export FIRECRAWL_API_KEY=fc-your-key

Both npm packages, firecrawl and @mendable/firecrawl-js, are published at the same version and carry identical contents. Newer code uses the short name, older guides the name carrying the organisation prefix.

The first call in Python looks like this. Note the field names: the Python SDK uses underscore notation, while its JavaScript counterpart uses the same names written in camel case.

from firecrawl import Firecrawl

app = Firecrawl(api_key="fc-your-key")

doc = app.scrape(
    "https://docs.firecrawl.dev",
    formats=["markdown", "links"],
    only_main_content=True,
    exclude_tags=["nav", "footer", ".cookie-banner"],
    wait_for=2000,
    timeout=60000,
    max_age=3600000,
)

print(doc.markdown[:500])
print(len(doc.links))

The max_age field states, in milliseconds, how old a previously stored version of a page may be for it to be returned without fetching again. Setting zero forces a fresh fetch. That single field can cut both the bill and the response time in jobs that visit the same addresses repeatedly.

Output formats and structured data

The formats field accepts a list, so a single request can return several representations at once. Available values include markdown, html, rawHtml, links, images, screenshot, summary, changeTracking, json, and attributes.

The most interesting one is json, which returns data matched to a schema instead of text. You supply the schema as a JSON Schema object or as a Pydantic model, or alternatively describe the need in a sentence through the prompt field. That convenience carries a price: the json format costs four credits more per page than a plain fetch.

from pydantic import BaseModel
from firecrawl import Firecrawl

class Posting(BaseModel):
    title: str
    company: str
    location: str
    remote: bool

app = Firecrawl(api_key="fc-your-key")

result = app.scrape(
    "https://example.com/jobs/12345",
    formats=[{"type": "json", "schema": Posting}],
    only_main_content=False,
)

print(result.json["title"], result.json["company"])

For pages that reveal content only after interaction there is the actions field. It accepts a list of steps performed in the browser before the document is read, and the available types are wait, click, write, press, scroll, screenshot, executeJavascript, pdf, and scrape. That covers modal windows, buttons expanding text, and pagination hidden behind a control.

The fetching layer has a few more fields that turn out to be needed in production. The proxy field accepts basic, stealth, enhanced, and auto, controlling how requests leave the network. The blockAds field cuts off advertising scripts, removeBase64Images strips images embedded in the source, and location with its country and languages subfields sets the region the page should be seen from.

Crawling and cost control

Fetching a whole site is the operation that most easily slips out of control, which is why the scope-limiting options belong in the first run rather than after the first invoice.

import { Firecrawl } from 'firecrawl'

const app = new Firecrawl({
  apiKey: process.env.FIRECRAWL_API_KEY,
  maxRetries: 3,
  timeoutMs: 120000
})

const job = await app.crawl('https://docs.example.com', {
  limit: 500,
  maxDiscoveryDepth: 3,
  includePaths: ['^/docs/.*'],
  excludePaths: ['^/docs/changelog/.*', '^/docs/.*\\.pdf$'],
  sitemap: 'include',
  crawlEntireDomain: false,
  allowSubdomains: false,
  ignoreQueryParameters: true,
  deduplicateSimilarURLs: true,
  delay: 1,
  maxConcurrency: 5,
  scrapeOptions: {
    formats: ['markdown'],
    onlyMainContent: true,
    blockAds: true
  }
})

console.log(job.status, job.completed, '/', job.total, 'credits:', job.creditsUsed)

The limit field is the only hard brake and should always be set. The maxDiscoveryDepth field caps how deep new addresses may be discovered, counting from the starting page. The includePaths and excludePaths fields accept regular expressions matched against the path by default rather than the full address, unless you set regexOnFullURL.

The sitemap field accepts skip, include, and only. The only value is often the best choice for documentation, because it takes addresses from the sitemap alone and skips link-based discovery, which removes accidental descents into archives and filters.

The delay and maxConcurrency fields decide how hard you hit the server on the other side. Hammering somebody else's site with fifty parallel connections is the behaviour that lands you on a blocklist, and occasionally in a legal department's inbox. Firecrawl respects robots.txt directives by default, and the ignoreRobotsTxt option lets you turn that off. The mere availability of the switch does not mean using it is appropriate, and the project's own documentation notes that responsibility for complying with the policies of visited sites rests with the user.

For long jobs there is a watcher object that emits events carrying documents as they arrive, instead of forcing a status polling loop. The alternative is the webhook field, which posts notifications to an address you supply.

Firecrawl inside a RAG pipeline

Firecrawl is rarely the last piece of the puzzle. In a typical pipeline it fetches content, something else splits it into chunks, and something else again computes vectors and stores them.

The LangChain integration lives in a separate package, langchain-firecrawl, released under MIT. It provides FirecrawlLoader with a mode field accepting scrape, crawl, map, extract, and search, plus a full set of agent tools: FirecrawlScrape, FirecrawlCrawl, FirecrawlMap, FirecrawlExtract, and FirecrawlSearch.

from langchain_firecrawl import FirecrawlLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter

loader = FirecrawlLoader(
    url="https://docs.example.com",
    mode="crawl",
    params={"limit": 200, "scrapeOptions": {"formats": ["markdown"]}},
)

documents = loader.load()

splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=150)
chunks = splitter.split_documents(documents)

print(len(documents), "pages ->", len(chunks), "chunks")

On the LlamaIndex side the counterpart is FireCrawlWebReader from the llama-index-readers-web package, accepting api_key, an optional api_url for your own deployment, and the same working modes. The package requires firecrawl-py at version 4.3.3 or newer.

If you are building a pipeline where some sources are web pages and others are office documents and PDFs sitting on disk, splitting the work usually makes more sense. Firecrawl takes the network, while Unstructured breaks down local files. A ready-made solution with an interface and a store included comes from RAGFlow, which you can feed with Firecrawl output instead of building your own fetching layer.

Self-hosting: what you get and what you do not

The project ships a docker-compose.yaml file and documentation pinned to a specific release. Getting to the first successful fetch takes a quarter of an hour, provided you do not try to enable everything at once.

git clone https://github.com/firecrawl/firecrawl.git
cd firecrawl
git checkout v2.11.162

cat > .env <<'EOF'
USE_DB_AUTHENTICATION=false
POSTGRES_USER=postgres
POSTGRES_PASSWORD=replace-with-at-least-32-random-characters
POSTGRES_DB=postgres
EOF

docker compose up --build -d

# a heartbeat only, it checks neither the queue nor the browser
curl --fail --silent --max-time 5 http://localhost:3002/v0/health/readiness

# the real test: one genuine fetch
curl --fail-with-body --silent --max-time 75 \
  -X POST http://localhost:3002/v2/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","formats":["markdown"],"timeout":60000}'

The stack runs the API server along with worker processes, Playwright, Redis, RabbitMQ, and PostgreSQL acting as the queue. Only port 3002 is published outward by default. The NUQ_BACKEND variable switches the queue to FoundationDB, and BULL_AUTH_KEY unlocks the queue administration panel, but both should be left alone on a first run.

Now the part you need before promising anything to the team. The documentation states outright what the default stack does not support. Screenshots and the actions field do not work, because both available fetching paths report no support and require a separate Fire-engine service that is not in the repository. The agent, browser, and interact capabilities, along with specialised formats such as product, menu, audio, and video, are cloud only. Formats backed by a language model, json included, require connecting your own OpenAI-compatible provider or a local Ollama.

Operational matters come on top of that. The default configuration starts with API authentication switched off, and the Compose file defines no durable volumes for PostgreSQL, Redis, or RabbitMQ. That means data disappears when a container is replaced. Exposing such a deployment beyond a trusted network without your own authentication layer and TLS termination is an invitation to trouble.

Self-hosting makes sense when you need control over the code or over where data goes. It makes no sense as a way to dodge the bill, because machine cost plus team time on this stack easily exceeds the subscription while the feature set is smaller.

Cloud pricing and the credit model

Billing runs on credits rather than requests, so two calls with the same request count can cost very differently.

The free plan gives a thousand credits a month, two parallel jobs, and asks for no card. The Hobby plan costs 19 dollars a month or 16 dollars billed annually, for five thousand credits and five parallel jobs. The Standard plan costs 99 dollars a month or 83 dollars annually for a hundred thousand credits. The Growth plan costs 399 dollars a month or 333 dollars annually for five hundred thousand credits. The Scale plan costs 749 dollars a month or 599 dollars annually for a million credits, with a 397 dollar surcharge for each additional 350 thousand. The Enterprise plan is priced individually.

The credit conversion is straightforward. Fetching a page, fetching a page inside a crawl, and one monitor check each cost one credit per page. Mapping is one credit per call. Search is two credits per ten results. The json format adds four credits to every page. A browser session costs two credits per minute. Queries against the research paper index are free, and agent mode is in preview with five free runs a day and pricing driven by tokens consumed.

Three things are worth checking before deciding. Credits do not roll over to the next month on self-serve plans; rollover starts at Scale and Enterprise. There is no pay-per-use billing without a subscription. Failed requests are not charged.

There is also a discrepancy in the concurrency figures that is better cleared up before signing anything. The pricing page lists twenty five parallel requests for Standard and fifty for Growth. The rate limits documentation, at the same moment, lists fifty and one hundred concurrent browsers for those same plans. For Free and Hobby the two sources agree, at two and five. If concurrency is a critical parameter for you, ask for confirmation in writing.

Request rate limits operate separately. On the free plan that means ten scrape calls, ten map calls, ten search calls, and only two crawl calls per minute.

Firecrawl against the alternatives

The choice usually comes down to two questions. The first concerns licensing: if AGPL is banned in your organisation for software running on your own infrastructure, Crawl4AI under Apache-2.0 removes the problem, though you then operate the browsers and the anti-blocking work yourself. The second concerns scale: for a dozen addresses a day, a homegrown script on Playwright is enough and needs nobody's subscription. For thousands of addresses a day across different domains, maintaining a browser pool and rotating exit addresses alone costs more than the service.

Common mistakes

The first is assuming that because the SDK is MIT, the whole project is permissive. The server sits under AGPL-3.0, and that is what matters the moment you move it onto your own infrastructure. The layers have to be separated and described individually in a dependency audit.

The second is running crawl without a limit field. Without that brake the job goes as far as the links allow, and on a site with filters and pagination the address count can grow exponentially. Start with map and see the size of the site for one credit.

The third is using the json format where markdown would do. A fivefold difference in per-page cost is invisible at ten addresses and very visible at ten thousand. If you pass the text through your own model anyway, extracting the data on your side is often cheaper.

The fourth is promising screenshots or actions steps on a self-hosted deployment. Those features require the Fire-engine service, which is not in the repository, so on the default stack they simply will not work.

The fifth is confusing the max_age and store_in_cache fields. The first says how old a stored version may be before it stops satisfying you, the second decides whether the result gets stored at all. A job meant to see page changes within minutes needs max_age set low or to zero, otherwise it receives an answer from hours ago.

The sixth is treating the returned markdown as ready input for a model. A documentation page can run to tens of thousands of characters, which when naively pasted into a prompt eats the context window and degrades results. Chunking and vector search are a separate stage that Firecrawl does not perform.

FAQ

Does the AGPL-3.0 licence block commercial use?

It does not block it, but it attaches conditions. Using the cloud service through an SDK falls outside that licence, since the SDKs are MIT. Running the server yourself falls inside AGPL, whose section thirteen requires making source available to users connecting over a network to a modified version. There is no self-hosting exception in the licence, so for a commercial product the decision belongs to a lawyer.

Are the Python and JavaScript SDKs also under AGPL?

No. The apps/js-sdk and apps/python-sdk directories carry their own LICENSE files with MIT text. Package metadata on npm and PyPI declares MIT, and the unpacked firecrawl archive at version 4.34.1 genuinely contains MIT text. All three sources agree here.

Does a self-hosted deployment have every cloud feature?

It does not. The documentation states outright that screenshots and actions steps are missing without the Fire-engine service, and that the agent, browser, and interact modes are unavailable. Formats backed by a language model require connecting your own OpenAI-compatible provider or Ollama.

What does fetching a thousand pages cost?

A thousand credits for a plain fetch to markdown, which is exactly what the free plan grants per month. Those same thousand pages with the json format come to five thousand credits, since each page costs one credit plus four for the extraction.

Does Firecrawl respect robots.txt?

By default it does, and the ignoreRobotsTxt option in the crawl settings lets you disable that. The project documentation notes that responsibility for honouring the policies of visited sites rests with the user, so switching the protection off is a legal decision rather than a technical one.

Do I need Firecrawl when I already have Playwright?

For a handful of known sites with stable structure, a homegrown script is cheaper and gives full control. Firecrawl starts paying off across many different domains, where maintaining a browser pool, rotating exit addresses, and keeping content cleaning rules current becomes a project of its own.

The source code and the licence file live in the GitHub repository, the documentation at docs.firecrawl.dev, and the current pricing on the plans page.