https://crawlee.dev/python/

Skip to main content
[?] If you like Crawlee for Python, give it a star on GitHub 
 
[crawlee-li][crawlee-da]
Crawlee for PythonDocsExamplesAPIChangelogBlog
Python

  * Node.js
  * Python

GitHubDiscord
Search

Crawlee is a web
scraping and browser
automation library

Crawlee is a web
scraping and browser
automation library

It helps you build reliable crawlers. Fast.

 Crawlee for Python is open to early adopters!
Get Started
[logo-blur-]

pipx run crawlee create my-crawler

Reliable crawling [?]

Crawlee won't fix broken selectors for you (yet), but it helps you
build and maintain your crawlers faster.

When a website adds JavaScript rendering, you don't have to rewrite
everything, only switch to a browser crawler. When you later find a
great API to speed up your crawls, flip the switch back.

Crawlee is built by people who scrape for a living and use it every
day to scrape millions of pages. Meet our community on Discord.

Python with type hints

Crawlee for Python is written in a modern way using type hints,
providing code completion in your IDE and helping you catch bugs
early on build time.

Headless browsers

Switch your crawlers from HTTP to a headless browser in 3 lines of
code. Crawlee builds on top of Playwright and adds its own features.
Chrome, Firefox and more.

Automatic scaling and proxy management

Crawlee automatically manages concurrency based on available system
resources and smartly rotates proxies. Proxies that often time-out,
return network errors or bad HTTP codes like 401 or 403 are
discarded.

Try Crawlee out 

before you start
Crawlee requires Python 3.9 or higher.

The fastest way to try Crawlee out is to use the Crawlee CLI and
choose one of the provided templates. The CLI will prepare a new
project for you, and add boilerplate code for you to play with.

pipx run crawlee create my-crawler

If you prefer to integrate Crawlee into your own project, you can
follow the example below. Crawlee is available on PyPI, so you can
install it using pip. Since it uses PlaywrightCrawler, you will also
need to install crawlee package with playwright extra. It is not not
included with Crawlee by default to keep the installation size
minimal.

pip install 'crawlee[playwright]'

Currently we have Python packages crawlee and playwright installed.
There is one more essential requirement: the Playwright browser
binaries. You can install them by running:

playwright install

Now we are ready to execute our first Crawlee project:

import asyncio

from crawlee.playwright_crawler import PlaywrightCrawler, PlaywrightCrawlingContext


async def main() -> None:
    crawler = PlaywrightCrawler(
        max_requests_per_crawl=5,  # Limit the crawl to 5 requests.
        headless=False,  # Show the browser window.
        browser_type='firefox',  # Use the Firefox browser.
    )

    # Define the default request handler, which will be called for every request.
    @crawler.router.default_handler
    async def request_handler(context: PlaywrightCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url} ...')

        # Enqueue all links found on the page.
        await context.enqueue_links()

        # Extract data from the page using Playwright API.
        data = {
            'url': context.request.url,
            'title': await context.page.title(),
            'content': (await context.page.content())[:100],
        }

        # Push the extracted data to the default dataset.
        await context.push_data(data)

    # Run the crawler with the initial list of URLs.
    await crawler.run(['https://crawlee.dev'])

    # Export the entire dataset to a JSON file.
    await crawler.export_data('results.json')

    # Or work with the data directly.
    data = await crawler.get_data()
    crawler._logger.info(f'Extracted data: {data.items}')


if __name__ == '__main__':
    asyncio.run(main())

Docs

  * Examples
  * API reference

Community

  * Blog
  * Discord
  * Stack Overflow
  * Twitter

More

  * Apify Platform
  * Docusaurus
  * GitHub

Crawlee is free and open source
Built by