https://github.com/mishushakov/llm-scraper

Skip to content
Toggle navigation
 
Sign in

  * Product
      +  
        Actions
        Automate any workflow
      +  
        Packages
        Host and manage packages
      +  
        Security
        Find and fix vulnerabilities
      +  
        Codespaces
        Instant dev environments
      +  
        Copilot
        Write better code with AI
      +  
        Code review
        Manage code changes
      +  
        Issues
        Plan and track work
      +  
        Discussions
        Collaborate outside of code
    Explore
      + All features
      + Documentation
      + GitHub Skills
      + Blog
  * Solutions
    For
      + Enterprise
      + Teams
      + Startups
      + Education
    By Solution
      + CI/CD & Automation
      + DevOps
      + DevSecOps
    Resources
      + Learning Pathways
      + White papers, Ebooks, Webinars
      + Customer Stories
      + Partners
  * Open Source
      +  
        GitHub Sponsors
        Fund open source developers
      +  
        The ReadME Project
        GitHub community articles
    Repositories
      + Topics
      + Trending
      + Collections
  * Pricing

Search or jump to...

Search code, repositories, users, issues, pull requests...

Search
[                    ]
Clear

Search syntax tips

Provide feedback

We read every piece of feedback, and take your input very seriously.

[                    ] [ ] Include my email address so I can be
contacted
Cancel Submit feedback

Saved searches

Use saved searches to filter your results more quickly

Name [                    ] 
Query [                    ]

To see all available qualifiers, see our documentation.

Cancel Create saved search
Sign in
Sign up
You signed in with another tab or window. Reload to refresh your
session. You signed out in another tab or window. Reload to refresh
your session. You switched accounts on another tab or window. Reload
to refresh your session. Dismiss alert
{{ message }}
mishushakov / llm-scraper Public

  * Notifications
  * Fork 5
  * Star 21
  * 

Turn any webpage into structured data using LLMs

License

MIT license
21 stars 5 forks Branches Tags Activity
Star
Notifications

  * Code
  * Issues 1
  * Pull requests 0
  * Actions
  * Security
  * Insights

Additional navigation options

  * Code
  * Issues
  * Pull requests
  * Actions
  * Security
  * Insights

mishushakov/llm-scraper

This commit does not belong to any branch on this repository, and may
belong to a fork outside of the repository.
 main
BranchesTags
  
Go to file
Code

Folders and files

      Name              Name           Last commit      Last commit
                                         message           date
Latest commit

 

History

24 Commits
 
examples          examples                             

src               src                                  

.gitignore        .gitignore                           

.prettierrc       .prettierrc                          

LICENSE.md        LICENSE.md                           

README.md         README.md                            

package-lock.json package-lock.json                    

package.json      package.json                         

tsconfig.json     tsconfig.json                        

View all files

Repository files navigation

  * README
  * MIT license

LLM Scraper

 
Screenshot 2024-04-20 at 23 11 16

LLM Scraper is a TypeScript library that allows you to convert any
webpages into structured data using LLMs.

Tip

Under the hood, it uses function calling to convert pages to
structured data. You can find more about this approach here

Features

 

  * Uses OpenAI chat models
  * Schemas defined with Zod
  * Full type-safety with TypeScript
  * Based on Playwright framework
  * Supports 3 operating modes: html, text, image
  * Streaming when crawling multiple pages

Make sure to give it a star!

Screenshot 2024-04-20 at 22 13 32

Getting started

 

 1. Install the required dependencies from npm:

    npm i zod playwright llm-scraper

 2. Get an OpenAI API key and set it in your environment variables:

    export OPENAI_API_KEY=***

 3. Optional. Create a new browser instance and attach LLMScraper to
    it:

    import { chromium } from 'playwright'
    import LLMScraper from 'llm-scraper'

    const browser = await chromium.launch()
    const scraper = new LLMScraper(browser)

Example

 

In this example, we're extracting top stories from HackerNews:

import z from 'zod'
import { chromium } from 'playwright'
import LLMScraper from 'llm-scraper'

// Create a new browser instance
const browser = await chromium.launch()

// Initialize the LLMScraper instance
const scraper = new LLMScraper(browser)

// Define schema to extract contents into
const schema = z.object({
  top: z
    .array(
      z.object({
        title: z.string(),
        points: z.number(),
        by: z.string(),
        commentsURL: z.string(),
      })
    )
    .describe('Top stories on Hacker News'),
})

// URLs to scrape
const urls = ['https://news.ycombinator.com']

// Run the scraper
const pages = await scraper.run(urls, {
  model: 'gpt-4-turbo',
  schema,
  mode: 'html',
  closeOnFinish: true,
})

// Stream the result from LLM
for await (const page of pages) {
  console.log(page.data)
}

Contributing

 

As an open-source project, we welcome contributions from the
community. If you are experiencing any bugs or want to add some
improvements, please feel free to open an issue or pull request.

About

Turn any webpage into structured data using LLMs

Topics

scraper browser ai openai gpt llm langchain

Resources

Readme

License

MIT license
Activity

Stars

21 stars

Watchers

4 watching

Forks

5 forks
Report repository

Releases

No releases published

Packages 0

No packages published

Languages

  * TypeScript 100.0%

Footer

 (c) 2024 GitHub, Inc.

Footer navigation

  * Terms
  * Privacy
  * Security
  * Status
  * Docs
  * Contact
  * Manage cookies
  * Do not share my personal information

You can't perform that action at this time.