https://github.com/alexellis/openfaas-puppeteer-template Skip to content Sign up * Why GitHub? Features - + Code review + Project management + Integrations + Actions + Packages + Security + Team management + Hosting + Mobile + Customer stories - + Security - * Team * Enterprise * Explore + Explore GitHub - Learn & contribute + Topics + Collections + Trending + Learning Lab + Open source guides Connect with others + Events + Community forum + GitHub Education + GitHub Stars program * Marketplace * Pricing Plans - + Compare plans + Contact Sales + Nonprofit - + Education - [ ] [search-key] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} alexellis / openfaas-puppeteer-template * Sponsor Sponsor alexellis/openfaas-puppeteer-template * Watch 2 * Star 57 * Fork 1 OpenFaaS template for headless Chrome and Puppeteer www.openfaas.com/blog/puppeteer-scraping/ MIT License 57 stars 1 fork Star Watch * Code * Issues 0 * Pull requests 0 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights master 1 branch 0 tags Go to file Code Clone HTTPS GitHub CLI [https://github.com/a] Use Git or checkout with SVN using the web URL. [gh repo clone alexel] Work fast with our official CLI. Learn more. * Open with GitHub Desktop * Download ZIP Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Go back Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Go back Launching Xcode If nothing happens, download Xcode and try again. Go back Launching Visual Studio If nothing happens, download the GitHub extension for Visual Studio and try again. Go back Latest commit @alexellis alexellis Add additional links ... 2fe565f Jan 5, 2021 Add additional links Signed-off-by: Alex Ellis (OpenFaaS Ltd) 2fe565f Git stats * 13 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time template/puppeteer-node12 Remove log statements from template Jan 5, 2021 LICENSE Initial commit Oct 28, 2020 README.md Add additional links Jan 5, 2021 View code README.md openfaas-puppeteer-template This OpenFaaS template uses docker-puppeteer by buildkite to give you access to Puppeteer. Puppeteer is a popular tool that can automate a headless Chrome browser for scraping fully-rendered web pages. Use-cases: * Run your end to end tests with mocha/jest against a real website * Capture screenshots of sites and diff them * Capture / scrape text from sites which have no API or are only rendered in the DOM * Automate websites which have no API * Create visual assets from HTML/CSS - like social sharing banners Why do we need an OpenFaaS template? Templates provide an easy way to scaffold a microservice or function and to deploy that at scale on a Kubernetes cluster. The faasd project also gives a way for small teams to get on the experience curve, without learning anything about Kubernetes. OpenFaaS benefits / features: * Extend timeouts to whatever you want * Run asynchronously, and in parallel * Get a callback with the result when done * Limit concurrency with max_inflight environment variable in stack.yml * Trigger from cron, or events * Get metrics on duration, HTTP exit codes, scale out across multiple nodes * Start small with faasd See the full tutorial on the OpenFaaS blog Web scraping that just works with OpenFaaS with Puppeteer Quickstart Get OpenFaaS Deploy OpenFaaS to Kubernetes, or to faasd (single-node with just containerd) Create a function with the template and deploy it to OpenFaaS faas-cli template pull https://github.com/alexellis/openfaas-puppeteer-template faas-cli new --lang puppeteer-node12 scrape-title --prefix alexellis2 faas-cli up -f scrape-title.yml Example functions and invocations Get the title of a webpage passed in via a JSON body 'use strict' const assert = require('assert') const puppeteer = require('puppeteer') module.exports = async (event, context) => { let browser let page browser = await puppeteer.launch({ args: [ // Required for Docker version of Puppeteer '--no-sandbox', '--disable-setuid-sandbox', // This will write shared memory files into /tmp instead of /dev/shm, // because Docker's default for /dev/shm is 64MB '--disable-dev-shm-usage' ] }) const browserVersion = await browser.version() console.log(`Started ${browserVersion}`) page = await browser.newPage() let uri = "https://inlets.dev/blog/" if(event.body && event.body.uri) { uri = event.body.uri } const response = await page.goto(uri) console.log("OK","for",uri,response.ok()) let title = await page.title() const result = { "title": title } browser.close() return context .status(200) .succeed(result) } echo '{"uri": "https://inlets.dev/blog"}' | faas-cli invoke scrape-title \ --header "Content-type=application/json" Alternatively run async: echo '{"uri": "https://inlets.dev/blog"}' | faas-cli invoke scrape-title \ --async \ --header "Content-type=application/json" Run async, post the response to another service or receiver function: echo '{"uri": "https://inlets.dev/blog"}' | faas-cli invoke scrape-title \ --async \ --header "Content-type=application/json" \ --header "X-Callback-Url=https://en98kppbwx32.x.pipedream.net" Take a screenshot and return it as a binary response 'use strict' const assert = require('assert') const puppeteer = require('puppeteer') const fs = require('fs').promises module.exports = async (event, context) => { let browser let page browser = await puppeteer.launch({ args: [ // Required for Docker version of Puppeteer '--no-sandbox', '--disable-setuid-sandbox', // This will write shared memory files into /tmp instead of /dev/shm, // because Docker's default for /dev/shm is 64MB '--disable-dev-shm-usage' ] }) const browserVersion = await browser.version() console.log(`Started ${browserVersion}`) page = await browser.newPage() let uri = "https://inlets.dev/blog/" if(event.body && event.body.uri) { uri = event.body.uri } const response = await page.goto(uri) console.log("OK","for",uri,response.ok()) let title = await page.title() const result = { "title": title } await page.screenshot({ path: `/tmp/page.png` }) let data = await fs.readFile("/tmp/page.png") browser.close() return context .status(200) .headers({"Content-type": "application/octet-stream"}) .succeed(data) } echo '{"uri": "https://inlets.dev/blog"}' | \ faas-cli invoke screenshot-page \ --header "Content-type=application/json" > screenshot.png open screenshot.png Produce homepage banners and social sharing images You can also produce homepage banners and social sharing images by rendering HTML locally, and then saving a screenshot. Unlike a SaaS service, you'll have no month fees to pay, and get unlimited use, you can also customise the code and trigger it however you like. The execution time is very quick at under 0.5s per image and could be made faster by preloading the Chromium browser and re-using it. if you cache the images to /tmp/ or save them to a CDN, you'll have single-digit latency. # Set to your Docker Hub account or registry address export OPENFAAS_PREFIX=alexellis2 faas-cli new --lang puppeteer-node12 banner-gen --prefix $OPENFAAS_PREFIX Edit ./banner-gen/handler.js 'use strict' const assert = require('assert') const puppeteer = require('puppeteer') const fs = require('fs'); const fsPromises = fs.promises; module.exports = async (event, context) => { let browser = await puppeteer.launch({ args: [ '--no-sandbox', '--disable-setuid-sandbox', '--disable-dev-shm-usage' ] }) const browserVersion = await browser.version() console.log(`Started ${browserVersion}`) let page = await browser.newPage() let title = "Set your title" let avatar = "https://avatars2.githubusercontent.com/u/6358735?s=160&v=4" console.log("query",event.query) if(event.query) { if(event.query.url) { url = event.query.url } if(event.query.avatar) { avatar = event.query.avatar } if(event.query.title) { title = event.query.title } } let html = `

TITLE

Avatar` html = html.replace("TITLE", title) html = html.replace("AVATAR", avatar) await page.setContent(html) await page.setViewport({ width: 1720, height: 460 }); await page.screenshot({ path: `/tmp/page.png` }) let data = await fsPromises.readFile("/tmp/page.png") await browser.close() return context .status(200) .headers({"Content-type": "image/png"}) .succeed(data) } Deploy the function: faas-cli up -f banner-gen.yml Example usage: curl -G "http://127.0.0.1:8080/function/generate-banner" \ --data-urlencode "avatar=https://avatars2.githubusercontent.com/u/6358735?s=160&v=4" \ --data-urlencode "title=Time for your favourite website to get social banners" \ -o out.png Note that the inputs are URLEncoded for the querystring. You can also use the event.body if you wish to access the function programmatically, instead of from a browser. This is an example image generated for my GitHub Sponsors page which uses a different HTML template, that's loaded from disk. Generated image HTML: sponsor-cta.html You may also like The full tutorial on the OpenFaaS blog * Web scraping that just works with OpenFaaS with Puppeteer Serverless Node.js that you can run anywhere Serverless doesn't have to mean using a function, bring your favourite micro HTTP framework with you: Serverless Node.js that you can run anywhere faasd with TLS on DigitalOcean * Bring a lightweight Serverless experience to DigitalOcean with Terraform and faasd About OpenFaaS template for headless Chrome and Puppeteer www.openfaas.com/blog/puppeteer-scraping/ Resources Readme License MIT License Releases No releases published Sponsor this project * * Learn more about GitHub Sponsors Packages 0 No packages published Languages * JavaScript 74.6% * Dockerfile 25.4% * (c) 2021 GitHub, Inc. * Terms * Privacy * Security * Status * Help * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.