https://www.ycombinator.com/companies/foundry/jobs/xtwLIsF-founding-engineer-large-scale-web-scraping-crawling

Y Combinator
About
What Happens at YC?ApplyYC Interview GuideFAQPeopleYC Blog
Companies
Startup DirectoryFounder DirectoryLaunch YC
Startup Jobs
All Jobs* Engineering* Operations* Marketing* SalesStartup Job Guide
YC Startup Jobs Blog
Find a Co-Founder
Library
SAFE
Resources
Startup SchoolNewsletterFor InvestorsHacker NewsBookface
Open main menu
Apply for X2025 batch.Apply
[58c5d9a291]

Foundry

RL Gym for Browser Agents

Founding Engineer: Large-Scale Web Scraping & Crawling

$100K - $200K / 1.00%
Location
San Francisco, CA, US / Remote (US)
Job Type
Full-time
Experience
3+ years
Connect directly with founders of the best YC-funded startups.
Apply to role >
Manil Lakabi
Manil Lakabi
Founder

About the role

About Us

We're building the first end-to-end testing platform for web agents,
including a Browser Gym for RL-driven optimization. Our platform
helps teams evaluate, benchmark, and improve web agents before they
go live, ensuring they can handle real-world, dynamic environments.

With synthetic user simulations, automated evaluations, and
large-scale benchmarking, we're setting a new standard for web agent
testing.

We're a YC-backed team, and this is a founding engineering
role--you'll be one of the first hires defining how we crawl,
structure, and analyze the open web at scale.

The Role

We need a Founding Web Scraping Engineer to build internet-scale web
crawling infrastructure--not just scraping a single site, but handling
millions of domains and evolving anti-bot defenses.

You'll be responsible for designing robust, distributed crawling
systems that adapt dynamically to web changes, optimize for
efficiency, and ensure reliable data extraction.

What You'll Do

  * Build large-scale, distributed crawlers that intelligently
    prioritize, schedule, and optimize requests across millions of
    domains.
  * Develop adaptive web scraping systems that handle DOM changes,
    WebSockets, AJAX-heavy sites, and dynamically loaded content.
  * Optimize scraping performance and resilience, ensuring
    high-throughput data extraction with proxy/network optimizations
    and behavior-driven stealth tactics.
  * Solve captchas at scale, integrating third-party solvers,
    heuristic-based workarounds, and behavior-driven bypass
    techniques.
  * Manage proxy and identity rotation, implementing session-aware
    scraping, JA3/TLS fingerprint spoofing, and request signature
    control.
  * Structure and clean extracted data for downstream analytics, AI
    training, and benchmarking applications.

What We're Looking For

  * Expert-level experience in large-scale web scraping & crawling
    (Selenium, Puppeteer, Playwright, Scrapy,
    undetected-chromedriver).
  * Deep knowledge of anti-bot detection strategies (TLS
    fingerprinting, JA3 signatures, request header anomalies, and bot
    behavior tracking).
  * Hands-on expertise with captcha-solving strategies, including
    leveraging APIs, OCR-based approaches, and behavior-driven
    evasion.
  * Proven experience building efficient proxy management systems,
    including rotating IPs across residential, datacenter, and mobile
    networks.
  * Proficiency in Python, Go, or JavaScript, with experience in
    high-performance, parallelized scraping frameworks.
  * Understanding of HTTP/2, HTTP/3, WebSockets, GraphQL, and
    browser-based fingerprinting.
  * Experience designing scalable, fault-tolerant scraping
    infrastructure that adapts to changes in real time.

Bonus Points

  * Experience with search engine-scale crawling.
  * Background in LLM-driven web extraction or RL-enhanced adaptive
    crawling.
  * Contributions to open-source scraping tools or web automation
    projects.

Why Join?

  * Founding role--you'll define and own our web crawling
    infrastructure from day one.
  * Work at internet scale--building a system that dynamically adapts
    and scales across millions of domains.
  * YC-backed--we're building something that doesn't exist yet, and
    you'll be part of the core team making it happen.

About Foundry

At Foundry, we're building the first end-to-end testing platform for
web agents, including a Browser Gym for RL-driven optimization. Our
platform helps teams run pre-production evaluations, benchmarking,
and reinforcement learning to make web agents more reliable. With
synthetic user simulations, automated evaluations, and large-scale
benchmarking, we make sure web agents can handle dynamic environments
with confidence.

Foundry
Foundry
Founded:2024
Team Size:2
Status:
Active
 
Founders
Manil Lakabi
Manil Lakabi
Founder
 
Pranav Raja
Pranav Raja
Founder
 

Footer

Y CombinatorY Combinator

Programs

  * YC Program
  * Startup School
  * Work at a Startup
  * Co-Founder Matching

Company

  * YC Blog
  * Contact
  * Press
  * People
  * Careers
  * Privacy Policy
  * Notice at Collection
  * Security
  * Terms of Use

Resources

  * Startup Directory
  * Startup Library
  * Investors
  * SAFE
  * Hacker News
  * Launch YC
  * YC Deals

Make something people want.

Apply
TwitterFacebookInstagramLinkedInYoutube

(c) 2025 Y Combinator