[HN Gopher] Asking AI to build scrapers should be easy right?
___________________________________________________________________
Asking AI to build scrapers should be easy right?
Author : suchintan
Score : 54 points
Date : 2025-10-17 19:03 UTC (3 hours ago)
(HTM) web link (www.skyvern.com)
(TXT) w3m dump (www.skyvern.com)
| showerst wrote:
| A point orthogonal to this; consider whether you need browser
| automation at all.
|
| If a website isn't using Cloudflare or a JS-only design, it's
| generally better to skip playwright. All the major AIs understand
| beautifulsoup pretty well, and they're likely to write you a
| faster, less brittle scraper.
| pavel_lishin wrote:
| If.
| Etheryte wrote:
| The vast majority of the modern internet falls into one of
| those two buckets though, no?
| showerst wrote:
| I mostly scrape government data so the sites are a little
| 'behind' on that trend, but no. Even JS heavy sites are
| almost always pulling from a JSON or graphql source under the
| hood.
|
| At scale, dropping the heavier dependencies and network
| traffic of a browser is meaningful.
| suchintan wrote:
| Yeah, reverse engineering APIs is another fantastic
| approach. They aren't enough if you are dealing with
| wizards (eg typeform), but they can work really well
| suchintan wrote:
| IF you can use crawlers, definitely do.
|
| They aren't enough for anything that's login-protected, or
| requires interacting with wizards (eg JS, downloading files,
| etc)
| philipbjorge wrote:
| We had a similar realization here at Thoughtful and pivoted
| towards code generation approaches as well.
|
| I know the authors of Skyvern are around here sometimes -- How do
| you think about code generation with vision based approaches to
| agentic browser use like OpenAI's Operator, Claude Computer Use
| and Magnitude?
|
| From my POV, I think the vision based approaches are superior,
| but they are less amenable to codegen IMO.
| suchintan wrote:
| I think they're complementary, and that's the direction we're
| headed.
|
| We can ask the vision based models to output _why_ they are
| doing what they are doing, and fallback to code-based
| approaches for subsequent runs
| ahstilde wrote:
| this matches our personal experience, too
| franze wrote:
| In AI First workshops. By now I tell them for the last exercise
| "no scrappers". the learning is to separate reasoning (AI) from
| data (that you have to bring.) and ai coded scrappers seem a
| logical, but always fail. scrapping is a scaling issue, not
| reasoning challenge. also the most interesting websites are not
| keen for new scrappers.
| pyuser583 wrote:
| Over the past few days I've spent a lot of time dealing with
| terribly designed UIs. Some legitimate and desired use cases are
| impossible because poor logic excludes them.
|
| Is AI capable of saying, "This website sucks, and doesn't work -
| file a complaint with the webmaster?"
|
| I once had similar problems with the CIA's World Factbook. I
| shudder to think what an I would do there.
| suchintan wrote:
| It's funny, one time we had a customer that wanted to use us to
| test their website for bugs..
|
| Skyvern kept suggesting improvements unrelated to the issue
| they were testing for
| nithril wrote:
| The same day, a post on reddit was about: "We built 3B and 8B
| models that rival GPT-5 at HTML extraction while costing 40-80x
| less - fully open source" [1].
|
| Not fully equivalent to what is doing Skyvern, but still an
| interesting approach.
|
| [1]
| https://www.reddit.com/r/LocalLLaMA/comments/1o8m0ti/we_buil...
| suchintan wrote:
| This is really cool. We might integrate this into Skyvern
| actually - we've been looking for a faster HTML extraction
| engine
|
| Thanks for sharing!
| guluarte wrote:
| the hardest part of scrapping is bypassing
| Cloudflare/captchas/fingerprinting etc
| suchintan wrote:
| Definitely. What are your thoughts on the CloudFlare agent
| identity
| herpdyderp wrote:
| I'd be all over Skyvern if only they had enterprise compliance
| agreements available.
| suchintan wrote:
| We do have them! We are HIPAA compliant, have soc-2 type 2 and
| offer self hosted deployments
| whinvik wrote:
| I feel like this is how normal work is. When I have to figure out
| how to use a new app/api etc, I go through an initial period
| where I am just clicking around, shouting in the ether etc until
| I get the hang of it.
|
| And then the third or fourth time its automatic. Its weird but
| sometimes I feel like the best way to make agents work is to
| metathink about how I myself work.
| suchintan wrote:
| I have a 2yo and it's been surreal watching her learn the
| world. It deeply resembles how LLMs learn and think. Crazy
| Retric wrote:
| Odd, I've been stuck by how different LLMs and kids learn the
| world.
|
| You don't get that whole uncanny valley disconnect do you?
| goatlover wrote:
| How so? Your kid has a body that interacts with the physical
| world. An LLM is trained on terabytes of text, then modified
| by human feedback and rules to be a useful chatbot for all
| sorts of tasks. I don't see the similarity.
| _pdp_ wrote:
| This is exactly the direction I am seeing agent go. They should
| be able to write their own tools and we are soon launching
| something about that.
|
| That being said...
|
| LLMS are amazing for some coding tasks and fail miserably at
| others. My hypothesis is that there is some sort of practical
| limit to how many concepts an LLM can hold into account no matter
| the context window given the current model architectures.
|
| For a long time I wanted to find some sort of litmus test to
| measure this and I think I found one that is an easy to
| understand programming problem, can be done in a single file, yet
| complex enough. I have not found a single LLM to be able to build
| a solution without careful guidance.
|
| I wrote more about this here if you are interested:
| https://chatbotkit.com/reflections/where-ai-coding-agents-go...
| pennaMan wrote:
| Yes, it is easy. LLMs have reduced my maintenance work on
| scraping tasks I manage (lots of specialized high-traffic adfield
| sites) by 99%
|
| What used to be a constant almost daily chore with them breaking
| all the time at random intervals is now a self-healing system
| that rarely ever fails.
| claysmithr wrote:
| I misread this as 'sky scrapers'
___________________________________________________________________
(page generated 2025-10-17 23:00 UTC)