[HN Gopher] Crawl Order and Disorder
___________________________________________________________________
Crawl Order and Disorder
Author : ingve
Score : 56 points
Date : 2025-03-27 13:07 UTC (9 hours ago)
(HTM) web link (www.marginalia.nu)
(TXT) w3m dump (www.marginalia.nu)
| keyle wrote:
| Maybe something worth exploring, run it like an inverted
| generational garbage collector.
|
| Separate the problematic domains and run them in a lower priority
| background crawl; then do the opposite as well, have a higher
| priority list for crawling sites that have regular updates more
| often, based on their last update gap.
|
| Basically like a generational garbage collector but in this case
| it's generational content collector, although arguably, still
| mostly garbage.
| marginalia_nu wrote:
| Yeah there's definitely more that can be done in this regard.
| Right now I have fairly limited statistics on the rate of
| change of the domains. There's probably improvements to be made
| by making more informed choices based on that sort of data.
| nthingtohide wrote:
| Simplifying Systems with Elixir * Sasa Juric * YOW! 2020
|
| https://www.youtube.com/watch?v=EDfm2fVS4Bo
|
| A noob question, is Elixir good at crawling tasks in the process
| simplying the system architecture.
| marginalia_nu wrote:
| I'm not well versed enough in Elixir to give a good answer.
|
| The hard problems in crawling is mostly dealing with a very
| large and potentially highly mutable state. It's a concurrent
| problem, but a fairly easy one and a good use case for a
| traditional thread pool since there is latency is mostly a non-
| issue.
| muscomposter wrote:
| based on what you said, and my assumption that Elixir (thanks
| to Erlang's VM) is pretty good at concurrency, then yes
| Elixir would be good for crawling
| jerf wrote:
| Elixir is one of a reasonably large number of acceptable
| languages.
|
| A crawler, and especially a very large one like Marginalia, is
| going to implement its own queuing logic, its own retry logic,
| its own management of same, and so on and so forth. As a
| result, things like "BEAM has supervisor trees" are actually
| not that useful; as they are nowhere near enough for this use
| case and need to be augmented anyhow, they don't really help a
| lot, and can even get in the way. At most the BEAM OTP
| infrastructure might help you bootstrap something up somewhat
| more quickly but I'd expect single-digit weeks into the dev
| process before it isn't really that helpful anymore. None of
| the BEAM-specific strengths strike me as hugely helpful here,
| mostly for similar reasons; they don't quite match what a
| crawler wants per se and the crawler is going to reimplement
| them anyhow.
|
| For a project of this scale, you also don't want to be doing
| the raw indexing in Elixir, as it is not a very fast language,
| and at this scale that adds up quickly, so you're going to pull
| in another language anyhow.
|
| All in all, while I might call it "acceptable", there's a solid
| half-dozen languages (plus runtimes, as appropriate) that I'd
| put solidly in front of Elixir for this use case and another
| several I'd rate as roughly ties. It certainly is not the case
| that it offers some sort of amazing, blow-me-away advantage
| that makes it the only sensible answer or anything.
| marginalia_nu wrote:
| I think the demands on concurrency are relatively basic. It's
| nice to have something a bit more robust than raw pthreads,
| but the main thing that makes or breaks a crawler is access
| to a robust HTTP client library and HTML parser, solid I/O
| performance, that sort of thing. Because the problem domain
| looks the way it looks, concurrency is one of the easier
| parts.
| outer_web wrote:
| Are you running into those autogen crawler labyrinths that have
| been posted about recently? Certainly the .edu and .gov ones are
| probably just huge sites.
|
| Am I correct in assuming you are recrawling your entire corpus to
| get fresh results? Would there be a downside to replacing crawl
| epochs with a continuous crawl that is random but with age
| priority?
| marginalia_nu wrote:
| Crawler labyrinths generally tend to be on paths disallowed by
| robots.txt, or behind nofollow links. I haven't seen any
| indication that they're playing much part in this.
|
| Seems the bigger problem is very large domains with slow
| response times and long crawl delays.
|
| > Am I correct in assuming you are recrawling your entire
| corpus to get fresh results?
|
| For known links, I'm sampling them and based on whether I find
| changes (first via if-none-match and if-modified-since or
| alternatively via locality sensitive hashing), I'm recrawling
| only a part or all of the links.
|
| > Would there be a downside to replacing crawl epochs with a
| continuous crawl that is random but with age priority?
|
| The drawbacks to this is a much more mutable crawl data (being
| able to read or write crawl data top down in an append-only
| format is a huge performance improvement), as well as problems
| with the indexing software which takes ~1 day to complete, and
| can't currently be partially rebuilt but is rebuilt from
| scratch every time at significant computational expense.
___________________________________________________________________
(page generated 2025-03-27 23:01 UTC)