[HN Gopher] ARIN Public Incident Report - 4.10 Misissuance Error
___________________________________________________________________
ARIN Public Incident Report - 4.10 Misissuance Error
Author : immibis
Score : 127 points
Date : 2025-12-21 15:19 UTC (7 hours ago)
(HTM) web link (www.arin.net)
(TXT) w3m dump (www.arin.net)
| gbil wrote:
| A couple of years ago ARIN increased their fees considerably -
| way higher than fees paid to RIPE for way less resources - and
| had a call with their management to express my frustration, not
| because I was paying from my pocket but because of the high
| discrepancy of the what they wanted to get and the
| quantity/quality of their services. Now I can see that their
| backbone services haven't really improved while their income for
| sure has.
|
| On a sidenote, what I appreciate in both RIPE and ARIN is that
| you can have at least a proper discussion when you have valid
| arguments with their support teams.
| rmoriz wrote:
| Now ARIN is much cheaper than RIPE for small entities.
| rmoriz wrote:
| fee schedules FYI
|
| - ARIN 2026 PDF: https://www.arin.net/resources/fees/images/2
| 026feeschedule.p...
|
| - RIPE 2026 : https://www.ripe.net/membership/payment/
|
| Enthusiasts, trainees and small orgs are paying a lot more
| with RIPE.
| icedchai wrote:
| Not necessarily. Many have their RIPE registrations through
| an existing, "sponsoring" LIR. They're not paying that 1800
| Euro, the LIR is.
| rmoriz wrote:
| A single AS resource and a single PI assignment cost more
| than the ARIN fee.
| icedchai wrote:
| Are you sure? For RIPE I see a 50 ASN plus 75 euro PI
| fee. ARIN is $275. Maybe I'm looking at it wrong.
|
| It's cheaper as a hobbyist to use a RIPE LIR. Even in the
| US. That's what I've been doing for years.
| rmoriz wrote:
| afaik that's +VAT and also for LIRs only. LIRs apply
| markup, see https://www.lir.services/lir-sponsoring/ they
| charge 200EUR per resource, so ASN + PI would be at last
| 400EUR/year that's way above the price of ARIN and you
| have a middleman.
|
| You _must_ have a sponsoring LIR for your resources or
| become a LIR yourself. The only exception is LEGACY
| resources (IPv4, no ASN) but that 's a different story.
| icedchai wrote:
| There are more competitive LIRs out there. Example:
| https://lagrange.cloud/products/lir
|
| It's also cheaper for me because I have legacy ARIN
| space. All I really needed was an ASN. The LIR gives me
| some PA v6 space for cheap, too.
| rmoriz wrote:
| Okay, but that is not enough to operate independently. PA
| v6 is another dependency. With ARIN you get your personal
| IPv6 assignment.
| progbits wrote:
| I like how frank the report is, no sugarcoating. "We relied on
| manual error prone verification and made a mistake. We have to
| automate the process."
|
| As ARIN block owner this situation is kinda scary but reading
| this actually makes me think it's less likely to happen again .
| anonnon wrote:
| You don't find this part
|
| > We have to automate the process.
|
| to be ominous?
| Aurornis wrote:
| I don't. The report says part of this process relied on flat
| files and spreadsheets. Automating that with software is a
| good idea.
|
| "Automate the process" doesn't mean feeding everything to an
| LLM.
| aaomidi wrote:
| Certificate issuance was once only possible manually.
| qingcharles wrote:
| Domains too, well into the 90s.
| netfortius wrote:
| The road to automation is always full of outages.
| stefan_ wrote:
| I'm curious how these fellas took something like _IP block
| allocation_ and turned it into an Excel based workflow.
| jonathanlydall wrote:
| "Workflow" is probably a bit generous to describe how they
| probably use Excel.
|
| Having worked at a mom and pop ISP a couple of decades ago
| where we used Excel to track a lot of things, I can see how
| this might have happened.
|
| To actually know who is allocated what is ultimately just a
| list.
|
| And when there are only a few people who edit the list (and
| probably no more than 1 person at a time) you can get by with
| even a plain text file, but Excel is quite a bit nicer as you
| can do things like filtering and sorting easily, maybe even
| some formulas to help with things.
|
| Building a program backed by a database might be nice, but
| hard to justify when the manual system has never been a
| problem before.
|
| They've probably been thinking for a while they should, but
| it's just never been enough of a pain point for them to
| invest the effort.
|
| Looks like they see this incident as justification that they
| need a system with hard coded rules and constraints, no more
| manual checking.
| autoexec wrote:
| I can't remember a screw up by ARIN this bad before. I'm not too
| concerned about it. I understand that mistakes can happen. That
| said, I'm a little surprised at how easy it was to make this one.
|
| I'm entirely unsurprised that this mistake involved an excel
| spreadsheet. Out of all the databases and IP management software
| they could be using which would have prevented this the first
| thing the employee reached for was excel. Almost every company
| I've worked for has employees using excel for data that would be
| better managed/stored/presented outside of an office document.
| simonjgreen wrote:
| All the RIRs are, in my experience, a very consistent and safe
| set of hands. This sort of things is vanishing rare to the point
| of borderline inconsequence by many providers of major internet
| infrastructure. The fact they care enough to take it seriously
| and publish shows how much they care about getting it right.
|
| I just completed a fairly major reorganisation of resources with
| RIPE, and I've interacted with them for two decades, and my
| experience is they remain as steady and consistent as ever.
|
| Sure, you may not like a particular policy at some moment, or may
| not agree with the charging structure at some point in time when
| it's not advantageous to you, but they do at least do what they
| say and say what they do.
| mlhpdx wrote:
| So at least a good chunk of the Internet does indeed operate on a
| spreadsheet. Good to know.
| 12_throw_away wrote:
| All data begins life in a spreadsheet and dies in a
| spreadsheet. Automation is an illusion; databases are
| illusions. Only Excel is real.
| ang_cire wrote:
| This reads like a joke, but I've known two DBAs who don't use
| database management tools beyond exporting whole tables to
| excel, making manual changes, and importing to update the
| tables. Scary stuff.
| aftbit wrote:
| I've considered setting up an ASN and grabbing an IPv6 block for
| myself for a while now, but have never had the gumption, time,
| and funds at the same time.
| galaxygate wrote:
| Affected customer here, if you're curious on our original NANOG
| post on the whole situation:
|
| Hey NANOG,
|
| After receiving a BGPAlerter notification that one of our subnets
| (23.150.164.0/24) had been hijacked, I checked and noticed the
| prefix in question was missing RPKI. Assuming I had fat fingered
| something and butchered the ROA, I logged into ARIN and found
| that the prefix was missing from our resource list entirely, and
| had been reallocated to another organization and announced from
| their network. I created a ticket in ARIN and called immediately.
|
| They confirmed that our subnet had been accidentally reallocated
| to another customer, and that they are currently working on
| returning it to us. After a couple hours, they told us the other
| organization will stop announcing the prefix, and WHOIS will be
| returned shortly.
|
| I'm guessing there's no way to prevent this kind of thing on our
| side if the RPKI ROA itself is removed along with the allocation?
| I'm planning on adding checks to look for missing ROAs (in
| addition to invalid/expiring ones), which I'm guessing would've
| caught this earlier.
|
| Have any of you had anything like this happen with ARIN or
| another RIR? I'm especially curious what might have happened if
| we'd only noticed and reached out a few weeks later instead of
| within a few minutes.
| thaumaturgy wrote:
| Off-topic, but: I see you've got a green username (new
| account). How did you know this post was on the HN front page?
| ARIN's writeup doesn't mention your service by name. I looked
| it up out of curiosity from the CIDR they mentioned, before
| clicking over into the comments here. Unless you've got a
| regular HN account and just set up a new business-facing one
| for this?
|
| I periodically see people showing up early in comment threads
| posted about things they've written or articles where they're
| the subject. Usually I figure they've got a Google alert or
| some other whatsit, or they've got something monitoring
| referers in their web traffic. But this is a case where neither
| would apply.
| AndroTux wrote:
| Maybe some college of theirs on HN recognized the story and
| shared it with them.
| nateb2022 wrote:
| > Unless you've got a regular HN account and just set up a
| new business-facing one for this?
|
| This is likely; I can't imagine a regular HN user would
| appreciate having their subnet publicly available in their
| comment history.
| galaxygate wrote:
| Yup, another engineer that works on our team mentioned seeing
| the report here, I figured I'd make an account to add some
| further context
| Titan2189 wrote:
| The original report says
|
| > The incorrect state persisted for approximately seven days
| before detection
|
| However you're saying you've reached out "within a few minutes"
| ?
| BlueMatt wrote:
| It was re-allocated to the new/wrong ARIN customer for seven
| days before they started announcing it, at which point the OP
| detected the issue. Prior to that their prefix was routing to
| them just fine, just without RPKI protection.
| teraflop wrote:
| The "incorrect state" being talked about is the IP prefix
| being misregistered in ARIN's database.
|
| The "hijacking" happened later, when the IP prefix was
| announced via BGP by the registrant who it was incorrectly
| assigned to. Those are two different events.
| yoan9224 wrote:
| The transparency in this incident report is refreshing. "We
| relied on manual Excel-based verification and screwed up" - no
| corporate speak, just honest assessment.
|
| What's scary is that IPv4 allocations are literally internet
| infrastructure. Having your /24 suddenly reassigned to someone
| else could be catastrophic for a business.
|
| The fact that RPKI didn't catch this is interesting. The ROA was
| deleted along with the allocation, so from RPKI's perspective
| everything was valid. This is a good reminder that RPKI protects
| against hijacking but not against the RIR itself making mistakes.
|
| Glad they're automating this. Anything involving copy-pasting IP
| ranges in Excel is an accident waiting to happen.
| squigz wrote:
| This is a bit beyond my paygrade, but... this is as serious as it
| sounds, right? I'm just a bit surprised/confused by the response
| in these comments, especially compared to outages like when CF
| goes down. It's like that Gordon Ramsay meme. Is ARIN the 8 year
| old in this situation?
___________________________________________________________________
(page generated 2025-12-21 23:00 UTC)