[HN Gopher] Gatekeepers of Law: Inside the Westlaw and LexisNexi...
___________________________________________________________________
Gatekeepers of Law: Inside the Westlaw and LexisNexis Duopoly
Author : toomuchtodo
Score : 105 points
Date : 2026-01-05 18:35 UTC (4 hours ago)
(HTM) web link (www.thebignewsletter.com)
(TXT) w3m dump (www.thebignewsletter.com)
| showerst wrote:
| Lexis especially makes even scraping public state laws and admin
| codes difficult, which is extra frustrating because they are the
| legal publisher of record in a number of states.
|
| I've been considering trying to launch an OpenStates style
| scraper project for US laws and admin codes, but haven't had the
| time to attack 100 more scrapers. Even with AI help, the volume
| is significant.
| shicholas wrote:
| Also the states themselves can't update their own laws. In
| Nevada the code (NRS) from Nevada's website is out of date.
| Very embarrassing imo and hard to get it to work bc AI can't
| have a trusted source of data.
| rockskon wrote:
| Why can't Nevada update its own laws on its own website?
| google234123 wrote:
| Google scholar tried and gave up
| toomuchtodo wrote:
| https://openlaws.us/
|
| https://public.resource.org/
|
| https://www.eff.org/press/releases/publicresourceorg-can-kee...
|
| https://blog.archive.org/2022/10/19/2022-internet-archive-he...
| showerst wrote:
| Neither openlaws nor public.resource actually let you just
| pull the laws in a common format (or the scrapers) as far as
| I can tell?
|
| I was thinking something more along the lines of a git repo
| per state.
| toomuchtodo wrote:
| Who will maintain the git repo per state [1] [2]? There is
| value in a pipeline that continually ingests this data from
| various sources and pushes it into the Internet Archive,
| but if you wish to treat it as authoritative, it must have
| a human minding it, because of entropy and decay. Even the
| Python Software Foundation has a budget of ~$5M/year. Hence
| my openlaws.us example.
|
| If it was as easy as writing a scraper and dumping it all
| in a bucket or repo, it'd already be done. It's just the
| usual thankless hard work over time grind.
|
| [1] https://xkcd.com/2347/
|
| [2] https://en.wikipedia.org/wiki/Free-rider_problem
| showerst wrote:
| I know a thing or two about that, 2,400 commits to the
| scrapers powering openstates over the past 9 years.
|
| Even with openstates, we have an API but don't "just"
| dump the bills to git for legacy nerd reasons.
|
| The nice thing about laws is that the host websites (or
| PDFs) don't change templates _that_ often, so generally
| you can rescrape quarterly (or in some states, annually)
| without a ton of maintenance. With administrative codes
| you need to scrape more often, but the websites are still
| pretty stable.
|
| The downside is that codes in particular are often big,
| so a single scrape might need to make 20,000 or more
| requests, so you have to be very careful about rate
| limiting and proxies, which goes to my original point
| that it sucks that accessing this stuff is such a mess.
| toomuchtodo wrote:
| Fair, I stand corrected. Thanks for your work. All
| Openness efforts are welcome.
| shicholas wrote:
| Free Law project is amazing and the best org to counter this
| duopoly. Please check them out if you haven't!
| wcarss wrote:
| https://www.courtlistener.com/ is run by freelaw.
| divbzero wrote:
| PlainSite is another project worth checking out. I have come
| across both when exploring in the space.
|
| [1]: https://free.law/
|
| [2]: https://www.plainsite.org/
| chaps wrote:
| I do a lot of IL criminal courts research and it's really, really
| fucking bad out there.
|
| Transcripts: Multiple dollars per pages. Want that expedited?
| Multiply that amount by four. Don't know the court date? Can't
| get your transcript. Clerks put in the wrong date? Tough luck.
| Payment for those transcripts? Over Zelle because the court
| reporters themselves are contractors and get paid independently.
|
| FOIA: IL Judicial system isn't FOIAable. There's a "data request"
| process that you can go down, but last time I tried by requesting
| through the chief judge of Cook County, they told me to request
| the data from the Clerk of Courts. The Clerk of Courts told me I
| need to speak to the chief judge's office. That went on for 6
| months. The docket-level data I've seen from them is... beyond
| indecipherable. Literally never seen a dataset so indecipherable
| in my life -- a combination of esoteric codes, hidden encoding
| schemes, and length-delimited lines.
|
| Court docs: Gotta go downtown if you're not a lawyer or if you're
| not a "criminal justice partner" to get any, then you have to
| print them (don't print past 4 or you'll get yelled at). Some
| journalists have access, but the IL Supreme Court's policies
| disallow journalists from getting it. So we have a system where
| only some journalists get access to court documents and fuck
| everyone else, I guess.
|
| Civil Asset Forfeiture cases: the courts and state's attorneys do
| _everything_ in their part to separate it from the initiating
| criminal case. So peoples ' cars are being taken by the cops
| under dubious constitutional grounds and they can't get it back
| until the criminal case is over. And I've yet to find a
| consistent way to figure out the initiating criminal case tied to
| a civil asset forfeiture. I've yet to find anything like a policy
| doc that says it's intentional to obfuscate the criminal case
| from the civil case, but I can't think of any reason why it would
| be like this if it wasn't intentional. FOIA through the State's
| Attorney? Nah, they don't respond, or they give nonsense denials
| that require litigation.
|
| I could go on, but. Fuck. It's bad out there and it hides so much
| suffering. I _highly_ encourage folk to give court watching a
| try. You 'll be shocked how much you'll learn.
| mothballed wrote:
| Family court is even worse. You can't even know what's illegal
| for your kid to do because the civil standards on things like
| 'neglect' are incredibly nebulous and left for agencies like
| CPS and family law judges to review, so to find out how they're
| actually interpreted you would need to see cases and how they
| played out. Those are ~all wrapped up in secret court hearings.
|
| So basically you have no fucking idea what the law actually is,
| how it's being enforced, what will happen to you if you don't
| follow their interpretation, and god forbid you wind up there
| they can use a lot of methods to keep you from talking about
| it.
| toomuchtodo wrote:
| Consider reaching out to https://www.reclaimtherecords.org/ if
| any headway can be made legally on these scenarios. They are
| aggressive, highly recommend. If the legal scaffolding exists
| to enable FOIA requests, https://www.muckrock.com/ can then do
| the rest.
| chaps wrote:
| Great recs.
|
| Reclaim the Records and I do a loooot of the same work but
| it's separate enough that we don't really interact much. But
| I'm a big, big fan of their work and have had a couple light
| chats with the founder a couple times. She's great and yeah,
| wonderfully aggressive. Her work's the sort of foundational
| work that enables so much.
|
| Muckrock and I go back quite a long way! Sending about 200
| requests through them later this month :)
| toomuchtodo wrote:
| I'll tip my hand a bit, I suggested Reclaim the Records
| because while I am time poor, if they take up fighting
| anything you suggested, I can send a directed donation
| towards that work. If you have someone else in mind who
| will take on these legal challenges in Illinois, I am
| interested.
|
| Love the Muckrock love, they are a treasure.
| chaps wrote:
| Nice, thank you. It's really appreciated. For
| Chicagoland:
|
| Lucy Parsons Labs: https://www.lucyparsonslabs.com/donate
|
| Chicago Appleseed: https://www.chicagoappleseed.org/ways-
| to-give/
|
| Invisible Institute: https://invisible.institute/donate
|
| Unraveled Press: https://unraveledpress.com/support-
| unraveled/
|
| All four are great. One is my employer. Unraveled Press
| would benefit the most from donations.
| jandrese wrote:
| > Transcripts: Multiple dollars per pages. Want that expedited?
| Multiply that amount by four. Don't know the court date? Can't
| get your transcript. Clerks put in the wrong date? Tough luck.
| Payment for those transcripts? Over Zelle because the court
| reporters themselves are contractors and get paid
| independently.
|
| Is this because a person has to literally transcribe the
| stenographer's notes into plain English? Is it expensive
| because it is labor intensive?
| chaps wrote:
| Yes and yes. I'm sure there's historical or possibly ethical
| reasons for it being this way, but I don't understand why
| it's not the clerk of court's job. Something to figure out!
| yxhuvud wrote:
| Over here in Sweden they justify the fees for printed cases
| by the paper cost. Cheaper than the number you mentioned
| but still quite costly if you want all nontrivial verdicts.
| And quite silly when they deliver pdfs..
|
| Then every single case needs to have all PII redacted
| before it is made searchable, but that is a different
| story.
| TZubiri wrote:
| >"Tough luck. Payment for those transcripts? Over Zelle because
| the court reporters themselves are contractors and get paid
| independently."
|
| That doesn't sound right, are you sure that wasn't a bribe?
| chaps wrote:
| You seem to have faith in the courts still. It was not a
| bribe. Here's a short exchange between me and the court
| reporter who did indeed sent me the transcripts after
| payment: "Out of curiosity, is it normal to
| pay court reporters over venmo like this? To be honest I was
| pretty surprised when I heard this. And funny enough when I
| asked folk on twitter, a scammer tried to "help out" with
| some "scam recovery" service for $500." "That's
| hilarious! Since the pandemic, it really took off because we
| were working remotely, and some of the Cook County judges and
| reporters remain remote to this day and may be permanently
| working on Zoom. I work at 26th Street so we've worked
| through most of the pandemic in person, but the courthouse
| was not open to the public. Zelle is a way you can get
| payment to us quickly to get your transcript faster.
| Most private attorneys use Zelle now because they want the
| reporter to start transcription sooner rather than later, but
| we also accept checks or if you prefer to send a check in the
| mail, you can do that. Very rarely does an attorney come
| into our office these days. It's just a matter of
| preference."
| aadraple wrote:
| Here's a great X thread with additional reporting giving an
| inside look on this:
| https://x.com/ROSSIntel/status/2006429363034824709?s=20
| atlanta90210 wrote:
| The state of Georgia tried to copyright its public written laws.
| They lost.
|
| https://www.bfvlaw.com/supreme-court-rules-georgia-cannot-cl...
| akudha wrote:
| How does one decide what should be in the public domain for the
| good of society and what should be commercialized? These are a
| couple of examples that I learned recently
|
| 1. In the UK, Royal Mail owns the postal addresses data. I was
| looking at UK's open datasets - apparently lot of datasets that
| have addresses can't be used without paying Royal Mail. There
| are some exceptions - but I am no lawyer. It is depressing to
| learn that Royal Mail is no longer a public institution, it was
| sold against public will by the UK government to a private
| entity, and sold again and as of last year it is owned by a
| Czech billionaire. Similarly, Canadian postal code database is
| also not free.
|
| 2. CPT code descriptions are owned by AMA (apparently they're
| super litigious?). Sure they took the time to write them, they
| should be compensated - but imagine how many interesting
| projects can be built if this data was freely available
|
| On one hand, multi Billion dollar companies like Bloomberg
| exist, thanks to free and open data. But also things that
| should be free (dictionaries, postal codes etc) aren't.
| AlotOfReading wrote:
| It's not a question of "how do you decide which public
| standards should be freely accessible". That's easy: all of
| them. The public benefits more from freely accessible
| standards whether they're building codes, legal codes, ISO
| standards, or HDMI. The effect of not having them publicly
| available is that people make-do without having read the
| standard and the public has no way to validate things against
| the standard afterwards.
|
| The question we don't have an easy answer for is how to
| incentivise the people behind these things without locking
| their work behind paywalls? Compliance marks, homologation
| regimes, copyright, and other strategies all have their own
| downsides.
| doctorpangloss wrote:
| Do you think it's good for UpToDate that OpenEvidence scrapes
| and paraphrases UpToDate and sells the same information in a
| GPT wrapper to make big investor bucks? I don't know what the
| answer is. Go for it, tell me.
| crazygringo wrote:
| I'm not sure what point you're trying to make?
|
| First, laws cannot be copyrighted. They're in the public
| domain.
|
| Second, the case you link to was not about laws, but
| _annotated_ law. The _annotated version_ (OCGA) was written by
| LexisNexis, and all that work paid for by LexisNexis.
| Obviously, normally that would mean LexisNexis has the
| copyright on the annotations. This is unproblematic.
|
| Another organization (Public.Resource.Org) posted the OCGA
| online. Georgia sued, arguing the annotations were the work of
| LexisNexis and therefore obviously copyrighted. But the problem
| was that Georgia had _declared the annotations to be official_.
|
| The Supreme Court found this act of making them official made
| them essentially the law itself, and therefore uncopyrightable.
| Which meant LexisNexis had done all that expensive work for
| free, whoops.
|
| So it's a weird edge case where Georgia should either not have
| accorded any special status to the annotations, or should have
| paid for annotations themselves if they wanted "official" ones,
| and made them public domain.
| esbranson wrote:
| If you're trying to fix anything relating to public policy,
| politics, &c., without fixing this you are wasting your time. You
| will make a bigger difference adopting a local stretch of highway
| and picking up litter. It is maddening to see generation after
| generation learn this the hard way.
|
| FLP's CourtListener RECAP hits hard, but it has no counterparts
| either worldwide or within the US. If it goes offline, democracy
| and rule of law across the entire world go offline with it.
| Federal statutes and regulations are a very small part of black
| letter law.
| rockskon wrote:
| There is little worse action you can take than discouraging
| others from doing anything.
|
| Far worse problems than PACER have been solved. Just because
| you've been unsuccessful doesn't necessarily mean others will
| be.
| TZubiri wrote:
| "The fees the federal government charges for court documents are
| significant. PACER charges users $0.10 for every page users
| search and access, and $30 per name or item searched plus $0.10
| per page per document per month, albeit with a max of $3 per
| individual document."
|
| It's funny how these numbers sound very reasonable on their face,
| especially if used for human review.
|
| But if you consider programmatic searches, 10 cents per page is a
| lot. Consider the CPU cost of cycling through each pdf, assume
| 1000 letters, that's 1Khz which is like a billionth of a cent, in
| terms of electricity and hardware ammortization.
|
| I'd bet that this incentivizes a 'hoarding' and reuse of
| information, rather than a "query when necessary" practice, which
| would in fact give place to monopolies like OP argues.
|
| A newcomer legal analysis company would have to pay, say 200$ for
| research from their first customer. If they save those documents,
| then it is unlikely that their second customer will benefit from
| those documents.
|
| However huge firms would make a business out of the byproduct of
| searches, they spend 200$ on documents, and they serve a client,
| but they add those documents to their own internal database,
| which of course has costs much lower than the nominal fees
| stipulated by the federal steward (PACER).
|
| An argument could be that you can't eliminate the fees because
| there's costs that need to be paid. But isn't that what taxes are
| typically for? Especially corporate income tax. The costs are
| probably a nominal fee to avoid overbearing systems, similar to
| franchise taxes and incorporation clerical fees.
|
| I don't think the idea of charging for documents is bad, who said
| that the system would need to be adopted for huge data hungry
| systems, I'm all for giving human analysis an advantage instead
| of adapting systems to play into the data munching AI fad. I'd
| rather see balancing that rewards human analysis rather than
| bringing data munching to the masses.
| burner420042 wrote:
| If someone wanted to do business research for Mexico or any other
| LATAM country, while in the US and using English source data what
| resources would you look to?
| yxhuvud wrote:
| Depends on what kind of business research you need. But there
| are at least collections of all tax treaties that exist
| available from a single source somewhere. I don't remember what
| the source is off the top of my head unfortunately.
| pyuser583 wrote:
| And this doesn't include the messiness in the decisions
| themselves. Judges often edit the decisions for some time after
| they are published.
|
| For example, Souter was editing his published opinion in Lee v.
| Weisman nine years after the case was decided.
|
| https://balkin.blogspot.com/2025/05/a-justice-souter-anecdot...
| bicepjai wrote:
| Its crazy insane that citizens who are expected to follow the law
| cannot have access to law ? what does that even mean ? Do we at-
| least know some high level numbers on how many law/opinions/cases
| texts are not freely available to citizens ?
| pstuart wrote:
| On a tangential note, this seems ripe for LLM
| analysis/summarization to digest the legal code down so mere
| mortals could better understand it.
|
| It would also be great for pointing out all the loopholes that
| have been built in by our owners.
| yxhuvud wrote:
| There is, but that depends a whole lot on access to content
| that can be hard to get - both case data and books in law are
| essential.
___________________________________________________________________
(page generated 2026-01-05 23:00 UTC)