[HN Gopher] OpenAlex: The Promising Alternative to Microsoft Aca...
___________________________________________________________________
OpenAlex: The Promising Alternative to Microsoft Academic Graph
Author : tokai
Score : 74 points
Date : 2022-05-05 10:03 UTC (12 hours ago)
(HTM) web link (library.smu.edu.sg)
(TXT) w3m dump (library.smu.edu.sg)
| sva_ wrote:
| First I thought having a REST API isn't that "open", but
| discovered you can mirror the complete dataset (~200G):
| https://docs.openalex.org/download-snapshot
| jasonpriem wrote:
| We emphasize the API because imho a well-documented, high-
| throughput API is something really lacking in the ecosystem
| right now. Dealing with a dataset this size (200M works, 200M
| authors) is a pain, especially since many end users for this
| data don't have a lot of technical expertise. Often people have
| really basic questions (like the ones from the linked post) or
| they want to build simple applications like monitoring
| dashboards, recommender systems, and scholarly search engines.
|
| With the API, folks can outsource the heavy data engineering to
| us for free, and just do the fun parts themselves. We want to
| make building real-world apps on the global research graph fun
| and easy, the kind of thing you can do as a hackathon project,
| instead of with a six-figure grant.
|
| That said, I agree that it's absolutely essential the entire
| dataset be easy to download and mirror as well. It's called
| OpenAlex because it's _open_, soup to nuts (the "Alex" part is
| homage to the ancient Library of Alexandria). All the data is
| open, the code is open, and our governance is as open as we can
| make it. [1]
|
| [1] https://ourresearch.org/transparency
| tokai wrote:
| And the data is licensed under cc0.
| lmc wrote:
| Anyone know of any nice front-ends or visualizations for this?
|
| I guess a Python or R Notebook and network analysis libraries
| would be a good place to start?
| jasonpriem wrote:
| If you go that route, you may also want to check out these
| community-contributed Python and R libraries:
| https://docs.openalex.org/api#client-libraries
| fluidcruft wrote:
| I don't know if it's related to this database or does what
| you're looking for but I've recently learned that
| https://www.connectedpapers.com and
| https://www.researchrabbit.ai exist and have started using them
| (particularly connectedpapers) pretty heavily.
| lmc wrote:
| Thanks, both seem like good options!
| shishy wrote:
| You can also do visualizations with https://scite.ai! The
| edges of the graphs show exactly what the papers (nodes) said
| about each other :)
|
| Here's a video: https://www.youtube.com/watch?v=P5J9EgAaai4
| fluidcruft wrote:
| That's pretty cool. It made me suddenly think that it would
| be great if there was a way to use this safely and without
| going broke while performing peer-review.
| zozbot234 wrote:
| Scholia https://scholia.toolforge.org/ is a frontend to
| comparable data on Wikidata, a general-purpose knowledge graph.
| Edit: The OpenAlex website states that a web interface is
| planned for Summer 2022.
| tomthe wrote:
| OpenAlex seems to be a really useful alternative to Scopus and
| Web of Science. Although I found that their author-disambiguation
| is not as good as the one from Scopus.
|
| I would like to see that they make it easier to download database
| dumps. Their json-based format seems to contain a lot of
| redundant information. A compressed dump should not be much
| bigger than 20...30 GB (Theirs is 200GB, according to
| https://docs.openalex.org/download-snapshot/download-to-your... )
| jasonpriem wrote:
| Agreed, the author disambiguation isn't quiet as good as
| Scopus'...they have a bit of a head start on us. But we're
| improving it quickly.
|
| Thanks for the suggestion about the data dump. A lot of that
| weight is abstracts, which come in at over 30GB just by
| themselves. But it's true that the JSON format has some
| redundancies. For now we think those are worth it, because the
| denormalized schema is very compatible with the API and easy
| for beginners to get started with. Plus you only have to
| download it once (for free! HT to AWS Open Data sponsorship),
| and after that the updates are very light.
|
| We'll certainly consider offering a smaller, normalized format
| in the future though, if we get more requests for it.
| jasonpriem wrote:
| Jason here, cofounder of OurResearch, the nonprofit behind
| OpenAlex. I appreciate everyone's feedback! Happy to answer any
| questions.
| legerdemain wrote:
| This looks very similar to what meta.org developed before it got
| shut down to pave the way for the Facebook renaming. I wonder if
| SMU inherited meta.org's archives and data pipelines.
| jessfyi wrote:
| Meta only focused on biomedical information. This is explicitly
| based on Microsoft Academic Graph which was all encompassing.
| They did remove patents, tag papers with fewer
| topics/subtopics, and don't index as many conferences, but they
| did add a couple of nice things like ORCID. [0]
|
| [0] https://docs.openalex.org/download-snapshot/mag-
| format/mag-m...
| jasonpriem wrote:
| To be clear, this is a project isn't affiliated with SMU, they
| just did the blog post...it's from a nonprofit called
| OurResearch (source: I'm a cofounder).
|
| We did have some good conversations with folks at Meta before
| they closed up shop, but didn't end up using any of their data.
| shishy wrote:
| We use this at scite [1] to power some of our features, kudos to
| Jason, Heather and the rest of the team for their awesome work on
| maintaining this infrastructure (and the rest of their work on
| Unpaywall, etc.)
|
| For context, many companies, including us, used Microsoft
| Academic Graph but it was shuttered last year [2]. So OpenAlex
| came in to replace, but they also made it backwards compatible
| with the MAG data, making it super easy for us to migrate.
|
| Overall there are some problems with the data that we've found
| and reported / fixed (and continue to do so), but it feels well
| supported to use moving forward.
|
| I believe the OpenAlex team put out a preprint about this
| recently [3].
|
| 1. https://scite.ai
|
| 2. https://www.microsoft.com/en-
| us/research/project/academic/ar...
|
| 3. https://arxiv.org/ftp/arxiv/papers/2205/2205.01833.pdf
| jasonpriem wrote:
| Thanks for the kudos! I'm very glad we've been able to help
| out!
___________________________________________________________________
(page generated 2022-05-05 23:01 UTC)