[HN Gopher] Hundreds of millions of stars turned into a map of G...
___________________________________________________________________
Hundreds of millions of stars turned into a map of GitHub projects
Author : anvaka
Score : 429 points
Date : 2023-05-13 18:47 UTC (1 days ago)
(HTM) web link (anvaka.github.io)
(TXT) w3m dump (anvaka.github.io)
| topspin wrote:
| Is Vue over represented in this? Seems like there would be a
| Reactistan island somewhere before Vue would show up.
| tomschlick wrote:
| Vue is a pretty big deal in the PHP/Laravel community. Evan the
| creator even gave a talk at Laracon a few months in, showing
| the hockey stick growth after it started to get traction in the
| community.
| anvaka wrote:
| True. Though I also expected to "Reactistan" somewhere more
| explicitly, I wonder if it is scattered in Fronterra
| continent?
|
| I am also puzzled myself why the algorithm has separated
| Fronterra from another large island to the west, seemingly
| related to node and some other JS libraries. Decided to keep
| it all as is in case I'm missing some reason
| silverwind wrote:
| My feeling is that the React community is at least double to
| triple the size of the Vue one, so yes, I feel React should
| definitely be it's own island.
| bluefishinit wrote:
| This is really neat! Are there any insights you gleaned from
| doing this that were unexpected?
| pvaldes wrote:
| Somebody has been doing a lot of Covid-19 plots in Gnu-R
| anvaka wrote:
| I knew GitHub is not a tiny website, but I didn't imagine how
| big it actually is. Each of those dots are giant parts of
| someone's life.
|
| There are a lot of interests that I didn't know exist. For
| example https://github.com/cat-milk/Anime-Girls-Holding-
| Programming-... - someone collects anime girls holding
| programming books.
|
| https://github.com/tylertreat/Comcast - and here is someone who
| is amazing at coming up with funny project names =)
| tomthe wrote:
| What else did you use to calculate similarities? When I tried
| something similar with HN co-commenters, I just got one big
| blob with dang and other prolific commenters in the middle.
| That was not very interesting or insightful.
|
| How long did it take to calculate the Leiden clustering and
| the force-layout? Do you think it would be somehow possible
| to compute a force-layout of the whole graph?
| matt_trentini wrote:
| Very cool visualisation!
|
| Though I noticed a couple of odd groupings - like that
| MicroPython was clustered in Arduinoria rather than the adjacent
| MicroPythonia... :P
| shon wrote:
| Wow.. that's cool
| junon wrote:
| The data seems a bit dated, but this is really cool.
|
| I know this because I moved visionmedia/debug to debug-js/debug
| in November of 2021.
| anvaka wrote:
| I've got data from 2020 to end of March 2023
| junon wrote:
| Ah okay, that makes more sense. Did you scrape it yourself?
| Incredible work by the way :)
| zX41ZdbW wrote:
| I recommend checking https://ghe.clickhouse.tech/
|
| It explains the full pipeline - how to download, collect,
| and analyze this sort of data.
| anvaka wrote:
| Thank you! Google BigQuery has it readily available, so
| probably took 10 minutes to fetch it =)
|
| https://cloud.google.com/blog/topics/public-
| datasets/github-...
| sdflhasjd wrote:
| If you type a capital letter into the search box, it gets stuck
| on "downloading index" indefinitely.
|
| I was waiting for about 10 minutes wondering how big these
| indexes were...
| shadycuz wrote:
| I also experienced this looking up DontShaveTheYak
| anvaka wrote:
| Oof, sorry! This should be fixed soon thanks to
| https://github.com/anvaka/map-of-github/pull/1
| pmontra wrote:
| The Ruby world is part of an island labeled Art of node, west of
| Fronterra. Strange choice.
| anvaka wrote:
| I agree. I'm not sure why they are together. Maybe all that
| tooling stuff that was happening in early days?
| _nalply wrote:
| The title is clickbaity in a very clever way, and I mean this as
| a compliment! I thought about what it says for almost a minute
| before clicking. Stars being made into Github projects? Would I
| find Orion and Alpha Centauri? I thought hard on about how to
| pull this off.
|
| Then I clicked and I realized that it is the opposite way, in
| reverse. And then this realization brought a smile on my face.
| anvaka wrote:
| You are very kind and your compliment put smile on my face too
| :)
| metalliqaz wrote:
| Did it break under the load? I can't search
| amrb wrote:
| Group names are 10/10
| Abimelex wrote:
| just came here to give it my upvote. That's so insane cool!
| OmarShehata wrote:
| This is amazing! I love this effect of showing me that my little
| project that I was working on fairly solitarily is actually part
| of a community of other people doing similar work that I can
| reach out to/collaborate with etc!
|
| What does it mean if, I click on a repo, and it shows 5-6 links
| to specific projects? Does that just mean the jaccard similarity
| index was below a threshold?
| anvaka wrote:
| Yes! I picked only the highest scores to form an edge in
| relationships graph. Typically a sigma (std deviation) or two
| away from the mean. So if there is a direct link between your
| project and others - the similarities are abnormally high.
|
| Note that I'm not rendering direct links outside of the country
| yet, there might be more there. Will probably add a "focused"
| view to see those better
| PedroBatista wrote:
| AILandia was big before but now has been growing at an astounding
| rate almost surpassing Frontera.
|
| Some say it's a cancer others say it's inevitable and here to
| stay. Probably both are right.
|
| Also, why is Swiftoria so big?
| anvaka wrote:
| I've been watching changelog nightly emails for a while - they
| summarize most starred daily repositories, and growth of AI
| there (subjectively) seems to be even higher than frontend tech
| speak wrote:
| W'
| phirschybar wrote:
| incredible and so inspiring. thank you!!
| jxf wrote:
| I'm embarrassed to admit that I read "stars" in the title and
| spent far too long trying to figure out how these were arranged
| into constellations.
| __MatrixMan__ wrote:
| This is so fun, thanks for making it.
|
| I bet that several of these regions have a common image in their
| readme (the python logo, the nix logo, etc). Imagine little flags
| popping out of each region...
| anvaka wrote:
| haha, I love the flags idea :). I wish I had the prowess to
| implement it in a way that is visually appealing and not
| obscuring the map.
|
| I would also love to have a giant octocat hugging the
| archipelago, with some radial gradient emitting inside of it.
| Alas my design-gl foo is not there yet
| irrational wrote:
| Is there a page that explains what the island names mean? I
| understand Minecraftria, but Fluttopia (and most others)? No
| clue.
| zX41ZdbW wrote:
| I found many ClickHouse-related projects in a dedicated cluster
| - "House of Clicks" :)
|
| But ClickHouse itself somehow appears in Kubernation, and
| ClickBench - in Datapolis. Nice names btw.
| anvaka wrote:
| Thank you!
|
| It could be a curse or popularity? Some projects are so
| popular that you can place them nearly anywhere, and they
| would still find a lot of densely connected group of
| neighbors
| detaro wrote:
| it usually becomes clear when zooming in and looking at the
| examples. E.g. "Fluttopia" is Flutter-related repos.
| anvaka wrote:
| Yes! Thank you
|
| Note that some country are missing their names because I
| didn't have enough knowledge to assign a name to them.
|
| Here is my naming process if anyone wants to help with the
| rest: https://github.com/anvaka/map-of-github#country-names
| swyx wrote:
| very cool! how did you decide on the boundaries and
| placement of the continents?
| speedgoose wrote:
| From the project readme on GitHub:
|
| > A lot of country labels were generated with help of ChatGPT.
| If you find something wrong, you can right click it, edit, and
| send a pull request - I'd be grateful.
| VitoVan wrote:
| I'm quite unhappy with SBCL located in Schemaria:
|
| https://anvaka.github.io/map-of-github/#12/13.469/-8.175
|
| It should be Lispana.
| anvaka wrote:
| Should I rename Schemaria to Lispania?
| ReleaseCandidat wrote:
| No, as there are all/most/some (Haskell, OCaml, Purescript,
| Idris, Coq, ...) functinal languages. Lambdania would be a
| possibility. Although Prolog and Forth are located there too.
| VitoVan wrote:
| The Scheme people would be unhappy then.
| anvaka wrote:
| True... Scheming Lispania :)?
| andreareina wrote:
| Parentensia
| defanor wrote:
| ML-family languages are placed in Schemaria as well; apparently
| it is a catch-all for functional languages.
| paulddraper wrote:
| Nice!!
|
| But how do I zoom in/out?
| bluefishinit wrote:
| Scroll wheel works for me.
| internetguy wrote:
| This is frickin awesome. Made me really see how the programming
| community can come together and contribute to something powerful.
| arthurcolle wrote:
| Reminds me of dotlan Eve maps
| data_ders wrote:
| 1. this is so cool. i'd love to see more name for the regions! 2.
| nit: one repo dear to my heart dbt-labs/dbt-core was previously
| dbt-labs/dbt as well as fishtownanalytics/dbt. they show up as
| unique nodes on your map. what most interesting to me is that
| each name of this repo links to a different set of related repos?
| anvaka wrote:
| Thank you!
|
| The connections are inferred from stargazers. If they lead to
| different set of related projects it might be a sign that
| different group of people gave stars different things at that
| time.
|
| Github of course has much more dimensions than a flat 2d
| surface can show
| danking00 wrote:
| This reveals an interesting dichotomy in the project I work on,
| Hail. I think of hail as a serverless workflow, relational, and
| linear algebra system most similar to Dask, BigQuery, Snowflake,
| Spark, etc. but this map is constructed from the perspective of
| the _user_ so Hail lands squarely in the world of bioinformatics.
|
| Also neat to see how bioinformatics is such a splitbrained
| community. They land next to R but are filled with Python
| projects.
| rmorey wrote:
| This is amazing
| pizza wrote:
| Damn that's good
| paxo wrote:
| Haha, very nice. I was able to locate multiple projects even ones
| with less than 1k stars easily.
| casey2 wrote:
| Amazing way to see related projects in an environment you aren't
| familiar with.
| geokon wrote:
| This gives a really cool top level view into the whole Chinese
| opensource community - which has always been very mysterious to
| me. Bc it's absolutely massive and almost entirely on its own
| doing its own thing due to the language barrier
| anvaka wrote:
| Indeed! A lot of amazing things happens there. A few are
| getting world wide adoption (echarts are awesome if you are
| into frontend viz, so is element-plus library for vue
| components). There are giant communities of ML too
| paulddraper wrote:
| IDK how I feel about Kubernation: Firecracker, Grafana,
| Localstack, Podman, Terraform, Vault?
|
| Seems like a really overreaching power. Wait, never mind, that
| makes sense.
| anvaka wrote:
| :D btw, if you find any country label to be missing or could be
| improved - right click it, change it, and then send a PR (it's
| two clicks away)
| swyx wrote:
| how exactly did you name and cluster these things? i love how
| you presented it and dont know of any existing quantitative
| methodology
| anvaka wrote:
| Check this out https://github.com/anvaka/map-of-github#how-
| was-it-made - I gave a short overview there - does this
| help?
| samsquire wrote:
| This is absolutely awesome. Such a good job well done! The search
| is fast! The sidebar is a great feature. I also love how you
| highlight the repositories you search for and draw red lines
| between things.
|
| I found my journal repositories in the "Land of Node" in
| "Frontartia". I am surprised by that because I didn't realise I
| was associated with the node community!
|
| I am so impressed with your visualization, it is intuitive and
| interesting the different communities of GitHub.
| anvaka wrote:
| Thank you so much!
|
| I'm puzzled by some countries in Frontartia's island too. I'm
| not sure why it was even pulled away, as if there is something
| I'm missing.
| rightbyte wrote:
| This is really neat. Clustering of data like this makes for so
| interesting graphics.
| specproc wrote:
| This is a beautiful project. It's useful, aesthetically pleasing,
| and based on a clever methodology. Absolutely love it. Great
| work.
| anvaka wrote:
| Thank you so much for your compliment!
| edrxty wrote:
| This is amazing, the GitHub ham radio and hardware scenes are way
| bigger than I previously thought. Shout-out to OpenRTX and M17
| along with all the awesome SDR applications!
| mftb wrote:
| To me this is proof that doing a good visualization is really an
| art. I saw this on hn for a couple hours, but really didn't think
| much of it. As soon as I clicked through the funny names, and
| "continents" brought a smile to my face.
|
| I guess my little project is too small to make it, but now I too
| aspire to join the great nation of Golandia.
| anvaka wrote:
| Thank you!
| kelsolaar wrote:
| This is very cool and was pleasant albeit surprising to see
| colour in Gamedonia!
| sethkim wrote:
| Very cool!
|
| If you work on a dynamic version that allows users to understand
| changes in the open-source topography over time and
| detect/predict new clusters, this could be a very powerful tool
| for investment intelligence.
| coffeeshopgoth wrote:
| I love these things - to see everything sort of classified, and
| all at once, helps me find things I would never think to look
| for. Forgive me if this has been posted somewhere else, but if
| you want to see a reddit map, here it is (maybe you found it
| already in their github, but I just have it as a bookmark):
| https://anvaka.github.io/map-of-reddit/?x=18083.096950551575...
| terrycody wrote:
| Yeah I just wanna paste this and found your answer :P, now we
| have another similar tool about Github, much more useful for
| many people, kudos to the author!
| anvaka wrote:
| thank you :)
| mellosouls wrote:
| This is great, not just nice to look at but useful too. Top work!
|
| How it was constructed (also really interesting) is described at
| the repo:
|
| https://github.com/anvaka/map-of-github#readme
| anvaka wrote:
| thank you!
| divan wrote:
| Check out other anvaka's projects, there are gems:
| https://news.ycombinator.com/from?site=anvaka.github.io
| tomthe wrote:
| Or check them out directly on the map:
| https://anvaka.github.io/map-of-github/#12.36/-31.74456/14.8...
___________________________________________________________________
(page generated 2023-05-14 23:02 UTC)