[HN Gopher] Grep.app - Search across a half million Git repos
___________________________________________________________________
Grep.app - Search across a half million Git repos
Author : thunderbong
Score : 171 points
Date : 2023-02-07 12:24 UTC (10 hours ago)
(HTM) web link (grep.app)
(TXT) w3m dump (grep.app)
| ptannor wrote:
| Haha I can share my first use case - searching for repos I'm
| involved in, kind of like searching for myself on Google Scholar
| NoZebra120vClip wrote:
| This is a great idea and I commend it and I hope others find it
| useful. However, I have a rather narrow use-case for GitHub
| searches, and in my initial experiments, this tool comes up 100%
| empty on common searches I use every day for work.
|
| Most of my GitHub searches don't actually revolve around actual
| code-code. I search for text strings in prose that turn up in
| code searches. So the file types may be text or MarkDown instead.
|
| Perhaps grep.app only indexes and presents actual code-code and
| filters out all the prose I'd be interested in. Sad, because
| there are actually a few instances where my searches involve code
| with many intricate special characters, and GitHub's current code
| search ignores all those.
| nikolahristov wrote:
| There's Sourcegraph - https://sourcegraph.com/search, which is
| far more powerful and feature wide.
| sebastianconcpt wrote:
| Yet it doesn't have a clue about the existance of one of my 17
| star GitHub public repos
| dang wrote:
| Recent and probably related:
|
| _The technology behind GitHub's new code search_ -
| https://news.ycombinator.com/item?id=34680903 - Feb 2023 (168
| comments)
| barbinbrad wrote:
| At first glance, this is great!
| voytec wrote:
| GitHub can be searched directly without any JS bloat.
| birriel wrote:
| Missed opportunity to call it "Grit."
| [deleted]
| [deleted]
| bcjordan wrote:
| I'd be curious to hear how the ingestion/search pipeline works --
| the searches are incredibly fast!
| Scaevolus wrote:
| It uses Solr: https://news.ycombinator.com/item?id=22397392
| xigoi wrote:
| Search across a half million _GitHub_ repos. It doesn 't support
| any other git hosts than the proprietary one.
| khaledh wrote:
| Previous discussions:
|
| - https://news.ycombinator.com/item?id=28424845 (September 5,
| 2021 -- 54 points, 6 comments)
|
| - https://news.ycombinator.com/item?id=22396824 (February 23,
| 2020 -- 614 points, 155 comments)
| IncRnd wrote:
| Please don't check your keys or passwords into github.
|
| This was my first search to show AWS API Keys. [1] and my second
| to show google API keys. [2] Unfortunately, there are many more
| of these types of searches - and far too many results.
|
| [1]
| https://grep.app/search?q=AKIA%5B0-9A-Z%5D%7B16%7D®exp=tr...
|
| [2]
| https://grep.app/search?q=AIza%5B0-9A-Za-z%5C%5C-_%5D%7B35%7...
| TYMorningCoffee wrote:
| Your search has a lot of false positives like
| AKIAIOSFODNN7EXAMPLE
| teaearlgraycold wrote:
| _Hacker noises_ - "I'm in"
| capableweb wrote:
| As far as I can tell, all the git repos come from GitHub, and
| GitHub automatically sends you a warning if you commit access
| keys of various kinds (and I seem to remember they even de-
| activate some, maybe AWS keys, automatically? Not 100% sure
| about this). Most of the keys shown are unlikely to be working,
| especially the ones older than just a couple of days (either
| owner notice, or someone else was faster than you, got access
| and regenerated new keys for themselves)
| IncRnd wrote:
| > or someone else was faster than you, got access and
| regenerated new keys for themselves
|
| There are many people who do these types of searches -
| regularly. If there is any key material checked into gh,
| consider it immediately compromised, even if the repo is
| removed.
| capableweb wrote:
| Tons of automated scraping for these sort of things as
| well, with the whole flow of "check if key is working,
| check permissions, if authorized, generate new key and
| deactivate old one".
| IncRnd wrote:
| Yes. There are indeed several mitigations in place.
| However, my point was that within 5 seconds of looking at
| this tool I was able to uncover hundreds of potential
| issues. There are many other regexs I have on my list.
| Please don't fall into the trap of "we have anti-virus,
| so we can't get attacked."
| capableweb wrote:
| Sorry if I was unclear, my previous comment was about
| attackers doing that in order to compromise
| infrastructure, not that companies are doing it in order
| to defend themselves. But companies probably do it to
| defend themselves as well, but I'm not as familiar with
| that as with the other side of things.
| greysteil wrote:
| GitHub PM here, just confirming that that's correct (re: us
| sending detected secrets to the relevant service provider,
| who take action automatically). There's more detail here:
| https://docs.github.com/en/code-security/secret-
| scanning/abo....
| iambateman wrote:
| This is probably one of the single most important security
| measures on the whole internet.
|
| Thanks for keeping an eye out.
| zikohh wrote:
| https://sourcegraph.com/search?q=context:global+AKIA%5B0-9A-...
|
| Sourcegraph is much better than this site.
| madeofpalk wrote:
| Most of these actually look like fake keys to mock data in unit
| tests.
| arp242 wrote:
| Many of these are tests, false positives (e.g. base64 images or
| "Migration1599570560FixSlovakiaDisplayedAsSloveniaTest"), or
| examples from documentation.
|
| Can't find a single match for an actual API key.
| codalan wrote:
| These are probably invalid by now.
|
| https://grep.app/search?current=2&q=api_key%20%3D%20%22%5Ba-.
| ..
|
| As others have posted, Github implemented an alerting system
| for keys/passwords/etc. that have been checked into the
| codebase. My guess is most of these will be invalidated by
| now, either by the developer requesting a new key (and not
| publishing it), or the provider invalidating it for them.
| abraham wrote:
| Most of the Google API keys look like they are client side keys
| and should be configured to only work on specific origins or
| apps.
| kristopolous wrote:
| ----- BEGIN RSA NOT SO PRIVATE KEY -----
| asicsp wrote:
| Previous discussion: "Show HN: Search code in GitHub repos using
| regular expressions"
|
| https://news.ycombinator.com/item?id=22396824 _(614 points | Feb
| 23, 2020 | 155 comments)_
| moneywoes wrote:
| How Is this different than GitHubs search?
| 100k wrote:
| Hi, I work on GitHub code search. Grep.app is based on Solr and
| indexes about 500,000 public repositories. As far as I know,
| the author has not shared their tokenization strategy, but
| based on the the results I expect they are indexing trigrams to
| perform regular expression matching.
|
| The current GitHub code search is based on Elasticsearch and
| indexes more than 100M repositories. Its tokenization is based
| on whitespace, case changes (like CamelCase) and punctuation
| (like kebab-case) and strips out non-letter characters like <
| or { which is why it can't do exact match search.
|
| Our new code search, currently in beta and indexing about 45
| million repositories, uses an search engine we built in house
| that indexes content using a technique we call sparse ngrams.
| This allows us to execute searches faster than a trigrams
| index, while also being smaller than a positional trigram
| index. My teammate discussed some of the technology behind it
| in our blog post that was published yesterday:
| https://github.blog/2023-02-06-the-technology-behind-githubs...
___________________________________________________________________
(page generated 2023-02-07 23:02 UTC)