[HN Gopher] Show HN: Send a GitHub webhook to a private URL
___________________________________________________________________
Show HN: Send a GitHub webhook to a private URL
I work on the OpenZiti project and I have a CI server that accepts
GitHub webhooks, but I don't want to expose my server to the
internet with open ports. I used the Python SDK for OpenZiti
(overlay networking platform) to create a GitHub Action that sends
the webhook to my private server via overlay instead of via the
open internet. This GitHub repo is a template that I made to show
you how it all works. You can use it right away to run the sample
server (httpbin-go) and see the GitHub Action in...action. Relevant
threads include https://news.ycombinator.com/item?id=32596212 .
Author : qrkourier
Score : 60 points
Date : 2022-09-21 12:43 UTC (10 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| caseysoftware wrote:
| Have you considered ngrok? You can open up a private tunnel from
| your local environment (local machine, container, device,
| whatever) to the public internet in seconds. Then you can layer
| on IP restrictions and even webhook verification to block any
| traffic _not_ from your provider of choice. It 's as simple as:
|
| _ngrok http 80 --verify-webhook=slack --verify-webhook-
| secret=[secret]_
|
| with a ton of providers out of the box:
| https://ngrok.com/docs/cloud-edge#webhook-verification
|
| Also, we recently launched https://webhooks.fyi/ to serve as a
| community resource to capture patterns & practices around webhook
| implementations. That's a github pages site so pull requests
| welcome!
|
| Disclosure: I work at ngrok and helped create webhooks.fyi :)
| gz5 wrote:
| ngrok free is great for smaller scale, as you describe.
|
| openziti simplifies scale:
|
| + mTLS
|
| + zero trust w/ inbound firewall rule of deny-all (rather than
| ACLs)
|
| + private DNS w/ wildcard domains
|
| note: mTLS, wildcard domains etc are in ngrok $900/user annual
| plan but these are free for foss like the OpenZiti solution
| used by OP (and maybe free for other solutions too?)
| qrkourier wrote:
| That's useful that ngrok has centralized webhook verification!
| It's meaningful security for the first hop from GitHub to
| ngrok.
| ChicagoBoy11 wrote:
| Disclosure: I have never paid for ngrok but have used it
| sporadically for many years for some quick tunneling use-cases.
| It feels... magical, and a joy to use. OP, if it fits your use-
| case, would recommend giving them a try!
| [deleted]
| ethomson wrote:
| Neat stuff - certainly this problem crops up quite a lot where an
| internal server needs to get GitHub webhook data.
|
| In the past, I've had good luck using a webhook proxy. I've
| mostly just used https://smee.io/ which is simple and lightweight
| although seems to be mostly abandonware at this point. I
| dockerized it so that it could be used in a Kubernetes cluster,
| which was very useful for my GitHub Actions build cluster:
| https://github.com/ethomson/smee-client
|
| There's also Hookdeck, which I haven't used in production, but
| have played around with, and it seems conceptually the same, but
| can be made more Enterprisey. Whether that's a bug or a feature
| is probably up to you.
| qrkourier wrote:
| The proxy idea is interesting too. Does a webhook proxy entail
| a polling model for events? That is, does the private server
| have to poll the proxy to receive the webhook? I wanted the
| GitHub event to push to trigger actions on the private server.
| ethomson wrote:
| No - your local server will still listen for webhooks, but
| they'll come from the proxy's client software.
|
| Basically, you set up your GitHub webhook URL as the proxy
| server (for example, smee.io). Then you run a client on your
| local machine that connects to the proxy server. When a
| webhook is fired, it will be sent to the proxy, then
| delivered to the connected client, which will then pass it
| along as a webhook to whatever machine you've configured.
|
| There's disadvantages to having all this stuff running, of
| course, so I think that handling this at the networking layer
| instead of putting a proxy just for webhooks into place is an
| interesting strategy. Certainly, it sounds like the right
| solution if you're already using OpenZiti.
| hangonhn wrote:
| We ended up implementing an API Gateway in AWS that just proxies
| the request to our CI server after a simple authentication (and
| also checking the signature of the request as suggested by GH).
|
| Another related option is that you can run GitHub Runners in your
| own environment and they will connect to GitHub to accept "jobs"
| from GitHub Actions. This is another thing we've started doing as
| we look to pare down our self managed CI. This is a very solid
| choice I think. In case anyone is interested:
| https://docs.github.com/en/actions/hosting-your-own-runners/...
| qrkourier wrote:
| That's a neat angle. I reckon you could position the self-
| hosted runner network-adjacent a private CI server and poke it
| directly from behind the firewall with a GitHub Actions job.
| scottydelta wrote:
| _> but I don 't want to expose my server to the internet with
| open ports._
|
| One simple approach I take to solve this issue is by whitelisting
| (using ufw/VPS firewall) Github's webhook IPs listed at
| https://api.github.com/meta
|
| This works flawlessly while keeping your CI server secure.
| qrkourier wrote:
| Did you find a way to auto-update your firewall from the
| dynamic allow list in the GitHub API?
| corytheboyd wrote:
| I was doing this with Cloudflare IPs, and iptables as the
| firewall. Pretty simple bash script scheduled with crontab
| worked just fine. Was doing this on a DD-WRT flashed netgear
| router. There's probably something similar that can be done
| here
| scottydelta wrote:
| Github rarely changes it's hooks IPs.
|
| The current list has 4 IPv4 IP range and upon checking my
| server firewall(last updated 3 years ago), I can see I have
| the first 3 entries in there.
|
| So in the last 3 years, Github has added 1 new IP range which
| is missing from my server but even then, no webhook call has
| ever failed to my CI server.
|
| As a precaution I just updated my server firewall right now.
|
| You could of course write a cron script to regularly check
| Github hooks IPs and update firewall if Github changes it's
| webhooks IPs.
| qrkourier wrote:
| Glad you got it updated before you missed an event! That's
| the worry that made me look for something flexible and
| software-defined that I could run in GitHub Actions.
| scottydelta wrote:
| from my previous comment:
|
| _> You could of course write a cron script to regularly
| check Github hooks IPs and update firewall if Github
| changes it 's webhooks IPs. _
|
| This is way easier and simpler than any other solution.
| It will be a mere 6 line script.
| Yeroc wrote:
| The bigger worry would be if they removed some IP
| addresses from their list. Those IP addresses would be
| juicy targets for hackers to scoop up and attempt an
| attack knowing that people have whitelisted them and that
| they allow access to what is likely relatively poorly
| protected infrastructure.
| Spivak wrote:
| It's totally trivial in almost all setups. Here's Linux.
| ipset create ghwebhooks hash:net iptables -A INPUT -m
| set --match-set ghwebhooks src -m tcp --dport 443 -j ALLOW
| # in /etc/cron.daily ipset add ... ipset del
| ... ipset list ... ipset save
| >/etc/ipset.conf
| linsomniac wrote:
| I do something similar for AWS IPs in our iptables firewalls.
| I create an ipset for "aws", and then create rules that match
| that ipset. I then have a script that runs periodically and
| downloads the set of AWS IPs (AWS publishes that) and writes
| then runs the script that creates the ipset.
| 0xbadcafebee wrote:
| Hi @qrkourier, you mention using the Python SDK, but it is not
| shown in the list of SDKs here, FYI: https://openziti.github.io/
| And also there's only a couple listed here:
| https://openziti.github.io/api/index.html I therefore assumed
| there was no Python SDK...
| qrkourier wrote:
| I see what you mean. I'll make sure to suggest adding it in
| those places, then! There's another post about the Python SDK
| in https://news.ycombinator.com/item?id=32923851 and the repo
| is https://github.com/openziti/ziti-sdk-py/#readme
| qrkourier wrote:
| Also relevant, sending a GitHub webhook to private Jenkins server
| with custom params: https://news.ycombinator.com/item?id=31503429
___________________________________________________________________
(page generated 2022-09-21 23:01 UTC)