[HN Gopher] Vxlan over WireGuard (On OpenBSD)
___________________________________________________________________
Vxlan over WireGuard (On OpenBSD)
Author : hucste
Score : 65 points
Date : 2023-02-14 13:33 UTC (9 hours ago)
(HTM) web link (rob-turner.net)
(TXT) w3m dump (rob-turner.net)
| blakesterz wrote:
| I love how this starts off "What is a VXLAN..." that was going to
| be my first question! So often posts like this seem to assume
| everyone knows what the topics are at the start. I know what
| WireGuard is, and I've at least heard of VXLAN before, but I
| couldn't remember what it was.
| candiddevmike wrote:
| Just do static routes or BGP over Wireguard. Simpler, scalable,
| less error prone.
| vxxzy wrote:
| Does this get you Layer 2 connectivity?
| candiddevmike wrote:
| No, because most applications don't need layer 2 adjacency.
| vxxzy wrote:
| You may consider VoIP phones. When a phone boots up on a
| network segment the DHCP process comes into play. This
| process is a Layer 2 process. The DHCP packet could contain
| a boot server field (this is typical) so the VoIP phone can
| grab configuration. One may want the boot server info
| "isolated" from other network segments. Utilizing VLANs is
| one way to do this. Additionally, it is typical that QoS is
| applied at Layer 2 (something necessary for real-time
| protocols like VoIP).
| candiddevmike wrote:
| I hope you're not trying to run VoIP phones via vxlan
| over wireguard because you don't want to setup a local
| DHCP server
| chungy wrote:
| Maybe he'll write a blog post about it :)
| iso1631 wrote:
| Or dhcp relay to a central dhcp server
| vxxzy wrote:
| Over VXLAN yeah... You won't believe how many small-
| medium companies are moving from on-prem to small managed
| private clouds. VXLAN allows them to maintain their
| existing network configuration. "You hope" ha! I suppose
| by this you mean that the wireguard + vxlan may not be a
| "mature" combination for something as mission-critical as
| voice traffic? VXLAN implementations by mature companies
| (Netgate, Fortinet, etc...) those work. Maybe this
| wireguard+vxlan would work too.. who knows!?
|
| Edit: I see you changed your comment to consider remote
| DHCP which causes my comment above to be irrelevant. Oh
| well. The fact still holds that VXLAN is meant to handle
| Layer 2 so saying BGP is an alternative is like putting a
| square peg in a round hole.
| [deleted]
| gorkish wrote:
| I appreciate that you are trying to give examples of "L2
| things" but none of this is really good advice.
|
| For QoS, what do you think double-encapsulating all of
| the QoS bits inside of two tunnels is going to do for
| you? If you pay your carrier to respect QoS, you have to
| actually give them the bits! QoS goes on the OUTSIDE of
| the onion.
|
| DHCP for VLANs should be handled by snooping for it on
| the switch and forwarding over L3 (ip helper-address),
| not by running vxlan over a VPN tunnel. And really it
| shouldn't ever be done over a potentially unreliable WAN
| link either; you should have a local agent for DDI at
| each site.
|
| The only story here is that L2-over-L3 VPN have their
| very specific niches, and VXLAN+Wireguard is an
| alternative to L2TP+IPSec that is (IMO) both easier to
| configure and more reliable when such a thing is needed.
| SoftTalker wrote:
| > VXLAN+Wireguard is an alternative to L2TP+IPSec that is
| (IMO) both easier to configure and more reliable
|
| I'm sorry to ask but which alternative do you find easier
| and more reliable? I seem to be able to parse that either
| way, maybe it's just me.
| gorkish wrote:
| Sorry for the dangling preposition. "That" refers to
| VXLAN+Wireguard being easier and more reliable. It has
| the drawback though of having very high overhead at 130
| bytes/packet, and it can be very tricky to use over the
| public Internet without paying lots of special attention
| to tuning the MTU of all devices on the bridged segment.
| I'm glad to know that there are geeks who have not been
| exposed to the horrors of IPSec, though. And yeah, you
| should pretty much never do L2VPN. It is perilous in the
| extreme.
| justsomehnguy wrote:
| >maybe it's just me.
|
| Nope, here with the same problem and I _don 't_ treat
| L2TP/IPSec as an easier to configure and reliable.
| [deleted]
| lormayna wrote:
| As someone that has fight for years with L2 in the WAN, I
| will not advice that at all. Just installa a DHCP server
| and transport the traffic over L3. You have lot more
| control both on routing both on QoS
| crest wrote:
| I've found that they work just fine. In my case all that
| was required was setting a DHCP option pointing them at
| their controller. The QoS to make it work well under load
| would've been the same for a L2 tunnel.
| zymhan wrote:
| Then it's not actually an equivalent solution.
|
| L2 connectivity is still quite useful, even if you don't
| have a need for it.
| johnklos wrote:
| "Here's how I did this thing."
|
| "You don't need to do that thing."
|
| How do you know?
| sophacles wrote:
| I've been in or very close to networking for a long time.
| Here's a certainty: there's lots of ways to do anything.
| Here's a corollary to that: someone will be sure to tell you
| how bad any choice you make is.
| iso1631 wrote:
| But doesn't provide layer 2 between networks. Think of devices
| that are hardcoded to communicate with broadcast or multicast
| with TTL of 1, you either need some active reflector, and cope
| with any perculiarities of the device, or you simply extend a
| single vlan between two routers (using vxlan or another
| solution)
|
| I sometimes need to extend a system like this from one site to
| another. One is a calrec system (an audio mixer, I think it's
| the control traffic that needs to be sent), and I don't have
| enough access or time to see if I could build some kind of
| transparent proxy -- it won't work with multicast routing.
|
| I do however have enough time to create a layer2 network
| between two nics. I tend to use mikrotiks for that, create an
| eoip tunnel (GRE with proprietary addons to cope with
| fragmentation) between the two endpoints and pop the interface
| in a bridge with a physical port, and move on.
| icedchai wrote:
| I'm doing OSPF over Wireguard (running BIRD on Debian, not
| OpenBSD though.) It works pretty well.
| yokem55 wrote:
| The feature I would be interested in is if this can do link state
| toggling on the on the vxlan interface if the wg handshake timer
| goes stale. If that works, then it becomes practical to do things
| like run ospf routing over the vxlan interface.
| [deleted]
| gorkish wrote:
| What exactly would you be intending to accomplish? OSPF already
| has state timers, and furthermore runs just fine on a wireguard
| interface without having to introduce a vxlan tunnel.
| jmclnx wrote:
| I could not get into the site, so from archive:
|
| https://web.archive.org/web/20230214134248/https://rob-turne...
|
| A little bit over my head, but an interesting read.
| tristor wrote:
| There's a weird font-rendering bug on this site that causes the
| text in the code blocks to be unreadable unless you highlight it
| with your mouse. If you enable Javascript, it seems to fix it.
|
| Not sure if the author is reading this thread, but it's something
| you may find worth investigating fixing.
| [deleted]
| Arnavion wrote:
| Works for me with JS disabled, unless the author fixed it in
| the two minutes since you posted your comment. Code blocks
| have: style="color:#f8f8f2;background-
| color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"
|
| ... in the HTML itself.
| tristor wrote:
| The bug still appears present for me:
|
| Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0)
| Gecko/20100101 Firefox/109.0
|
| Code blocks without JS: https://i.imgur.com/Tcq34IK.png
|
| Code blocks without JS under highlight:
| https://i.imgur.com/Lbvk2LS.png
|
| Code blocks with JS enabled: https://i.imgur.com/kDCW4Q0.png
| generalizations wrote:
| This can cause massive packet fragmentation. I'd be most
| interested in the performance degradation due to the L2
| encapsulation. Are there any benchmarks available for this kind
| of project?
| jacob019 wrote:
| That's what I was thinking, unless you have jumbo frames you're
| going to have a hard time stuffing ethernet frames into IP
| payloads. Does Vxlan mitigate this somehow?
| Denatonium wrote:
| I've tunneled VXLAN over Wireguard on Linux. In my setup, my
| WAN's MTU was 1500 bytes, and my Wireguard tunnel's MTU was
| 1550, with the VXLAN's MTU being 1500. Surprisingly, traffic
| and iperf3 tests going over the VXLAN had much better
| throughput than traffic going directly over the Wireguard
| connection. IIRC, over the VXLAN, I was pulling ~800Mbps over
| the VXLAN/WG setup with iperf3.
|
| Where this would fall apart is if there are firewalls in
| between that silently drop UDP fragments. In a case like that,
| it may be necessary to do VXLAN/Wireguard/Wireguard to conceal
| the fragmented packets with MTUs of 1500/1550/1440
| respectively, assuming IPv4 and WAN MTU of 1500. I bet this
| would come with a significant performance hit though.
| anton5mith2 wrote:
| I took this to extremes last year: I used it to run MAAS from
| Australia to sweden (which requires layer 2). Granted I used
| tailscale to make the WireGuard part even easier, but it was a
| lot of fun.
|
| https://medium.com/@antongslismith/bare-metal-cloud-provisio...
| floatinglotus wrote:
| The difficulty isn't setting it up, it's in determining how much
| traffic you can put through it before it breaks.
|
| How does it handle segmenting jumbo frames? Etc.
| noobface wrote:
| This is fun, but applications requiring L2 adjacency do it to
| limit latency/distance. Creating a L2 domain between here and the
| moon, what are you gonna use it for? Certainly not anything other
| than fun.
| skullone wrote:
| There's a number of specific scenarios this could be useful,
| like, some SANs can only replicate to L2 adjacent units. Say
| you wanted a replica off-site, and your gear is
| older/proprietary, you used to have to buy enterprise network
| gear to encap L2 and ship VLANs to remote sites. I wouldn't be
| dismissive of using VXLAN over wireguard to accomplish that.
| anton5mith2 wrote:
| Indeed check out my other post here - and it definitely was
| fun!
| miller_joe wrote:
| This is great! One small suggestion: try it with the newer veb
| bridge device. It should be a bit faster.
| crest wrote:
| IP over Ethernet over VXLAN over UDP over IP over WireGuard over
| UDP over IP over Ethernet... _sigh_
|
| OpenBSD does support both routing domains and multiple routing
| tables and includes multiple routing daemons in the base system.
| I would recommend to the author to stop hacking at the keyboard,
| grab whatever not to structured visualisation tool works for them
| (e.g. a whiteboard, a block of paper, a random drawing app,
| Visio) and (re-)phrase the problem. Are you solving a problem or
| showing of how many acronyms you can expand without looking them
| up? This n layer encapsulation can work and can even be required
| to reproduce some (problematic) organisational structure, but
| it's far from elegant. Given the chance I would vastly prefer to
| just use multiple routing domains for the WireGuard tunnel
| interfaces and the underlay. It would result in far less
| complexity to manage as well as less overhead.
|
| Why do so many people insist on tunneling Ethernet over IP?
| What's keeping operators from using IP routing (and just one
| layer of encapsulation) instead? Is IP routing so scary or
| everyone that indispensable applications that only work over
| Ethernet?
| systems_glitch wrote:
| I end up having to run basically this very setup (on OpenBSD,
| too) because I have a customer who has a Novell NetWare 5 setup
| and runs IPX only. Bad times.
| gerdesj wrote:
| NetWare 5 can do IP fine.
|
| I used to run a cluster of DNS/DHCP servers that were the
| first on site to run 5. The rest were 4.11 until we binned
| them for 6. Three cream coloured Compaq 3U lumps.
| aroulin wrote:
| Using vxlan you can also connect L3 networks, not just L2
| networks. i.e., virtualize an L3 network
| jacob019 wrote:
| Wireguard virtualizes L3 out of the box.
| [deleted]
| supertrope wrote:
| vMotion needs L2 adjacency to make live migrating VMs easy.
| Some software rely heavily on broadcast discovery messages and
| are thus designed for LAN usage not Internet connectivity but
| businesses try to stuff a square peg into a round hold.
| wyager wrote:
| There are a ton of protocols that don't work using cross-subnet
| IP routing, e.g. anything that uses multicast.
| lstodd wrote:
| Because stuff that requires this circus of encapsulation is
| usually so brain-dead that it can't be gotten to work in any
| other less horrible way.
|
| (also I think you lost one 'over UDP')
| crest wrote:
| You're right. I forgot that WireGuard sits on top of UDP.
| jeroenhd wrote:
| Just the grateful that nobody has tried to wrap the entire
| thing in JSON over HTTP yet! I wouldn't be surprised if we get
| Wireguard over websockets for "enterprise" applications soon.
|
| Sometimes you just need an L2 tunnel. Most of the time you
| don't, but when you do, you do. For example, if you use IPv6
| over SLAAC in a private network, you'll need to route NDP.
|
| In the rare cases that you do need an L2 tunnel between two
| different locations, you probably want some kind of
| authorisation and authentication of the traffic to prevent
| injection/spoofing attacks and to make life just a but harder
| for the NSA (Google's use of HTTP was one way the NSA managed
| to tap connections that were otherwise encrypted by HTTPS).
| After all, this isn't just any traffic, these are internal
| subnets.
|
| In terms of authorised traffic, Wireguard is quite lightweight
| and foolproof. Perhaps IPSec is even more lightweight but it's
| a pain to set up. The alternative would be to wrap all internal
| network traffic in an encrypted protocol and set up the
| necessary whitelists in the upstream ISPs.
|
| The impact of such layering depends on the network connection
| between the data centers. If you can get jumbo packets across,
| fragmentation won't be a problem at all. If you run your own
| fiber between data centers, there's basically no downside until
| you're reaching very high saturation network saturation.
___________________________________________________________________
(page generated 2023-02-14 23:01 UTC)