[HN Gopher] Speedbump - a TCP proxy to simulate variable network...
___________________________________________________________________
Speedbump - a TCP proxy to simulate variable network latency
Author : sph
Score : 232 points
Date : 2024-01-16 12:47 UTC (10 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| Uptrenda wrote:
| Checkout also shopify's awesome tool called toxiproxy:
| https://github.com/Shopify/toxiproxy
|
| It turns out to be also a very good way to test a networking
| library by implementing it. Since your stack needs to be able to
| basically handle most adverse events properly.
|
| The idea behind 'chaos engineering' is cool.
| sph wrote:
| Yes, I found toxiproxy first, but the client-server model
| didn't suit me, while speedbump is perfect for my use case --
| simulating HTTP latency.
|
| I am developing a progress bar for a web crawler, and testing
| on localhost is too fast to notice if there is any issue. With
| speedbump, I just do `podman run --net=host
| kffl/speedbump:latest --latency=1s --port=8001 localhost:8000`
| and test my crawler on http://localhost:8001
|
| Neat tool.
| miry wrote:
| I believe a similar approach could be achieved using
| toxiproxy and configuration. The documentation at
| https://github.com/Shopify/toxiproxy#2-populating-toxiproxy
| provides insights into the process. While it may not be as
| straightforward as with speedbump, the flexibility of
| toxiproxy makes it a viable option.
|
| Toxiproxy primarily serves the purpose of integration into
| tests. It proves particularly useful when testing features
| like the progress bar directly from code. Notably, for Go
| applications, integration is seamless, eliminating the need
| for an additional application. Instead, you can run the
| Toxiproxy server part directly from the codebase. An
| illustrative example can be found at https://github.com/Shopi
| fy/toxiproxy/blob/main/_examples/tes....
| londons_explore wrote:
| I'd like this to go one layer deeper... Simulate a _network_ with
| a variable latency, and see how typical TCP implementations
| behave over it.
|
| Far too many realtime things, like games and videoconferencing
| (both of which frequently use TCP, despite it being badly suited
| to the application), perform really badly when the bandwidth
| increases and decreases, for example as I walk around a building
| with wifi and pass concrete pillars.
|
| I _want_ not a single dropped frame in those circumstances. Sure
| - you can have some lower res frames, but I don 't expect to see
| 30 frames dropped in a row and a big glitch.
| clbrmbr wrote:
| I wrote such a tool[1] for my own use some years ago. It uses
| libpcap to grab all the packets on one interface and then re-
| emit them from another interface after some delay, with
| optional packet loss.
|
| If there's real interest I could dust it off... maybe rewrite
| in Rust.
|
| [1] https://github.com/chrismerck/lagsim
| ajsnigrutin wrote:
| You can do this with a linux router and 'tc'
|
| https://www.pico.net/kb/how-can-i-simulate-delayed-and-dropp...
|
| (just a short example)
| laserbeam wrote:
| Can't you just do it locally with tc without a router? I have
| artificially throttled / simulated delays with tc before and
| I don't see why you need a router.
|
| PS. My knowledge of tc is very rudimentary. I only had to use
| it once.
| jrockway wrote:
| You don't need a router. I've used it on `lo` before.
| ysleepy wrote:
| IIRC FreeBSD had dummynet which allows creating slow, laggy and
| lossy virtual networks.
| derhuerst wrote:
| AFAIK mininet [0] can help you with this. It allows configuring
| latency and/or jitter dynamically using Python [1].
|
| [0] https://mininet.org [1]
| https://mininet.org/api/classmininet_1_1link_1_1TCIntf.html#...
| oneplane wrote:
| GNS3 can do that
| jsnell wrote:
| If you're specifically looking to probe the behavior of
| different TCP implementations, I think you want something
| slightly different than this. First, you want the faults to be
| deterministic (e.g. not random 1% packet loss but packet loss
| at a certain point in the connection); and you want the limits
| to be per TCP connection, not per link. If you don't have that
| determinism or connection isolation there are just going to be
| too many confounding variables in addition to the TCP stacks
| you're supposed to be testing.
|
| I wrote a tool like that a few years back[0], but didn't
| publish most of what I discovered about e.g. how different CDNs
| tuned their TCP stacks (just a couple of anonymized
| examples[1]).
|
| [0] https://github.com/teclo/flow-disruptor
|
| [1] https://www.snellman.net/blog/archive/2015-10-01-flow-
| disrup...
| lelanthran wrote:
| > I want not a single dropped frame in those circumstances.
| Sure - you can have some lower res frames, but I don't expect
| to see 30 frames dropped in a row and a big glitch.
|
| This is quite difficult, I think. You only know how bad the
| connection is _after_ dropping a frame or two, and the
| intervening network will detect it before the endpoints do.
|
| From an application level, I suspect that on the socket layer
| level you can't do anything about TCP _anyway_ , the underlying
| layer will retransmit after a delay.
|
| [EDIT: Just realised you meant _video_ frames, and not _network
| frames_. I think my comment still applies though - dropping a
| single transmission unit that is part of an I video frame still
| means discarding that entire frame, whilst dropping a B or P
| frame is going to result in glitches.]
| londons_explore wrote:
| With closer integration between the application and the
| network layers, you can see things like the RSSI dropping
| (basically signal strength). This can trigger the application
| to start sending less data before any actual data loss
| happens - and the application at the local end can respond in
| under a millisecond rather than a whole network roundtrip.
|
| For the traffic from the remote end, data packets can consist
| of "here is the low res data" and another packet for "and
| here is the extra data to turn the low res into higher res".
| The high res would be sent with QoS markers so that network
| hardware knows that if anything is to be dropped, drop that
| first. The different QoS streams could even be sent with
| different transmit powers and modulation schemes on the WiFi
| network.
| withinboredom wrote:
| RSSI goes up and down all the time for a variety of
| reasons. That's not really an indication of network speed
| though, only link speed. You can have a wifi connection
| over several km's if nobody else has wifi routers (in real
| life, they do, so you can't go much further than your
| house) and still have a really good link speed (from
| experience).
|
| Network speed is a function of link speed AND the success
| of packets to reach you. When you are on wifi, your device
| gets a very tiny slice of time to send packets. If you have
| more devices, your device gets less time to send packets.
| Further, a "loud router" nearby (or self-interference ...
| or radar in 5G space -- or a microwave, or Bluetooth in 2G
| space) can cause a frame to be interfered with and never be
| received. Thus it has to be retransmitted.
|
| RSSI is only a very small part of your overall network
| conditions and doesn't really tell anyone anything.
| ahoka wrote:
| What do you think TCP should do when bandwidth lowers?
| jrockway wrote:
| This little tool is showing its age:
| https://gfblip.appspot.com/ but as you walk around you'll
| probably notice that you aren't getting any packets through to
| the Internet at all. That's why your video drops out, you
| aren't connected to the Internet anymore.
|
| What Zoom does is notice this and play the video frames in its
| buffer back more slowly in an attempt to smooth over the "no
| data coming from the Internet" period. I don't find this that
| helpful because if you comment on something someone just said,
| they've moved on to a new topic by the time Zoom unbuffers and
| displays all the frames. Clever hack, I prefer the complete
| dropout.
| westurner wrote:
| Very many apps poorly perform with intermittent network
| connectivity, in Diaster Relief scenarios.
|
| More app developers could help others by testing with simulated
| intermittent connectivity.
|
| From "Toxiproxy is a framework for simulating network conditions"
| (2021) https://news.ycombinator.com/item?id=29084277#29088775 :
|
| > _Many apps lack 'pending in outbox' functionality that we
| expect from e.g. email clients._
|
| > _- [ ] Who could develop a set of reference toxiproxy 'test
| case mutators' (?) for simulating typical #DisasterRelief
| connectivity issues?_
| withinboredom wrote:
| My favorite is not filling buffers before sending the packet
| and then everything with a crappy internet connection (dropped
| packets + high latency = TCP retransmissions) is suddenly doing
| 120kps because you're only sending 50 byte packets and every
| 10th one is getting lost. Meanwhile, that's a thread on your
| server not doing useful work.
| apitman wrote:
| Would you mind describing this problem more fully? I'm
| currently learning TCP more deeply and this sounds
| interesting but I don't quite understand the issue.
| diggan wrote:
| I looked into doing something similar for testing various
| ActivityPub implementations in various network sizes and
| conditions.
|
| Turns out, I already had everything installed on my machine to do
| it, via `tc` (Explained a bit here:
| https://wiki.archlinux.org/title/advanced_traffic_control), which
| apparently came with the iproute2 package on my distribution.
|
| With that, you'd be able to run something like this to add
| latency on a specific interface: tc qdisc add
| dev eth0 root netem delay 100ms
|
| Really easy to use, works well in docker container too, comes
| with a bunch of different conditions you can apply (delay, packet
| loss, duplication) and you might just have it installed already.
| lttlrck wrote:
| The nice thing about speedbump is the ability to modulate the
| impairments. tc cannot do that.
|
| That could be really useful for maybe simulating weather
| effects on satellite/RF over time?
| tussa wrote:
| tc can simulate various network issues:
| https://man7.org/linux/man-pages/man8/tc-netem.8.html
| camtarn wrote:
| You could easily build something that could repeatedly call
| the tc command and feed it different levels of impairment.
| Doing so to simulate weather is a great idea. If you had a
| wave-rider buoy measuring wave height every half-second or
| so, you could even simulate the impairment of an RF
| connection over water due to waves.
| camtarn wrote:
| Yeah, tc/netem/tbf is amazing. I built a simple Python GUI over
| the top of it and ran it on a Pi in a touchscreen case - just a
| little black box with "Drop packets: [0%] [1%] [10%] [50%] /
| Corrupt packets: ..." etc. Clients were very impressed.
|
| I'm actually quite surprised that a similar frontend doesn't
| seem to exist as a commercial hardware product, unless we
| missed one in our search.
| BiteCode_dev wrote:
| You have a product here.
| lifeisstillgood wrote:
| Yeah - I was looking around this week and think there is a
| "gap" for a good simple proxy - speed bump is nice but I am
| thinking just let my application do it's thing, then let me
| replay certain traffic, that JSON needs to
|
| I think a programmable proxy with simple hooks is what I
| want
| jandrese wrote:
| The downside of tc is that it is kinda weird and tricky to get
| it to operate on incoming packets.
|
| I had to write my own emulator once to simulate a specific
| commercial satellite terminal. The terminal had the behavior of
| queuing up packets until they hit a certain threshold or
| exceeded a time limit and then bursting them out in a big blob.
| It also "helpfully" reordered small packets to the front of the
| queue for better latency, which made TCP stacks very cross.
| jmspring wrote:
| It has been inactive for awhile, but the name says a lot -
| https://github.com/tylertreat/comcast
| kayyyy wrote:
| amazing name
| RajT88 wrote:
| A similar tool for Windows I have used:
|
| https://jagt.github.io/clumsy/
| ComputerGuru wrote:
| Looks cool but judging purely by the screenshot it seems to be
| systemwide w/ filters rather than per-adapter?
| RajT88 wrote:
| Correct, but you can use filters to determine which packets
| it targets. Source IP is one of the filter criteria, so
| presumably that would get you your adapter targeting.
| baq wrote:
| +1, used it like a decade ago to test out various
| intercontinental scenarios, results matched reality, would
| recommend.
| walth wrote:
| You can do the same thing on Mac with built-in tools
|
| ``` # Setup pipe sudo dnctl pipe 1 config bw 1Kbit/s delay 800
|
| # Setup matching pf rule echo "dummynet out proto tcp from any to
| 127.0.0.1 port 11211 pipe 1" | sudo pfctl -f -
|
| # Turn on firewall sudo pfctl -e
|
| # Test time nc -vz 127.0.0.1 11211 Connection to 127.0.0.1 port
| 11211 [tcp/*] succeeded! nc -vz 127.0.0.1 11211 0.01s user 0.00s
| system 0% cpu 1.333 total ```
| rhizome wrote:
| Dummynet and the rest of this functionality comes from FreeBSD,
| where it has existed for a long time. I was doing packet loss
| testing with it 15+ years ago. Works great!
| qainsights wrote:
| Great utility, will test this today.
| dmarinus wrote:
| years back I've used linux netem for such a purpose, back then it
| was already fabulous
| NelsonMinar wrote:
| Every software engineer working on interactive Internet
| applications should be required to use a tool like this in their
| daily work. Need QUIC as well as TCP, ideally all UDP to catch
| DNS too.
|
| I'm convinced 90% of webapp bloat would go away if the people
| building them didn't have a gold plated Cadillac computing
| experience.
| galleywest200 wrote:
| You can do this in Firefox with their Development Tools! I am
| sure Chrome has some form of equivalent in its Development
| Tools.
|
| https://firefox-source-docs.mozilla.org/devtools-user/networ...
|
| Admittedly this only works for front-end, browser based
| testing.
| irisgrunn wrote:
| Yeah, Chrome has the same feature
| Flimm wrote:
| This only throttles the speed, it doesn't drop packets,
| AFAIK.
| hunter2_ wrote:
| In a browser context, doesn't the HTTP stack sort out any
| issues in lower layers such as those stemming from packet
| loss (e.g., packets arriving out of order) such that what's
| presented to the client app (and debuggable by the
| developer thereof) is just an HTTP response or lack
| thereof? That lack thereof (or additional latency) _could_
| be due to packet loss, but it doesn 't have to be, and it
| doesn't really matter because it's abstracted away.
| youngtaff wrote:
| Chrome throttle's at the individual request level, and not at
| the network level so can't simulated loss due to contention
| e.g. downloading a multi-MB page over a 3G connection
| wang_li wrote:
| FreeBSD also has dummynet as part of ipfw that allows injection
| of latency, bandwidth caps, queue sizes, and packet loss. Same
| thing as in MacOS.
| e145bc455f1 wrote:
| Is that the same as tc in linux?
| INTPenis wrote:
| I'll never forget at my first job back in 2004 my manager
| configured our FreeBSD IPFW firewall so that it slowed down ICMP
| responses. So whenever someone pinged us it looked like we had
| the highest response times.
|
| He was a joker that guy.
| anthk wrote:
| Not latency, but tritty can simulate a 56k/ISDN-like connection
| speed.
| jedberg wrote:
| This is what we built at Netflix and called latency monkey.
|
| It turns out determining that a downstream service is slow is a
| lot harder than determining if it's unavailable, so it was an
| important way for us to test how services handle slowdowns and
| network problems as well.
|
| It was really simple, it just dropped some configureable
| percentage of packets, which would force a resend, causing the
| other side to get packets delayed and out of order.
|
| It ended up finding a lot of problems in our error handling code
| for network access.
| girishso wrote:
| Recently wanted to simulate slow network on Mac and came across
| Network Link Conditioner, it's pretty good. No need to setup
| proxy or anything else. It needs to be installed from Xcode
| additional tools.
|
| https://nshipster.com/network-link-conditioner/
___________________________________________________________________
(page generated 2024-01-16 23:00 UTC)