[HN Gopher] Analysis and reverse-engineering of the original Sta...
       ___________________________________________________________________
        
       Analysis and reverse-engineering of the original Starlink router
        
       Author : codewiz
       Score  : 122 points
       Date   : 2021-12-26 03:55 UTC (1 days ago)
        
 (HTM) web link (olegkutkov.me)
 (TXT) w3m dump (olegkutkov.me)
        
       | smoldesu wrote:
       | I like this router a lot, a few notes from using it for almost a
       | year now:
       | 
       | 1. It gets _really_ hot. Like, it kinda makes the plastic routers
       | seem well-ventilated.
       | 
       | 2. Congestion is, sadly, a real concern here. Too many devices
       | will definitely cause the router to choke a bit, and if you're
       | going to share it with a family that's just something to be aware
       | of.
       | 
       | 3. Latency is fine wired, but a little shaky over WiFi.
       | Especially recently; I'm not sure if this is a firmware issue or
       | what, but there's some funky QoS issues that I've noticed lately
       | where latency is wildly unpredictable. It almost shut down a Zoom
       | interview the other day...
       | 
       | 4. You get one LAN port. That's it.
       | 
       | Besides those gripes, it's _fine_ , but it leaves a lot to
       | improve on for the next iterations. Hopefully Starlink is taking
       | notes here, because as nice of a device as it is, it does feel a
       | little rushed for the purpose of getting it out the door.
        
       | richarme wrote:
       | > Also, there are multiple u-boot binaries with separate
       | environments. It could be done for redundancy or different modes.
       | I'm not sure.
       | 
       | This is typically done in order to be able to safely update the
       | u-boot bootloader, retaining the ability to boot the previous
       | version if the upgrade fails. Also called A/B firmware update.
       | The previous bootloader stage ("SBL" or "TZ", depending on secure
       | boot mechanism) might check a flag in a configuration partition
       | to see which loader should be started and whether a previous boot
       | to that loader was attempted, and will revert back to the
       | previously active one after an upgrade that fails to boot up, or
       | fails to pass secure boot integrity checks.
       | 
       | A failure could for instance happen if the power drops while
       | writing the update to the u-boot partition. Without such a
       | mechanism, an update failure would brick the device. Alternatives
       | are "never update the bootloader", or "hope for the best".
        
         | colejohnson66 wrote:
         | Some x86 motherboards (especially high-end ones) are now adding
         | this feature as well. They're sometimes referred to as "dual
         | BIOS" motherboards.
        
           | oblak wrote:
           | Minor correction: many motherboards, especially high-end
           | ones, have had this feature a decade or so
        
           | pixl97 wrote:
           | In this way BIOS's are so much better these days.
           | 
           | You can even boot and update your BIOS without a working
           | processor. Did this over Christmas when the mainboard/CPU
           | combo I got for a family member wasn't compatible. Just put
           | in a USB drive with the correct file name and press a button
           | on the board and it updated and I was good to go.
           | 
           | A little of topic, but just wanted to state how much better
           | these things are than the old days where it was easy to brick
           | a board.
        
       | andoma wrote:
       | Depressing to see a "reboot every 20 days" hack / workaround for
       | whatever stuff doesn't work correctly. OTOH the xfinity Arris
       | modem I used to have rebooted almost daily whenever it felt a bit
       | sad (I think if the BER crossed a threshold or something), so I
       | suppose it's an improvement if that's your baseline.
        
         | squarefoot wrote:
         | It wouldn't be a huge issue if the user had control on when the
         | reboot would occur. I mean, for many users it would be much
         | better to have a reboot at 4AM say every sunday morning than
         | randomly and possibly in the middle of something important. An
         | external timer power cycling the router at the desired moment
         | before the 20th day can solve the problem as the counter would
         | then restart, but it would have been trivial to implement it in
         | software.
        
           | IgorPartola wrote:
           | I think it absolutely is an issue. Don't drop my connection
           | more than once a year.
        
             | [deleted]
        
         | spmurrayzzz wrote:
         | This is very often due to upstream vendor issues. In the case
         | of routers, even if you're the one designing the IC's, building
         | all the layer 7 services, industrial design of the plastics,
         | etc; there's likely still a vendor providing your radios and
         | their drivers are almost always a black box for you.
         | 
         | Sometimes, while you go back and forth with the vendor in a
         | hapless effort to repro anomalous bugs, the best thing for the
         | customer is reboot-every-N-days instead of degraded-
         | performance-forever.
         | 
         | It sucks, but its commonplace.
         | 
         | (N.B. I work in this space and its definitely been a pain point
         | that we've had to accept to be able to make progress)
        
         | mschuster91 wrote:
         | The norm in embedded world, particularly if the component
         | giving you hard-to-debug issues is a "black box" where you have
         | to deal with binary blob firmware - and almost everything
         | dealing with WiFi or BT is closed source.
        
           | randomhacker123 wrote:
           | It is very likely that most Qualcomm customers like Starlink
           | have access to the source code of the Qualcomm proprietary
           | Wifi driver for the QCA wifi Access point SoCs. Some vendors
           | also have access to the source code of the proprietary Wifi
           | firmware running on the Tensilica CPUs inside the Wifi IP
           | core of the SoC. (Linux runs on an ARM CPU in this SoC, the
           | wifi IP cores are an extra realtime FW)
           | 
           | End consumers normally do not have access to some source code
           | or any documentation about these chips, sometimes even the
           | competitors are getting access to source code to integrate
           | their solution better. I do not know whom they protect
           | against, probably all people who did not sign a NDA.
           | 
           | These thinks are only shipped to end customers in binary only
           | version to protect the IP from someone. Often it is pretty
           | easy to tell the system it is now operating in a different
           | country (e.g. setting in Web UI) and then it will not comply
           | to the local radio regulatory requirements any more. From my
           | experience it is not the FCC which really demands it, please
           | blame the chip vendors for binary only driver first.
           | 
           | The worst hacker for a silicon vendor, where they normally
           | protect most against, is some guy like the author of this
           | blog post who analyses his device in much detail and tries to
           | run own software on it which was not verified by the vendor
           | first. Such software modifications could cause worse customer
           | experience because it is performing worse (which is funny if
           | they have to reboot the vendor software every 20 days) or
           | could cause extra support effort.
        
             | spmurrayzzz wrote:
             | In my experience, its exceptionally rare to have vendors
             | release code like that, if for no other reason than opsec
             | concerns. Obviously larger clients have much more leverage,
             | but even in those situations its usually because they're
             | getting a bespoke branch of whatever mainline firmware is
             | produced for their other retail products (assuming its not
             | from-scratch for a client that size in the first place).
        
             | baybal2 wrote:
             | > It is very likely that most Qualcomm customers like
             | Starlink have access to the source code of the Qualcomm
             | proprietary Wifi driver for the QCA wifi Access point SoCs
             | 
             | No. They are jus that hard to work with.
             | 
             | MTK, Qcom, BCom are all terrible with software support, no
             | matter who you are.
        
               | randomhacker123 wrote:
               | Candela Technologies for example has access to the QCA
               | wifi firmware source code for the QCA Wifi 5 AP chips.
               | They provide custom builds here:
               | https://www.candelatech.com/ath10k.php I know of one
               | other company. I do not know if they also have access to
               | the driver source code, but I assume so.
               | 
               | Silicon vendors often do not care about small customers,
               | small is probably less than 250k units per year. If you
               | are a customer which is expected to generate more than 5%
               | of the revenue for the complete product line or the
               | business unit then you get good support from these
               | companies.
               | 
               | The proprietary Broadcom Wifi driver was leaked multiple
               | times by some companies which did not run the script to
               | clean a GPL tar, but did something on their own and
               | forgot to remove the Boardcom wifi driver source code.
               | Last time I saw this was already 8 years ago. The
               | Broadcom proprietary Wifi driver also contained the
               | source code of the firmware running on the ARM chip
               | inside the Wifi IP, at least it did this some time ago.
               | 
               | For the Qualcomm 5G modem and the graphics driver this is
               | probably different.
        
               | baybal2 wrote:
               | I am talking about world brands, 5m+ units a month
        
               | mschuster91 wrote:
               | Unless you're Apple, I'd make an exception there - they
               | absolutely require support from their vendors for all
               | their non-standard stuff: AirPlay WiFi-based screen
               | mirroring, Handoff, Thunderbolt-based Target Display Mode
               | / Target Disk Mode.
               | 
               | Apple _always_ demands to be in control of what is
               | running on their devices, no matter what - even where it
               | 's pure software that's concerned, like with the NVidia
               | fallout - and they (usually) insist on quality... there's
               | a reason why Apple hardware feels so painless to
               | interoperate, compared to the Windows or Linux world.
        
               | mikevin wrote:
               | I doubt Apple ever had real access to something like
               | Intel ME though.
        
       | tyingq wrote:
       | Similar article reverse-engineering the embedded linux server in
       | the antenna. Though they didn't get as far. It also has an
       | STMICRO STSAFE chip:
       | 
       | https://www.esat.kuleuven.be/cosic/blog/dumping-and-extracti...
        
       | kybernetyk wrote:
       | I don't understand why star link is country bound. Here in the eu
       | you can't take your starlink equipment with you when you travel
       | to another eu country. It just won't work. Even if it's a
       | neighboring country. And even when starlink is offered in said
       | country. You need a country specific kit.
       | 
       | This is the only reason I didn't get a starlink subscription.
       | Seems like a bullshit limitation. Reminds me kinda of Tesla
       | stink.
        
         | Denvercoder9 wrote:
         | SpaceX has to allocate bandwidth on the satellites to
         | consumers. If consumers move, they will have overallocated
         | bandwidth in some places, degrading network performance.
         | They'll probably come out with a mobile product at a higher
         | price point in the future.
        
           | tpmx wrote:
           | Kind of doubtful that consumers moving their terminals around
           | would be statistically significant compared to all the other
           | factors that cause regional variations in aggregated
           | bandwidth usage.
        
             | rtkwe wrote:
             | If it were offered as a mobile service it would be very
             | attractive to people on the go so it would skew the number
             | of mobile nodes higher than expected. I think base station
             | discovery is another issue blocking mobile nodes from
             | happening so an in between step could be to provide a way
             | to easily move your base station and register it to a new
             | cell.
        
         | rtkwe wrote:
         | There's a couple reasons this is currently happening:
         | 
         | 1) The Starlink Constellation only looks for your base station
         | in it's allocated cell and isn't setup to scan and discover
         | mobile stations
         | 
         | 2) Right now there is no communication inside the constellation
         | (they might have launched a few with the laser links but it's
         | not turned on broadly) so the satellites are linking down to
         | another base station that has access out to the wider internet.
         | They need to provision access on those so they don't get
         | overwhelmed.
         | 
         | 3) The base station itself needs to know where it is well
         | enough to hit the active satellite for it's cell. You might be
         | able to do that without GPS but it is an issue you have to
         | address.
         | 
         | Both of these are supposed to be solved eventually and they've
         | said they are planning to allow roaming base stations
         | eventually.
        
           | BoxOfRain wrote:
           | I wonder when it'll be available for boats? It'd be brilliant
           | for yachtsmen etc.
        
         | xoa wrote:
         | > _I don't understand why star link is country bound._
         | 
         | First and primarily, the legal front: All regulated
         | electromagnetic spectrum transmission is country-bound, since
         | the regulators are by country. Private entities cannot legally
         | transmit in any given country without authorization, and there
         | is a fairly in-depth set of national and international legal
         | frameworks for this that are pretty well respected. In
         | principle SpaceX could negotiate the proper licenses and
         | agreements internationally to allow roaming, and they certainly
         | will eventually for certain applications like maritime and
         | aircraft usage. But it's perfectly understandable in their
         | bootstrapping phase of pure fixed point service (which is the
         | easiest for a host of reasons) that they just put a blanket ban
         | on it. Those who want mobile just have to wait.
         | 
         | Second, there are technical reasons. Starlink will always face
         | some fundamental density limitations in how small a cell any
         | sat can talk to and how much bandwidth there will be per cell
         | and for the overall sat. And in its initial version, satellites
         | also act purely as "bent pipes", mediating directly between a
         | ground station which then talks to the internet at large and
         | the end customer CPE. And SpaceX is being careful and
         | deliberate about building up all the infra needed to scale its
         | network, and wants to ensure that everyone joining gets a good
         | experience. So they're also being fairly careful about allowing
         | anyone out of their assigned cell.
         | 
         | Eventually a number of developments will alleviate this _to
         | some extent_. Intersat optical links are now launching on new
         | birds, and IIRC this year (maybe early next?) they 'll have the
         | in-space routing network up. That will dramatically increase
         | their flexibility in routing traffic and dealing with
         | chokepoints (as well as allowing Starlink usage out of range of
         | ground stations, again critical for ships/aircraft). Simple
         | increasing raw numbers of sats (and particularly VLEO v-band
         | with Starship) will improve their density margins a bit as
         | well. And then there is just plain old software infra stuff.
         | Portal for customers to update addresses, backend to deal with
         | handling allocations and looking for cell overload including
         | across varying regulatory regimes, figuring out pricing at the
         | margins and what to allocate for fixed vs mobile usage in a
         | given area, letting people see where there is room and where
         | there isn't, and putting a nice GUI over all of that is work
         | that isn't needed for a 1.0 MVP but will be important to
         | getting stuff to work right.
         | 
         | > _Seems like a bullshit limitation. Reminds me kinda of Tesla
         | stink._
         | 
         | No, this one at least is pretty reasonable. Easy to forget that
         | Starlink is doing something very different, new, with more to
         | come, and is very, very early. And still a chance it could all
         | fail, they haven't actually hit their core minimum economic
         | targets yet (that was what Elon's "bankruptcy" warning last
         | month was about). It's much more capex intensive and opex
         | intensive then traditional offerings. They've got an excellent
         | product though and are bootstrapping it impressively but
         | they've got to be very careful nonetheless. Plenty of
         | fundamentally incredibly promising businesses have foundered on
         | the shoals of scaling.
        
       | alufers wrote:
       | Woah, really in-depth write-up. It's sad that so many vendors are
       | locking-up their Linux devices, preventing technically savy
       | people from running their own software there (and also not
       | releasing drivers to their hardware, for example it is really
       | hard to find a 802.11ax router that is supported by OpenWrt).
       | 
       | > This app is written in Go, and it's really hard to disassemble.
       | Shouldn't it be easier than an app written in C/C++? As far as I
       | know Go embeds reflection metadata, including type information,
       | which should help with decompiling. I have never tried though, so
       | I might be wrong.
        
         | tyingq wrote:
         | I think SpaceX/Starlink is specifically trying to protect the
         | geo-fencing so they can sell a more expensive mobile product
         | for RVs, etc.
         | 
         | Edit: Maybe not?
         | https://publish.twitter.com/?query=https%3A%2F%2Ftwitter.com...
         | ... though it is already "later this year".
        
           | DenseComet wrote:
           | Currently the satellites don't have intersat links. There is
           | a ground station in each geofenced region, which makes
           | mobility difficult. I'd expect it to start working once the
           | intersat links are functional.
        
             | trompetenaccoun wrote:
             | When will that be?
             | 
             | Also how do they update hardware on a satellite? Or are
             | there other problems keeping them from establishing
             | intersat links for now?
        
               | wmf wrote:
               | They don't update hardware; they have to launch new
               | satellites that have laser links.
        
             | wmf wrote:
             | I don't think that's the issue. Starlink won't even let you
             | move from one cell to another even when the cells are
             | served by the same ground station. And moving from one
             | ground station to another is no harder than handing off
             | from one cell tower to another, which I assume was solved
             | 30 years ago. They just aren't allowing it.
        
             | dboreham wrote:
             | You do not need intersat links for mobility.
        
         | randomhacker123 wrote:
         | The MediaTek wifi AX platform is supported by OpenWrt master
         | and also supported by upstream Linux including AX wifi.
         | (MediaTek contributes a lot to upstream kernel)
         | * MT7622 SoC (2x Cortex-A53, including 4x4 ieee80211n wifi
         | integrated in MT7622 and supported by upstream kernel too)
         | * MT7915 Wifi 6 (ieee80211ax) 4x4       * MT7530 5 port 1G
         | switch (2.5G uplink)
         | 
         | You can find this hardware in the following devices:
         | * Linksys E8450 / Belkin RT3200 (same device just different
         | name and color of casing)       * Ubiquiti UniFi 6 LR       *
         | Also some others
         | 
         | See here the open source wifi driver for MT7915:
         | https://github.com/torvalds/linux/tree/master/drivers/net/wi...
        
           | alufers wrote:
           | Unfortunately the Linksys E8450 is unavailable in my country
           | (Poland) or is ridiculously expensive. I settled on a
           | Totolink X5000R which supposedly is supported by OpenWRT and
           | has a Mediatek MT7621A. I will receive it in 2 days and test
           | it.
        
             | paulcarroty wrote:
             | https://en.wikipedia.org/wiki/Mail_forwarding
        
       | ncmncm wrote:
       | If they put a few GB of RAM in the router, they could run a proxy
       | in it to play video out of a cache of traffic broadcast to all
       | terminals in range of a satellite and held by terminals that
       | actually want it, obviating need to upload and download multiple
       | logically-identical copies of content being watched (at possibly
       | different points in the playback sequence) from multiple
       | terminals in the area. This would reduce loading on the channels,
       | enabling more effective bandwidth from the very limited available
       | spectrum.
       | 
       | It would need adaptation at the application layer, but the data
       | management needed is minimal. It would also need some attention
       | at the central terminus / edge node serving the area, to identify
       | duplicate requested streams and shift a corresponding stream to
       | the broadcast channel. In turn, this would typically depend on
       | cooperation from the original source of a stream, typically
       | mediated by their contract with an edge cache service.
       | 
       | The extremum of usefulness for this facility would be when
       | hundreds or thousands of terminals in an area are used to watch
       | the same World Cup football match. In this case, the cache
       | storage footprint would be minimal, as almost all viewers would
       | be watching at the same point in the broadcast.
       | 
       | More commonly, a few blockbuster movies and local news might
       | dominate broadcast traffic. But no advance planning or personal
       | attention would be needed; any stream found being fed to more
       | than one downlink could be shifted to broadcast according to the
       | amount of bandwidth that would thereby be saved.
       | 
       | As laser inter-satellite links are added to the constellation, a
       | single uplink stream could serve simultaneous broadcast from
       | multiple satellites in a chain.
        
         | Scoundreller wrote:
         | You may want to look into Othernet, which does satellite
         | broadcast data distribution to rasp pi based terminals. But
         | they only have 2kbps to work with, so it's largely the day's
         | news and some broadcast messages.
         | 
         | But it's free and it works!
         | 
         | I understand some Canadian arctic cities want to do what you
         | say because they depend on x$/gb satellite links, but not big
         | enough for a Netflix or whatever appliance. But every time
         | there's an iPhone or windows update, everyone is downloading it
         | independently. It would also make sense to load a box of SD
         | cards with their supply flights and put them on-net. Dunno if a
         | formal system exists for that.
        
           | reaperducer wrote:
           | Thanks for the Othernet mention. I just looked into the
           | Wikipedia article and the company's web site, and it's
           | exactly the sort of tech I enjoy.
           | 
           | Unfortunately, even though its receiver was supposed to enter
           | production in November, it's still not available. Ordinarily
           | I'd blame 'rona/supply chain/whatever, but according to
           | Wikipedia, it failed to deliver a previous kickstarter.
           | 
           | When it's available, I'll buy one. But I won't be the first,
           | in case this turns into vaporware.
        
             | Scoundreller wrote:
             | Keep an eye on the forums. It seems to be a real thing, but
             | exactly how anyone got setup without the kickstarter
             | delivering, iunno. Be suspicious.
             | 
             | https://forums.othernet.is/
        
         | tyingq wrote:
         | The satellite based WiFi used on commercial aircraft do this,
         | with some on-demand content cached on the aircraft. I think
         | they also "tape-delay" some of the live stuff to deal with
         | short fade-outs. Google around for DVB-S2 and DVB-S2X.
        
         | fuzzy2 wrote:
         | No need for any caching, just do proper multicast like "real"
         | IPTV networks do. Other than that, I don't think that enough
         | people would watch the same shows within a reasonable timeframe
         | to justify the costs.
        
           | ncmncm wrote:
           | Multicast would save _exactly zero_ Hz of spectrum bandwidth,
           | which a moment 's thought would reveal to you. Allow me
           | encourage a habit of that.
           | 
           | Multicast routing uses, exclusively, point-to-point links.
           | So, a packet could go up to the satellite once, but would
           | need to be copied down to every terminal individually.
           | 
           | But internet video is in any case not distributed by
           | multicast UDP. Even if it were, you still would need caching
           | for viewers not watching identically the same frames at the
           | same time, because a multicast router does not do time-
           | shifting: all copies of each packet are queued to send
           | immediately.
        
             | fuzzy2 wrote:
             | Is that right. Do enlighten me then!
             | 
             | Of course the transport link would have to cooperate and
             | open a group-encrypted channel for every multicast group a
             | client is subscribed to. Basically how regular satellite TV
             | works, but with dynamically allocated bandwidth.
        
       ___________________________________________________________________
       (page generated 2021-12-27 23:01 UTC)