[HN Gopher] The Miserable State of Modems and Mobile Network Ope...
       ___________________________________________________________________
        
       The Miserable State of Modems and Mobile Network Operators
        
       Author : hasheddan
       Score  : 114 points
       Date   : 2025-02-26 11:55 UTC (11 hours ago)
        
 (HTM) web link (blog.golioth.io)
 (TXT) w3m dump (blog.golioth.io)
        
       | PaulHoule wrote:
       | > The number one reason we see customers opt to not use cellular
       | when > another connectivity option is on the table is due to
       | concerns around > cost and reliability.
       | 
       | For "reliability" read "availability". Cellular networks are
       | profitable because carriers serve customers that are easy to
       | serve and leave the rest alone. Vehicle 2 Vehicle communications
       | mediated by cellular is a non-starter because people drive a lot
       | in places where cell phones don't work. Wireless internet has
       | been a false hope that's spoiled realistic plans for universal
       | service because it's only affordable in _markets that are already
       | served_. Large-scale government and corporate IoT programs find
       | that 20% or more of stations are in places without service, etc.
        
         | amluto wrote:
         | On top of this, there are markets that are not well covered due
         | to local politics. I can go just a few blocks away into a
         | neighborhood that has plenty of density to be interesting to
         | the major carriers but has basically no usable cellular data. A
         | couple years ago, there was a big fight between Crown Castle
         | (big cellular network contractor) and the local government.
         | Crown Castle wanted to build cell towers. The residents wanted
         | service but didn't want cell towers in their back yard. The
         | government tried, and failed, to avoid issuing permits, because
         | federal law sensibly prevents local government from effectively
         | prohibiting the construction of cell towers.
         | 
         | Win? No! The local government is very proud of itself for
         | having pulled a rabbit out of its hat. It turns out that a cell
         | tower is useless without backhaul, and the government rolled
         | out so much red tape that Crown Castle determined that they
         | would never recover the costs of installing the backhaul and
         | abandoned the entire project. The local government somehow
         | considers this to be a victory.
        
           | PaulHoule wrote:
           | What I remember in the 3G era was that my MVNO phone [1]
           | worked great in medium sized cites like Rochester, NY but
           | poorly in places with lower density and even worse in the
           | highest density places like NYC and Hollywood. 4G had the
           | capacity to serve super density urban areas really well.
           | 
           | [1] I've got a suspicion that Tracfones ride the back of the
           | bus and get worse service than you'd get with a premium-
           | priced plan.
        
       | Delphiza wrote:
       | As part of our IoT offerings, we tried quite hard to build
       | devices with mobile connectivity about 7 years ago. This was just
       | as low power 5G, NB-IoT and similar technologies were going to
       | become a thing. We gave up because it was too much effort for
       | little return, and it was better to focus on doing things with
       | data, rather than collecting it. Even getting prototypes up with
       | powered fanless PCs (SBCs) and Mini-PCIe or M.2 modems was harder
       | than it should have been - you would think it would be easy with
       | off-the-shelf devices and drivers.
       | 
       | Small Internet connected devices are still needed, despite the
       | perception that IoT is dead. Mobile networks and the modem supply
       | chain are definitely holding the market back. On the plus side,
       | for stationary powered devices most people are happy to connect
       | to wi-fi. For low power devices, LoRa, with private gateways,
       | seems to be a standard. Mobile that is used in outdoor vehicle
       | and asset tracking is still stuck with fighting with modems as
       | per OP.
        
         | jon-wood wrote:
         | You might want to take another look, I went through a similar
         | process a couple of years ago, and am now doing it again
         | because our original equipment supplier decided they'd like a
         | life (RIP PC Engines). There's a thriving industry of companies
         | providing fanless ARM based machines either shipping with LTE
         | modems, or with M2 & SIM slots on the board to provide your
         | own.
         | 
         | We install in industrial environments where an accessible
         | internet connection is far from a given, having LTE on all our
         | devices means that we can almost always give the device a way
         | to call home. I can strongly recommend Compulab's devices,
         | which you can purchase as a fully assembled unit that just
         | needs a SIM card put in it (I can't recommend the Linux
         | distribution they run, but you're free to flash them with
         | whatever you'd like).
        
           | sumtechguy wrote:
           | > There's a thriving industry of companies providing fanless
           | ARM based machines either shipping with LTE modems, or with
           | M2 & SIM slots on the board to provide your own
           | 
           | When I did this about 10 years ago we had quite a number to
           | pick from. Build your own was possible. But that was only
           | because of our organization had the capability and expertise
           | to do it. We settled on 3 off the shelf ones. That was
           | dependent on cost and number of I/O the customer was needing.
           | 
           | DiY basically is 'first make your own computer with the
           | ARM/MIPS/x86 chipsets' then 'spin your own special firmware
           | for it'. Then 'build your own ground up linux distro or
           | similar with compiler chain and SDK to work with it'. You may
           | be able to get someone to sell/give you a reference
           | layout/SDK. Then after all of that. You are now ready to add
           | in a modem. Also prepare for the certifications of all the
           | mobile networks you want to run on. Plus software for you to
           | interact with the cell modem. Oh also you need to work on
           | getting yourself provisioned correctly in the mobile
           | networks. When you do for five devices it is a couple hours
           | of playing with an API. But you probably want hundreds of
           | them so be prepared for managing that, plus billing. Oh also
           | you need to manage EoL for your parts. Many IoT installations
           | are looking at you hanging around for 10+ years.
        
             | astrobe_ wrote:
             | > Also prepare for the certifications of all the mobile
             | networks you want to run on
             | 
             | AFAIK the modem maker actually does this. You would rather
             | check that the chip is certified for what you need before
             | you buy.
             | 
             | > also you need to work on getting yourself provisioned
             | correctly in the mobile networks
             | 
             | AFAIK also, this is a service provided by "virtual
             | operators", companies that pass deals with the main
             | operators and provide advanced services for machine-to-
             | machine uses, for instance.
             | 
             | But perhaps these didn't exist 10 years ago. Things really
             | started to move where I operate when they EoL'd copper
             | wires.
        
           | Delphiza wrote:
           | Thanks... I'll have another look. I always assumed that
           | device/modem supply would catch up, but they always seemed
           | held back by the established mobile network operators. Also,
           | covid-related supply-chain issues stopped a lot of products
           | in their tracks. We would waited up to a year for modems in
           | some cases. I suppose that has all flushed through the system
           | by now.
        
           | throwway120385 wrote:
           | Seconding this about Compulab. If they made a box with a
           | Marvell CN9130 it would be really nice.
        
         | jlarocco wrote:
         | Why mobile connectivity instead of Bluetooth?
         | 
         | As an end user, I prefer my devices use Bluetooth and a hub
         | device like a phone or laptop, rather than each one have a
         | direct mobile connection.
        
           | crote wrote:
           | Because these aren't regular consumer gadgets. Think more
           | like "lamppost mounted traffic counter", Bluetooth
           | connectivity to a smartphone would be completely useless.
        
       | kmeisthax wrote:
       | > The patents are referred to as Standards Essential Patents, or
       | SEPs. Because of this arrangement, a vendor like Nordic probably
       | can't open source their modem firmware even if they wanted to.
       | 
       | Why would a SEP license forbid publishing modem firmware? The
       | IP[0] involved with the modem firmware is copyrighted and trade
       | secret material; publishing the firmware does not (and cannot)
       | grant a patent license to those licensed standards-essential
       | patents. It only imperils your implementation of the modem
       | firmware, not the whole patent spec.
       | 
       | To wit, Cisco publishes an H.264 codec with BSD-licensed code,
       | even though H.264 is under a shitload of patents that they are
       | licensing. OpenH264 _does_ ship with an additional patent
       | sublicense that applies if you are using their build and
       | installing it in a specific way, but that is particular to MPEG-
       | LA 's licensing structure[1]. Publishing an implementation of a
       | patent you are licensing does not automatically sublicense the
       | patent.
       | 
       | [0] Laws that grant the ability to dictate the conduct of your
       | competitors
       | 
       | [1] Cisco is abusing(?) the MPEG-LA royalty fee cap to shield
       | FOSS entities from having to take a license, basically, by paying
       | for them.
        
         | gjsman-1000 wrote:
         | I suspect a big reason is that there's middleware: some of this
         | stuff is so complex, that like video games, there's probably a
         | cottage industry of companies selling partially implemented
         | bits and pieces, or licensing across companies of source code,
         | or companies only licensing the patents if you agree to never
         | release code. There's also the IP angle: if you do manage to
         | figure it out, there's no honor helping Huawei.
        
           | baby_souffle wrote:
           | > I suspect a big reason is that there's middleware
           | 
           | I'd bet it's simpler. Saying "yes" to the 'can we share this
           | outside of this company?' Question means a lot of work and
           | meetings and red tape and somebody is going to have to rope
           | in legal and ... and ... and ...
           | 
           | Or, "no, get back to work".
        
             | gjsman-1000 wrote:
             | True, but who owns most of the patents? Some of the most
             | litigious companies on earth (Qualcomm, Broadcom, Cisco).
             | How many legal cases are more expensive than patent
             | litigation?
             | 
             | Also, it's very simple: Imagine you approach Qualcomm. The
             | script probably goes something like this:
             | 
             | "Sure, we'll license you that patent, for $0.92 per device,
             | a $25K per month maintenance fee, and compliance with our
             | terms and conditions - chiefly, don't license this patent
             | to anyone else, and don't share the source code of your
             | implementation."
        
           | AnotherGoodName wrote:
           | I honestly just figured it was spectrum licensing. You can't
           | sell devices that can broadcast at power on just any old
           | frequency. At a hardware level these modems are capable of
           | going well outside the bounds of mobile device regulations.
           | See all the 'turn xyz into a software defined radio' hacks
           | that have been done before on other devices on the market.
           | 
           | So the firmware is locked down. To do otherwise will break
           | spectrum licensing as the device then becomes capable of
           | doing things outside the original licensed purpose.
        
             | ndiddy wrote:
             | That's probably it, the FCC won't let you sell a device
             | that has a user-accessible method to violate wireless
             | regulations. At most, they could release the source code
             | but not the signing keys so end-users could inspect but not
             | change the firmware.
        
               | crote wrote:
               | Wouldn't the FCC come into play at a later level? We're
               | talking about providing source code to _device
               | developers_ , not end users. As long as the device
               | developers ensure the products they release are locked
               | down, the FCC shouldn't care about it - they only look at
               | the final product as sold to end users.
        
               | PinkSheep wrote:
               | I'm not too keen on this topic, but router firmware
               | usually asked YOU, what country you (it, the router)
               | resides in to allow or block additional frequencies. No
               | source required.
               | 
               | And then Intel still goes and makes their "automatic
               | geolocation for frequency policy based on neighboring
               | APs" mechanism the default and only policy in their
               | wireless Wi-Fi drivers.
        
             | mschuster91 wrote:
             | > So the firmware is locked down. To do otherwise will
             | break spectrum licensing as the device then becomes capable
             | of doing things outside the original licensed purpose.
             | 
             | So what. Take pretty much any older transceiver, swap out
             | the quartz(es) and suddenly what used to be a ham radio can
             | now communicate with police radios, or a commercial radio
             | can be converted for ham use.
             | 
             | Yes, of course the former case is illegal and the latter
             | legal as long as you don't transmit without a callsign and
             | a ham license. But still, radio manufacturers were not
             | required to stop people from doing such mods - to the
             | contrary, up until the 90s it was commonplace to have
             | detailed schematics and BOMs in radios, and it was just as
             | commonplace for people to mod their radios.
             | 
             | I don't get why the FCC is putting up so much pressure on
             | locking down devices ffs.
        
         | burnte wrote:
         | > Why would a SEP license forbid publishing modem firmware?
         | 
         | Contracts and money.
         | 
         | > The IP[0] involved with the modem firmware is copyrighted and
         | trade secret material; publishing the firmware does not (and
         | cannot) grant a patent license to those licensed standards-
         | essential patents. It only imperils your implementation of the
         | modem firmware, not the whole patent spec.
         | 
         | Publishing the source to the firmware would allow people to
         | make use of patented tech without paying for it even though it
         | would be illegal to do so. Also, your license may simply
         | stipulate that you cannot publish or share your code in anyway
         | outside the company.
         | 
         | > To wit, Cisco publishes an H.264 codec with BSD-licensed
         | code, even though H.264 is under a shitload of patents that
         | they are licensing. > Cisco is abusing(?) the MPEG-LA royalty
         | fee cap to shield FOSS entities from having to take a license,
         | basically, by paying for them.
         | 
         | As you point out here, Cisco is paying for it. They publish
         | open code, and to ensure the LA doesn't come after them,
         | they're paying additional money.
        
           | gjsman-1000 wrote:
           | There's also the possibility of trade secrets being involved.
           | An example hypothetical: Maybe Qualcomm has tons of lab data
           | showing signal performance and integrity for any given
           | channel, band, power level, general environment (urban,
           | suburban, rural), device position, device altitude, etc.
           | 
           | You might want to use that data, and algorithms derived from
           | it, to improve your device's battery life. That data
           | obviously cannot be patented, and while the lab data would be
           | copyrighted and not part of the finished product, Qualcomm
           | would understandably be mighty particular about any open-
           | source code implementing the conclusions.
        
           | p_l wrote:
           | MPEG-LA also explicitly _used to_ (I do not know current
           | state) consider open source  "some assembly required"
           | releases to be non-infringing.
           | 
           | It was product incorporating the patented tech, i.e. ready to
           | use binaries, that required licensing.
           | 
           | Cisco paid the licensing fees to enable easily downloadable
           | h.264 codec for certain projects, but you could legally build
           | it from source yourself.
           | 
           | IIRC h.265 had similar licensing
        
       | aftbit wrote:
       | Wow that was quite the technical deep dive! I bet there's not one
       | person in the world who understands all the interactions in this
       | system.
        
         | linuxguy2 wrote:
         | I too greatly enjoyed the article. Love those deep
         | troubleshooting write-ups!
        
       | awelkie wrote:
       | I think the miserable state of cellular modems ultimately comes
       | from the power dynamics of private spectrum. The 3GPP protocols
       | are complex because they're not really beholden to the
       | implementers, but rather to the network operators. The operators
       | are fine with additional complexity if it serves their interests,
       | and the cost will just be passed on to the OEMs. The network
       | operators have all the power because they're the ones with the
       | licenses.
       | 
       | WiFi tends to be simpler in part because the protocol authors are
       | working more in the interest of the implementers, since it's
       | really they who decide whether to adopt or not. Obviously a gross
       | simplification but I think it's at the heart of the problem.
        
         | dilyevsky wrote:
         | Vendors wrote those standards though, not the operators. They
         | are complex because it's a defensive moat against competition
         | imho
        
           | tinktank wrote:
           | They are complex because, usually, they are a union of
           | everything every vendor in the consortium has done and/or
           | wants to do.
        
             | astrobe_ wrote:
             | Looking at the _overview_ of the architecture on the
             | network side [1] tells the tale, I think.
             | 
             | My take is that telecom operators forcefully had to evolve
             | from Morse and copper to IP and radio, hacking stuff left
             | and right every step of the way.
             | 
             | [1] https://en.wikipedia.org/wiki/IP_Multimedia_Subsystem#A
             | rchit...
        
               | dilyevsky wrote:
               | Long time ago i worked on WiMAX devices which was a
               | competing standard for the term "4g" from ieee and while
               | it was simpler it wasn't a whole lot simpler and
               | performance sucked, so there's that
        
       | hasheddan wrote:
       | Hey folks, author here. Appreciate the comments and discussion on
       | the post! Happy to continue the discussion or answer any follow-
       | on questions folks have about our investigation and resolution.
        
         | sharpshadow wrote:
         | Good read thanks. To the point of having a open platform, they
         | would probably lose their monopoly. I could then just spin up
         | my own mobile network and provide service like a wifi network
         | with any credentials and do what I want.
         | 
         | I guess there needs to be a big initial effort with lots of
         | maintaining afterwards. If everything goes as expected the AIs
         | could provide that in the future.
        
       | ForOldHack wrote:
       | The complaint is that people cannot connect, and that hardware
       | vendors dont want to go down the pit of other peoples sh*(blobs).
       | 
       | Magnificent article. Clearly shows his expertise going down the
       | sh*/rabbit hole, the more I read, the more dismal it got.
       | 
       | Clearly the way to get around the technical side of the DNS bug,
       | is to do what cell providers do: Start the device in the office
       | of the provider, and let the DNS resolve there, and if it does
       | not work, get a ticket started right then and there.
       | 
       | Out in the field... use another devices Wifi, and do the same
       | thing.
       | 
       | If you cannot get cell phone service or Wifi, then ( I literally
       | took about 45 mins on the phone to T-Mobile, which immediately
       | refused to take responsibility, and also refused to put
       | _anything_ in writing: Then their recommendation is to get a
       | satellite link with, StarLink!. I would rather be slapped in the
       | face with a cold fish.
       | 
       | Cell phone service providers all let this happen, until of course
       | they almost all sold out to T-Moble. ( LifeLine, Assurnace, etc..
       | )
        
       ___________________________________________________________________
       (page generated 2025-02-26 23:01 UTC)