[HN Gopher] The Curse of Dialup World
___________________________________________________________________
The Curse of Dialup World
Author : tyoma
Score : 154 points
Date : 2023-10-01 09:54 UTC (1 days ago)
(HTM) web link (allenpike.com)
(TXT) w3m dump (allenpike.com)
| manicennui wrote:
| I've never had an ISP that I've liked since the dialup days. I
| remember liking a couple of the ISPs that I had at that time a
| lot, but EnterAct in Chicago is the one I remember most
| distinctly. They were very transparent about what was going on,
| provided great service, and did things like offer different
| numbers for different 56k protocols so you could get the best
| speeds and stability depending on which modem you owned.
|
| Unfortunately EnterAct was eventually sold to some larger company
| and DSL became more popular.
| bArray wrote:
| What would truly keep you up at night is how many small/medium
| businesses still do stuff like this.
|
| One small business I have seen just has tonnes of word documents
| sitting in a single folder, with a mix of customer details,
| invoices, etc. To write a new invoice they copy an old one, then
| change the details, save it with a new name with a "system" (that
| changed over the years).
|
| Up until semi-recently I saw a small-medium sized business (hired
| quite a few people) that did all of their accounting _by hand_,
| for hundreds to thousands of transactions. They had an old lady
| that came in to manually run their accounts on a calculator and
| it took several days.
|
| A small business I once dealt with had a small black book (A6?)
| that was all written in note form, scattered, not dated, random
| important scribbles everywhere, mix of cash and card (not
| documented) - it was a nightmare. Somehow they had a good
| reputation in the local area, but behind the scenes it was pure
| chaos.
| GiorgioG wrote:
| I consulted for a Blue Cross Blue Shield affiliate (big health
| insurance group for our non-US readers) that used Excel
| spreadsheets for managing all their plans and driving their
| claims processing. This was a company that did 2+ billion
| dollars a year in business back in 2007.
|
| It was mind-boggling. They had a team of 20 people updating
| spreadsheets. Hopefully they've gotten their stuff together by
| now - but I doubt it. We tried to get them to buy a product
| data management suite (they were bad at building software in-
| house)...they finally agreed...then the recession hit, and they
| reversed course.
| didgetmaster wrote:
| I can relate. I was building a new kind of data management
| system when I approached a small mortgage company to be one
| of my first customers. Startups will bend over backwards to
| get those first few customers on board, so I worked closely
| with them to show how my software could make their business
| run much smoother.
|
| They had a separate spreadsheet for every state (all 50 of
| them). Some states just had a small number of customers in
| them (e.g. Wyoming). Others had large numbers of customers
| (e.g. California).
|
| Their support team could only update one spreadsheet at a
| time. So adding a customer in Texas required locking that
| spreadsheet until the transaction was complete. Doing reports
| required compiling data from all the relevant spreadsheets.
| Queries that could be done on a relational database in under
| a second, took many minutes (or hours).
|
| I tried to show them how my system could easily load all
| their data and make everything work much more smoothly (and
| saving them hundreds of man-hours); but they just couldn't
| see the value of spending a few hundred dollars a year for a
| license for my software.
| zdragnar wrote:
| I wonder if it wasn't the value (people-hours aren't hard
| to calculate) but a risk assessment. If their system has
| been working just fine, but the new system entailed major
| risks (startup going out of business, hosted off-site,
| backups not in their control, PII / security issues) the
| calculation gets a lot worse.
| zdragnar wrote:
| I can confirm a large retail company in the US ran all of
| their inventory purchasing through an Excel file that was
| managed by a small (<10 people) group.
|
| They were bought out by an even bigger company a few years
| back, but they were managing to chug along until the mid
| 2010's at least.
| jellyfishbeaver wrote:
| Large/medium businesses, too. In finance, we receive countless
| reports from 3rd parties. I can't tell you how many of them
| still rely on person A to manually log into our FTP server at a
| specific time every day and drop an Excel file into a folder.
| Sometimes it goes on like this for years. I always wonder if
| that person ever gets sick.. goes on vacation..
| ghaff wrote:
| A long ago company I worked for that subsequently went
| through a couple of acquisitions still basically had one
| person in finance somewhere who was basically who you
| contacted for anything related to our defined benefit
| pension. I assume the bus factor wasn't literally one as
| there were legally required annual statements etc. but I
| assume there would have been some manner of chaos if this
| person from a no longer existing company wasn't there one
| day. (About 7 years ago, the benefit administration--now a
| responsibility of Dell's--transitioned to one of the big
| benefit administration companies; OI expect the person who
| had been handling it retired.)
| ecshafer wrote:
| A scary amount of the finance world runs on batched csv and
| whatever those || column deliminited files are called, just
| being FTPd around, processed in batches. Which then trigger
| more FTP and more file processing.
| reassembled wrote:
| Hey, that's the same way my company wrote a ton of code in the
| past.
| mvandermeulen wrote:
| I have colleagues still running on this code
| [deleted]
| deely3 wrote:
| What will be alternative and not-so-expensive solution? Is
| there a lot of easy to install, support, customize solutions,
| cheap to use oriented on a small businesses?
| pjc50 wrote:
| Round here it seems like Quickbooks is the default. They even
| advertise on TV.
| SpaceNoodled wrote:
| Sadly, that's an Intuit product.
| inversetelecine wrote:
| and one that's filled with years of cruft with a
| subscription service tacked on but they don't really do
| anything and just milk it.
| nicolaslem wrote:
| GNUCash would be a step in the right direction.
| thsksbd wrote:
| ... I'll add reasonably user friendly to OP's requirements.
| nicolaslem wrote:
| It's not THAT bad. I'd argue that if after watching a few
| tutorials on Youtube and playing a bit with the software,
| someone is not able to grasp the concepts, it's a strong
| indication that this person shouldn't be responsible the
| accounting of a small business.
| ddmitriev wrote:
| Wave (https://www.waveapps.com/) works well for me.
| Admittedly, I have not explored things like accrual
| accounting (I just do everything on a cash basis), so I don't
| know how well this is supported, and I'm perplexed that there
| is no easy way to generate EBITDA reports from within Wave
| itself, but basic tracking of expenses and transfers across
| multiple vendors and accounts works smoothly.
|
| [Edit] Also, most CPAs I have talked with know and can
| integrate with Wave.
| dusted wrote:
| > otherwise "surf the web" as people honest to god called it,
|
| What else to call it ? browsing the web ?
| camtarn wrote:
| Cruising the Information Superhighway.
|
| (Not even a joke.)
| johnnyworker wrote:
| https://www.youtube.com/watch?v=yFPQbnraeVg
|
| "Surfing across the world with multimedia, surfing day and
| night on the data Autobahn"
|
| And the verses are so much worse.
| toast0 wrote:
| https://www.youtube.com/watch?v=FNNThHfBxTc
|
| Lazin' in the Shade (of the Information Superhighway)
| eddieroger wrote:
| I really miss the term Information Superhighway, and I don't
| know why besides nostalgia. It felt like there was so much
| possibility and road ahead. If only we'd have known where
| that road ultimately ended.
| Jigsy wrote:
| > If only we'd have known where that road ultimately ended.
|
| A Toll Booth called a "Paywall."
| hinkley wrote:
| Because Cable Television with "57 Channels and Nothin' On" had
| already taught us about channel surfing. Subsequently couch
| surfing became part of the vernacular.
| JohnBooty wrote:
| - When a customer submitted their signup form to Dialup World,
| they would add a row for that new customer to their
| 20,000-row Excel spreadsheet. Then, they would put that
| signup form in a pile with all the other signup forms of
| customers who had signed up on that day of the month.
| - Every day, they would find which one of the 31 piles of literal
| signup forms corresponded to that day of the month.
| - Then, they'd charge the customers in that pile, one by one, for
| a month of service.
|
| Well, if you hop into a time machine and need dial-up access from
| "Dialup World" back in the 1990s, I guess there's a sweet hack --
| sign up on the 31st of the month and you only get billed 7 times
| a year.
| meepmorp wrote:
| Or pick Feb 29, and it's just once every 4 years on average.
| brazzy wrote:
| The article appears to build up though a chain of bad business
| and technical decisions to something "delightfully and poetically
| dysfunctional", but what that thing actually is stays
| frustratingly implicit and doesn't actually make sense to me.
|
| As far as I can tell it's this: Their user management consisted
| of an excel sheet, but their billing was based on physical piles
| of filled out signup forms. A user cancelling their contract was
| implemented by (only) removing their physical from, but this was
| not communicated to the acquiring company, so they started
| billing customers who had cancelled.
|
| But then how did they stop cancelled customers from using the
| service? How did they _implement_ a user authentication mechanism
| in the first place? There must have been a software
| representation of user records somewhere in their system, which
| was kept up to date with cancellations. And the acquirer never
| learned about that, which would have just been a dub mistake, not
| exactly delightful or poetic.
|
| Unless... they somehow managed to have the dialup system query
| the excel sheet directly and never stopped cancelled customers
| from using the service because they assumed those had switched to
| broadband anway and wouldn't want to use free dialup even if they
| could.
|
| These would have been a much more interesting WTF than the mere
| existence of the Excel sheets and piles of physical forms, and I
| would expect them to be mentioned explicitly.
|
| So... what gives?
| belthesar wrote:
| Usually in these days, auth was done via RADIUS. It's possible
| that the company handoff was done by not-the-sysadmins, and the
| operators of the company only had the things they knew of how
| things worked, which was the spreadsheet, whereas the
| spreadsheet to the sysadmins was just the tool they used to
| create the RADIUS account information off of.
| russdill wrote:
| Yes, there's a ton of build up but very little tying up at the
| end of the story.
| lgeorget wrote:
| Is it possible that the subscription came with a modem lended
| by the company which the customer had to send back to cancel
| their subscription? I think nowadays in the US people tend to
| buy their router (people more knowledgeable than me, please
| correct me) but in France for instance, one usually rents their
| router from their ISP as part of their subscription. Maybe it
| Dial-up World used to do the same in 1999?
| brazzy wrote:
| Hm, interesting idea. And then they might not have done user
| authentication at all. Also an interestingly bad practice,
| plausible, but non obvious and not in any way hinted at by
| the article.
| omgmajk wrote:
| From the post:
|
| > one of the aggrieved chose to show their displeasure by
| posting the list of all Dialup World user accounts online -
| along with every password in clear text.
| relaxing wrote:
| No, subscription hardware was not done anywhere to my
| knowledge. You bought your modem at a local store like
| CompUSA or through a mail order catalog. Actually finding a
| service to dial in to was left as an exercise to the user.
| (Hence why AOL famously mailed out floppies (and later CDs)
| to every household in America on a regular basis.)
| Yhippa wrote:
| I like that they at least had _A_ process rather than none.
| hinkley wrote:
| SEI Capability Maturity Model has 5+1 Levels.
|
| Level Zero is "we have no idea what our process is". You get a
| cookie and promotion to level 1 just for writing down your
| batshit process, which some engineers find particularly
| cathartic and thus are all too eager to contribute. Pithy
| things like "and then we play phone tag and finger point for a
| week until someone gets fed up and volunteers to get it done,"
| can end up in the first draft.
| tptacek wrote:
| 20,000 phone lines in 12 cities would be like 70 PRIs per city?
| It's a big order, but doesn't seem like one that MCI wouldn't
| happily cover. Unless "city" means "suburb", and not like major
| metro market. Just nerding out, because I ran (tech for) a mid-
| sized ISP in the late 90's.
| h2odragon wrote:
| Little towns were fun. Had a buddy at the regional ISP (that
| became a CLEC but that was later) tell me how they had bought
| all the remaining capacity on the switches in 6 of the 10 towns
| they served. There's wasn't any ISDN out to these places yet,
| so they had to have old school closets full of modems in
| buildings near each CO in these towns. They had to do a
| microwave backhaul from one of the closets to their network
| because AT&T couldn't sell them _any_ data lines into or out of
| that CO.
|
| Helped them shuck and rack the USR courier modems for a couple
| closet expansions. they'd got a local carpenter to make them a
| standard 16 wide modem board carrier that was very slick.
| Still, punching down lines and getting all that stuff woven
| together was _tedious_.
| apike wrote:
| Yes, good point - I should have written towns, rather than
| cities. I just pushed a fix, thanks.
| kmbfjr wrote:
| In the early 2000s, it was pretty uncommon to bring in all
| those PRIs from the cities, most telcos supported foreign
| exchanges through a combination of DMS-100 (brand name Centrex
| in Ameritech world) features and circuits. Near the end of the
| retail dial-up ISP, it was not uncommon to cover entire states
| through that and have them come into groups of PRIs.
|
| The outfit I worked for started in 1990 and had actual POPs
| with aggregated 3Com, Livingston and Micom serial aggregators
| that went to a central location over leased lines. They had
| 1500 lines between retail SLIP/PPP and a couple bulletin
| boards. That was quite the monster.
| tptacek wrote:
| I don't know if this disagrees with you or not, but we were
| done with individual dial-up modems by early 1996, and spent
| most of that year on Ascend's platform (the Livingston PM3's
| the next year; I left in the middle of 1997 to go work for a
| vulnerability lab, and I definitely set up a bunch of PM3's,
| so that's the timing worked out for me).
| DamonHD wrote:
| I started and ran a small early (UK) ISP in the 90s! I can't
| remember how we did the billing but it was better than that. I
| suspect that it involved awk with : separated fields, because I
| remember my dyslexic colleague saying that all he could see was
| swimming : characters... And I (still) like awk.
|
| I had more incoming phone lines than the rest of the street
| sometimes (this business was run from home), at several addresses
| around London as we moved. And we broke each telco's billing
| system in a new and interesting way...
| gwbas1c wrote:
| Wow, so many screwups...
|
| 1: Did the billing system (and its problems) show up in due
| diligence?
|
| 2: Did they ever figure out who leaked all the passwords and sue
| them? (Honestly, given that this generally soured the
| acquisition, the perpetrator could have been on the hook for a
| lot of money.)
|
| (Now, before you jump to the leaker's defense, remember that in
| tech, things can change fast, and we all will end up working for
| an obsolete or acquired company at some time. Part of the game
| isn't pissing on the seat on the way out.)
| oofnik wrote:
| > If you don't work somewhere that rewards curiosity, find
| somewhere else to work.
|
| I wish everyone had this privilege.
| dgacmu wrote:
| I too ran a small ISP from 1995-1997 or so, and the biggest gift
| I got from one of my co-founders was the strong suggestion we use
| this new cheap database called mSQL [1] as the basis of our
| accounting system. And I did. I wrote a lot of terrible,
| undergrad-in-cs C code around it, but we never messed up on
| billing. Thanks, Kent!
|
| Now that time I hadn't verified that our backups were working and
| lost all of our data, that was a little less good.
|
| [1] not a typo, it still exists, though I doubt anyone would use
| it over MySQL, postgres, or sqlite.
| https://en.m.wikipedia.org/wiki/MSQL
| manicennui wrote:
| mSQL was my first introduction to open source databases as
| well.
| 300bps wrote:
| Me too! Although I did web hosting and company ISDN lines and
| not dialup 1996-1998.
|
| Used Windows NT for web hosting and Slackware Linux with qmail
| for mail hosting. Sold out at what I thought was the top -
| which was a year or two too early but that's OK.
| nyrikki wrote:
| We did our billing with great plains accounting software
| which still exists as Microsoft Dynamics GP.
|
| When 56k came around you had to do ISDN, but for many rural
| ISPs the tariffs for ISDN was cost prohibitive so POTS was
| the only option.
|
| I worked for a few ISPs and negotiated telcos to bring in
| redundant OC12s to support the need, allowing them to use our
| data centers for the local switching. We never needed that
| capacity ourselves, but as it was before they started using
| multiple colors per fiber that was the smallest they would
| run.
|
| I also started out with stacks of external modems and NT RAS.
|
| I remember that the plastic on stacked modems would blacken
| due to heat in about 6 months.
|
| I dealt with sendmail, bind, etc and didn't do the account
| provisioning but it was written in Delphi.
|
| As I had written an entire 7 digit double entry accounting
| system in college in two weeks backed by dbase, this stories
| excel and paper based system seems like an anomaly for the
| time.
|
| NT we hosting was a pain, but front page did make it a
| popular option you had to support.
|
| While I did introduce Linux, we were mostly on DEC
| Alphastations because you could support a lot more load for
| less money than the sun boxes at the time.
|
| It was a fun time. I remember hopping on #nanog on irc
| because I couldn't get uunet support to pull routs from us
| and had them pulling from us in less than an hour.
| toast0 wrote:
| > NT we hosting was a pain, but front page did make it a
| popular option you had to support.
|
| I don't remember the details, but at the micro-ISP I worked
| at, I got frontpage extensions working on Apache for
| customers.
|
| We had two T1s, one for upstream, one for modems (which I
| never got to play with), we also contracted out nationwide
| dialup through megapath? or someone... We ran a radius auth
| server and they did the rest. Some java accounting package;
| I did an integration for ach billing, and made a module so
| we could sell domains (via Tucows OpenSRS). At some point,
| the modem T1 stopped working and we couldn't get the telco
| to fix it, so we had to move all the customers to the
| outsourced dialup provider, which wasn't great for the
| business... I left because of scheduling/poor performance
| at school and they got purchased by a customer.
| [deleted]
| Aschebescher wrote:
| A lot better than one year late, I guess.
| daft_pink wrote:
| I was living in the Chicago and got one of the first DSL lines
| during the dot com bubble. It took 6 months to get it
| installed.
|
| They never charged us for it. Roughly a year later, we got a
| letter from their big four accounting firm saying that they
| were going bankrupt due to billing irregularities and our
| service would be terminated soon, but it was easy to get
| service then.
|
| We never paid a dime for it.
| grogenaut wrote:
| We were in the first 100 customers to get dsl in STL. It took
| 1.5 years and escalating to a vice president to get it to
| work as both a phone line and dsl. Their billing system was
| miffed as dsl was commercial. My friends worked there and had
| oracle access (swbell) and could easily see the mismatch, but
| couldn't fix it. They ended up giving us our old dialup
| second phone line for free and dsl for free for 3 years to
| solve it.
|
| Wife also worked for att and dsl was cheap for them except
| the billing system swore my apt was in another county and
| would send the installers there constantly. Took 8 months and
| was only fixed when wife went on a tour of a facility and
| mentioned it to an engineer and their VP.
| organsnyder wrote:
| Early DSL was such a shitshow. We went through two or three
| providers going under before it finally began to stabilize.
| giantrobot wrote:
| In 2000 or thereabouts PacBell was promising ADSL at my
| apartment. I was well within range of the CO/DSLAM and
| everything seemed set but the technician could not get the
| modem to sync with a signal. Turned out the building's
| phone wiring was a rat's nest of twisted and taped RJ-11.
|
| It really sucked not to get anything faster than dial-up.
| It also explained why my dial-up couldn't connect reliably
| at 28k and I'd often be stuck with a 19k connection.
|
| The experience did teach me a lot about Squid as a caching
| proxy and using wget and screen to do overnight downloads
| (I didn't learn about whey's background flag until a few
| years later).
| kmbfjr wrote:
| Dial-up World was consumed in a "roll-up", which was popular
| right up to about 2005 when the only dialup ISPs were rural.
|
| The problem with pivoting to broadband was, most could not. The
| only option was reselling DSL, becoming a CLEC or WISP, and
| taking the dirt nap. The company I worked for tried everything
| and eventually went out of business after the dialup was sold in
| a rollup.
|
| I was at a 123.net colo a few years ago, and off in a rack was
| half consumed by Livingston/Lucent and US Robotics Total Control
| dial-up T1 endpoints, running. The USR actually showed signs of
| use.
| kotaKat wrote:
| I at one point was taking calls in 2012 for a variety of ISPs
| including a small dial-up ISP in PA. [Also mixed in were weird
| other clients -- I could bounce from a dial-up call to someone
| asking about Bit9.]
|
| It broke my heart to be taking a call from someone _willingly
| asking to pay for 12 months of dialup in advance_. But they
| did, and it was like $130. Ow.
|
| National Rural Telecommunications Cooperative members were
| slowly but surely lighting up fiber, at least.
| belthesar wrote:
| This story hits home pretty hard. My first gig in the industry,
| so to speak, was working at one of the last few independently
| owned dialup ISPs in midwest in 2005. I first started as a phone
| tech, where I did everything from help folks get connected and
| configure their email clients to basic computer tech support.
| Sometimes I'd get dispatched to do an on-site install. There was
| also the dirty old man customer that was constantly getting his
| computer infected on shady porn sites, and we eventually turned
| on our web filter service for him gratis, because his wife was
| constantly embarrassed to keep paying to bring the computer in
| for service.
|
| Eventually became a Jr. Sysadmin, cut my teeth on managing Apache
| HTTPD, MySQL, qmail, built my first Linux server compiling Gentoo
| from Stage 1 on a dual 700 MHz Pentium 3 box. Learned so much
| from an awesome old school Linux sysadmin who, like me, got his
| start as a young kid at this very ISP. Oh, and I had access to 8
| bonded T1s, which when the fastest consumer bandwidth I could
| ever dream of was a consumer cable modem with 3 Mbit down, and
| 256kbit up, being able to surf day in and day out on 12 Mbit of
| bandwidth was blazing fast.
|
| However, like every other dialup ISP, much like the story goes in
| the OP, this little ISP was far too entrenched in their Dialup
| install. We did offer DSL service, but we were effectively a
| reseller to the larger ISPs in the area, so for customers that
| did switch over, we either offered poorer service, or they were
| paying more for the same service they could get direct from the
| bigger ISP. We were suffering from consumer attrition as the
| great ISP consolidation was happening, and the money was getting
| tight. A startup looked to be our savior - they wanted to deploy
| WiMAX (this was before Sprint bought up all the WiMAX spectrum
| for their form of 4G) to the region. What was originally pitched
| as a full acquisition became an acquisition of our customer list
| and some of our services like our shared hosting. Spoiler alert,
| those jokers didn't know what they were doing, people started
| leaving the company left and right, eventually equipment started
| failing and the people that knew how to fix those things took
| their knowledge with them, and eventually, my paychecks started
| bouncing, so I left.
|
| Personally one of the most rewarding jobs I ever had. I learned
| so much, got paid quite well for very little "work", and it
| really set a course for the rest of my professional career. But
| hoo, the death spiral story that OP showed gave me a stark
| reminder of the very bad end times of that time in my life.
|
| Also, I wonder if OP is talking about GlobalPOPS as the company
| that bought all the dialup ISPs that they worked for.
___________________________________________________________________
(page generated 2023-10-02 23:01 UTC)