[HN Gopher] A Plea for Lean Software (1995) [pdf]
       ___________________________________________________________________
        
       A Plea for Lean Software (1995) [pdf]
        
       Author : dvfjsdhgfv
       Score  : 128 points
       Date   : 2021-06-28 13:18 UTC (9 hours ago)
        
 (HTM) web link (blog.frantovo.cz)
 (TXT) w3m dump (blog.frantovo.cz)
        
       | andrewmcwatters wrote:
       | I don't know how you would do this, other than setting core
       | affinity, but I'm surprised I can't limit processing power or RAM
       | per process on any modern OS.
       | 
       | Both for setting restrictions per another comment here, but also
       | for testing purposes.
       | 
       | Like, I'd love to be able to run an executable with equivalent
       | processing power to that of a specific CPU generation.
        
         | delusional wrote:
         | On linux you could use cgroups and cfs_quota:
         | https://www.kernel.org/doc/html/latest/scheduler/sched-bwc.h...
        
         | Etheryte wrote:
         | On macOS, there is a legacy CPU throttler called AppPolice [0]
         | which as of right now still works on Catalina, despite the last
         | release being made in 2016 (!). The user experience is very
         | flaky, but it can help out when it's really needed. I haven't
         | tested it on Big Sur.
         | 
         | Definitely agree though, this is something that should be baked
         | into the OS as a first class citizen, especially in the modern
         | era of Electron-all-the-things.
         | 
         | [0] https://github.com/AppPolice/AppPolice
        
         | phaemon wrote:
         | Surely you can with cgroups? Isn't that exactly what they're
         | for?
        
         | opencl wrote:
         | On Linux there's a tool called cpulimit that does this, or you
         | can use cgroups as others have mentioned.
         | 
         | https://manpages.ubuntu.com/manpages/xenial/man1/cpulimit.1....
        
           | andrewmcwatters wrote:
           | Thank you for this insight!
        
       | dvfjsdhgfv wrote:
       | One of the key quotes: > Another important reason for software
       | complexity lies in monolithic design, wherein all conceivable
       | features are part of the system's design. Each customer pays for
       | all features but actually uses very few. Ideally, only a basic
       | system with essential facilities would be offered, a system that
       | would lend itself to various extensions. Every customer could
       | then select the extensions genuinely required fora given task.
        
         | sidlls wrote:
         | Such a system sounds nice in theory. In practice it leads to a
         | byzantine nightmare of configurable interfaces, support
         | systems, and so on, with a result that is ultimately fatter and
         | harder to maintain.
        
           | jcelerier wrote:
           | As an user, all the software that I take actual pleasure in
           | using and make me productive, are large plug-in based
           | software.
           | 
           | As a developer of a mainly plug-in based application, the
           | iterating ability it gives is pretty incredible - nowadays
           | for a large majority of my feature requests, I don't have to
           | touch the original codebase at all, just create a small plug-
           | in for the feature.
        
           | CraigJPerry wrote:
           | I disagree with that - what i'd call a plugin architecture -
           | leads to a very nice separation of components.
        
             | interactivecode wrote:
             | plugins works for things that are plugins. Most people seem
             | to equate separation of concerns with everything should be
             | a plugin. Otherwise you get massive issues actually working
             | across each plugin (or package).
             | 
             | So unless your component is actually used in multiple core
             | products it's not a component, it's just a part of your
             | monolith. Making it a separate component usually, just
             | spreads your monolith across several git repo's
        
           | fouric wrote:
           | Emacs is a system that adheres to this theory, being a small
           | C core that implements an elisp runtime, and then the rest of
           | the built-in functionality (as well as the entire plugin
           | ecosystem) implemented as easily-tweakable elisp on top of
           | that - but has not turned into "a byzantine nightmare of
           | configurable interfaces, support systems, and so on". It's
           | quite the counterexample to this claim.
        
             | taeric wrote:
             | I'd argue that emacs is more monolithic in is main focus of
             | text editing and elisp execution. It doesn't just build a
             | plugin architecture. It literally built a platform and does
             | most of it's work in the same way that user code can run.
        
           | Jtsummers wrote:
           | I don't know. It sounds similar to a lot of IDEs. Produce an
           | interface that plugins can make use of, then people can
           | extend the system to support new languages and capabilities.
           | Seems to work well in a lot of cases.
        
           | mysterydip wrote:
           | That was my first reaction, but I think it can work in some
           | cases. Take for example plugins in a paint program, or
           | dissectors in wireshark. Both of those can have any number of
           | user-created extensions for different packet types or editing
           | that the general populace doesn't need.
           | 
           | However, to your point, when wireshark changes their API
           | every few versions, all the plugins have to change as well or
           | you either break your dissectors or are stuck on an old
           | version. The base program should be designed with extension
           | maintainability in mind.
        
           | 7952 wrote:
           | Well it depends on the level of abstraction you use. You
           | could think of the operating system as the platform, and each
           | piece of software as a plugin. That can work well enough. I
           | can save a PNG in photoshop, and then open it in other
           | programs and it just works. But the file paradigm has lost
           | popularity and there are probably some good reasons for that.
        
           | vendiddy wrote:
           | If the interfaces are defined poorly, it can result in what
           | you are describing.
           | 
           | But I've seen systems do this well and the result is
           | something far less complex.
           | 
           | I agree with the author's view that good interfaces require
           | iteration. So expect that several irrational will have
           | incorrectly drawn boundaries and poorly defined interfaces.
           | 
           | This kind of design takes time, but if you expect the system
           | but if you expect that it will need to evolve over the years,
           | it pays for itself.
        
         | necrotic_comp wrote:
         | Isn't this more or less the idea behind Gentoo ?
         | 
         | I know that when I initially started setting my system up, it
         | was a bit of a mess trying to figure out what USE flags I
         | needed/etc., but once it stabilized, it was great to know that
         | I had a system that worked and there was no trace of software
         | or libraries I didn't want or need.
         | 
         | It definitely is a bit more legwork to do up-front, but I feel
         | like the idea of "this is what I want, please build only to
         | that spec" has saved me a bit of heartache as I've been using
         | my system and while the leanness doesn't matter per se, it
         | feels like it's helping my system be a bit more organized.
        
       | [deleted]
        
       | uniqueuid wrote:
       | Previously (with comments):
       | 
       | https://news.ycombinator.com/item?id=8301511
       | 
       | https://news.ycombinator.com/item?id=24059704
       | 
       | https://news.ycombinator.com/item?id=17872400
        
         | [deleted]
        
         | dang wrote:
         | Thanks! I've added one small one:
         | 
         |  _A Plea for Lean Software (1995) [pdf]_ -
         | https://news.ycombinator.com/item?id=24059704 - Aug 2020 (137
         | comments)
         | 
         |  _A plea for lean software (1995) [pdf]_ -
         | https://news.ycombinator.com/item?id=17872400 - Aug 2018 (100
         | comments)
         | 
         |  _A Plea for Lean Software (1995) [pdf]_ -
         | https://news.ycombinator.com/item?id=8537970 - Oct 2014 (3
         | comments)
         | 
         |  _Niklaus Wirth: A Plea for Lean Software (1995) [pdf]_ -
         | https://news.ycombinator.com/item?id=8301511 - Sept 2014 (12
         | comments)
        
       | lproven wrote:
       | This is such a classic paper that I felt it deserved a more
       | readable copy. So I copied the text into Word and cleaned it up a
       | bit. May be easier for those with visual impairments etc.
       | 
       | Here:
       | 
       | https://www.dropbox.com/s/kmpo4c2q9klqx1e/A%20Plea%20for%20L...
       | 
       | (MS Word format, old .DOC version. Should be readable in
       | anything.)
       | 
       | Feel free to reshare.
        
       | ape4 wrote:
       | Sorry if this is obvious - the article is by Niklaus Wirth -
       | author of Pascal and other important languages.
        
         | Koshkin wrote:
         | I feel like I can guess the name of the future lean programming
         | language that will conquer the world. (It'll be worth the
         | wait.)
        
           | nine_k wrote:
           | Very many ideas of Pascal, Modula, and Oberon live on in the
           | lean (to me, excessively lean) language named Go, which got
           | pretty popular during last decade.
        
             | titzer wrote:
             | Quite literally by pedigree. Robert Griesemer was a student
             | of Wirth.
        
       | coliveira wrote:
       | The big problem we have is that people don't care just about
       | software features, but about software bugs. The biggest example
       | is the web. We could have something that implements the features
       | of a web browser relatively clearly, but what about the bugs that
       | are treated as features? There are billions of pages that depend
       | on bugs of current browsers to display correctly. The same for
       | other areas such as word processors, operating systems, etc.
        
         | cogman10 wrote:
         | I don't know that I agree with this assessment. I think the
         | bigger issue isn't "bugs" per se but rather that we keep
         | accruing capabilities.
         | 
         | Take the web as an example. HTML3 was so simple that we had
         | hobbyist implementing it. It could be ran by 100MHz computers
         | with 16mb of ram. Now, the HTML5 spec is impossibly large. So
         | large that we are starting to see big major companies
         | consolidate parts of the browser because it's too complex and
         | too big for 1 multibillion dollar company to keep up to date.
         | 
         | It's so large that a team of engineers which worked primarily
         | on browsers never really finished a from scratch rendering
         | engine (Servo) even after several YEARS of effort.
         | 
         | The same thing happened to the PDF spec.
         | 
         | This happens with a bunch of desktop/consumer products that get
         | old.
         | 
         | Excel can still open xls docs from the 90s, even though there's
         | been about 30 different changes to the spec since it was first
         | created. Consider what that does to a product.
         | 
         | Even microsoft windows runs into exactly this problem.
         | Microsoft can still run games from the 90s. Considering they
         | completely changed out the kernel, imagine how crazy that is.
         | By and large, they accomplish this feat by keeping around dlls
         | from the 90s and reusing them. This is a primary reason why
         | windows installs are now pushing 10+gb. They can't throw out
         | support or there would be a corporate mutiny.
         | 
         | And before we give MS too much shade for doing this, consider
         | what we've done in the linux world. In order to get
         | applications running consistently across OS versions we've
         | decided the best way to do that is shipping almost all of an
         | operating system along with our runnable (Hello docker images)!
         | 
         | It's not a question of bugs. It's a question of behavior. And
         | as time goes on, new behavior and capabilities are added to
         | software. The longer it lives as a product, the more true that
         | is. This is where bloat comes from.
         | 
         | And consider this, as a software developer, arguing to remove
         | capabilities is almost always a losing proposition. Imagine,
         | for example, if photoshop decided "you know what, nobody really
         | uses JNG anymore, so we should remove it from our products."
         | Any PO worth there salt would say "Hold it, why should we
         | remove this? Is it costing us anything to have it? Just a few
         | MB? Then we keep it!"
        
           | Koshkin wrote:
           | > _I don 't know that I agree... per-say... hobbiest...
           | accomplish this feet_
           | 
           | I hole-hardedly agree, but allow me to play doubles advocate
           | here for a moment. For all intensive purposes I think you are
           | wrong. In an age where false morals are a diamond dozen, true
           | virtues are a blessing in the skies...
        
             | cogman10 wrote:
             | Fixed for you.
        
           | badsectoracula wrote:
           | > By and large, they accomplish this feat by keeping around
           | dlls from the 90s and reusing them. This is a primary reason
           | why windows installs are now pushing 10+gb.
           | 
           | Not really. I mean, sure, from my Win10 installation (which
           | is around 11GB), 6.8GB are DLL files _HOWEVER_ pretty much
           | all of them are inside the WinSxS which aren 't the "DLLs
           | from 90s", but duplicates of various DLLs that applications
           | bring along.
           | 
           | Outside of WinSxS the only other place with lots of DLLs is
           | System32, the bulk of which is taken by AMD's drivers and the
           | rest of the DLLs take around 800MB. Then there is SysWOW64
           | where a similar layout can be found with most of the space
           | being taken by AMD's drivers and around 500MB or so for the
           | rest of the DLLs. Inside other Windows directories there are
           | also around 900MB of various .NET installation DLL.
           | 
           | So in total, the Windows DLLs, including .NET are probably
           | around 2GB. However keep in mind that i have a ton of stuff
           | installed, with pretty much every VC and .NET runtime that
           | existed and my installation is a bit older than 3 years old,
           | on a fresh Win10 installation will probably have less stuff
           | in there.
        
         | klysm wrote:
         | I would say backwards compatibility and ossification of
         | terrible APIs is a larger concern.
        
       | slver wrote:
       | Pleas don't work.
       | 
       | What works is restrictions.
       | 
       | This is why when iOS restricts apps from eating up the entire
       | physical RAM on the device, and developers complain that this is
       | not like on the mac/PC, they should realize Apple is extremely
       | familiar that it's not. And it'll never be. When you let go of
       | restrictions, applications bloat up to take the available
       | resources while adding almost nothing in terms of functionality.
        
         | Cthulhu_ wrote:
         | It's one reason why iOS had five years head start on Android
         | when it comes to performance, perceived or otherwise, even with
         | much slower hardware - they made much better use of it.
         | Background apps? Yeah no, if you need any background activity,
         | use a push notification service. Whereas on Android, for years
         | it was the norm to just have your application run in the
         | background and poll every once in a while.
        
         | cle wrote:
         | To me an interesting question is _why_ they bloat up. Is it all
         | mindless waste or is there a tradeoff being made here?
        
           | dale_glass wrote:
           | We do more things, and have more complex graphics.
           | 
           | Eg, a 320x200x8 display has 64000 bytes of memory. A 4Kx32
           | bit display has 33177600 bytes of memory. That's 518 more
           | times memory used just for a single screen. Take into account
           | that these days we have compositors and each window is
           | present in memory on its own, before it's composed into the
           | result, and it's even worse.
           | 
           | We also work on far bigger projects. The project I work on as
           | a hobby has 400K lines of C++. Imagine the amount of
           | processing an IDE needs to do to parse all of that and
           | provide its various convenient functions. And of course a
           | good chunk of that is in RAM, because that's far too much
           | stuff to parse everything every time.
           | 
           | Old tools also don't really cut it anymore. If I try running
           | ctags on it, it comes up with a 14GB tags file. I don't think
           | it handles it quite right.
        
             | andrewmcwatters wrote:
             | Thank you for mentioning ctags, even if it's not useful
             | today. I wasn't aware of this tool.
        
               | dale_glass wrote:
               | I mean, it depends on what you use it for. My project
               | seems to confuse it in some way. But it should work
               | perfectly fine for a lot of stuff.
        
           | titzer wrote:
           | Human effort and attention span are still limited quantities.
           | There are so many more libraries, frameworks, utilities, and
           | doodads, that it's easier to plug something together from
           | lego blocks rather than craft (and debug!) a custom solution
           | that is both efficient and small.
        
           | slver wrote:
           | When something doesn't matter, you don't spend time
           | optimizing it. It's the natural behavior of systems.
        
           | mywittyname wrote:
           | It's so much faster to develop wasteful software. It's also a
           | lot easier.
           | 
           | There's no need to use high-performance, but difficult to
           | develop languages (c, c++). There's no need to be conscious
           | of data structure choices. There's no need to profile code,
           | etc.
           | 
           | Optimizing code can be a deep rabbit hole. If you remember a
           | few months ago, some random person dug into why GTA Online
           | was taking longer each day to load. The reason turned out to
           | be a string parsing inefficiency in the strlen method, a core
           | library. But it only manifested itself in certain situations.
        
           | legulere wrote:
           | Optimizing software costs developer time, so it's only done
           | when needed.
        
             | vendiddy wrote:
             | I don't fully buy this argument. The bloat we see in many
             | systems ends up slowing down development in the long run.
             | 
             | Well defined interfaces and good composition support the
             | goals of the author of lean software.
             | 
             | They also happen to improve developer comprehension and
             | reusability, both of which speed up software development in
             | the long run.
        
             | fredsir wrote:
             | So its about core values, it's about principles. It's
             | because people focus on something else than craftsmanship
             | and creating great things.
             | 
             | Obviously, it's because of how our society and culture is
             | oriented. But I'd like to see that change.
        
               | legulere wrote:
               | You can put almost infinitely much effort into
               | optimization. When do you stop? Do you test every user-
               | interaction takes less than 100 ms below which it can be
               | considered good?
        
               | [deleted]
        
               | loup-vaillant wrote:
               | I would hope so actually: _someone_ needs to test every
               | user interaction at least once, at which point we can
               | notice if it 's too slow.
        
               | legulere wrote:
               | It's not as easy as that. What's acceptable in test might
               | not be acceptable in your customers conditions. It's
               | difficult to correctly measure the time for every
               | interaction and humans are extremely bad at telling you
               | how fast something is. Then you also have the problems of
               | regressions that stem from other unrelated changes.
        
               | loup-vaillant wrote:
               | I believe it _can_ be made easy, if you have an easy way
               | to profile your GUI. Test the feature, get a timer. If it
               | took more than 20ms on your killer test machine, it might
               | be too slow for some users. (Adjust your heuristic with
               | the context of your application of course.)
        
               | blacktriangle wrote:
               | Most devs would love to focus on craftmanship and
               | creating great things.
               | 
               | Most customers want their immediate problem solved as
               | cheaply as possible.
               | 
               | Unless you're in a market where customer's pay for
               | craftsmanship, you will get undersold.
        
               | fredsir wrote:
               | It's a cultural thing and cultures can and do change.
        
           | kwertyoowiyop wrote:
           | Tell a Product Manager "I can do a preliminary version of
           | that feature for the next release, but it'll be really
           | inefficient" and they hear only "I can do...that feature for
           | the next release."
        
           | rosetremiere wrote:
           | In my "software development" course at university (so, not
           | programming, but every "management" thing around it: unit
           | testing, git, waterfall/scrum, etc), I remember the
           | instructor telling us that the first approach to making your
           | program faster was upgrading the hardware. The more time
           | passes, the more shocked I am at an instructor telling this
           | to his students, to the point that I'm now thinking I might
           | recall a cruder version of what was originally said. In any
           | case, that may be the cause!
        
             | mrweasel wrote:
             | To some extend I don't disagree with: "Just buy a bigger
             | server", but for software for laptops, home PCs or mobile
             | it's not a great option. If you're trying to tweak every
             | single query to get more performance from your SQL server,
             | then maybe the money is better spend on hardware than
             | developer time.
             | 
             | Weird counter point to "buy more hardware": I worked with a
             | client who have some code they developed, it takes a
             | dataset, run some calculation and returns the result. To
             | speed up part of the calculation they use CUDA.
             | 
             | We help migrate they system from older physical hardware to
             | AWS, pretty standard. They got more memory, more CPU cores
             | and jumped three generations of Nvidia GPUs. The result was
             | 10% speed increase, no where near what hardware alone
             | should have yielded, so clearly in this case there's
             | something in code that prevent the hardware from being
             | fully utilised.
        
           | hindsightbias wrote:
           | "Premature optimization is the root of all evil" - Knuth
           | 
           | So the only exposure most students get is not to do it. I
           | doubt any CS programs make performance analysis or
           | optimization a requirement. Just buy a faster system.
        
         | CharlesW wrote:
         | Absolutely. Your example, the App Store, and lots of other
         | aspects of the iOS ecosystem are how Apple prevents most (not
         | all) of the "tragedy of the commons" problems on that platform.
        
           | fouric wrote:
           | What does the App Store have to do with any of this? I can't
           | imagine any technical restrictions that the App Store
           | enforces that could _not_ be enforced by the OS alone.
           | 
           | For instance, "no background services" and "you can't use up
           | all the physical RAM" are both OS-level restrictions, not
           | app-store-level restrictions.
        
         | cartoonworld wrote:
         | This brings to mind the possibly apocryphal quotes of Arthur
         | Bloch or Hind's Laws of Computer Programming:
         | 
         | V. Any program will expand to fill available memory.
         | 
         | https://www.cs.oberlin.edu/~jwalker/humor/lawsOfComputerProg...
        
         | swiley wrote:
         | >What works is restrictions.
         | 
         | What works is not putting up walls between developers and power
         | users. Linux has no such restrictions and bloated software
         | isn't something I have to deal with on it (I have the option to
         | if I wanted to, but I don't.)
        
       | coding123 wrote:
       | Meh - it will always be complicated and get more complicated
       | going forward.
       | 
       | Also has anyone looked at what it takes to build a home, let
       | alone a hospital. I don't think software is any more complicated
       | than our society allows other processes to become complicated.
        
       | RcouF1uZ4gsC wrote:
       | A good counterpoint to this is Joel on Software's post about
       | "Bloatware"
       | 
       | https://www.joelonsoftware.com/2001/03/23/strategy-letter-iv...
        
         | jhgb wrote:
         | I don't see how this vulgar reduction of the problem in
         | question to just size of binaries has anything to do with what
         | Wirth is talking about in that article, which are among other
         | things the factors of maintainability, cognitive load,
         | reusability and such.
        
       | bigbillheck wrote:
       | The 'inflated' software Wirth complains about is the same as is
       | often fondly remembered in places such as here by people talking
       | about the Good Old Days.
        
       | fisensee wrote:
       | I've found the part where the author talks about "arbitrarily
       | overlapping windows" and "fancy icons" as "cute but not
       | essential" pretty telling. Find me someone on the street and ask
       | them if these things are "cute but not essential". Sometimes I
       | feel like software engineers forget what world we live in. In the
       | virtual world all these things are obviously not needed and
       | wasting precious computing power on them is a disgrace to the
       | "craftsmanship". In the real world most applications are made for
       | flawed humans with needs and preferences which stem not from
       | logic and reasoning but from emotions and experience. This
       | doesn't excuse shoddy engineering in any way (everybody has some
       | horror story of somebody not understanding runtime complexity or
       | something, so there certainly is a lot of it). But especially
       | with the examples the author brings up like text editors and
       | operating systems I find these critic/plea severly lacking. Are
       | the core features of ed and visual studio code the same?
       | Absolutely, you edit text. Are the extra ressources used for
       | visual studio code compared to ed a waste? For some people here
       | probably, but I would bet quite a lot that for most the answer
       | would be no. In the end I often find that discussions about
       | optimatization are often more for people to show off and prove
       | that they are a "real engineer" than any real concern about
       | performance.
        
         | joshuak wrote:
         | It is also common trope that people say "but consider the
         | typical user." I think both arguments are valid. At differing
         | points in user skill level.
         | 
         | For example I would not expect a "typical user" to know or want
         | to use a terminal to navigate directories, however as a more
         | experienced user I know one can navigate much more quickly
         | using a terminal than using the hunt and peck style of most
         | file system GUIs.
         | 
         | Some things are useful before they can be made easy. Some
         | things are easy that have more uses with more effort.
        
           | zozbot234 wrote:
           | A good TUI-based file manager is way quicker than typical
           | GUI's while still preserving much of the same intuition.
           | Whether it's also quicker than the CLI depends on your use
           | case; in some cases it can be.
        
             | joshuak wrote:
             | Even GUI's can be as fast or faster then terminal if it's
             | not oriented around visually searching and selecting each
             | step in the path.
        
               | Koshkin wrote:
               | Indeed, TUI and GUI are not mutually exclusive to begin
               | with.
        
         | II2II wrote:
         | Is it really telling? A couple of things to consider:
         | 
         | - The original Macintosh did not use icons as heavily as
         | software in 1995. From my vague recollections of 1995, the
         | computers of today may even use fewer icons. (It was a time
         | when there were icons on buttons, like a check-mark on an Okay
         | button, and icons on individual menu items.)
         | 
         | - Even arbitrarily overlapping Windows seem less common these
         | days. People have been maximizing applications for ages. These
         | days, tiling is increasingly popular on desktops (e.g. Windows
         | Snap) and people typically run apps full screen on mobile
         | devices (with split screen, rather than floating window
         | managers, being the only alternative). It's also worth noting
         | that 1995 was part of the era of multiple document interfaces,
         | which were arbitrarily overlapping windows _inside of_
         | arbitrarily overlapping windows.
         | 
         | Wirth seems to be taking a far more extreme stance than the
         | examples I have provided, yet the early years of the 1990's
         | were also an era of excess. While 1995 Wirth would probably
         | strongly disagree with the software of today, modern software
         | looks austere compared to some of the software of the 1990's
         | while the performance of modern hardware means the user is
         | going to feel less of an impact due to the user interface.
        
           | [deleted]
        
           | [deleted]
        
           | Groxx wrote:
           | > _It was a time when there were icons on buttons, like a
           | check-mark on an Okay button, and icons on individual menu
           | items._
           | 
           | tbh I wish we would go back to this. Even with the variation
           | in styles, those icons provided language-agnostic "names" for
           | things. If you're using a computer stuck in a different
           | language / with different layout patterns, a checkmark in a
           | primary position is much less ambiguous than having to guess
           | if this is a yes/no/ok/cancel/abort/retry/fail/ignore of some
           | kind.
        
         | zozbot234 wrote:
         | Icons and square-ish buttons are convenient in mobile UI's
         | because they make for good touch targets. At the same time,
         | arbitrarily overlapping windows are replaced by tiled screen
         | splits for much the same reason. So what's 'essential' will
         | often depend on what the software is being used for.
        
         | philipswood wrote:
         | For context: keep in mind that the personal computer that Wirth
         | and team built for ETH Zurich, project Oberon, was graphical,
         | but eschewed overlapping windows for a paned interface and
         | didn't buy into the skeuomorphic desktop WIMP metaphor.
         | 
         | So "arbitrarily overlapping windows" and "fancy icons" as "cute
         | but not essential" may be telling, but is probably not telling
         | what you thing it does.
        
           | fisensee wrote:
           | I agree, taking these quotes out of its time context is a bit
           | silly but then again this is the implied intention of posting
           | this here, right? Most people reading this and posting a
           | comment will not be talking about the state of software in
           | 1995 but today, using the article to show that the things
           | described in it are still true/got worse. My comment itself
           | was more of a quick reaction to the general sentiment in the
           | comment section than a serious critic of the article itself.
           | 
           | Since I'm a huge fan of tiling windows managers and spend a
           | lot of my time in terminal emulators as a power user I
           | believe to understand his vision about computer usage and
           | have a lot of admiration for what he and his colleagues were
           | working on. But for the general population I believe he did
           | not have the "right" idea (easy to say with almost 30 years
           | of hindsight).
        
           | nine_k wrote:
           | Was the computer science professor Niklaus Wirth, a creator
           | of several programming languages, a layman user of personal
           | computers?
           | 
           | Why did Amiga, the series of computers beloved by artists and
           | musicians, have overlapping windows and colorful icons?
        
             | jhgb wrote:
             | Amiga may have had some overlapping windows within a screen
             | but it also had multiple screens with significantly
             | restricted arrangement options, if memory serves me right.
        
             | zozbot234 wrote:
             | > Why did Amiga, the series of computers beloved by artists
             | and musicians, have overlapping windows and colorful icons?
             | 
             | To be fair, these things were computationally intensive at
             | the time. The Amiga had a hardware blitter and very good
             | color capabilities, so it could afford to show them off.
        
               | nine_k wrote:
               | The creators of Amiga introduced them, and added hardware
               | support to make it performant. The only thing which felt
               | sluggish in an Amiga was the floppy disk.
               | 
               | Likely these creators understood the importance of
               | overlapping, draggable, resizeable windows for the
               | audience, and the audience indeed enjoyed them. Same
               | thing with the classic Mac.
               | 
               | If you tried a tiling WM, you realize that it requires
               | learning, tweaking, and discipline. Those are expensive
               | qualities to require from users; one should ask for
               | limited amounts of them.
        
               | zozbot234 wrote:
               | Overlapping windows also require discipline, perhaps more
               | so than tiled screen splits. We are just more used to the
               | former over the latter.
        
           | badsectoracula wrote:
           | It also should be noted that Oberon's UI (which you can try
           | out by downloading the emulator from here[0]) might be an
           | initial culture shock if you are used to a Windows-like
           | interface, but when you learn how it works it actually is
           | both very powerful _and_ very consistent - which helps into
           | usability since everything works the same way.
           | 
           | AFAIK Oberon was actually used by "regular people" at ETH
           | Zurich for a long time.
           | 
           | (and a lot of people ran and still run their programs
           | maximized anyway)
           | 
           | [0] http://www.projectoberon.com/
        
             | Koshkin wrote:
             | Mouse chording, ouch. (I firmly believe it is the worst
             | idea in the history of personal computing.)
        
               | pjmlp wrote:
               | People that worship the ACME editor might be of different
               | point of view.
        
               | philipswood wrote:
               | Wasn't too bad to use.
               | 
               | Biggest gripe was finding a PC with a 3-buttoned mouse in
               | the uni computer lab when it was full.
               | 
               | Being able to select, copy, paste and execute without
               | touching a keyboard was cool.
               | 
               | (We used it a bit in a 3rd year computer science OS
               | course)
        
         | delusional wrote:
         | Do you disagree that they aren't essential? Can you not do the
         | computing without them?
         | 
         | Obviously you can, although you may not want to. That's exactly
         | the point of the article. You WANT arbitrarily overlapping
         | windows, but that comes at a cost. You HAVE to pay that cost if
         | you WANT the feature.
         | 
         | In the very next chapter, the author even says as much:
         | "Increase in complexity results in large part from our recent
         | penchant for friendly user interaction."
         | 
         | What makes such a realization interesting is that if you don't
         | need the "friendly user interaction", say you're making a POS
         | system or some embedded device, you don't need all that extra
         | complexity.
         | 
         | A lot of POS systems run full Windows systems. Do they NEED
         | arbitrarily overlapping windows? If not, then why do they carry
         | around that extra complexity?
        
           | Koshkin wrote:
           | "Essential" is such a loaded term, and talking about it can
           | easily get us to a slippery slope. Like much else, this
           | should be taken in context, and contexts abound.
           | 
           | For many use cases the (excellent) interface provided by,
           | say, AS/400 (a.k.a. IBM i) is totally sufficient, but this
           | fact alone does not mean that something like that should be
           | used universally.
        
           | tonyedgecombe wrote:
           | >What makes such a realization interesting is that if you
           | don't need the "friendly user interaction", say you're making
           | a POS system or some embedded device, you don't need all that
           | extra complexity.
           | 
           | Why would a POS system not require a friendly user interface.
           | 
           | This attitude towards users is why corporate software stinks
           | so much, it's based on the idea that the developers have a
           | captive audience so have no choice in the matter, they should
           | just accept what they are given and stop complaining.
        
             | delusional wrote:
             | The running example in my comment was overlapping windows.
             | a POS system might not need windows, much less arbitrary
             | overlapping ones. That was the specific "friendly user
             | interaction" i was talking about.
             | 
             | I think you're taking me a bit out of context in order to
             | talk about something you care about. I agree with you, so
             | please don't hold me up as something I'm not.
        
             | pc86 wrote:
             | There's a difference between a "friendly user interface"
             | (which is pretty clearly referring to the earliest points
             | about design-heavy skeuomorphic icons and complex window
             | management functionality) and a friendly, e.g. accessible
             | and functional, user interface.
             | 
             | A POS system doesn't need window management, it doesn't
             | need skeuomorphic design, and I'm sure there three or four
             | dozen other features that come along with a full-blown
             | Windows installation that are irrelevant to a functional,
             | usable POS system.
        
         | bitwize wrote:
         | This is pretty much the standard issue "Ignorant/arrogant code
         | jockeys have no idea how alone and scared us users are, and our
         | need to be mollycoddled" rant.
         | 
         | It's Niklaus Wirth. He's the creator of Oberon, a
         | language/OS/development environment which, while a bit
         | idiosyncratic by our standards, is far more powerful than ed.
         | 
         | Oh, and while we're talking about the "real world", why do we
         | never discuss the data entry clerks, sales associates, and
         | customer service reps who learned to operate terminal-based
         | F-key-driven UIs quite efficiently -- and were then forced on
         | to graphical MFC, Java, or Web-based UIs with little thought
         | given to efficient operation with the keyboard alone? The
         | standard Macintosh-based WIMP paradigm is great for onboarding
         | new users, but it's less good for efficient use by a trained
         | operator when speed or volume is important.
         | 
         | Personally I find overlapping windows and icons to be more
         | essential than... ugh... _composited desktops_. Translucent
         | windows, drop shadows, and the  "genie effect" are fun for half
         | an hour tops, but these desktops consume additional processing
         | power and memory _and_ introduce latency! I 'd rather go with a
         | 90s style stacking WM.
        
           | zozbot234 wrote:
           | > Translucent windows, drop shadows, and the "genie effect"
           | are fun for half an hour tops, but these desktops consume
           | additional processing power and memory and introduce latency
           | 
           | There's certainly some room for optimization in how desktops
           | work today. I can see the obvious case for tear-free
           | (v-synced) compositing when dealing expressly with full-
           | screen (or close to full-screen) animation or full-screen
           | video, since screen tearing is highly noticeable in these use
           | cases. Wrt. everything else though, there's nothing wrong
           | with rendering without vsync and avoiding that added latency.
           | These optimizations can however be hard to make precise
           | beyond very simple cases, like a single foregrounded full-
           | screen app being enabled to take over the whole display.
           | Tiled windows might be the next most obvious step.
        
         | wtetzner wrote:
         | > Find me someone on the street and ask them if these things
         | are "cute but not essential".
         | 
         | Non-technical people seem to actually do better on e.g. an iPad
         | without overlapping windows.
        
       | klelatti wrote:
       | > Time pressure is probably the foremost reason behind the
       | emergence of bulky software.
       | 
       | Indeed, but for startups now I'd add easy scaling. With the
       | public cloud why worry about efficiency now (vs time to market)
       | if you can easily scale up later.
        
       | [deleted]
        
       | bigdict wrote:
       | Niklaus Wirth on software girth.
        
         | [deleted]
        
       | econner wrote:
       | I feel this. I just upgraded MacOS 11 Big Sur and to XCode 12.5
       | (from 12.4). Both updates were 12 GB.
        
       | kwertyoowiyop wrote:
       | In the past, all programs expanded until they included a bad
       | email feature. Now, all programs expand until they include a bad
       | social network.
        
       | Flex247A wrote:
       | I mean, some software can really get rid of bloatware without
       | losing much functionality. For example: The AMD Adrenalin driver
       | takes about 1GB of disk space and comes bundled with a browser.
       | The control panel takes forever to start, even on a laptop with
       | NVME SSD. This is why I have switched to installing drivers
       | through Windows Settings.
        
       | ChrisMarshallNY wrote:
       | _> Another important reason for software complexity lies in
       | monolithic design, wherein all conceivable features are part of
       | the system 's design._
       | 
       | This is really a call for modularity. I'm a believer in
       | modularity, but it is _not_ a cure-all.
       | 
       | Modularity brings a lot of baggage with it, like duplication of
       | code and lifecycle management, along with increased resource
       | usage (ironic, as the article is mainly about reducing resource
       | usage).
       | 
       | But I am still a believer in it. In my experience, it makes for a
       | _massive_ increase in system stability, quality, agility, and
       | speed of development.
        
         | Jtsummers wrote:
         | > Modularity brings a lot of baggage with it, like duplication
         | of code and lifecycle management, along with increased resource
         | usage (ironic, as the article is mainly about reducing resource
         | usage).
         | 
         | I just can't conceive of how modularity, in general, causes
         | duplicate code rather than reduces it. And this is before
         | considering something like generic code which could potentially
         | further reduce duplication. Every time we've been deliberate
         | about using modularity in projects it ends up creating
         | libraries that reduce the total amount of code across projects.
         | Even if it's just in one project, good modularity has been used
         | to reduce a lot of duplication.
        
           | BBC-vs-neolibs wrote:
           | Others have already pointed to the distinction between _code_
           | duplication and runtime duplication.
           | 
           | Regarding the development experience:
           | 
           | I think it's a tug and back-and-forth kind of dynamic.
           | 
           | Modularity has exactly the kind of effect you talk about,
           | until it doesn't.
           | 
           | At some point, the cost of communication (between people!)
           | responsible for different modules can increase until it's a
           | net negative to try to adapt a module, or even find out it
           | exists and is suitable.
           | 
           | Of course, a monolithic project of comparable size has likely
           | collapsed from its weight into a ball of mud.
           | 
           | Development tools have come quite a ways, but the science of
           | getting teams to develop software together, perhaps not so
           | much.
        
             | ChrisMarshallNY wrote:
             | _> Development tools have come quite a ways, but the
             | science of getting teams to develop software together,
             | perhaps not so much._
             | 
             | Yup. I think I'll put that on a T-shirt.
        
           | ChrisMarshallNY wrote:
           | As I mentioned above, if the module is expressed as a code
           | extension, then it is, sort of, "free." It is compiled into
           | the main code body.
           | 
           | But, for example, say we are talking about a communication
           | module. One module may use a well-known library as a bundled
           | dependency, and another module may use the same one. They may
           | be smart enough to use dylibs, in which case, you have the
           | potential for "DLL hell," or they may use static libs, in
           | which case you have duplication of code, and maybe resource
           | contention.
        
             | Jtsummers wrote:
             | Ok, so it's a conflict of meaning.
             | 
             | I would not consider static libraries to be duplication of
             | code. Duplication of assemblies or compilation unit, sure,
             | but not _code_. The code itself has not been duplicated.
        
               | ChrisMarshallNY wrote:
               | Fair 'nuff.
        
           | madpata wrote:
           | > I just can't conceive of how modularity, in general, causes
           | duplicate code rather than reduces it.
           | 
           | Yeah, like my node_modules folder doesn't contain tons of
           | duplicate code.
           | 
           | /s
        
             | Jtsummers wrote:
             | Modularity doesn't cause that. Modularity doesn't
             | inherently reduce duplication either. But modular code
             | offers opportunities for reuse (or clearer opportunities
             | for reuse) that non-modular code does not (or where the
             | reuse potential is so obfuscated that it may as well be
             | non-existent).
        
         | numbsafari wrote:
         | > Modularity brings a lot of baggage with it, like duplication
         | of code and lifecycle management
         | 
         | If the operating system and standard libraries provide better
         | facilities for the common parts, that can go a long way to
         | solving these problems.
        
           | ChrisMarshallNY wrote:
           | Exactly. But these days, everyone likes to bundle in
           | dependencies.
           | 
           | Also, for cross-platform SDKs and libraries, the provider may
           | be forced to "reinvent the wheel." That may well be worth it,
           | as developer bandwidth, and time-to-market are important
           | variables.
        
             | numbsafari wrote:
             | > But these days, everyone likes to bundle in dependencies.
             | 
             | If the OS and standard libs provide those dependencies,
             | then I don't need to bundle them.
             | 
             | I'm thinking of things like the Go or Python standard
             | libraries, or OTP in Erlang. On iOS you have a rich set of
             | built-in frameworks, etc., etc.
        
               | ChrisMarshallNY wrote:
               | ...and you have a huge number of folks that program for
               | iOS in JavaScript, with dozens of dependencies in each
               | project.
               | 
               | This can lead to some rather... _overfed_...applications,
               | and four- or five-digit bug counts.
        
               | numbsafari wrote:
               | That sounds like a problem for those developers and their
               | approach, not with the platform on which they are
               | building.
        
               | ChrisMarshallNY wrote:
               | Yup, but these have become so common, in the industry,
               | that they are now synonymous.
               | 
               | For example, I no longer trust people that describe
               | themselves as "iOS developers" to know the first thing
               | about actually developing for the Apple ecosystem. In
               | some cases, they don't even know how to use Xcode.
        
         | Tainnor wrote:
         | Maybe I'm misunderstanding, but it sounds like you talk about
         | modularity at the unit of deployments, i.e. separate services.
         | 
         | But you can have modularity within a single codebase and some
         | languages / build-systems allow you to define very clear
         | boundaries between your different modules. In that case, I
         | don't think you have necessary a lot of code duplication (you
         | might have slightly increased consumption due to the use of
         | more indirection). But you might have to code to interfaces and
         | explicitly eliminate coupling etc.
        
           | ChrisMarshallNY wrote:
           | Oh, no. I use it all the time, for code components.
           | 
           | If you look at my portfolio[0], you'll see it packed with
           | things like widgets and adornments. Most of these, I use in
           | my own projects. I eat my own dog food.
           | 
           | But today's languages, with their reliance on extensibility
           | and interfaces, makes writing standalone code assets a lot
           | less cumbersome. I write in Swift, and there are some pretty
           | big Swift "extension suites" out there[1]. The Apple Swift
           | Package Manager is nice, too[2].
           | 
           | It's just that when I write a code module, I do it as a
           | standalone _product_ , with a full test suite (usually a
           | harness[3], which often takes the form of a "full-fat"
           | application), and a unique release management cycle.
           | 
           | My modules are pretty independent, so they aren't usually
           | custom-tailored for all of their implementations; acting in a
           | fairly general capacity.
           | 
           | [0] https://stackoverflow.com/story/chrismarshall
           | 
           | [1] https://swifterswift.com
           | 
           | [2] https://swiftpackageindex.com
           | 
           | [3] https://littlegreenviper.com/miscellany/testing-harness-
           | vs-u...
        
           | taeric wrote:
           | Modularity is, by definition, a feature of the code. No
           | feature is free. Some do have major benefits. More often,
           | some are required by tradeoffs or circumstances of
           | deployment.
        
           | papito wrote:
           | They could have just said "distributed systems". It's not
           | like the Node kids suddenly invented "microservices".
        
           | tabtab wrote:
           | If the "modules" are source code you don't know or can't
           | control, then it's a dependency that can create dependency
           | headaches. For example, a new OS or browser version comes out
           | and the existing components are not compatible with the
           | upgrade. Happens all the time. External components are thus
           | not a free lunch.
           | 
           | What's really missing is the willingness to say "no". There's
           | pressure to put fancy or trendy toys into applications to
           | look up to date or to get experience on the latest fads. KISS
           | and YAGNI are often ignored for eye candy or Fear of Falling
           | Behind.
           | 
           | For example, fairly early in the 2000's I had a web form that
           | only did server-side date validation. It worked just fine.
           | But client-side date validation was becoming "expected"
           | because everyone started doing it and the manager asked me to
           | put it in. So I found a JavaScript component on the web, and
           | it worked fine at first. But a few months later a bug was
           | found in it that appeared on a certain date, and it generated
           | a lot of phone calls from users. Those phone calls would have
           | never existed had I gone with my gut and left it out. Few
           | users are going to spend all day making date errors such that
           | client-side date validation is NOT necessary. It's a me-too
           | toy.
           | 
           | Warren Buffett often cites his ability to say "no" as to why
           | he's so wealthy. He looks at the actual tradeoff numbers, not
           | peer pressure.
        
         | xbar wrote:
         | This is actually a call for restrained Product Management.
        
           | ChrisMarshallNY wrote:
           | _> This is actually a call for restrained Product
           | Management._
           | 
           | I would adjust that, like so:
           | 
           | This is actually a call for restrained _, jargon-free,_
           | Product Management.
        
         | markus_zhang wrote:
         | One thing I observed as a data analyst (prev job) was that
         | modularity could make data anlysis pretty difficult.
         | 
         | Example: In a monolistic application you can easily send out
         | telemetrics of different "components", e.g. player hitpoints,
         | coins in inventory, days in game, etc. But in a microservice
         | structure (at least the one I saw), since hitpoint service
         | doesn't care about coins, and coins service doesn't care about
         | days in game, to get all three in one telemetry, developers
         | have to break the barrier and connect them in one way or
         | another.
         | 
         | So essentially you still get to write them into sort of
         | monolithtic, but uglier and needs more arguments (but still
         | have to write them at the end because CXO needs the report).
        
           | 3pt14159 wrote:
           | That's what data warehouses are good for though. So long as
           | you have something you can join on, you don't need to worry
           | about how the application developers structure their systems.
        
             | markus_zhang wrote:
             | Theoretically it's true (but again the burden is simple
             | transferred to someone else).
             | 
             | But in reality, as a BI Developer (prev as a data analyst),
             | what I really don't want to do is a join by timestamp (and
             | a userId), pretty much the only way to join two
             | "subsystems" that are not interconnected.
             | 
             | Example: Analysts want to know what's the coin balance when
             | player starts purchasing energy bundles. Now that they
             | don't know about each other so the telemetry only has one
             | common field: userId. Eventually either BI developer (who
             | develops DWH/View/etc.) or DA has to write a join by userId
             | and timestamp to get the most accurate coin balance when
             | player purchases an energy bundle.
        
           | pjc50 wrote:
           | Microservices are not the only way to modularize. But yes,
           | cross-cutting concerns tend to wreck modularization; in
           | desktop environments it's usually global settings or events
           | that cause this.
           | 
           | (Long ago, the software industry wanted to copy the
           | electronics industry, where people would assemble software
           | from components listed in catalogs. We never quite got there,
           | but you can see the influence of this metaphor all over the
           | place when people talk about "components". It required the
           | development of libre+gratuit Free online libraries to get
           | there, causing an absolute Cambrian explosion and the left-
           | pad problem)
        
             | tabtab wrote:
             | Most things are cross-cutting concerns. For example,
             | database access and security are issues that entity-
             | specific modules have to deal with, and it's best to share
             | those features rather than have every entity reinvent it.
             | 
             | Early in the GUI days, you had UI components that were
             | pretty independent. But now they are expected to match the
             | look and feel of given OS versions or brands such that they
             | can no longer be truly isolated: they are expected to
             | integrate somewhat with the OS. If people had the guts to
             | live with "ugly", a lot of things could be simplified. I
             | discuss saying "no" to the IT Fashion Show nearby. It may
             | read like a git-off-my-lawn rant, but software could be
             | cheaper if you kick fadsters off your lawn. Beauty ain't
             | free.
        
               | delusional wrote:
               | That's an interesting take. My observation is quite the
               | opposite, with each application seemingly overriding
               | systems style willy nilly. I'd say that application
               | styling has become MORE differentiated.
        
           | dafelst wrote:
           | I mean that is exactly what correlation IDs and joins are
           | for.
        
             | markus_zhang wrote:
             | It's not that easy. Join by what? In a lot of my work I
             | have to join by common userId (sure) and timestamp which is
             | pretty heavy. Something like this:
             | 
             | SELECT *, ROW_NUMBER() OVER(PARTITION BY userId ORDER BY
             | timestamp DESC) AS rowNum FROM A LEFT JOIN B ON A.timestamp
             | <= B.timestamp and A.date = B.date
             | 
             | And then write another query to take the top row based on
             | timestamp, and for sure this method may lose some data
             | (e.g. what if userId doesn't have data in table B that
             | day?).
        
               | dafelst wrote:
               | (warning: old man rant ahead, apologies in advance)
               | 
               | My original answer was perhaps a bit terse and should
               | have been more along the lines of "that is what a good
               | schema design should fix". Your original position that
               | "microservices make things more difficult for analysis"
               | is absolutely true, but I guess my observation is that
               | while you might get lucky and have everything you need in
               | one place in your raw data, for any sufficiently complex
               | system (especially if there are 3rd party components
               | involved) you're going to need to do some real work to
               | transform that data to build a repository with a flexible
               | schema to support your analyses and make yours and your
               | colleagues work easier and more productive. You are 100%
               | right in that every layer of indirection makes this
               | slightly more difficult, but unfortunately that is
               | something you need to design for in all but the most
               | simple systems - this is true today the same as it was
               | true 20-30 years ago.
               | 
               | Hopefully you're already doing that sort of design, but
               | my experience is that disciplined schema design for
               | effective analysis is incredibly rare, despite it being
               | an old, crufty, well established and documented area ala
               | Kimball and old school Enterprise Data Warehousing and
               | BI. Not all of it translates perfectly to the world of
               | Big Data(TM) but the fundamental concepts of designing
               | such that your data can be turned into easily queryable
               | and useful measures and dimensions still very much
               | applies.
               | 
               | For your example, the correct correlation ID completely
               | depends on the context in which you need to do your
               | analysis, and that should be ideally baked into your
               | telemetry schema where applicable (and where it makes
               | sense from an architectural/performance/data volume
               | perspective), but if not, then baked into your
               | intermediate ETL schema such that it is easily accessible
               | for analysis. Perhaps it's a contextual or global
               | incrementing index that more easily allows you to get the
               | "latest" transaction per context (maybe a session or a
               | match in your game, or an encounter, or outside of games,
               | a shopping cart checkout flow), maybe your client passes
               | more contextual information to the microservice to allow
               | you to more easily hierarchically partition into your
               | context. Maybe if you don't have that level of control
               | downstream (like in a 3rd party service for example) you
               | have an ETL or streaming job generating the global or
               | local context from the raw telemetry that makes it easy
               | to join against.
               | 
               | And lack of context for analysis isn't even just a
               | microservices issue, it can even happen in any kind of
               | modular software - in the game I'm working on right now,
               | for example, the crafting system has no need to know
               | about overall state from the player, hence the crafting
               | module has no (easy) access to player state that might be
               | relevant for downstream analysis - so even in what would
               | be considered a monolith, getting access to player
               | health/armor/xp/etc in the context of crafting analyses
               | (e.g. do players craft medkits at low health more often
               | than at high health?) is difficult and really uglies
               | things up, much like you imply. But making available a
               | playerStateIndex that can be used to join back to that
               | state (managed by the player state system) is doable and
               | introduces no real problematic module coupling, and is a
               | repeatable and extensible pattern to use across other
               | modules and even external services.
               | 
               | I've spent the better part of half of my 20+ years in the
               | industry doing some form of data analytics, and my
               | message to the analysts/data scientists I work with is
               | always "if you have to do crazy gymnastics to do your
               | analyses then our schema design needs work". You don't
               | want to be in the (all too common) world of analysts
               | spending 80-90% of their time doing data transformations
               | and cleaning and 10-20% of their time doing analysis.
        
               | markus_zhang wrote:
               | >if you have to do crazy gymnastics to do your analyses
               | then our schema design needs work.
               | 
               | Yeah I figured out that if the query is over say 50 lines
               | long and has weird marks everywhere (say DISTINCT with
               | some windows function) then it's pretty much a bad
               | database design.
               | 
               | Then I move to the data side and figured out that it
               | actually also has something to do with the original raw
               | data sent by the app.
               | 
               | So eventually _someone_ has to take the burden and do the
               | gymnatics and frankly it totally depends on who is closer
               | to the CXO. In our company the developers are in favor so
               | they can do whatever they want.
               | 
               | >Hopefully you're already doing that sort of design, but
               | my experience is that disciplined schema design for
               | effective analysis is incredibly rare, despite it being
               | an old, crufty, well established and documented area ala
               | Kimball and old school Enterprise Data Warehousing and
               | BI. Not all of it translates perfectly to the world of
               | Big Data(TM) but the fundamental concepts of designing
               | such that your data can be turned into easily queryable
               | and useful measures and dimensions still very much
               | applies.
               | 
               | EXACTLY what I'm feeling now. We are trying to work out a
               | DWH on top of the raw telemetries (basically some simple
               | transformations and routing done by Spark from HQ).
               | However I really think that Kimball doesn't fit very well
               | with the big data thing. I don't know how to say it
               | clearly because I just moved to the data team for a few
               | months so I'm a total newbie and everything I _feel_
               | could be 100% BS. I think we should start from business
               | requirement (i.e. from our data analytics lords), and
               | then bend the rules (Kimball will require a LOT of extra
               | joins if what I'm reading it correctly) and keep the DWH
               | tables sort of flattened but in a way that makes more
               | sense than the original raw telemetry tables.
               | 
               | I'm not in control so I don't know what it will result
               | into.
               | 
               | >For your example, the correct correlation ID completely
               | depends on the context in which you need to do your
               | analysis, and that should be ideally baked into your
               | telemetry schema where applicable (and where it makes
               | sense from an architectural/performance/data volume
               | perspective), but if not, then baked into your
               | intermediate ETL schema such that it is easily accessible
               | for analysis. Perhaps it's a contextual or global
               | incrementing index that more easily allows you to get the
               | "latest" transaction per context (maybe a session or a
               | match in your game, or an encounter, or outside of games,
               | a shopping cart checkout flow), maybe your client passes
               | more contextual information to the microservice to allow
               | you to more easily hierarchically partition into your
               | context. Maybe if you don't have that level of control
               | downstream (like in a 3rd party service for example) you
               | have an ETL or streaming job generating the global or
               | local context from the raw telemetry that makes it easy
               | to join against.
               | 
               | It's...not really easy to come up with an intermediate
               | "common context" id sometimes. Session Id doesn't work
               | because each session takes a lot of time and nothing else
               | really are real-time. Since I'm the only guy in the
               | company for requirements taking (I really think
               | developers should have their own business analyst), it's
               | pretty painful that each time I have to fight both sides.
               | I have to ask the DA team to relax their requirements
               | (realtime? No it's not possible, how about same session?)
               | and ask the developers to bestow more connection fields
               | (if you have this field DA will be really happy), thus
               | the vent above :D
        
           | Droobfest wrote:
           | This is what api's are for. A telemetry service asks each
           | service for the specific data, combines them and outputs the
           | combined metrics. This doesn't really break a barrier, but it
           | does increase overhead.
        
             | markus_zhang wrote:
             | Yeah but other teams don't really have control over that.
        
         | Koshkin wrote:
         | Keep in mind that modularity is a complex notion: the space of
         | components can be rather non-trivial, in that the dependency
         | graph can have a pretty interesting topology - if you include,
         | in this notion, for instance, inheritance, generics
         | (templates), and all the related usage patterns that have been
         | invented over the past 40 years. (I believe Modula and Oberon
         | both failed to address this complexity, having restricted the
         | programmer's understanding of the concept to a very basic use
         | case.)
        
       | ryaan_anthony wrote:
       | 1995 - the year JS was first released
        
       ___________________________________________________________________
       (page generated 2021-06-28 23:01 UTC)