[HN Gopher] The Opposite of Documentation is Superstition (2020)
___________________________________________________________________
The Opposite of Documentation is Superstition (2020)
Author : BerislavLopac
Score : 51 points
Date : 2024-11-27 08:32 UTC (3 days ago)
(HTM) web link (buttondown.com)
(TXT) w3m dump (buttondown.com)
| smitty1e wrote:
| I have also used the phrase "tribal knowledge" in the
| documentation-vacuum context.
| m348e912 wrote:
| There could be a business to be made in capturing and
| disseminating tribal knowledge for organizations. Even orgs
| with the best documentation don't have a great mechanism to
| capture and make available tribal knowledge which is un-
| authoritative, sometimes conflicting, and constantly changing.
|
| In the past I made an effort to address this by creating a
| unofficial but quickly searchable document repository
| documenting every question I have gotten from a teammate by
| either answering it or pointing to where the authoritative
| answer could be found.
|
| What I found was the following:
|
| 1. Questions are repetitive, you can quickly build a base of
| tribal knowledge and point people to the answer. 2. If the
| document tool is quick and efficient, team members will
| generally check it first instead of asking. 3. Sometimes
| document tribal knowledge conflicts with someone else's tribal
| knowledge. This is a great opportunity to hash out the more
| correct answer and update the document. 4. Tribal knowledge
| gets stale fast and needs constant curating. 5. The less
| friction to find an answer, eg fast access and fast/accurate
| searching, greatly increases success.
|
| NB. GPT is a perfect front end for such an effort.
| wjholden wrote:
| I would have loved to train an LLM on my old technical blog.
|
| In my last job, I used a blog to write informal reports. I
| set pretty loose rules for the blog for my team. We would
| post meeting minutes, outage postmortems, scripts, tricky
| one-liners, general lessons learned, and so on. We used
| SharePoint, which is kinda meh but it works. The "Alert Me"
| feature in SharePoint made it easy to subscribe my teammates
| to the blog via email, so you got the benefits of email (push
| notifications) without the drawbacks (employee mailboxes are
| unsearchable once they leave).
|
| I believe email contributes to this tacit ("tribal")
| knowledge that the OP names. I think my blog went a long way
| to counter this effect. A major goal in our blog posts was to
| capture the "why" behind our decisions. Often, we fail to
| remember the nuances of our own decisions in hindsight. I
| always thought it was important to name our present
| priorities and known-unknowns.
| nox101 wrote:
| Seems to me documentation can also be superstition if it's out of
| date.
|
| I wonder if LLMs can generate docs at any reasonable level of
| usefulness and accuracy
|
| Speaking of which, I just tried using ChatGPT to create a React
| app with typescript and rollup. What it generated didn't work
| (got errors). Of course I could tell it the errors and it fixed
| them but it suggests they aren't yet ready to write accurate docs
| iterateoften wrote:
| Maybe LLMs make docs irrelevant if they are just generating
| static pages or doc strings. Why spend the time generating
| something static when everyone has their own LLMs they can just
| ask directly? Anything you can generate as docs someone can
| generate themselves anyways.
| danihh wrote:
| We actually ran this train of thought with our startup right
| now. The problem? Not everybody knows what questions to ask.
| For non-power users it's usually better to present digestible
| information and create docs that answer questions they didn't
| even know they had.
| blharr wrote:
| I mean, LLMs can be regenerated as needed. With every major
| release, you might regenerate the documentation. Then, you
| don't need to have everyone taking time to ask the same
| questions, reducing the amount of times you need to call the
| LLM. And thereby you can use a more expensive LLM, like o1 to
| generate the documentation where you wouldn't want to use it
| normally
| AshamedCaptain wrote:
| Why would you bring ML up? The entire example problem brought
| up by TFA is because the ink-to-shape transform likely uses
| some ML behind the scenes (plain interpolation likely) , and as
| consequence, it is an _indocumentable_ blackbox even if you
| wanted to document it. Superstition is the only way you can
| make it work.
|
| In fact, "superstition" very neatly defines most prompt
| engineering for LLMs.
| ChrisMarshallNY wrote:
| I tend to use in-code documentation[0].
|
| Works pretty well. Could still [theoretically] go out of date,
| but that hasn't happened, so far.
|
| The one thing that does happen, though, is copy/pasta errors.
| I'll copy a bunch of code, including the headerdocs, and paste,
| but forget to adjust the docs.
|
| [0] https://littlegreenviper.com/leaving-a-legacy/
| closed wrote:
| This is an interesting case, since the pigeon study is about what
| happens when the underlying process is random.
|
| But if the shape drawing process isn't random, I think the
| author's experience of feeling unable to articulate the rules AND
| gravitating to a set of behaviors is a good example of procedural
| memory (implicit vs explicit).
|
| Explicit rules would probably help speed things up, though!
| ChrisMarshallNY wrote:
| He's really talking about the user's mental model, here.
|
| I've found that it is sometimes a good idea to reinforce mental
| models that don't actually reflect the operation of the software
| (in my case) under it.
|
| Documentation is good, but, in my experience, almost no one ever
| reads it. I can tell, because of all the questions that I get,
| that are in the docs.
|
| That's totally my fault. A really good tech writer could probably
| do a better job of conveying information.
|
| That said, I think that it's important to present good,
| discoverable UI, that doesn't punish users for exploring.
| pfix wrote:
| Both is important. And related. Documentation needs to be
| discoverable. I was amazed when I tried jj (jujutsu, the "new
| git") and it popped me into some kind of weird textual user
| interface after executing `jj split` and I felt lost. I
| guessed, pressing `?` won't hurt and it told me just to use the
| mouse. The menues showed the related hot keys.
|
| But the actual documentation of the tool has room for
| improvement. I needed a YouTube video to get started and that's
| rare for me.
|
| So what I want to say is, that you need an intuitive,
| discoverable UI, but also a documentation that has each case
| (and if it's just for linking in 1st level support cases) _and_
| is discoverable. And by that I mean both easy to grasp (e.g.
| following https://diataxis.fr/) and also can actually be found.
| I've had cases where a tool had good documentation, but
| actually finding it was the hard part.
| throwaway14356 wrote:
| our architects love building the house before making the
| drawings. i imagine we will probably figure it out eventually
| when the feature set can be strictly defined.
|
| (maybe you eventually want a bath tub and a toilet in each
| room? maybe not?)
| vincent-manis wrote:
| I can't remember the number of times I have given up on software
| because there is no "Concepts and Facilities" document that gives
| me a mental model of what the program will do and how I operate
| it. Instead, one sees a website with a mass of unrelated
| documents that I'm supposed to read in some order, and divine how
| everything works.
|
| Maybe because I'm a child of the 20th century that I value
| documentation that gives me a path from understanding what and
| why to learning how I can use it, with a master index so I can
| look up exactly what the "Mung Until No Good" command does, and
| see which other commands relate to it.
|
| Perhaps one day generative AI will be good enough that we can
| feed in one of these websites, with its pages with titles like
| "Migrating from V4.7.2 to V4.7.3" and "Building for OS/2 Warp" (I
| exaggerate, but only slightly) to documentation that is useful
| for learning, use, and troubleshooting. I live in hope.
| a_t48 wrote:
| Thanks for the reminder to make the concepts section of my site
| easier to read. :)
| shizzy0 wrote:
| Some of the best technical documentation I've ever written has
| straight up lies in it. Granted they were placed there in the
| service of distilling the most germane portions of the API I was
| trying to explain. But it made a huge difference once I decided
| my code in the library didn't have to be my code in its
| documentation. I felt a little like a biographical writer who
| makes a composite character. But honestly I recommend trying it.
| Lie in your docs if it serves the greater goal of communicating
| what's most important. Let the computer labor in the "truth" and
| its tediums.
| Noumenon72 wrote:
| Do you mean you just left out required args, try-catches for
| checked exceptions, and so on, so the example would be easier
| to read? Did you put an asterisk on, like "* illustrative code:
| will not compile"?
| Animats wrote:
| I've called this ritual-taboo programming for decades. It happens
| for user interfaces and APIs when the documentation is absent, or
| only consists of examples. If there's no reference documentation,
| everything is a copy of something someone else did. Nobody
| understands how it really works.
|
| Now, for some interfaces, this isn't too bad. Most people don't
| know why US AC power plugs are polarized, or what the ground
| prong is for. Electricians have to, but users do not.
|
| For more complex interfaces, it means that many functions will
| either be misused or undiscovered. This is the source of the
| plaint that only 10%-20% of a product's features are used.
|
| On user interface design, the classic is "Tog on Interface",
| Bruce "Tog" Tognazzini, 1992. That's from the Mac UI era. A more
| modern take is "The Gamer's Brain", by Celia Hodent, designer of
| Fortnite's UX.
|
| Software internal documentation seems to suffer today from a
| mindset that comments are unnecessary and waste space. Especially
| in the Javascript era. It's worst in languages that don't have
| data declarations. There's no place to properly document the
| data.
|
| Rust has good conventions and tools for documenting data. There's
| a well defined place where the documentation for each structure
| and field goes, and reasonable tools for checking it and turning
| that into documentation. If you fail to do this, when you publish
| your crate on crates.io, the documentation pages will come up
| blank, which screams "loser".
|
| Rust is weak on function parameter documentation. There should
| have been a defined place where each formal parameter gets a
| comment, which then appears in the documentation of the call.
|
| Most other languages don't take such a hard line. More should.
| fiforpg wrote:
| > ritual-taboo programming
|
| I like to think of it as "cargo-cult programming", along the
| lines of the well-known essay by Feynman.
| Animats wrote:
| Cargo cults don't work at all. Ritual-taboo systems sort of
| work but are suboptimal.
| IgorPartola wrote:
| How do you view self-documenting interfaces?
|
| Have you ever read the manual for your browser which you are
| using to make this comment? The OS on which the browser is
| running? The device on which they are running?
|
| I think there are interfaces that need to be usable assuming
| documentation will not be read. These are of course user
| interfaces, not necessarily APIs. But if you are creating
| something like a library of basic math functions I would rather
| read your code than your docs.
|
| Some of the worst docs I've read lately have been for ffmpeg
| and mp4box. The former sometimes straight up has things like
| "so one guy tried this and it worked for him so we recommend
| you try it and see how it works for you". The latter has like 4
| examples with no explanation of what you are looking at or how
| to make anything else work. The project used Stack Overflow for
| their docs because nobody can really figure out what to do with
| it right away. I'd rather take readable code in this case over
| the terrible docs.
| throwaway14356 wrote:
| writing great documentation is extremely hard. sometimes it
| is so well done you only notice it in how fast and easy you
| progress. the funniest opposite was documentation i wrote
| myself but turned out to complicated for the future self. i
| kept thinking, what is this guy on about?? it assumed the
| reader knew all kinds of things i didn't and it made effort
| to explain the obvious.
| w10-1 wrote:
| The opposite of docs is also expertise: learn through experience,
| then have a career moat. It's why Microsoft and Apple get away
| with secret API, which also gives them some leverage. Docs tend
| to be good only in the post-prototype, pre-dominance phase of
| rapid growth. Or when the author cares about users.
| throwaway14356 wrote:
| the fun of that study is that the pigeon behavior they think is
| to get food is superstition it self.
| GuB-42 wrote:
| One also could say that the opposite of open source is
| superstition.
|
| It works with the example given in the article: OneNote is
| proprietary software, and it is documented, though incompletely.
| Had it been open source, one could see exactly how shape
| recognition works, bugs, quirks and all. No documentation will
| give that amount of detail.
|
| Sure, not everyone has the time and skills for that, but maybe
| someone on stackoverflow does and have answered the question
| _with proof_ , that is, not a superstition.
| codr7 wrote:
| Isn't this more likely an example of LLM-crappification?
|
| As in, the reason it isn't documented is that no one has a
| clue...
| akira2501 wrote:
| Technically the opposite of documentation is anecdote.
|
| Literal interpretation of anecdotes lead to superstition.
|
| Ironically this leads to them becoming documented.
___________________________________________________________________
(page generated 2024-11-30 23:00 UTC)