jackkelly.name.rss.xml - sfeed_tests - sfeed tests and RSS and Atom files
(HTM) git clone git://git.codemadness.org/sfeed_tests
(DIR) Log
(DIR) Files
(DIR) Refs
(DIR) README
(DIR) LICENSE
---
jackkelly.name.rss.xml (150211B)
---
1 <?xml version="1.0" encoding="utf-8"?>
2 <rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"
3 xmlns:dc="http://purl.org/dc/elements/1.1/">
4 <channel>
5 <title>Blog | jackkelly.name</title>
6 <link>http://jackkelly.name</link>
7 <description><![CDATA[Latest posts on jackkelly.name]]></description>
8 <atom:link href="http://jackkelly.name/blog/rss.xml" rel="self"
9 type="application/rss+xml" />
10 <lastBuildDate>Fri, 09 Jul 2021 13:00:00 UT</lastBuildDate>
11 <item>
12 <title>Free Software's Relevance in 2021</title>
13 <link>http://jackkelly.name/blog/archives/2021/07/09/free_softwares_relevance_in_2021/index.html</link>
14 <description><![CDATA[<p>2020 was a bad year at the end of a bad decade for software freedom. This post discusses how software rules everyone’s lives, how bad the situation is for software freedom, why software freedom matters, and explains why I believe that free software developers should direct their efforts toward emerging language ecosystems.</p>
15 <!--more-->
16 <h2 id="why-software-freedom-matters">Why Software Freedom Matters</h2>
17 <p>Over the past eighteen months, there have been incredible restrictions placed on people’s freedoms of movement, speech, and association. To have any social contact at all, many people have been restricted to online interactions as their only social outlet, almost exclusively through proprietary software.</p>
18 <p>In Australia, we are required by law to report our personal movements to one or more “contact-tracing” platforms, even to visit the supermarket or have something like a normal social life. This forces people to interact with an enormous pile of proprietary surveillance software just to participate in society. Despite earlier promises to only be used for contact tracing, these databases have been <a href="https://www.abc.net.au/news/2021-06-29/queensland-coronavirus-check-in-app-police-data-search/100249624">used by police</a> (as always happens). It should be possible to build these check-in systems in a privacy-preserving way, and for the public to verify they have done so. A sketch: store location history on the device, upload to contact tracers <em>only</em> with the person’s consent and <em>only</em> in response to contact tracing queries. Unlink PII (used for contacting exposed people) from user IDs (used to build the graph of exposures) and store it separately. Free software then allows citizens at least some confidence that the software has in fact been built in a privacy-preserving way.</p>
19 <p>We live our lives at the mercy of software, and have been doing so well before the pandemic. Some non-COVID examples:</p>
20 <ul>
21 <li><p>Inflexible automated systems make human compassion and discretion impossible.</p></li>
22 <li><p>The <a href="https://en.wikipedia.org/wiki/Robodebt_scheme">“Robodebt” fiasco</a> applied nonexistent debts to welfare recipients, driving some to suicide.</p></li>
23 <li><p>“Smart home” devices in general. All software is haunted; a house that runs on software becomes a haunted house.</p></li>
24 </ul>
25 <p>Being able to inspect, modify and run software on your own terms is valuable to everyone as a matter of public interest. The Free Software Foundation outlines <a href="http://www.gnu.org/philosophy/free-sw.en.html">four freedoms</a> for a program to be considered “free software”:</p>
26 <ul>
27 <li>The freedom to run the program for any purpose;</li>
28 <li>The freedom to study the program, and to change it;</li>
29 <li>The freedom to redistribute copies; and</li>
30 <li>The freedom to distribute your modified copies.</li>
31 </ul>
32 <h2 id="the-four-freedoms-are-not-enough">The Four Freedoms are not Enough</h2>
33 <p>These days, the FSF’s four freedoms are not enough to ensure software freedom for users:</p>
34 <ul>
35 <li><p>Having the ability to see/change/build the source code to a program is no use if you cannot actually deploy and run that software:</p>
36 <ul>
37 <li><p>The large body of BSD/MIT/X11-licenced software allows proprietary programs to be deployed on phones, consoles, and other locked-down devices. Now that locked-down computing is accepted by the public, this trend is creeping into PCs.</p>
38 <ul>
39 <li>Example: Windows certification on x86/amd64 used to require the computer owner to be able to disable his or her motherboard’s “secure boot” feature. This is no longer true for Windows 11 and has never been true for Windows on ARM. How long until motherboard vendors just stop bothering to give users that option?</li>
40 </ul></li>
41 <li><p>Software running on a server or “in the cloud” is never “distributed” or “conveyed” to users, bypassing the protections of all GPL-style licences except the <a href="https://www.gnu.org/licenses/why-affero-gpl.html">Affero GPL</a>.</p></li>
42 </ul></li>
43 <li><p>The rise of software powered by machine learning models means that full source code can be useless, unless you have both the training data and enough computing power to re-train the models.</p></li>
44 </ul>
45 <h2 id="the-market-will-not-save-you">The Market will not Save You</h2>
46 <p>For years, privacy advocates have been saying <a href="https://duckduckgo.com/?t=ffab&q=if+it%27s+free%2C+you%27re+the+product&ia=web">“if it’s free, you’re the product”</a>. These days, not even paying will protect you:</p>
47 <ul>
48 <li><p>Windows gets a special dishonourable mention:</p>
49 <ul>
50 <li><p>Windows 10 includes advertising all over the OS, even down to <a href="https://news.ycombinator.com/item?id=4826334">minesweeper</a>. All the UI will let you do is turn off your “advertising ID”.</p></li>
51 <li><p>Windows <a href="https://www.windowslatest.com/2020/11/15/windows-10-is-now-nagging-users-with-microsoft-edge-recommendations/">nags users to try Microsoft’s Edge browser</a>.</p>
52 <ul>
53 <li>Seen <a href="https://www.windowslatest.com/2021/05/31/windows-10-is-now-nagging-users-with-microsoft-bing-alerts/">yet again</a> as I was writing this post.</li>
54 </ul></li>
55 <li><p>Windows has a bad habit of restarting to install “updates”, which are actually <a href="https://www.theverge.com/2020/10/17/21520315/microsoft-install-office-pwa-web-app-without-permission-update-word-powerpoint-excel">unwanted programs</a> (or <a href="https://www.tenforums.com/software-apps/110391-unwanted-apps-install-automatically-again.html">crappy games</a>).</p></li>
56 </ul></li>
57 <li><p>Windows isn’t the only offender of this type. Google recently installed a health notification app onto people’s phones without their knowledge or consent (<a href="https://news.ycombinator.com/item?id=27558500">HN</a>, <a href="https://play.google.com/store/apps/details?id=gov.ma.covid19.exposurenotifications.v3&showAllReviews=true">Google Play</a>). You bought the phone, but everyone except you gets to run their code on it.</p></li>
58 <li><p>Even <a href="https://news.ycombinator.com/item?id=24502768">mouse drivers</a> come with ads these days.</p></li>
59 <li><p>Have you tried to buy a non-“smart” TV recently? You pretty much can’t, and “smart” TVs have ads in their menu systems now, and who knows what data they exfiltrate? Low-priced TVs definitely make <a href="https://www.engadget.com/vizio-q1-earnings-inscape-013937337.html?guccounter=1">nearly as much money from ads and data as they do from selling the TV itself</a>, and I’m sure that even premium models use spying as an additional revenue source.</p></li>
60 </ul>
61 <h2 id="copyleft-software-is-not-attractive-enough">Copyleft Software is not Attractive Enough</h2>
62 <p>The FSF’s proposed solution is to release software under <a href="https://www.gnu.org/licenses/copyleft.en.html">copyleft</a> licences, so that user freedom is preserved as people share copyleft software with each other. In the old old days, system software from the <a href="https://www.gnu.org">GNU Project</a> was a significant improvement over the UNIX vendor’s proprietary software. The software the GNU Project is most well-known for (system-level tools like <a href="https://www.gnu.org/software/coreutils/"><code>coreutils</code></a>) has since been thoroughly commoditised, and people these days don’t really say “I’d really love to use <code>$AWESOME_LIBRARY</code>; I’ll copyleft my own software so that I can.”</p>
63 <h2 id="suggestions">Suggestions</h2>
64 <p>I’d love to set out a clear and simple plan to liberate the world through the power of copyleft. Unfortunately, all I can offer are a few suggestions. The overall idea: make copyleft software more powerful and easier to contribute to, so that it becomes more attractive for users and developers.</p>
65 <h3 id="one-make-it-easier-to-contribute-to-copyleft-projects">One: Make it Easier to Contribute to Copyleft Projects</h3>
66 <p>The GCC Project recently relaxed its contributing requirements from “requires copyright assignment to the FSF”, to “<a href="https://lwn.net/Articles/857791/">provide Developer Certificate of Origin</a>”. I am optimistic: the easier it is to contribute to copyleft projects, the more people will do it.</p>
67 <p>If you run a project and proceed on this path, I think it’s most future-proof to license individual contributors (and possibly the whole project) under the “or any later version”-style terms. This provides a mechanism to evolve the project licence without tracking down every single contributor each time you want to change.</p>
68 <h3 id="two-copyleft-unique-software">Two: Copyleft Unique Software</h3>
69 <p>If you publish a copyleft library in a crowded domain, people can flinch away and select an alternative. If you build something unique and different, people are more likely to pay the copyleft “price of entry” because the supply of alternatives is smaller.</p>
70 <p>Famous example in the other direction: <a href="https://www.xiph.org/vorbis/doc/libvorbis/"><code>libvorbis</code></a> switched from the <a href="https://lists.xiph.org/pipermail/vorbis/2001-February/001758.html">LGPL to a BSD-style licence</a> with RMS’ approval, because the space of sound libraries was already very crowded. Having <code>libvorbis</code> under LGPL hurt the main goal — adoption of a non-patent-encumbered audio codec — more than licensing via LGPL helped normalise copyleft.</p>
71 <h3 id="three-develop-for-emerging-languages">Three: Develop for Emerging Languages</h3>
72 <p>A more extreme version of point two: because there are relatively few copyleft developers compared to BSD/MIT/X11-licence FOSS developers, copyleft developers must be proportionally more powerful in order to stay competitive. Writing everything in C is no longer going to cut it; copyleft authors should be using the most powerful tools possible.</p>
73 <p>I think that if copyleft software wants to carve a niche for itself in 2021, emerging language ecosystems might be a good place to consider:</p>
74 <ul>
75 <li><p>Emerging languages often aim to give developers more power, which fits the “most powerful tools” goal.</p>
76 <ul>
77 <li>Developer tooling is another potential target area, as it’s <a href="http://www.pathsensitive.com/2021/03/developer-tools-can-be-magic-instead.html">massively powerful and massively underdeveloped</a>. But Freedom Zero makes it impossible to say “you can only use these tools on copyleft projects”. If the tool requires the developer to link his or her program against a runtime library, perhaps then it is strategically useful to build and release as copyleft.</li>
78 </ul></li>
79 <li><p>There are often fewer developers in an emerging language overall, which means that a copyleft project has more time to get established before a “BSD-licenced alternative” pops up to compete with it.</p></li>
80 <li><p>People working on a new niche may be happy just to “work on their thing” regardless of licensing. (Example: the FSF was <a href="https://lists.gnu.org/archive/html/emacs-devel/2015-02/msg00594.html">offered the LLVM copyrights</a>. Astonishingly, the offer slipped through the cracks.)</p></li>
81 <li><p>There are often large gaps in the libraries available for emerging languages. Contributing a JSON or XML library helps a lot of projects.</p></li>
82 </ul>
83 <p>It could be interesting to see what happens to an emerging language with a high-quality copyleft-by-default ecosystem instead of a BSD/MIT/X11-licence-by-default ecosystem. Perhaps such an ecosystem could make copyleft developers more productive than non-copyleft open source developers, which could then make writing copyleft software more attractive.</p>
84 <h3 id="four-build-ecosystems">Four: Build Ecosystems</h3>
85 <p>Generalising point three: a single copyleft library is easily worked around. An ecosystem of copyleft libraries, all depending on each other? It becomes easier to copyleft than to reinvent the wheel and the axle and the transmission and the chassis…</p>
86 <h3 id="five-use-stronger-licenses-when-strategically-possible">Five: Use Stronger Licenses when Strategically Possible</h3>
87 <p>Especially on unique software (point two), use licenses that provide stronger protections for users and developers. The AGPL is a start, but a critical mass of novel “or any later version” copyleft software could be used to enforce protections relating to machine learning, ensuring interoperability, or ensuring installability. The licences also need to evolve to defend against new attacks (such as locked bootloaders), but can only do so if there’s enough projects supporting those transitions.</p>
88 <h2 id="conclusion">Conclusion</h2>
89 <p>I think that free software is vitally important to public society, and that copyleft software is still a useful way to ensure its continued existence. But that software needs to be more powerful for users and more attractive for developers, otherwise it will never reach critical mass. Instead, it will be routed around.</p>]]></description>
90 <pubDate>Fri, 09 Jul 2021 13:00:00 UT</pubDate>
91 <guid>http://jackkelly.name/blog/archives/2021/07/09/free_softwares_relevance_in_2021/index.html</guid>
92 <dc:creator>Jack Kelly</dc:creator>
93 </item>
94 <item>
95 <title>Town of ZZT - Remixed</title>
96 <link>http://jackkelly.name/blog/archives/2021/01/05/town_of_zzt_-_remixed/index.html</link>
97 <description><![CDATA[<p><strong>TL;DR:</strong> ZZT is now 30, there’s still a community, we’ve remixed the shareware world, it’s pretty fun, and you can <a href="https://museumofzzt.com/play/t/TOWNRMIX.zip">play it online</a>.</p>
98 <p>Thirty years ago, on the 15th of January 1991, Tim Sweeney released his first game to the world: a little arcadey game called <a href="https://museumofzzt.com/about-zzt">ZZT</a>, for the IBM PC and compatibles.</p>
99 <p><img src="/blog/images/zzt/zzt.png" alt="Original TOWN.ZZT Title Screen" /></p>
100 <!--more-->
101 <p>Despite being technially dated even when it came out (<a href="http://www.shikadi.net/keenwiki/Keen_3:_Keen_Must_Die!">Keen 3: Keen Must Die!</a> came out a month before, and <a href="http://www.shikadi.net/keenwiki/Keen_4:_Secret_of_the_Oracle">Commander Keen 4 - “Goodbye Galaxy”</a> would come out at the end of 1991), it sold well enough to fund Tim’s next game, which funded the one after that, which… ends up with Tim and his company Epic Games trying to <a href="https://www.matthewball.vc/all/epicgamesprimermaster">build the Metaverse</a>, apparently.</p>
102 <p>But back to ZZT. ZZT came with an editor, and the game’s simplicity meant that nearly anyone could make their own games with it. My friends and I had a lot of fun making crappy games with it back in the day (sadly/thankfully lost to time), and I kept half an eye on it over the years. A thriving online scene in the BBS and AOL era declined slowly through the “forum era” of the WWW (2000-2010?). In the late 2010s, the community had an <a href="https://museumofzzt.com/article/494/we-are-still-out-here">unexpected renaissance</a> thanks to the <a href="https://museumofzzt.com/">Museum of ZZT</a>, the <a href="https://zeta.asie.pl/">Zeta x86 emulator</a> (which can even run in the browser) and a <a href="https://blog.asie.pl/2020/08/reconstructing-zzt/">reconstruction of ZZT’s lost source code</a>.</p>
103 <p>During 2020, the Discord of ZZT community decided to remix the first world, Town of ZZT. People claimed one or more boards (screens), and reworked them as they saw fit. I snagged <a href="https://museumofzzt.com/file/t/TOWNRMIX.zip?file=TOWNRMIX.ZZT&board=20">The Mixer</a>. With a bit of a push during November/December, everyone submitted their boards in time for a new year’s release, to kick off ZZT’s 30th birthday celebrations:</p>
104 <p><a href="https://museumofzzt.com/file/t/TOWNRMIX.zip"><img src="/blog/images/town-remix/remix-title.png" alt="Remixed TOWN.ZZT Title Screen" /></a></p>
105 <p>This thing is seriously impressive. Most boards had a substantial visual upgrade, some really push ZZT’s capabilities, and others reference the game’s long history. The result somehow feels both fresh and familiar, and I had an absolute blast seeing what everyone else created. Tim Sweeney also retweeted the release announcement; it’s cool to see that he noticed.</p>
106 <p><a href="https://twitter.com/worldsofzzt/status/1345202964667875329"><img src="/blog/images/town-remix/tim-rt.png" alt="Tim Sweeney Retweeted Release Announcement" /></a></p>
107 <p>The community was a delight to work with, which helped the project feel really coherent in ways that a less-organised remix wouldn’t. People would say “I have this idea, who wants to be the other half?” and someone would run with it. This coordination also meant small details were done <em>right</em>, like matching colour fades between boards and having NPCs drop hints for other people’s puzzles. This does a lot to tie the world together, and there are also a couple of longer cross-board links and gags that are just too fun to spoil.</p>
108 <p>The museum has a ZIP file explorer and can run ZZT in your browser, so <a href="https://museumofzzt.com/play/t/TOWNRMIX.zip">give it a go</a> or just <a href="https://museumofzzt.com/file/t/TOWNRMIX.zip">have a poke around</a>. You could also explore the <a href="https://museumofzzt.com/file/z/zzt.zip?file=TOWN.ZZT">original Town</a> at the same time, if you want to see what’s been changed.</p>]]></description>
109 <pubDate>Tue, 05 Jan 2021 13:00:00 UT</pubDate>
110 <guid>http://jackkelly.name/blog/archives/2021/01/05/town_of_zzt_-_remixed/index.html</guid>
111 <dc:creator>Jack Kelly</dc:creator>
112 </item>
113 <item>
114 <title>Vaccine Deployment</title>
115 <link>http://jackkelly.name/blog/archives/2021/01/04/vaccine_deployment/index.html</link>
116 <description><![CDATA[<p><strong>TL;DR:</strong> Despite earlier reservations, I will probably be taking a vaccine when offered, but it would be for different reasons if I was outside Australia. Also, continue to watch out for manipulation and fudging from authorities.</p>
117 <p>(Note: Twitter links have screenshots next to them, as it’s notoriously difficult to predict what Twitter shows on the other end of a link.)</p>
118 <p>Emergency approvals for the major vaccine candidates (Pfizer/BioNTech, Moderna, Oxford/AstraZeneca) have been issued, and it’s time to scale up and deploy. Then we reach herd immunity and get back to normal life, right? Well, first we need to make sure we’re all talking about the same “herd immunity”, and watch closely for shifting definitions. Between November and December 2020, the WHO’s website changed its definition of “herd immunity” at least twice (emphasis in the quotes is mine):</p>
119 <!--more-->
120 <ul>
121 <li><p><a href="https://web.archive.org/web/20201112033233/https://www.who.int/news-room/q-a-detail/coronavirus-disease-covid-19-serology">2020-11-12</a>:</p>
122 <blockquote>
123 <p>Herd immunity is the indirect protection from an infectious disease that happens when a population is immune <strong>either</strong> through <strong>vaccination</strong> or immunity developed through <strong>previous infection</strong>.</p>
124 </blockquote></li>
125 <li><p><a href="https://web.archive.org/web/20201114155111/https://www.who.int/news-room/q-a-detail/coronavirus-disease-covid-19-serology">2020-11-24</a>:</p>
126 <blockquote>
127 <p>‘Herd immunity’, also known as ‘population immunity’, is a concept used for vaccination, in which a population can be protected from a certain virus if a <strong>threshold of vaccination</strong> is reached.</p>
128 </blockquote></li>
129 <li><p><a href="https://web.archive.org/web/20201231221345/https://www.who.int/news-room/q-a-detail/coronavirus-disease-covid-19-serology">2020-11-31</a>:</p>
130 <blockquote>
131 <p>‘Herd immunity’, also known as ‘population immunity’, is the indirect protection from an infectious disease that happens when a population is immune <strong>either</strong> through <strong>vaccination</strong> or immunity developed through <strong>previous infection</strong>. WHO supports achieving ‘herd immunity’ through vaccination, not by allowing a disease to spread through any segment of the population, as this would result in unnecessary cases and deaths.</p>
132 </blockquote></li>
133 </ul>
134 <p>The 2020-11-24 definition attempts to <a href="/blog/archives/2020/06/08/enforcing_the_consequent">enforce the consequent</a> — the actual goal is to reach herd immunity through vaccination instead of exposure. Instead of just <em>saying that</em>, the WHO instead claimed that “herd immunity” <em>requires</em> vaccination. I suspect the 2020-11-31 change only happened because people noticed and kicked up a stink, and this is why it’s always important that we keep paying close attention and hold institutions to account. They’ll take lazy shortcuts otherwise.</p>
135 <p>What else is changing? Well, we have <a href="https://www.nytimes.com/2020/12/24/health/herd-immunity-covid-coronavirus.html">Dr. Fauci shifting the herd immunity goalposts</a> (emphasis in quotes mine):</p>
136 <blockquote>
137 <p>In a telephone interview the next day, Dr. Fauci acknowledged that he had slowly but <strong>deliberately</strong> been <strong>moving the goal posts</strong>. He is doing so, he said, partly based on new science, and partly on his <strong>gut feeling</strong> that the country is finally <strong>ready to hear what he really thinks</strong>.</p>
138 <p>[…]</p>
139 <p>Dr. Fauci said that weeks ago, he had hesitated to publicly raise his estimate <strong>because many Americans seemed hesitant about vaccines</strong>…</p>
140 <p>[…]</p>
141 <p>“When polls said only about half of all Americans would take a vaccine, I was saying herd immunity would take 70 to 75 percent,” Dr. Fauci said. “Then, when newer surveys said 60 percent or more would take it, I thought, <strong>‘I can nudge this up a bit,’</strong> so I went to 80, 85.”</p>
142 </blockquote>
143 <p>It is despicable to gripe that people aren’t “trusting the science” and then admit that you’re making decisions based on gut feelings. The new mutations from UK and South Africa are more transmissible and are spreading almost world-wide. This will increase the herd immunity threshold (which is <span class="math inline">1 − 1/<em>R</em></span>, so if <span class="math inline"><em>R</em></span> increases, so does the threshold). But this manipulation is disgusting. It’s no wonder people are hesitant about the vaccines when they can see the manipulation taking place at other levels (remember “masks don’t work”? <a href="/blog/archives/2020/04/26/what_i_remember">I do</a>).</p>
144 <h2 id="hows-the-rollout-going">How’s the Rollout Going?</h2>
145 <p>We now need more people than ever to get jabbed, and the faster the better. We have the emergency approvals, so all we need to do is crank up production, get doses into people’s arms and this is over? Right? I mean, that’s <a href="https://twitter.com/segal_eran/status/1344570377469628416">what Israel’s doing</a> (<a href="/blog/images/vaccine-deployment/israel.png">screenshot</a>): vaccinating <a href="https://marginalrevolution.com/marginalrevolution/2020/12/vaccinate-24-7.html">24 hours a day, 7 days a week, including Shabbat</a>. This is what <em>actually trying</em> looks like.</p>
146 <p>How about the rest of the world? Here’s a <a href="https://www.bloomberg.com/graphics/covid-vaccine-tracker-global-distribution/">vaccine rollout tracker</a>. Zvi has <a href="https://thezvi.wordpress.com/2020/12/31/covid-12-31-meet-the-new-year/">another giant post</a> about this week in COVID. The stuff about rollouts starts at “All I Want For Christmas is a Covid Vaccine, But They Somehow Underpriced Them So Much No One’s Even Bothering To Sell Out”, and nobody is <em>actually trying</em>:</p>
147 <ul>
148 <li>Canada: vaccinators took several days off during holidays.</li>
149 <li>UK: a retired GP was unable to volunteer as a vaccinator, because she didn’t have her bueraucratic box-ticking certificates in order (e.g., “Conflict Resolution”, “Fire Safety”, “Preventing Radicalisation”).</li>
150 <li>US: some doses are being given out to the general public, because they’d expire otherwise. Deployment rates are a disappointing fraction of total doses shipped.</li>
151 </ul>
152 <h2 id="will-people-take-a-vaccine">Will People take a Vaccine?</h2>
153 <p>It doesn’t matter how bad the rollout machinery is if nobody is taking the doses. It doesn’t look good so far (links go to twitter threads):</p>
154 <ul>
155 <li><a href="https://twitter.com/HHH_Report/status/1340098612487983105">‘In America, almost 72% of “surveyed certified nursing assistants who work in long-term care facilities” don’t want to take the vaccine.’</a> (<a href="/blog/images/vaccine-deployment/hhh-report.png">screenshot</a>)</li>
156 <li><a href="https://twitter.com/conorduffy_7/status/1345175306202722309">“Speaking to several normies over the Christmas has given me the impression that a lot of this mightn’t be anti-vax per se, but more unease at how quickly it was tested and approved, and that the best way to deal with this would just be to vaccinate lots of the less hesitant”</a> (<a href="/blog/images/vaccine-deployment/connorduffy.png">screenshot</a>)</li>
157 </ul>
158 <p>Would I take a vaccine? I’ve been hesitant about the SARS-CoV-2 vaccines (the mRNA ones in particular), and I know I’m not alone among my friends. Not because we’re anti-vaxxers, but because the vaccine approvals seem somehow too fast and too slow: the trials don’t feel long enough, but when it was time for the FDA to actually issue an emergency approval, they took Thanksgiving off and scheduled a meeting to look over Pfizer’s data <a href="https://thedispatch.com/p/fda-career-staff-are-delaying-the"><em>three weeks</em> after submission</a> (NB: the comment section is probably better than the article).</p>
159 <p>So why have I changed my mind? I’m concerned by these “<a href="https://twitter.com/PaulBieniasz/status/1345195420033691648">musings of an anonymous, pissed-off virologist</a>” (repeated below because screenshots of text suck):</p>
160 <details>
161 <summary>
162 Musings of an anonymous, pissed-off virologist
163 </summary>
164 <blockquote>
165 <p>
166 As viruses go, SARS-CoV-2, is quite easy to neutralize with antibodies and, it turns out, straightforward to generate effective vaccines based on the spike protein. Perhaps, even probably, those two properties are causally related. Moreover, it appears that it is quite hard (albeit not impossible) to generate resistant spike variants that evade the polyclonal antibody responses elicited by said vaccines. This is all excellent news.
167 </p>
168 <p>
169 However, if I had a nefarious nature and wanted to ensure that the new SARS-CoV-2 vaccines were rendered impotent, these are a few things I would try.
170 </p>
171 <p>
172 First, we’d want to maximize the viral population size and diversity. Because SARS-CoV-2 has a proof-reading polymerase, we might have to work hard to do this. The four measures outlined below might help accomplish this, assisting the virus to explore as much genetic diversity as possible, generating every conceivable point mutation as frequently as possible.
173 </p>
174 <ol>
175 <li>
176 Delay the rollout of testing, so that the virus could spread undetected, seeding outbreaks in geographically, demographically and culturally diverse host populations, rendering it virtually impossible to quash with test-trace-isolate approaches.
177 </li>
178 <li>
179 Implement partial and patchy restrictions on movement and social interactions, thus maintaining consistently large pools of infected individuals.
180 </li>
181 <li>
182 Keep schools open, claiming that children don’t frequently transmit SARS-CoV-2. Because children have generally mild and perhaps more frequently asymptomatic infections, diversifying viral populations are more likely to spread undetected.
183 </li>
184 <li>
185 Start a rumor-mill, making full use of social media and other outlets, with topics such as masks are unnecessary or don’t work, that PCR tests are too sensitive or unreliable, that infection-induced ‘herd immunity’ is a reasonable strategy, or even that SARS-CoV-2 isn’t real. Undermining already inadequate public health measures helps keep viral population sizes large.
186 </li>
187 </ol>
188 <p>
189 Second, during or after the establishment of large and diverse viral populations, we’d begin to apply selection pressure to enrich antibody resistance mutations. For that, we would elicit the help of the medical establishment to implement measures 5 and 6. They, laudably, want to help as many people as possible as quickly as possible — we could exploit this.
190 </p>
191 <ol start="5">
192 <li>
193 Treat tens of thousands of people with uncharacterized convalescent plasma of weak/unknown potency, without proper clinical trials, to get the ball rolling in applying some selection pressure to enrich for antibody resistant variants. (Again, I don’t know how effective this would be since it is mostly done in hospitals, where onward transmission would presumably be rare, but it would certainly be worth a try) Immunocompromised individuals with persistent infection might be especially helpful here.
194 </li>
195 <li>
196 Finally, and here’s the kicker: having developed a remarkable two-dose vaccine, that is extraordinarily effective, ADMINISTER IT TO MILLIONS OF PEOPLE — BUT DELAY THE SECOND DOSE. Generating a pool of hosts with just the right amount of neutralizing antibody to apply selection pressure, but also maintain sufficient levels of partially antibody-resistant virus to allow onward transmission is key here. We might not achieve this shortly after the first dose, but if we let immunity wane for a little while, say 4 to 12 weeks, we just might hit the sweet spot. Of course, I don’t know if the above would be successful, but that’s what l’d try if I wanted to generate vaccine-resistant SARS-CoV-2 variants.
197 </li>
198 </ol>
199 </blockquote>
200 </details>
201 <p>The world’s pathologically (heh) bad response creates a rich environment for mutations (e.g., the new UK strain, similar mutations in South Africa) and potential for vaccine escape. If I was living overseas, I would seriously consider taking any of the offered vaccines, for the following reasons:</p>
202 <ol type="1">
203 <li><p>The risk trade-off is much higher because there’s a lot more virus going around. Containment of the UK strain has failed, and it’s been detected in the US and all over Europe.</p></li>
204 <li><p>Faster vaccination rollout limits the amount of time the virus has to mutate and get around vaccine-derived antibodies.</p></li>
205 <li><p>The current degree of lockdown is extremely unpleasant, compliance is waning, and the lockdown required to contain the new strain is unthinkable (and probably unachievable).</p></li>
206 </ol>
207 <p>However, I have seen a couple of things that make me a bit cautious. I mention them for completeness but still think getting the jab (when offered) is correct overall:</p>
208 <ol type="1">
209 <li><p>Obviously and badly stage-managed vaccinations. I can’t find the links, but I remember seeing VIPs getting jabbed with empty or spring-loaded syringes, and a nurse who fainted after receiving the vaccine. She later claimed to be “prone to fainting spells” — why would you put someone like that in front of the cameras?</p></li>
210 <li><p>Four participants in the vaccine group of one of the Pfizer trials <a href="https://www.fda.gov/media/144245/download">developed Bell’s Palsy</a> (page 6), compared to zero in the placebo group. Even so, this is lower than the incidence of Bell’s Palsy in the general population.</p></li>
211 </ol>
212 <p>But in Australia, we’re not expecting to see rollout until March or so, and so my thoughts are a bit different. Here, I would probably take the Oxford/AstraZeneca vaccine, if offered. By the time any mRNA vaccines are offered, we should know from the overseas rollout whether there’s anything to watch out for. Here are my reasons why I’m okay with that rollout timeline, and willing to stay a bit cautious around the mRNA vaccines:</p>
213 <ol type="1">
214 <li><p>We don’t have widespread community transmission because our hotel quarantine (mostly) works.</p>
215 <p><strong>Note:</strong> This is not a one-sided trade-off. As demonstrated by the 2020-21 outbreak, our quarantine system is brittle, and nobody wants a repeat of what Melbourne went through. If we had widespread vaccination, we wouldn’t be creating so much waste (e.g., discarded masks), and domestic tourism would be much more stable. The mismanaged outbreak in Sydney <a href="https://www.abc.net.au/news/2021-01-01/victorians-race-to-get-home-before-nsw-border-closure/13024514">killed regional tourism over NYE</a>, which was a brutal follow-up for those places hit by the 2019-20 bushfires.</p></li>
216 <li><p>We probably won’t start seeing the vaccine rollout for a couple of months. This means we get to watch the world for more safety data (particularly for the mRNA vaccines, which are being deployed for the first time against COVID-19).</p></li>
217 <li><p>Australians are more likely to be vaccinated using the Oxford/AstraZeneca vaccine than an mRNA vaccine. We have domestic manufacturing capacity (<a href="https://www.abc.net.au/news/2020-11-24/oxford-astrazeneca-covid19-vaccine-gives-australia-pathway-out/12914340">CSL will manufacture 30M</a> of the 33.8M Oxford/AstraZeneca doses on order, but we can’t make mRNA vaccines), it’s cheaper per dose ($6-7 vs $40-60 per person for the mRNA vaccines), has less strict cold chain requirements (Pfizer needs storage at -70°C, Moderna at -20°C for long-term storage), and is based on proven techniques (an <a href="https://www.theguardian.com/world/2020/dec/30/how-well-does-the-oxford-vaccine-work-what-we-know-so-far">altered virus that doesn’t grow inside human cells</a>).</p></li>
218 <li><p><a href="https://www.9news.com.au/national/coronavirus-vaccine-australia-astra-zeneca-oxford-university-greg-hunt-when-can-i-be-vaccinated/66e3916a-7a47-4705-9e03-96d074d4fb6f">It is not being made mandatory</a>.</p>
219 <blockquote>
220 <p>“We will be making the vaccine free and voluntary,” Mr Morrison said.</p>
221 </blockquote>
222 <p>After a year of noble lies and deceitful manipulation of the public, this says that the vaccine can stand on its own merits, and that’s a massive breath of fresh air.</p></li>
223 </ol>
224 <p>We are not at the end, we haven’t yet seen the light at the end of the tunnel, but we should soon see the first pinprick of that light. It’s time to close this out.</p>]]></description>
225 <pubDate>Mon, 04 Jan 2021 13:00:00 UT</pubDate>
226 <guid>http://jackkelly.name/blog/archives/2021/01/04/vaccine_deployment/index.html</guid>
227 <dc:creator>Jack Kelly</dc:creator>
228 </item>
229 <item>
230 <title>Profunctor Decoders? Optical Decoders?</title>
231 <link>http://jackkelly.name/blog/archives/2020/11/03/profunctor_decoders_optical_decoders/index.html</link>
232 <description><![CDATA[<p>I’ve been wrangling with an interesting decoding problem over the weekend. I started out fiddling with profunctors and arrows, but it morphed into something more lenslike for a while. Since Chris Penner has just published his article about <a href="https://chrispenner.ca/posts/witherable-optics">fitting <code>Witherable</code> into an optical framework</a>, I thought I’d write up what I have so far, and see if I can get some useful thoughts out while this stuff is flavour of the week.</p>
233 <!--more-->
234 <h2 id="the-problem">The Problem</h2>
235 <p><a href="https://hackage.haskell.org/package/amazonka-dynamodb"><code>amazonka-dynamodb</code></a> uses an <a href="https://hackage.haskell.org/package/amazonka-dynamodb-1.6.1/docs/Network-AWS-DynamoDB-Types.html#t:AttributeValue"><code>AttributeValue</code></a> type to represent data stored in a DynamodDB table. Underneath all the lenses, this is morally a sum type: an <code>AttributeValue</code> can be a string, number, boolean, binary, null, string set, binary set, &c.</p>
236 <p>When storing or retrieving data (e.g., with the <a href="https://hackage.haskell.org/package/amazonka-dynamodb-1.6.1/docs/Network-AWS-DynamoDB-PutItem.html"><code>PutItem</code></a> and <a href="https://hackage.haskell.org/package/amazonka-dynamodb-1.6.1/docs/Network-AWS-DynamoDB-Query.html"><code>Query</code></a> operations), amazonka often uses <code>HashMap Text AttributeValue</code> in its request and response types. We’re looking for a solution with the following properties:</p>
237 <ol>
238 <li>
239 It needs to be composable. There should be little difference between (de)serialising a primitive <code>AttributeValue</code> and an <code>AttributeValue</code> at some specific column in the table.
240 </li>
241 <li>
242 <p>(De)serialising should be as close to bidirectional as possible, but only when that makes sense. Basic use cases treat a DynamoDB table as a key-value store, and deserialisation becomes very similar to serialisation (read/write <em>these</em> fields at <em>those</em> keys). DynamoDB wizards pack many access patterns into a single table, and each access pattern could well need a different (de)serialiser.</p>
243 <details>
244 <summary>
245 Aside: Single-Table Design in DynamoDB
246 </summary>
247 <p>Alex DeBrie has a great article on <a href="https://www.alexdebrie.com/posts/dynamodb-single-table/">single-table design in DynamoDB</a>. In it, he quotes <a href="https://www.trek10.com/blog/dynamodb-single-table-relational-modeling/">Forrest Brazeal</a> (whose article is also pretty good):</p>
248 <blockquote>
249 In fact, a well-optimized single-table DynamoDB layout looks more like machine code than a simple spreadsheet — despite all the bespoke, human finagling it took to create it.
250 </blockquote>
251 Rick Houlihan has also given some excellent talks about DynamoDB: <a href="https://www.youtube.com/watch?v=HaEPXoXVf2k">one in 2018</a>, and <a href="https://www.youtube.com/watch?v=jzeKPKpucS0">one in 2017</a>. Watch the 2018 talk first, because it does a better job of talking through technology shifts, and motivating the use of denormalised data. The 2017 talk has a lot of overlap with the 2018 talk, but there’s enough different material to make it worth watching too.
252 </details>
253 </li>
254 <li>
255 It should be symmetric. Where it makes sense, encoding and decoding should have similar interfaces.
256 </li>
257 <li>
258 It should be ergonomic. This is somewhat subjective, but important. Bad ergonomics is one reason I didn’t pursue <a href="/blog/archives/2020/08/19/abstracting_over_applicative_alternative_divisible_and_decidable">monoidal functors</a> any further — the <code>newtype</code>s that select which bifunctor to use added a lot of noise.
259 </li>
260 <li>
261 It should have good error reporting. It’s unacceptable to return <code>Nothing</code> on failure.
262 </li>
263 </ol>
264 <h2 id="non-solution-tofrom-classes">Non-Solution: <code>To</code>/<code>From</code> Classes</h2>
265 <p><a href="https://hackage.haskell.org/package/aeson"><code>aeson</code></a>-style encoder/decoder typeclasses are well-understood. We need to deserialise both <code>AttributeValue</code> and <code>HashMap Text AttributeValue</code>, which means we need two classes for each direction:</p>
266 <div class="sourceCode" id="cb1"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true"></a><span class="kw">data</span> <span class="dt">DecodeError</span> <span class="ot">=</span> <span class="dt">Something</span> <span class="co">-- Placeholder</span></span>
267 <span id="cb1-2"><a href="#cb1-2" aria-hidden="true"></a></span>
268 <span id="cb1-3"><a href="#cb1-3" aria-hidden="true"></a><span class="kw">class</span> <span class="dt">FromAttributeValue</span> a <span class="kw">where</span></span>
269 <span id="cb1-4"><a href="#cb1-4" aria-hidden="true"></a><span class="ot"> fromAttributeValue ::</span> <span class="dt">AttributeValue</span> <span class="ot">-></span> <span class="dt">Either</span> <span class="dt">DecodeError</span> a</span>
270 <span id="cb1-5"><a href="#cb1-5" aria-hidden="true"></a></span>
271 <span id="cb1-6"><a href="#cb1-6" aria-hidden="true"></a><span class="kw">class</span> <span class="dt">FromAttributeValues</span> a <span class="kw">where</span></span>
272 <span id="cb1-7"><a href="#cb1-7" aria-hidden="true"></a><span class="ot"> fromAttributeValues ::</span> <span class="dt">HashMap</span> <span class="dt">Text</span> <span class="dt">AttributeValue</span> <span class="ot">-></span> <span class="dt">Either</span> <span class="dt">DecodeError</span> a</span>
273 <span id="cb1-8"><a href="#cb1-8" aria-hidden="true"></a></span>
274 <span id="cb1-9"><a href="#cb1-9" aria-hidden="true"></a><span class="kw">class</span> <span class="dt">ToAttributeValue</span> a <span class="kw">where</span></span>
275 <span id="cb1-10"><a href="#cb1-10" aria-hidden="true"></a><span class="ot"> toAttributeValue ::</span> a <span class="ot">-></span> <span class="dt">AttributeValue</span></span>
276 <span id="cb1-11"><a href="#cb1-11" aria-hidden="true"></a></span>
277 <span id="cb1-12"><a href="#cb1-12" aria-hidden="true"></a><span class="kw">class</span> <span class="dt">ToAttributeValues</span> a <span class="kw">where</span></span>
278 <span id="cb1-13"><a href="#cb1-13" aria-hidden="true"></a><span class="ot"> toAttributeValues ::</span> a <span class="ot">-></span> <span class="dt">HashMap</span> <span class="dt">Text</span> <span class="dt">AttributeValue</span></span></code></pre></div>
279 <p>This is not acceptable for several reasons:</p>
280 <ul>
281 <li><p>Type classes for serialisation become awkward across package boundaries. Libraries that provide data structures end up depending on serialisation libraries (potentially pulling in large dependency trees) or are forced to create a separate package with either orphan instances or newtypes just to hold instances. None of these are satisfactory options.</p></li>
282 <li><p>Type-class-based serialisation libraries sometimes provide instances that use an element’s (de)serialisers to (de)serialise a collection (e.g., <code>instance FromJSON a => FromJSON [a]</code>). This is convenient, but presupposes a single canonical transformation, which is not necessarily the case.</p></li>
283 <li><p>There’s no symmetry between encoders and decoders. Decoders may fail, encoders always succeed.</p></li>
284 </ul>
285 <h2 id="partial-solution-profunctorarrow-coders">Partial Solution: Profunctor/Arrow Coders</h2>
286 <p>We want to be able to abstract our inputs over <code>AttributeValue</code> and <code>HashMap Text AttributeValue</code>, so the obvious thing to do is to have a type variable for the input as well as the output. We’d also like to deal with different ways of failing (<code>Maybe</code>, <code>Either SomeError</code>, accumulating errors into a <code>Validation</code>, <code>These</code>, …). Let’s hack up some types:</p>
287 <div class="sourceCode" id="cb2"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true"></a><span class="kw">class</span> <span class="dt">Failing</span> f <span class="kw">where</span></span>
288 <span id="cb2-2"><a href="#cb2-2" aria-hidden="true"></a><span class="ot"> warn ::</span> <span class="dt">String</span> <span class="ot">-></span> a <span class="ot">-></span> f a</span>
289 <span id="cb2-3"><a href="#cb2-3" aria-hidden="true"></a><span class="ot"> err ::</span> <span class="dt">String</span> <span class="ot">-></span> f a</span>
290 <span id="cb2-4"><a href="#cb2-4" aria-hidden="true"></a></span>
291 <span id="cb2-5"><a href="#cb2-5" aria-hidden="true"></a><span class="kw">newtype</span> <span class="dt">Coder</span> f i o <span class="ot">=</span> <span class="dt">Coder</span> {<span class="ot"> runCoder ::</span> i <span class="ot">-></span> f o }</span>
292 <span id="cb2-6"><a href="#cb2-6" aria-hidden="true"></a> <span class="kw">deriving</span> stock <span class="dt">Functor</span></span>
293 <span id="cb2-7"><a href="#cb2-7" aria-hidden="true"></a> <span class="kw">deriving</span> <span class="kw">newtype</span> (<span class="dt">Semigroup</span>, <span class="dt">Monoid</span>)</span>
294 <span id="cb2-8"><a href="#cb2-8" aria-hidden="true"></a> <span class="kw">deriving</span> (<span class="dt">Applicative</span>, <span class="dt">Monad</span>) via (<span class="dt">Star</span> f i)</span>
295 <span id="cb2-9"><a href="#cb2-9" aria-hidden="true"></a></span>
296 <span id="cb2-10"><a href="#cb2-10" aria-hidden="true"></a><span class="ot">hoistCoder ::</span> (<span class="kw">forall</span> a <span class="op">.</span> f a <span class="ot">-></span> g a) <span class="ot">-></span> <span class="dt">Coder</span> f i o <span class="ot">-></span> <span class="dt">Coder</span> g i o</span>
297 <span id="cb2-11"><a href="#cb2-11" aria-hidden="true"></a>hoistCoder nt (<span class="dt">Coder</span> f) <span class="ot">=</span> <span class="dt">Coder</span> <span class="op">$</span> nt <span class="op">.</span> f</span></code></pre></div>
298 <p>(<code>Star</code> is <a href="https://hackage.haskell.org/package/profunctors-5.5.2/docs/Data-Profunctor.html#t:Star"><code>Data.Profunctor.Star</code></a>, and is representationally equivalent to <code>Coder</code>.)</p>
299 <p>This gives us a whole pile of useful instances, and we can even use instances from <code>Control.Arrow</code> or <a href="https://hackage.haskell.org/package/product-profunctors-0.11.0.0"><code>product-profunctors</code></a> to construct analogues to <a href="https://hackage.haskell.org/package/contravariant-1.5.2/docs/Data-Functor-Contravariant-Divisible.html"><code>Divisible</code> and <code>Decidable</code></a>. (I gave a talk about this at <a href="https://www.meetup.com/FP-Syd/">fp-syd</a> the other week, but it was quite exploratory and I’m not sure if there’s good video.)</p>
300 <p>Let’s assume the existence of a simple decoder of <code>Int</code> from <code>AttributeValue</code>, and write a lifting function that can extract from a map key:</p>
301 <div class="sourceCode" id="cb3"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true"></a><span class="ot">decodeInt ::</span> <span class="dt">Failing</span> f <span class="ot">=></span> <span class="dt">Coder</span> f <span class="dt">AttributeValue</span> <span class="dt">Int</span></span>
302 <span id="cb3-2"><a href="#cb3-2" aria-hidden="true"></a>decodeInt <span class="ot">=</span> <span class="fu">undefined</span> <span class="co">-- Not important</span></span>
303 <span id="cb3-3"><a href="#cb3-3" aria-hidden="true"></a></span>
304 <span id="cb3-4"><a href="#cb3-4" aria-hidden="true"></a>decodeAtKey</span>
305 <span id="cb3-5"><a href="#cb3-5" aria-hidden="true"></a><span class="ot"> ::</span> (<span class="dt">Eq</span> k, <span class="dt">Hashable</span> k, <span class="dt">Show</span> k, <span class="dt">Failing</span> f)</span>
306 <span id="cb3-6"><a href="#cb3-6" aria-hidden="true"></a> <span class="ot">=></span> k <span class="ot">-></span> <span class="dt">Coder</span> f v o <span class="ot">-></span> <span class="dt">Coder</span> f (<span class="dt">HashMap</span> k v) o</span>
307 <span id="cb3-7"><a href="#cb3-7" aria-hidden="true"></a>decodeAtKey k (<span class="dt">Coder</span> f) <span class="ot">=</span> <span class="dt">Coder</span> <span class="op">$</span> \i <span class="ot">-></span></span>
308 <span id="cb3-8"><a href="#cb3-8" aria-hidden="true"></a> <span class="fu">maybe</span> (err <span class="op">$</span> <span class="st">"missing key: "</span> <span class="op"><></span> <span class="fu">show</span> k) f <span class="op">$</span> HashMap.lookup k i</span></code></pre></div>
309 <p>So far, so good. We could go even further and decode the <code>[HashMap Text AttributeValue]</code> that the <code>Query</code> operation returns. What about encoding? Let’s suppose the existence of an encoder from <code>Int</code> to <code>AttributeValue</code> and try to lift it into encoding at a map key:</p>
310 <div class="sourceCode" id="cb4"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true"></a><span class="ot">encodeInt ::</span> <span class="dt">Coder</span> <span class="dt">Identity</span> <span class="dt">Int</span> <span class="dt">AttributeValue</span></span>
311 <span id="cb4-2"><a href="#cb4-2" aria-hidden="true"></a>encodeInt <span class="ot">=</span> <span class="fu">undefined</span> <span class="co">-- Not important</span></span>
312 <span id="cb4-3"><a href="#cb4-3" aria-hidden="true"></a></span>
313 <span id="cb4-4"><a href="#cb4-4" aria-hidden="true"></a>encodeAtKey</span>
314 <span id="cb4-5"><a href="#cb4-5" aria-hidden="true"></a><span class="ot"> ::</span> (<span class="dt">Functor</span> f, <span class="dt">Eq</span> k, <span class="dt">Hashable</span> k)</span>
315 <span id="cb4-6"><a href="#cb4-6" aria-hidden="true"></a> <span class="ot">=></span> k <span class="ot">-></span> <span class="dt">Coder</span> f i v <span class="ot">-></span> <span class="dt">Coder</span> f i (<span class="dt">HashMap</span> k v)</span>
316 <span id="cb4-7"><a href="#cb4-7" aria-hidden="true"></a>encodeAtKey k (<span class="dt">Coder</span> f) <span class="ot">=</span> <span class="dt">Coder</span> <span class="op">$</span> \i <span class="ot">-></span></span>
317 <span id="cb4-8"><a href="#cb4-8" aria-hidden="true"></a> f i <span class="op"><&></span> \v <span class="ot">-></span> HashMap.singleton k v</span></code></pre></div>
318 <p>Here’s where the cracks creep in. We’d like to <code>mconcat</code> our outputs together to make a larger structure, but that can be <a href="/blog/archives/2020/10/16/accidentally-quadratic_hashmaps">surprisingly slow</a>. We actually want the <code>Monoid</code> instance of <a href="https://hackage.haskell.org/package/base-4.12.0.0/docs/Data-Monoid.html#t:Endo"><code>Endo m</code></a>:</p>
319 <div class="sourceCode" id="cb5"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true"></a>encodeAtKey</span>
320 <span id="cb5-2"><a href="#cb5-2" aria-hidden="true"></a><span class="ot"> ::</span> (<span class="dt">Eq</span> k, <span class="dt">Hashable</span> k)</span>
321 <span id="cb5-3"><a href="#cb5-3" aria-hidden="true"></a> <span class="ot">=></span> k <span class="ot">-></span> <span class="dt">Coder</span> <span class="dt">Identity</span> i v <span class="ot">-></span> <span class="dt">Coder</span> <span class="dt">Endo</span> i (<span class="dt">HashMap</span> k v)</span>
322 <span id="cb5-4"><a href="#cb5-4" aria-hidden="true"></a>encodeAtKey k (<span class="dt">Coder</span> f) <span class="ot">=</span> <span class="dt">Coder</span> <span class="op">$</span> \i <span class="ot">-></span></span>
323 <span id="cb5-5"><a href="#cb5-5" aria-hidden="true"></a> <span class="dt">Endo</span> <span class="op">$</span> HashMap.insert k (runIdentity <span class="op">$</span> f i)</span>
324 <span id="cb5-6"><a href="#cb5-6" aria-hidden="true"></a></span>
325 <span id="cb5-7"><a href="#cb5-7" aria-hidden="true"></a><span class="ot">collectHashMap ::</span> [<span class="dt">Coder</span> <span class="dt">Endo</span> i (<span class="dt">HashMap</span> k v)] <span class="ot">-></span> <span class="dt">Coder</span> <span class="dt">Identity</span> i (<span class="dt">HashMap</span> k v)</span>
326 <span id="cb5-8"><a href="#cb5-8" aria-hidden="true"></a>collectHashMap coders <span class="ot">=</span> <span class="dt">Coder</span> <span class="op">$</span> \i <span class="ot">-></span></span>
327 <span id="cb5-9"><a href="#cb5-9" aria-hidden="true"></a> <span class="dt">Identity</span> <span class="op">$</span> appEndo (runCoder (<span class="fu">mconcat</span> coders) i) HashMap.empty</span></code></pre></div>
328 <p>To do this, we’ve had to abandon any pretense of generality in the “<code>Functor</code>” type parameter <code>f</code>. We can’t use <code>Compose Endo f</code> because it doesn’t have the <code>Monoid</code> instance we want.</p>
329 <h2 id="what-about-optics">What About Optics?</h2>
330 <p>Let’s go back to when things were going well, and look more closely at <code>decodeAtKey</code>. What happens if we strip away the <code>newtype</code>s?</p>
331 <div class="sourceCode" id="cb6"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb6-1"><a href="#cb6-1" aria-hidden="true"></a>decodeAtKey</span>
332 <span id="cb6-2"><a href="#cb6-2" aria-hidden="true"></a><span class="ot"> ::</span> (<span class="dt">Eq</span> k, <span class="dt">Hashable</span> k, <span class="dt">Show</span> k, <span class="dt">Failing</span> f)</span>
333 <span id="cb6-3"><a href="#cb6-3" aria-hidden="true"></a> <span class="ot">=></span> k <span class="ot">-></span> <span class="dt">Coder</span> f v o <span class="ot">-></span> <span class="dt">Coder</span> f (<span class="dt">HashMap</span> k v) o</span>
334 <span id="cb6-4"><a href="#cb6-4" aria-hidden="true"></a> <span class="co">-- k -> (v -> f o) -> (HashMap k v -> f o)</span></span>
335 <span id="cb6-5"><a href="#cb6-5" aria-hidden="true"></a> <span class="co">-- k -> LensLike f (HashMap k v) o v o</span></span></code></pre></div>
336 <p>This seems promising, and if we step into the <code>lens</code> universe we can use <code>Prism</code>s for our primitives. This also gives us a start on our “bidirectional where it makes sense” goal:</p>
337 <div class="sourceCode" id="cb7"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb7-1"><a href="#cb7-1" aria-hidden="true"></a><span class="ot">int ::</span> <span class="dt">Prism'</span> <span class="dt">AttributeValue</span> <span class="dt">Int</span></span>
338 <span id="cb7-2"><a href="#cb7-2" aria-hidden="true"></a>int <span class="ot">=</span> <span class="fu">undefined</span> <span class="co">-- Not important</span></span>
339 <span id="cb7-3"><a href="#cb7-3" aria-hidden="true"></a></span>
340 <span id="cb7-4"><a href="#cb7-4" aria-hidden="true"></a>preview<span class="ot"> int ::</span> m <span class="ot">-></span> <span class="dt">Maybe</span> <span class="dt">Int</span></span>
341 <span id="cb7-5"><a href="#cb7-5" aria-hidden="true"></a>review<span class="ot"> int ::</span> <span class="dt">Int</span> <span class="ot">-></span> <span class="dt">AttributeValue</span></span></code></pre></div>
342 <p>Trouble is, it’s very easy to lose prism-ness, and find ourselves stuck with a <code>Traversal</code> or <code>Fold</code>. To lift our primitives to work over map keys, we need to wrangle the prism itself:</p>
343 <div class="sourceCode" id="cb8"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb8-1"><a href="#cb8-1" aria-hidden="true"></a><span class="ot">atKey ::</span> (<span class="dt">Eq</span> k, <span class="dt">Hashable</span> k) <span class="ot">=></span> k <span class="ot">-></span> <span class="dt">Prism'</span> v a <span class="ot">-></span> <span class="dt">Prism'</span> (<span class="dt">HashMap</span> k v) a</span>
344 <span id="cb8-2"><a href="#cb8-2" aria-hidden="true"></a>atKey k pr <span class="ot">=</span> prism'</span>
345 <span id="cb8-3"><a href="#cb8-3" aria-hidden="true"></a> (HashMap.singleton k <span class="op">.</span> review pr)</span>
346 <span id="cb8-4"><a href="#cb8-4" aria-hidden="true"></a> (HashMap.lookup k <span class="op">>=></span> preview pr)</span></code></pre></div>
347 <p>We’ve found ourselves back at our first problem: we’re making singleton maps again. There might be ways around this (Yair Chuchem has some examples of using <code>Prism</code>s to <a href="https://yairchu.github.io/posts/codecs-as-prisms">implement codecs</a> and <a href="https://yairchu.github.io/posts/codecs-as-prisms-asts.html">“roundtrip” ASTs</a>), but even then you have to solve the error-reporting problem. Yair proposes a <a href="https://gist.github.com/yairchu/8f31122cf145985dcf8c4c5699c22bb2"><code>VerbosePointed</code> typeclass</a> to smuggle error information into the <code>f</code> parameter of an optic, but <a href="https://old.reddit.com/r/haskell/comments/eh4gpg/elegant_ast_parsing_and_building_with_prisms/fcjyzor/?context=1">Ed Kmett notes</a> that they ‘played with “coindexed” prisms which looked somewhat similar to these, but ran into problems with inference for them when folks would compose them with indexed traversals’, so that doesn’t sound promising.</p>
348 <h2 id="pickler-combinators">Pickler Combinators</h2>
349 <p>How else can we keep encoding and decoding together? I came across a 2004 Functional Pearl, <a href="https://www.microsoft.com/en-us/research/wp-content/uploads/2004/01/picklercombinators.pdf">“Pickler Combinators”</a>, which defines a “pickler/unpickler” type:</p>
350 <div class="sourceCode" id="cb9"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb9-1"><a href="#cb9-1" aria-hidden="true"></a><span class="kw">type</span> <span class="dt">St</span> <span class="ot">=</span> [<span class="dt">Char</span>]</span>
351 <span id="cb9-2"><a href="#cb9-2" aria-hidden="true"></a><span class="kw">data</span> <span class="dt">PU</span> a <span class="ot">=</span> <span class="dt">PU</span> {<span class="ot"> appP ::</span> (a, <span class="dt">St</span>) <span class="ot">-></span> <span class="dt">St</span>,<span class="ot"> appU ::</span> <span class="dt">St</span> <span class="ot">-></span> (a, <span class="dt">St</span>) }</span></code></pre></div>
352 <p>Many examples in the paper do the equivalent of <a href="https://hackage.haskell.org/package/invariant-0.5.4/docs/Data-Functor-Invariant.html#v:invmap"><code>invmap</code>ing</a> with a partial function on one side, which definitely doesn’t give us the safety we’re after. We also want to extend this to support varying input types, which means we have a <code>data PU i o = ...</code> type where <strong>both</strong> <code>i</code> and <code>o</code> are invariant. This quickly leads down a similar road as the monoidal functor stuff, but two invariant type variables make it twice as awkward.</p>
353 <h2 id="what-next">What Next?</h2>
354 <p>I’m not really satisfied by any of the options in this post, but the <code>Coder</code> profunctor/arrow experiment dissatisfies me least. I’ll probably develop this further at some point, and it might even be worth putting a “<code>Functor</code>-shaped parameter” on the input side, to allow a <code>Coder</code> to consume input from a streaming library:</p>
355 <div class="sourceCode" id="cb10"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb10-1"><a href="#cb10-1" aria-hidden="true"></a><span class="kw">newtype</span> <span class="dt">Coder</span> f g a b <span class="ot">=</span> <span class="dt">Coder</span> {<span class="ot"> unCoder ::</span> f a <span class="ot">-></span> g b }</span></code></pre></div>
356 <p>While I haven’t found a good story for bidirectional {en,de}coding, the <code>g</code> parameter enables a bunch of neat tricks:</p>
357 <ul>
358 <li>With <code>g ~ Either AnError</code>, you can report failure;</li>
359 <li>With <code>g ~ Endo</code>, you can build up maps or other data structures;</li>
360 <li>With <code>g ~ Compose IO (Const ())</code>, you can execute actions on the coded structure; or</li>
361 <li>With <code>g ~ ((,) i)</code>, you can pass unconsumed input to a downstream <code>Coder</code>, giving it the flavour of a parser combinator library.</li>
362 </ul>
363 <p>There’s a few things I’m hoping to do, to see whether or not this actually has legs:</p>
364 <ul>
365 <li>Build a vocabulary of combinators to lift <code>Coder</code>s in and out of maps, lists, other monoids; sequence <code>Coder</code>s and build up structures using <code>Applicative</code>/<code>Divisible</code>/<code>Decidable</code>-style combinators &c.;</li>
366 <li>Build out a vocabulary of primitive {en,de}-coders for DynamoDB <code>AttributeValue</code>s;</li>
367 <li>Work out how to make the different functors for the <code>g</code> parameter compose well. I suspect something MTL-ish will emerge, as we’ll need to consider (e.g.) <code>Coders</code> that pass input through <strong>and</strong> report errors;</li>
368 <li>Decide whether the <code>f</code> parameter on the input is worth the ergonomic cost. I suspect it is not; while decoding from an input stream is appealing, setting <code>f ~ Stream (Of a) m</code> severely constrains the type of the <code>g</code> parameter, and the a in <code>f a</code> won’t even refer to the stream item type (it names the type of the final value returned by the streaming computation).</li>
369 </ul>]]></description>
370 <pubDate>Tue, 03 Nov 2020 13:00:00 UT</pubDate>
371 <guid>http://jackkelly.name/blog/archives/2020/11/03/profunctor_decoders_optical_decoders/index.html</guid>
372 <dc:creator>Jack Kelly</dc:creator>
373 </item>
374 <item>
375 <title>Accidentally-Quadratic HashMaps</title>
376 <link>http://jackkelly.name/blog/archives/2020/10/16/accidentally-quadratic_hashmaps/index.html</link>
377 <description><![CDATA[<p><strong>TL;DR:</strong> Unless you’re careful, using <code>foldMap</code> to build <code>HashMap</code>s (from <a href="https://hackage.haskell.org/package/unordered-containers"><code>unordered-containers</code></a>) can take <span class="math inline"><em>O</em>(<em>n</em><sup>2</sup>)</span> time. This can be avoided by:</p>
378 <ol type="1">
379 <li><p>Building a <code>Map</code> (from <a href="https://hackage.haskell.org/package/containers"><code>containers</code></a>) instead;</p></li>
380 <li><p>Using <code>foldr</code> or <code>foldl</code> with <code>insert</code>; or</p></li>
381 <li><p>Using <code>foldMap</code> to build an intermediate list <code>[(k, v)]</code> and then building the <code>HashMap</code> with <code>fromList</code>.</p></li>
382 </ol>
383 <p>Shout-out to <a href="https://luke.worth.id.au/">Luke Worth</a> who found this out while we were chasing down performance problems.</p>
384 <!--more-->
385 <h2 id="foldmap"><code>foldMap</code></h2>
386 <p>The <code>foldMap</code> function is a very convenient way to build up maps, especially if you use the newtype wrappers from <a href="https://hackage.haskell.org/package/monoidal-containers"><code>monoidal-containers</code></a>:</p>
387 <div class="sourceCode" id="cb1"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true"></a><span class="fu">foldMap</span><span class="ot"> ::</span> (<span class="dt">Foldable</span> t, <span class="dt">Monoid</span> m) <span class="ot">=></span> (a <span class="ot">-></span> m) <span class="ot">-></span> t a <span class="ot">-></span> m</span>
388 <span id="cb1-2"><a href="#cb1-2" aria-hidden="true"></a></span>
389 <span id="cb1-3"><a href="#cb1-3" aria-hidden="true"></a><span class="co">-- Specialise m ~ MonoidalMap k v:</span></span>
390 <span id="cb1-4"><a href="#cb1-4" aria-hidden="true"></a><span class="fu">foldMap</span></span>
391 <span id="cb1-5"><a href="#cb1-5" aria-hidden="true"></a><span class="ot"> ::</span> (<span class="dt">Foldable</span> t, <span class="dt">Eq</span> k, <span class="dt">Ord</span> k, <span class="dt">Monoid</span> v)</span>
392 <span id="cb1-6"><a href="#cb1-6" aria-hidden="true"></a> <span class="ot">=></span> (a <span class="ot">-></span> <span class="dt">MonoidalMap</span> k v) <span class="ot">-></span> t a <span class="ot">-></span> <span class="dt">MonoidalMap</span> k v</span></code></pre></div>
393 <p>You map each <code>a</code> to a map (usually with the <code>singleton</code> function), and the <code>Monoid</code> instance combines each map into a single final result. This is a convenient idiom, but it takes quadratic time if you try to use it to build <code>HashMap</code>s. Let’s look at why.</p>
394 <p>The <code>Monoid</code> instance for <code>HashMap</code> is implemented in terms of <a href="https://hackage.haskell.org/package/unordered-containers-0.2.13.0/docs/Data-HashMap-Strict.html#v:unionWith"><code>union</code></a> (<code>MonoidalHashMap</code> uses <code>unionWith (<>)</code>, but is similar). <code>union</code> has <span class="math inline"><em>O</em>(<em>m</em> + <em>n</em>)</span> runtime, and the computation we get from our <code>foldMap</code> has the form:</p>
395 <div class="sourceCode" id="cb2"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true"></a>m1 <span class="ot">`union`</span> (m2 <span class="ot">`union`</span> (m3 <span class="ot">`union`</span> (m4 <span class="ot">`union`</span> (<span class="op">...</span>))))</span></code></pre></div>
396 <p>Assume each <code>m_i</code> is a singleton map (size <span class="math inline">1</span>). Then the total cost of the <code>union</code>s will add up like this:</p>
397 <ul>
398 <li><p><span class="math inline"><em>K</em><sub>1</sub></span>, the cost of the innermost <code>m_(n-1) `union` m_n</code>, costs <span class="math inline">1 + 1 = 2</span>;</p></li>
399 <li><p><span class="math inline"><em>K</em><sub>2</sub></span>, the cost of the two innermost <code>union</code>s (<code>m_(n-2) `union` (m_(n-1) `union` m_n)</code>) costs <span class="math inline">1 + 2 + 2 = 5</span>, because the map costs <span class="math inline">1 + 2 = 3</span> to build, plus another <span class="math inline">2</span> for the innermost <code>union</code>;</p></li>
400 <li><p><span class="math inline"><em>K</em><sub>3</sub></span>, the cost of the three innermost <code>union</code>s (<code>m_(n-3) `union` (m_(n-2) `union` (m_(n-1) `union` m_n))</code>) costs <span class="math inline">1 + 3 + 5 = 9</span>, because we pay <span class="math inline">1 + 3 = 4</span> to build the map, plus another <span class="math inline">5</span> to build the union of the three final entries;</p></li>
401 <li><p>Generally, the cost of the <span class="math inline"><em>i</em><sup><em>t</em><em>h</em></sup></span> union, <span class="math inline"><em>K</em><sub><em>i</em></sub></span>, will cost <span class="math inline">1 + <em>i</em> + <em>K</em><sub><em>i</em> − 1</sub></span>.</p></li>
402 </ul>
403 <p>If you expand out the <span class="math inline"><em>K</em><sub><em>N</em></sub></span> term, you will find a <span class="math inline">1 + 2 + 3 + … + <em>N</em></span> sum, which shows that <span class="math inline"><em>K</em><sub><em>N</em></sub> ∈ <em>O</em>(<em>N</em><sup>2</sup>)</span>.</p>
404 <h2 id="why-is-map-fine">Why is <code>Map</code> Fine?</h2>
405 <p><a href="https://hackage.haskell.org/package/containers-0.6.0.1/docs/Data-Map-Lazy.html#v:union"><code>union</code></a> for <code>Map</code> has running time <span class="math inline"><em>O</em>(<em>m</em> × log (<em>n</em>/<em>m</em> + 1)); <em>m</em> ≤ <em>n</em></span>. Since we’re dealing with singleton maps, we can say that <span class="math inline"><em>m</em> = 1</span> is the smaller side of the <code>union</code>, and the runtime simplifies to <span class="math inline"><em>O</em>(log <em>n</em>)</span>. From this, we can see that the runtime of all the <code>union</code>s will be <span class="math inline"><em>O</em>(<em>n</em>log <em>n</em>)</span>.</p>
406 <h2 id="what-if-i-need-hashmap">What if I Need <code>HashMap</code>?</h2>
407 <p>There are at least a couple of ways to turn a <code>Foldable</code> into a <code>HashMap</code> without hitting this performance trap:</p>
408 <ol type="1">
409 <li><p><code>HashMap</code>’s <a href="https://hackage.haskell.org/package/unordered-containers-0.2.13.0/docs/Data-HashMap-Strict.html#v:insert"><code>insert</code></a> has <span class="math inline"><em>O</em>(log <em>n</em>)</span> performance. Using it with the <code>Foldable</code> version of <code>foldl</code> or <code>foldr</code> will give you a <code>HashMap</code> that takes <span class="math inline"><em>O</em>(<em>n</em>log <em>n</em>)</span> time to build.</p></li>
410 <li><p>If you’re not building singleton maps, you could try building an intermediate list of <code>(k, v)</code> pairs, and calling <code>fromList</code> on it. The <code>Foldable</code> instance for <code>[]</code> right-associates the applications of <code>(<>)</code>, so concatenating singleton lists should be performant. <code>HashMap</code>’s <a href="https://hackage.haskell.org/package/unordered-containers-0.2.11.0/docs/Data-HashMap-Strict.html#v:fromList"><code>fromList</code></a> has <span class="math inline"><em>O</em>(<em>n</em>log <em>n</em>)</span> complexity, which is also fine.</p></li>
411 </ol>
412 <h2 id="conclusion">Conclusion</h2>
413 <p><code>Map</code> can be surprisingly fast, and <code>HashMap</code> can be surprisingly slow. None of this says, “never use <code>HashMap</code>”, but I was surprised to see an idiom as natural as <code>foldMap</code> become a performance trap. This also isn’t the first time I’ve been able to get better performance out of <code>Map</code> — functions like <a href="https://hackage.haskell.org/package/containers-0.6.0.1/docs/Data-Map-Lazy.html#v:fromDistinctAscList"><code>fromDistinctAscList</code></a> can exploit input structure for better performance — and I often find the <span class="math inline"><em>O</em>(log <em>n</em>)</span> lookups quite acceptable. I’m left wondering whether good ol’ <code>Map</code> should be my default mapping type, unless I have a demonstrated need for <code>HashMap</code>’s features.</p>]]></description>
414 <pubDate>Fri, 16 Oct 2020 13:00:00 UT</pubDate>
415 <guid>http://jackkelly.name/blog/archives/2020/10/16/accidentally-quadratic_hashmaps/index.html</guid>
416 <dc:creator>Jack Kelly</dc:creator>
417 </item>
418 <item>
419 <title>Building and Importing NixOS AMIs on EC2</title>
420 <link>http://jackkelly.name/blog/archives/2020/08/30/building_and_importing_nixos_amis_on_ec2/index.html</link>
421 <description><![CDATA[<p><strong>Update (2020-09-03):</strong> <a href="https://lobste.rs/u/zimbatm">/u/zimbatm</a> on <a href="https://lobste.rs">lobste.rs</a> suggested the <a href="https://github.com/nix-community/nixos-generators">nixos-generators</a> project. Added link and brief discussion.</p>
422 <p><strong>Update (2020-11-29):</strong> Added notes on the impact of instance limits on Packer builds.</p>
423 <p>The <a href="https://nixos.org">NixOS</a> project publishes Amazon Machine Images (AMIs) that are a great base for reproducible servers. This post describes how NixOS and EC2 work together, first showing how to build upon the NixOS project’s public AMIs; and then digging all the way into the scripts maintainers use to build, import and distribute new AMIs on AWS.</p>
424 <h2 id="getting-a-nixos-ami">Getting a NixOS AMI</h2>
425 <p>There are few good ways to get AMI IDs for NixOS project images:</p>
426 <ul>
427 <li><p>The NixOS project publishes its images with <code>owner-id=080433136561</code>; you can use this with an <code>aws ec2 describe-images</code> call. <a href="https://github.com/stedolan/jq"><code>jq</code></a> is a good way to select the interesting parts of the response, and is the easiest way to find an AArch64 (which AWS calls <code>arm64</code>) AMI:</p>
428 <pre><code># Return the most recent AArch64 NixOS image in the region
429 $ aws ec2 describe-images \
430 --region ap-southeast-2 \
431 --filters Name=owner-id,Values=080433136561 \
432 | jq '.Images | map(select(.Architecture == "arm64")) | sort_by(.CreationDate) | reverse | map({ ImageId, Description }) | .[0]'
433 {
434 "ImageId": "ami-05446a2f818cd3263",
435 "Description": "NixOS 20.03.2351.f8248ab6d9e aarch64-linux"
436 }</code></pre></li>
437 <li><p>The <a href="https://nixos.org/download.html">NixOS download page</a> lists AMI IDs for the most recent stable release, which I think are all <code>x86_64</code> images. Scroll down to the “Getting NixOS” section and click the “Amazon EC2” tab to find a list of AMIs, one for each region. The launch buttons take you straight to the launch wizard in the EC2 Management Console.</p></li>
438 <li><p>There is a list of AMIs going back to NixOS 14.04 in <a href="https://github.com/NixOS/nixpkgs/blob/2983c1b7167859e9560dc66a3605a003cdf66286/nixos/modules/virtualisation/ec2-amis.nix"><nixpkgs/nixos/modules/virtualisation/ec2-amis.nix></a>. You can retrieve a specific AMI ID with a <code>nix</code> command like:</p>
439 <pre><code>$ nix eval --raw '(import <nixpkgs/nixos/modules/virtualisation/ec2-amis.nix>)."20.03".ap-southeast-2.hvm-ebs'
440 ami-04c0f3a75f63daddd</code></pre></li>
441 </ul>
442 <!--more-->
443 <h2 id="configuring-nixos-inside-the-ami">Configuring NixOS inside the AMI</h2>
444 <p>The NixOS AMIs can rebuild themselves from NixOS configuration in <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-add-user-data.html">instance user data</a>. To do this, the user data should look something like this:</p>
445 <div class="sourceCode" id="cb3"><pre class="sourceCode nix"><code class="sourceCode bash"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true"></a><span class="co">### https://nixos.org/channels/nixos-unstable nixos</span></span>
446 <span id="cb3-2"><a href="#cb3-2" aria-hidden="true"></a><span class="co">### https://example.com/path/to/another/channel channel-name</span></span>
447 <span id="cb3-3"><a href="#cb3-3" aria-hidden="true"></a></span>
448 <span id="cb3-4"><a href="#cb3-4" aria-hidden="true"></a><span class="kw">{</span> <span class="ex">config</span>, pkgs, ... <span class="kw">}</span>:</span>
449 <span id="cb3-5"><a href="#cb3-5" aria-hidden="true"></a><span class="kw">{</span></span>
450 <span id="cb3-6"><a href="#cb3-6" aria-hidden="true"></a> <span class="co"># Normal NixOS config goes here</span></span>
451 <span id="cb3-7"><a href="#cb3-7" aria-hidden="true"></a><span class="kw">}</span></span></code></pre></div>
452 <p>On each boot the system refreshes its configuration:</p>
453 <ul>
454 <li><p><code>root</code>’s channels are replaced with the ones listed after the three-hash magic comments;</p></li>
455 <li><p>If any channels were found, <code>nix-channel --update</code> runs to fetch the latest version of each channel;</p></li>
456 <li><p><code>/etc/nixos/configuration.nix</code> is replaced with the entire user data; and</p></li>
457 <li><p><code>nixos-rebuild switch</code> runs, rebuilding the OS.</p></li>
458 </ul>
459 <p>If you only want this to happen once, you can set <code>systemd.services.amazon-init.enable = false;</code>. The first boot will always refresh the configuration from user data (because <code>amazon-init</code> is enabled in the AMI), but then turn off the service so it doesn’t happen on subsequent restarts.</p>
460 <h3 id="how-it-works">How it Works</h3>
461 <p>EC2 instance meta data and user data (if it exists) gets downloaded from the <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html">Instance Meta Data Service</a> (IMDS) and applied to a NixOS AMI by the following mechanism:</p>
462 <ul>
463 <li><p>A script on the initramfs will query the IMDS and download the user data (if it exists) and some of the instance meta data to <code>/etc/ec2-metadata</code>, if those files don’t already exist.</p></li>
464 <li><p>On each boot, a systemd service called <a href="https://github.com/NixOS/nixpkgs/blob/43612e8e1abc4ea473b5e10efc10500c865d99e3/nixos/modules/virtualisation/ec2-data.nix#L16-L69"><code>apply-ec2-data</code></a> runs to apply the downloaded data to the system. It:</p>
465 <ul>
466 <li><p>Sets the host name, if not set by the NixOS configuration (<a href="https://search.nixos.org/options?show=networking.hostName&query=networking.hostName&from=0&size=15&sort=relevance&channel=20.03#disabled"><code>config.networking.hostName</code></a>);</p></li>
467 <li><p>Sets root’s <code>authorized_keys</code> file to contain the first SSH key from the IMDS, unless <code>authorized_keys</code> already exists;</p></li>
468 <li><p>Checks for SSH host keypairs in user data, treating the user data as a pipe-separated list of key/value pairs, and setting SSH host keys if they aren’t already present. This seems to be part of instance bootstrapping for <a href="https://github.com/NixOS/nixops">NixOps</a> (which passes known keys so it can use strict host key checking, and immediately replaces them afterwards), and is a <a href="https://github.com/NixOS/nixops/issues/267#issue-57637898">bad idea otherwise</a>.</p></li>
469 </ul></li>
470 <li><p>On each boot, a systemd service called <a href="https://github.com/NixOS/nixpkgs/blob/43612e8e1abc4ea473b5e10efc10500c865d99e3/nixos/modules/virtualisation/ec2-data.nix#L71-L88"><code>print-host-key</code></a> dumps the SSH host key fingerprints to the system console, where they can be grepped for.</p></li>
471 <li><p>On each boot, a systemd service called <a href="https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/amazon-init.nix"><code>amazon-init</code></a> checks whether the user data looks like a nix expression, parses out nix channels and updates them, and calls <code>nixos-rebuild switch</code> on the user data.</p></li>
472 </ul>
473 <h2 id="customising-your-nixos-amis">Customising your NixOS AMIs</h2>
474 <p>Evaluating a full NixOS configuration on each boot can take a lot of CPU and network resources, particularly if it needs to build uncached derivations. This can cause runaway autoscaling if you’re not careful: if autoscaling starts in response to CPU usage and the new instances spend a lot of CPU trying to <code>nixos-rebuild</code>, further autoscaling can happen before the new instances have finished coming online. On T2 instances, it can also burn through your <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-performance-instances-standard-mode-concepts.html#launch-credits">launch credits</a> for no real benefit.</p>
475 <p>A tool like <a href="https://www.packer.io/">Packer</a> can help you build and distribute AMIs by customising the base NixOS AMI. The main steps to provision our image are very simple, NixOS gives us declarative OS configuration:</p>
476 <ul>
477 <li><p>Upload <code>configuration.nix</code> and replace <code>/etc/nixos/configuration.nix</code> with it;</p></li>
478 <li><p>Invoke <code>nixos-rebuild switch --upgrade</code> to build the new OS;</p></li>
479 <li><p>(Optional) Run <code>nix-collect-garbage -d</code> to remove old files from <code>/nix/store</code>; and</p></li>
480 </ul>
481 <p>There is one more <strong>very important</strong> step you must do at the end: make sure the new image responds to its instance meta data and user data when it boots, and not the meta data/user data from when <code>packer</code> booted the NixOS AMI. As the final provisioning action, you must remove all the files created by the EC2 metadata fetcher, any SSH host keys, and <strong>most importantly</strong> root’s <code>.ssh/authorized_keys</code> file. If you do not do this, you will be locked out of your image.</p>
482 <p>Here’s a simple packer configuration that provisions a NixOS AMI with <code>git</code> installed:</p>
483 <details>
484 <summary>
485 <code>nixos-packer-example.json</code>
486 </summary>
487 <div class="sourceCode" id="cb4"><pre class="sourceCode json"><code class="sourceCode json"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true"></a><span class="fu">{</span></span>
488 <span id="cb4-2"><a href="#cb4-2" aria-hidden="true"></a> <span class="dt">"builders"</span><span class="fu">:</span> <span class="ot">[</span></span>
489 <span id="cb4-3"><a href="#cb4-3" aria-hidden="true"></a> <span class="fu">{</span></span>
490 <span id="cb4-4"><a href="#cb4-4" aria-hidden="true"></a> <span class="dt">"type"</span><span class="fu">:</span> <span class="st">"amazon-ebs"</span><span class="fu">,</span></span>
491 <span id="cb4-5"><a href="#cb4-5" aria-hidden="true"></a> <span class="dt">"ami_name"</span><span class="fu">:</span> <span class="st">"nixos-packer-example {{timestamp}}"</span><span class="fu">,</span></span>
492 <span id="cb4-6"><a href="#cb4-6" aria-hidden="true"></a> <span class="dt">"instance_type"</span><span class="fu">:</span> <span class="st">"t2.micro"</span><span class="fu">,</span></span>
493 <span id="cb4-7"><a href="#cb4-7" aria-hidden="true"></a> <span class="dt">"ssh_username"</span><span class="fu">:</span> <span class="st">"root"</span><span class="fu">,</span></span>
494 <span id="cb4-8"><a href="#cb4-8" aria-hidden="true"></a> <span class="dt">"source_ami_filter"</span><span class="fu">:</span> <span class="fu">{</span></span>
495 <span id="cb4-9"><a href="#cb4-9" aria-hidden="true"></a> <span class="dt">"filters"</span><span class="fu">:</span> <span class="fu">{</span></span>
496 <span id="cb4-10"><a href="#cb4-10" aria-hidden="true"></a> <span class="dt">"architecture"</span><span class="fu">:</span> <span class="st">"x86_64"</span></span>
497 <span id="cb4-11"><a href="#cb4-11" aria-hidden="true"></a> <span class="fu">},</span></span>
498 <span id="cb4-12"><a href="#cb4-12" aria-hidden="true"></a> <span class="dt">"most_recent"</span><span class="fu">:</span> <span class="kw">true</span><span class="fu">,</span></span>
499 <span id="cb4-13"><a href="#cb4-13" aria-hidden="true"></a> <span class="dt">"owners"</span><span class="fu">:</span> <span class="ot">[</span></span>
500 <span id="cb4-14"><a href="#cb4-14" aria-hidden="true"></a> <span class="st">"080433136561"</span></span>
501 <span id="cb4-15"><a href="#cb4-15" aria-hidden="true"></a> <span class="ot">]</span></span>
502 <span id="cb4-16"><a href="#cb4-16" aria-hidden="true"></a> <span class="fu">}</span></span>
503 <span id="cb4-17"><a href="#cb4-17" aria-hidden="true"></a> <span class="fu">}</span></span>
504 <span id="cb4-18"><a href="#cb4-18" aria-hidden="true"></a> <span class="ot">]</span><span class="fu">,</span></span>
505 <span id="cb4-19"><a href="#cb4-19" aria-hidden="true"></a> <span class="dt">"provisioners"</span><span class="fu">:</span> <span class="ot">[</span></span>
506 <span id="cb4-20"><a href="#cb4-20" aria-hidden="true"></a> <span class="fu">{</span></span>
507 <span id="cb4-21"><a href="#cb4-21" aria-hidden="true"></a> <span class="dt">"type"</span><span class="fu">:</span> <span class="st">"file"</span><span class="fu">,</span></span>
508 <span id="cb4-22"><a href="#cb4-22" aria-hidden="true"></a> <span class="dt">"source"</span><span class="fu">:</span> <span class="st">"./configuration.nix"</span><span class="fu">,</span></span>
509 <span id="cb4-23"><a href="#cb4-23" aria-hidden="true"></a> <span class="dt">"destination"</span><span class="fu">:</span> <span class="st">"/tmp/"</span></span>
510 <span id="cb4-24"><a href="#cb4-24" aria-hidden="true"></a> <span class="fu">}</span><span class="ot">,</span></span>
511 <span id="cb4-25"><a href="#cb4-25" aria-hidden="true"></a> <span class="fu">{</span></span>
512 <span id="cb4-26"><a href="#cb4-26" aria-hidden="true"></a> <span class="dt">"type"</span><span class="fu">:</span> <span class="st">"shell"</span><span class="fu">,</span></span>
513 <span id="cb4-27"><a href="#cb4-27" aria-hidden="true"></a> <span class="dt">"inline"</span><span class="fu">:</span> <span class="ot">[</span></span>
514 <span id="cb4-28"><a href="#cb4-28" aria-hidden="true"></a> <span class="st">"mv /tmp/configuration.nix /etc/nixos/configuration.nix"</span><span class="ot">,</span></span>
515 <span id="cb4-29"><a href="#cb4-29" aria-hidden="true"></a> <span class="st">"nixos-rebuild switch --upgrade"</span><span class="ot">,</span></span>
516 <span id="cb4-30"><a href="#cb4-30" aria-hidden="true"></a> <span class="st">"nix-collect-garbage -d"</span><span class="ot">,</span></span>
517 <span id="cb4-31"><a href="#cb4-31" aria-hidden="true"></a> <span class="st">"rm -rf /etc/ec2-metadata /etc/ssh/ssh_host_* /root/.ssh"</span></span>
518 <span id="cb4-32"><a href="#cb4-32" aria-hidden="true"></a> <span class="ot">]</span></span>
519 <span id="cb4-33"><a href="#cb4-33" aria-hidden="true"></a> <span class="fu">}</span></span>
520 <span id="cb4-34"><a href="#cb4-34" aria-hidden="true"></a> <span class="ot">]</span></span>
521 <span id="cb4-35"><a href="#cb4-35" aria-hidden="true"></a><span class="fu">}</span></span></code></pre></div>
522 </details>
523 <details>
524 <summary>
525 <code>configuration.nix</code>
526 </summary>
527 <div class="sourceCode" id="cb5"><pre class="sourceCode nix"><code class="sourceCode bash"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true"></a><span class="kw">{</span> <span class="ex">pkgs</span>, ... <span class="kw">}</span>:</span>
528 <span id="cb5-2"><a href="#cb5-2" aria-hidden="true"></a></span>
529 <span id="cb5-3"><a href="#cb5-3" aria-hidden="true"></a><span class="kw">{</span></span>
530 <span id="cb5-4"><a href="#cb5-4" aria-hidden="true"></a> <span class="ex">imports</span> = [ <span class="op"><</span>nixpkgs/nixos/modules/virtualisation/amazon-image.nix<span class="op">></span> ]<span class="kw">;</span></span>
531 <span id="cb5-5"><a href="#cb5-5" aria-hidden="true"></a> <span class="ex">ec2.hvm</span> = true<span class="kw">;</span></span>
532 <span id="cb5-6"><a href="#cb5-6" aria-hidden="true"></a> <span class="ex">environment.systemPackages</span> = with pkgs<span class="kw">;</span><span class="bu"> [</span> git<span class="bu"> ]</span>;</span>
533 <span id="cb5-7"><a href="#cb5-7" aria-hidden="true"></a><span class="kw">}</span></span></code></pre></div>
534 </details>
535 <p>Save the two files to the same directory, and run <code>packer build nixos-packer-example.json</code> from inside it. Remember to clean up any registered AMIs and EBS snapshots when you’re done playing around, otherwise Amazon will charge you to host them.</p>
536 <h3 id="instance-resource-limits">Instance Resource Limits</h3>
537 <p><code>nixos-rebuild</code> can easily use all the instance’s disk space, especially when building against more recent <code>nixos</code> channels than the one used to build the base NixOS AMI. You can ask for additional space by adding a <code>launch_block_device_mappings</code> stanza to the <code>amazon-ebs</code> builder:</p>
538 <div class="sourceCode" id="cb6"><pre class="sourceCode json"><code class="sourceCode json"><span id="cb6-1"><a href="#cb6-1" aria-hidden="true"></a><span class="er">"launch_block_device_mappings":</span> <span class="ot">[</span></span>
539 <span id="cb6-2"><a href="#cb6-2" aria-hidden="true"></a> <span class="fu">{</span></span>
540 <span id="cb6-3"><a href="#cb6-3" aria-hidden="true"></a> <span class="dt">"delete_on_termination"</span><span class="fu">:</span> <span class="kw">true</span><span class="fu">,</span></span>
541 <span id="cb6-4"><a href="#cb6-4" aria-hidden="true"></a> <span class="dt">"device_name"</span><span class="fu">:</span> <span class="st">"/dev/xvda"</span><span class="fu">,</span></span>
542 <span id="cb6-5"><a href="#cb6-5" aria-hidden="true"></a> <span class="dt">"volume_size"</span><span class="fu">:</span> <span class="dv">10</span><span class="fu">,</span></span>
543 <span id="cb6-6"><a href="#cb6-6" aria-hidden="true"></a> <span class="dt">"volume_type"</span><span class="fu">:</span> <span class="st">"gp2"</span></span>
544 <span id="cb6-7"><a href="#cb6-7" aria-hidden="true"></a> <span class="fu">}</span></span>
545 <span id="cb6-8"><a href="#cb6-8" aria-hidden="true"></a><span class="ot">]</span></span></code></pre></div>
546 <p>Some builds (e.g., anything that triggers a rebuild of NixOS documentation) use a lot of memory, and can exhaust the RAM of a <code>t2.micro</code>. If this happens, you’ll see <code>nixos-rebuild</code> (or one of its children) fail with exit code <code>137</code> and no useful error message. To fix this, you’ll have to use a larger instance, or create a swap file as a temporary provisioning step.</p>
547 <h2 id="building-nixos-amis-from-scratch">Building NixOS AMIs from Scratch</h2>
548 <p>Customising NixOS AMIs with a tool like <code>packer</code> lets you prebuild almost-ready-to-go images, and delivering each instance’s <code>configuration.nix</code> via user data creates a very flexible configuration system with reasonable cold-start times. This is probably all you need unless you’re building images for multiple formats (e.g., ISO, EC2 AMI, OpenStack) or hacking on <code>nixpkgs</code>’ image-building support. But if you’re interested in the gory details, read on.</p>
549 <p>You can build a <code>.vhd</code> virtual HD image using the infrastructure in <code>nixpkgs</code>:</p>
550 <pre><code>$ nix-build '<nixpkgs/nixos/release.nix>' \
551 -A amazonImage.x86_64-linux \
552 --arg configuration /path/to/configuration.nix</code></pre>
553 <p>These builds boot a VM to finish the build, so you will want ample CPU, memory and storage. If building as root (which is the only user on a default NixOS AMI), you’ll probably want to set <code>NIX_REMOTE=daemon</code> so that the build takes place in <code>/tmp</code>.</p>
554 <p>The <a href="https://github.com/nix-community/nixos-generators">nixos-generators</a> project provides a nice wrapper around the expressions in nixpkgs, and a single command to build NixOS images in selected formats. If you’re looking to build the same NixOS config into multiple formats, consider looking into it.</p>
555 <p>Either way, once you’ve built the <code>.vhd</code> file, you’ll need to get it into S3 so you can import it with Amazon’s <a href="https://docs.aws.amazon.com/vm-import/latest/userguide/what-is-vmimport.html">VM Import/Export</a> service. It’s probably easiest to do the build on an EC2 instance, to avoid pushing gigabytes of data across the public internet. I used a <code>t3a.medium</code> spot instance when writing this post, and that was fast enough.</p>
556 <p>(It should also be possible to specify <code>-A amazonImage.aarch64-linux</code> to have <code>nix</code> build an AArch64 image, but I couldn’t make it work. Any tips?)</p>
557 <p>The <code>configuration</code> argument is not strictly necessary, but if you omit it, you will get a “blank” image like the ones published by the NixOS project. The only real difference is that it will be built against your version of nixpkgs.</p>
558 <p>Once the build finishes, the symlink <code>result</code> will point to a directory in the nix store that contains the <code>.vhd</code> image, along with a <code>nix-support</code> directory containing image metadata.</p>
559 <h2 id="importing-the-image-into-an-ami">Importing the image into an AMI</h2>
560 <p>Once you have built your image, you need to import it into EC2 as an AMI. The tool to do this is <a href="https://docs.aws.amazon.com/vm-import/latest/userguide/what-is-vmimport.html">VM Import/Export</a>.</p>
561 <p>VM Import/Export needs an S3 bucket to store the images before triggering the import, and a role specifically called <code>vmimport</code> for the service to use. If you’re just mucking around, you might want to try the following CloudFormation template to create an S3 bucket and the necessary role:</p>
562 <details>
563 <summary>
564 <code>template.yaml</code> for VM Import
565 </summary>
566 <div class="sourceCode" id="cb8"><pre class="sourceCode yaml"><code class="sourceCode yaml"><span id="cb8-1"><a href="#cb8-1" aria-hidden="true"></a><span class="fu">AWSTemplateFormatVersion</span><span class="kw">:</span><span class="at"> 2010-09-09</span></span>
567 <span id="cb8-2"><a href="#cb8-2" aria-hidden="true"></a><span class="fu">Description</span><span class="kw">:</span><span class="at"> Bucket and roles for VM import</span></span>
568 <span id="cb8-3"><a href="#cb8-3" aria-hidden="true"></a><span class="fu">Resources</span><span class="kw">:</span></span>
569 <span id="cb8-4"><a href="#cb8-4" aria-hidden="true"></a><span class="at"> </span><span class="fu">VMImportBucket</span><span class="kw">:</span></span>
570 <span id="cb8-5"><a href="#cb8-5" aria-hidden="true"></a><span class="at"> </span><span class="fu">Type</span><span class="kw">:</span><span class="at"> AWS::S3::Bucket</span></span>
571 <span id="cb8-6"><a href="#cb8-6" aria-hidden="true"></a><span class="at"> </span><span class="fu">Properties</span><span class="kw">:</span></span>
572 <span id="cb8-7"><a href="#cb8-7" aria-hidden="true"></a><span class="at"> </span><span class="fu">BucketEncryption</span><span class="kw">:</span></span>
573 <span id="cb8-8"><a href="#cb8-8" aria-hidden="true"></a><span class="at"> </span><span class="fu">ServerSideEncryptionConfiguration</span><span class="kw">:</span></span>
574 <span id="cb8-9"><a href="#cb8-9" aria-hidden="true"></a><span class="at"> </span><span class="kw">-</span><span class="at"> </span><span class="fu">ServerSideEncryptionByDefault</span><span class="kw">:</span></span>
575 <span id="cb8-10"><a href="#cb8-10" aria-hidden="true"></a><span class="at"> </span><span class="fu">SSEAlgorithm</span><span class="kw">:</span><span class="at"> AES256</span></span>
576 <span id="cb8-11"><a href="#cb8-11" aria-hidden="true"></a><span class="at"> </span><span class="fu">PublicAccessBlockConfiguration</span><span class="kw">:</span></span>
577 <span id="cb8-12"><a href="#cb8-12" aria-hidden="true"></a><span class="at"> </span><span class="fu">BlockPublicAcls</span><span class="kw">:</span><span class="at"> </span><span class="ch">true</span></span>
578 <span id="cb8-13"><a href="#cb8-13" aria-hidden="true"></a><span class="at"> </span><span class="fu">BlockPublicPolicy</span><span class="kw">:</span><span class="at"> </span><span class="ch">true</span></span>
579 <span id="cb8-14"><a href="#cb8-14" aria-hidden="true"></a><span class="at"> </span><span class="fu">IgnorePublicAcls</span><span class="kw">:</span><span class="at"> </span><span class="ch">true</span></span>
580 <span id="cb8-15"><a href="#cb8-15" aria-hidden="true"></a><span class="at"> </span><span class="fu">RestrictPublicBuckets</span><span class="kw">:</span><span class="at"> </span><span class="ch">true</span></span>
581 <span id="cb8-16"><a href="#cb8-16" aria-hidden="true"></a></span>
582 <span id="cb8-17"><a href="#cb8-17" aria-hidden="true"></a><span class="at"> </span><span class="fu">VMImportExportServiceRole</span><span class="kw">:</span></span>
583 <span id="cb8-18"><a href="#cb8-18" aria-hidden="true"></a><span class="at"> </span><span class="fu">Type</span><span class="kw">:</span><span class="at"> AWS::IAM::Role</span></span>
584 <span id="cb8-19"><a href="#cb8-19" aria-hidden="true"></a><span class="at"> </span><span class="fu">Properties</span><span class="kw">:</span></span>
585 <span id="cb8-20"><a href="#cb8-20" aria-hidden="true"></a><span class="at"> </span><span class="fu">RoleName</span><span class="kw">:</span><span class="at"> vmimport</span></span>
586 <span id="cb8-21"><a href="#cb8-21" aria-hidden="true"></a><span class="at"> </span><span class="fu">Description</span><span class="kw">:</span><span class="at"> Service role for VM import/export</span></span>
587 <span id="cb8-22"><a href="#cb8-22" aria-hidden="true"></a><span class="at"> </span><span class="fu">AssumeRolePolicyDocument</span><span class="kw">:</span></span>
588 <span id="cb8-23"><a href="#cb8-23" aria-hidden="true"></a><span class="at"> </span><span class="fu">Version</span><span class="kw">:</span><span class="at"> 2012-10-17</span></span>
589 <span id="cb8-24"><a href="#cb8-24" aria-hidden="true"></a><span class="at"> </span><span class="fu">Statement</span><span class="kw">:</span></span>
590 <span id="cb8-25"><a href="#cb8-25" aria-hidden="true"></a><span class="at"> </span><span class="kw">-</span><span class="at"> </span><span class="fu">Effect</span><span class="kw">:</span><span class="at"> Allow</span></span>
591 <span id="cb8-26"><a href="#cb8-26" aria-hidden="true"></a><span class="at"> </span><span class="fu">Principal</span><span class="kw">:</span></span>
592 <span id="cb8-27"><a href="#cb8-27" aria-hidden="true"></a><span class="at"> </span><span class="fu">Service</span><span class="kw">:</span><span class="at"> vmie.amazonaws.com</span></span>
593 <span id="cb8-28"><a href="#cb8-28" aria-hidden="true"></a><span class="at"> </span><span class="fu">Action</span><span class="kw">:</span><span class="at"> sts:AssumeRole</span></span>
594 <span id="cb8-29"><a href="#cb8-29" aria-hidden="true"></a><span class="at"> </span><span class="fu">Condition</span><span class="kw">:</span></span>
595 <span id="cb8-30"><a href="#cb8-30" aria-hidden="true"></a><span class="at"> </span><span class="fu">StringEquals</span><span class="kw">:</span></span>
596 <span id="cb8-31"><a href="#cb8-31" aria-hidden="true"></a><span class="at"> sts</span><span class="fu">:Externalid</span><span class="kw">:</span><span class="at"> vmimport</span></span>
597 <span id="cb8-32"><a href="#cb8-32" aria-hidden="true"></a><span class="at"> </span><span class="fu">Policies</span><span class="kw">:</span></span>
598 <span id="cb8-33"><a href="#cb8-33" aria-hidden="true"></a><span class="at"> </span><span class="kw">-</span><span class="at"> </span><span class="fu">PolicyName</span><span class="kw">:</span><span class="at"> vmimport</span></span>
599 <span id="cb8-34"><a href="#cb8-34" aria-hidden="true"></a><span class="at"> </span><span class="fu">PolicyDocument</span><span class="kw">:</span></span>
600 <span id="cb8-35"><a href="#cb8-35" aria-hidden="true"></a><span class="at"> </span><span class="fu">Version</span><span class="kw">:</span><span class="at"> 2012-10-17</span></span>
601 <span id="cb8-36"><a href="#cb8-36" aria-hidden="true"></a><span class="at"> </span><span class="fu">Statement</span><span class="kw">:</span></span>
602 <span id="cb8-37"><a href="#cb8-37" aria-hidden="true"></a><span class="at"> </span><span class="kw">-</span><span class="at"> </span><span class="fu">Effect</span><span class="kw">:</span><span class="at"> Allow</span></span>
603 <span id="cb8-38"><a href="#cb8-38" aria-hidden="true"></a><span class="at"> </span><span class="fu">Action</span><span class="kw">:</span></span>
604 <span id="cb8-39"><a href="#cb8-39" aria-hidden="true"></a><span class="at"> </span><span class="kw">-</span><span class="at"> s3:GetBucketLocation</span></span>
605 <span id="cb8-40"><a href="#cb8-40" aria-hidden="true"></a><span class="at"> </span><span class="kw">-</span><span class="at"> s3:GetObject</span></span>
606 <span id="cb8-41"><a href="#cb8-41" aria-hidden="true"></a><span class="at"> </span><span class="kw">-</span><span class="at"> s3:ListBucket</span></span>
607 <span id="cb8-42"><a href="#cb8-42" aria-hidden="true"></a><span class="at"> </span><span class="fu">Resource</span><span class="kw">:</span></span>
608 <span id="cb8-43"><a href="#cb8-43" aria-hidden="true"></a><span class="at"> </span><span class="kw">-</span><span class="at"> !GetAtt VMImportBucket.Arn</span></span>
609 <span id="cb8-44"><a href="#cb8-44" aria-hidden="true"></a><span class="at"> </span><span class="kw">-</span><span class="at"> !Sub "${VMImportBucket.Arn}/*"</span></span>
610 <span id="cb8-45"><a href="#cb8-45" aria-hidden="true"></a><span class="at"> </span><span class="kw">-</span><span class="at"> </span><span class="fu">Effect</span><span class="kw">:</span><span class="at"> Allow</span></span>
611 <span id="cb8-46"><a href="#cb8-46" aria-hidden="true"></a><span class="at"> </span><span class="fu">Action</span><span class="kw">:</span></span>
612 <span id="cb8-47"><a href="#cb8-47" aria-hidden="true"></a><span class="at"> </span><span class="kw">-</span><span class="at"> ec2:ModifySnapshotAttribute</span></span>
613 <span id="cb8-48"><a href="#cb8-48" aria-hidden="true"></a><span class="at"> </span><span class="kw">-</span><span class="at"> ec2:CopySnapshot</span></span>
614 <span id="cb8-49"><a href="#cb8-49" aria-hidden="true"></a><span class="at"> </span><span class="kw">-</span><span class="at"> ec2:RegisterImage</span></span>
615 <span id="cb8-50"><a href="#cb8-50" aria-hidden="true"></a><span class="at"> </span><span class="kw">-</span><span class="at"> ec2:Describe*</span></span>
616 <span id="cb8-51"><a href="#cb8-51" aria-hidden="true"></a><span class="at"> </span><span class="fu">Resource</span><span class="kw">:</span><span class="at"> </span><span class="st">"*"</span></span></code></pre></div>
617 </details>
618 <p>There are a few things to note before you use the <code>template.yaml</code> in your own environment:</p>
619 <ul>
620 <li><p>The template only creates a bucket for import. I didn’t need to export VMs so I stripped the <code>vmimport</code> permissions back from the set <a href="https://docs.aws.amazon.com/vm-import/latest/userguide/vmie_prereqs.html#vmimport-role">recommended in the VM Import/Export documentation</a>.</p></li>
621 <li><p>The role must be called <code>vmimport</code> for VM Import/Export to find it, so if you already have that role set up, you may get clashes if you try to deploy this template.</p></li>
622 <li><p>You will need to deploy the stack with <code>CAPABILITY_NAMED_IAM</code>, because of the explicitly-named <code>vmimport</code> role.</p></li>
623 <li><p>If you try to pull the stack down, CloudFormation will not delete the S3 bucket unless it is empty.</p></li>
624 </ul>
625 <p>Once you have the role and bucket set up, you can import your NixOS image as an EBS snapshot and then register it as an AMI. The NixOS maintainers use a script from nixpkgs at <a href="https://github.com/NixOS/nixpkgs/blob/c376f3ec1196c881e72fa0236ab5b04f766b675a/nixos/maintainers/scripts/ec2/create-amis.sh"><code>nixos/maintainers/scripts/ec2/create-amis.sh</code></a> to release the new AMIs, but it does more than we need for our experiments. It:</p>
626 <ul>
627 <li><p>Uploads the <code>.vhd</code> image to S3 if it doesn’t already exist, by calling <code>aws s3 ls</code> and <code>aws s3 cp</code>;</p></li>
628 <li><p>Imports the <code>.vhd</code> from S3 to an EBS snapshot, by calling <code>aws ec2 import-snapshot</code>;</p></li>
629 <li><p>Waits for the snapshot import to finish, by calling <code>aws ec2 describe-import-snapshot-tasks</code> in a loop;</p></li>
630 <li><p>Registers the EBS snapshot as an AMI, by calling <code>aws ec2 register-image</code>;</p></li>
631 <li><p>Waits for the registration to finish, by calling <code>aws ec2 describe-images</code> in a loop;</p></li>
632 <li><p>Makes the new AMI public, by calling <code>aws ec2 modify-image-attribute --launch-permission 'Add={Group=all}'</code>; and</p></li>
633 <li><p>Copies the AMI to all the other regions and makes them public, by calling <code>aws ec2 copy-image</code> and <code>aws ec2 modify-image-attribute</code> in a loop.</p></li>
634 </ul>
635 <p>For tinkering, it’s probably enough to comment out the calls to <code>make_image_public</code>, and also comment out the loop in <code>upload_all</code> that iterates across the regions and copies the AMI.</p>
636 <h2 id="conclusion">Conclusion</h2>
637 <p>I think customising the NixOS project’s images with a tool like <code>packer</code> and then configuring instances with custom <code>configuration.nix</code> user data is a very solid way to get started with NixOS on EC2. If you need to ship the same NixOS config in multiple image formats, or you have extremely unusual configuration needs, nixpkgs provides great tooling for fully-declarative image specifications. Odds are you probably won’t need this level of control, but it’s still interesting to see how the sausage is made.</p>]]></description>
638 <pubDate>Sun, 30 Aug 2020 13:00:00 UT</pubDate>
639 <guid>http://jackkelly.name/blog/archives/2020/08/30/building_and_importing_nixos_amis_on_ec2/index.html</guid>
640 <dc:creator>Jack Kelly</dc:creator>
641 </item>
642 <item>
643 <title>How to Follow an Issue</title>
644 <link>http://jackkelly.name/blog/archives/2020/08/26/how_to_follow_an_issue/index.html</link>
645 <description><![CDATA[<p>Roughly six months ago, the COVID-19 pandemic reached Australia, and began upending things down under (mid-March was when major events like the <a href="https://www.abc.net.au/news/2020-03-13/australian-formula-1-grand-prix-cancelled-over-coronavirus/12052142">F1 Grand Prix were first cancelled</a>). Do you consider yourself well-informed? How well do you remember everything that’s happened? Do you remember the rise and fall of hydroxychloroquine as a potential “miracle cure”? If pressed, could you back up the things you think you remember, even where the media has retreated from things they once said, quietly edited or retracted their stories, and deleted all their embarrassing tweets?</p>
646 <p>Most people reading this are already rightfully suspicious of the Murdoch press, but unfortunately the “respectable” sources are also terrible. Even from them, you must expect bad behaviour at all times: institutional bias, misleading wording, and cherry-picking. Don’t even expect real cherries, but you might find some if you’re lucky. Constant vigilance is necessary if you hope to engage with the media and come away with any truth at all.</p>
647 <p>I will use the tale of hydroxychloroquine as an example, because it’s recent and I have the links, but there’s no reason to expect anything different with any other issue, Hopefully you’ll understand the level of attention required to accurately track even one issue. When each new issue comes along, will need to decide whether it’s worth your time to follow it properly, or to let it pass. Anything less means you will be acting on bad information, which is worse than useless.</p>
648 <!--more-->
649 <h2 id="hydroxychloroquine">Hydroxychloroquine</h2>
650 <p>At the start of August, the ABC published an article about <a href="https://www.abc.net.au/news/science/2020-08-01/hydroxychloroquine-coronavirus-drug-now-right-wing-ideology/12510812">how hydroxychloroquine became so politicised</a>, touching on how hydroxychloroquine (HCQ) went from antimalarial drug, to possible miracle cure, to being tied up with support for President Trump and scepticism of “establishment science”. As usual, the media wilfully ignores its own role in making things worse. As soon as an issue has any connection to Trump at all, everyone in the media immediately loses their minds. If he were to tweet out “it’s going to be a beautiful day tomorrow”, news outlets would immediately start blasting out one of the following two messages:</p>
651 <ol type="1">
652 <li>GLORIOUS LEADER MAKES SUN RISE FOR AMERICAN PATRIOTS</li>
653 <li>Sunlight is deadly. Stay safe, stay inside.</li>
654 </ol>
655 <p>Trump is really not that great, but his critics undermine themselves with their obvious bias against him. To give the ABC some credit, they are one of the few news sources to get Trump’s “game changer” quote <em>almost</em> correct:</p>
656 <blockquote>
657 ‘On March 19, Trump said the drug could be a “game changer” at a White House news conference with his coronavirus task force.’
658 </blockquote>
659 <p>Here’s what the <a href="https://www.youtube.com/watch?v=ScdYrN3BWTM&t=160">US president actually said</a>:</p>
660 <blockquote>
661 “I think it’s going to be very exciting, I think it could be a game changer. And maybe not, and maybe not, but I think it could be based on what I see, it could be a game changer.”
662 </blockquote>
663 <p>Notice that the word “could” is literally spoken by Trump, but is not quoted in the ABC article. Why is that? It’s because the media have been using the “game changer” comment against him for months, quoting only those two words, as though he presented HCQ as the definitive answer to COVID-19. <a href="https://www.theguardian.com/world/2020/apr/06/hydroxychloroquine-trump-coronavirus-drug">Example from The Guardian</a>, published 2020-04-07, also referencing the 2020-03-19 press conference. When you see “game changer” in the more recent ABC article, you’re expected to just nod along.</p>
664 <p>If you want a decent article about the politicisation of HCQ, <a href="https://www.tabletmag.com/sections/science/articles/hydroxychloroquine-morality-tale">Tablet Magazine</a> did a proper piece. They note a lot of things that the ABC miss, correctly pointing out that the study which stopped a lot of the momentum behind HCQ was based on <a href="https://www.theguardian.com/world/2020/jun/03/covid-19-surgisphere-who-world-health-organization-hydroxychloroquine">fabricated data by a company called Surgisphere</a>, and that HCQ advocates were not promoting HCQ alone, but <em>specifically</em> promoting:</p>
665 <ul>
666 <li>The combination of HCQ and azithromycin;</li>
667 <li>Sometimes supplemented with zinc;</li>
668 <li>Delivered early in a person’s case of COVID-19.</li>
669 </ul>
670 <p>Nevertheless, most of the trials were run with HCQ alone as a last-resort treatment for patients with advanced cases of COVID-19.</p>
671 <p>I do not have an opinion on whether or not HCQ/AZ is effective — there’s too much politicisation for me to find truth in that tarpit. My point is this: the anti-Trump parts of the mainstream media (i.e., most of it) rushed to make him look bad and actively muddied the waters around HCQ/AZ to do so. If HCQ/AZ is an effective treatment, that means that these media outlets needlessly prolonged worldwide misery and death just to score points. If HCQ/AZ is not an effective treatment, then the media’s suppression makes it harder to tell whether it’s disfavoured because the research results are coming down against it, or because it’s connected with Trump. This makes it harder for researchers and organisations to correctly assess whether it’s worth their attention, fuels conspiracy theories, and generally prolongs the worldwide misery. Either way, it’s more journogenic harm of the sort that I <a href="/blog/archives/2020/04/26/what_i_remember">documented previously</a>, and utterly reprehensible behaviour.</p>
672 <h2 id="sneaky-edits">Sneaky edits</h2>
673 <p>Look closely at the ABC article I linked at the start of this essay. Did you notice that the article’s slug (that’s the text in the URL that identifies the article: <code>hydroxychloroquine-coronavirus-drug-now-right-wing-ideology</code>) doesn’t match the headline (“Hydroxychloroquine is a poor coronavirus treatment but a perfect parable for our times”)? This is a sign that the article’s title has changed since publication. Articles are quietly updated <em>all the time</em> with minimal notice, and sometimes even deleted outright. If you’re very lucky you get a “Posted Sat 1 Aug 2020 at 4:39pm, updated Sun 2 Aug 2020 at 9:37am” with no indication of what actually changed. We live in an age of cheap storage, and the <a href="https://www.man7.org/linux/man-pages/man1/diff.1.html"><code>diff(1)</code></a> tool has been around for <a href="https://en.wikipedia.org/wiki/Diff#History">nearly 50 years</a>. It is inexcusable for news outlets to omit proper change reporting. Posting detailed change histories is technically easy but inconvenient for outlets trying to control narratives, so they don’t.</p>
674 <h2 id="how-to-properly-follow-an-issue">How to properly follow an issue</h2>
675 <p>So how are you going to actually form an accurate view of an emerging issue? First, you’ll need to get a handle on the broad outlines of the issue and make some effort to compensate for the biases of each news source. Organisational bias doesn’t just affect how articles are written, it affects which articles get run at all. Each side will rush to print articles that fit their worldview; and will delay, minimise or avoid printing articles that run counter to their preferred narrative. Unless you’re keeping at least half an eye on less-pleasant sources like the Murdoch press, you will miss inconvenient details that “respectable” outlets don’t want you to know. (Of course, you then have to work out what the “inconvenient details” <em>actually</em> mean, because it’s almost never what the contrarian news says they mean.)</p>
676 <p>As you read articles, you need to build an archive to link everything together. At the bare minimum, collect URLs, article titles, dates visited and make notes about each article. Set up some kind of tagging system so you can track who pushed which narratives when. Where relevant, you will want to do this for official mouthpiece organisations as well (in the coronavirus context, this was organisations like WHO and CDC). Official mouthpieces can only speak to certify the completely obvious long after it’s become obvious (<a href="/blog/archives/2020/04/26/what_i_remember/">remember WHO dragging its feet</a>?), and it’s useful to be able to highlight this.</p>
677 <p>When you find older articles, check their edit history if you can. The <a href="http://web.archive.org/">Internet Archive’s Wayback Machine</a> may help, but will not always have snapshots of any given site. Embarrassed publishers can hide their pages from the archive using a <a href="http://blog.archive.org/2018/04/24/addressing-recent-claims-of-manipulated-blog-posts-in-the-wayback-machine/"><code>robots.txt</code> entry</a>, and the Internet Archive is currently being sued for <a href="https://www.npr.org/2020/06/03/868861704/publishers-sue-internet-archive-for-mass-copyright-infringement">mass copyright infringement</a> because of its its “National Emergency Library” project, so there’s a big question mark hanging over the whole operation.</p>
678 <p>The next level beyond “URLs and notes” is maintaining your own archive of entire pages. In an ideal world, you’d capture pages when you first visit them, and check back a few times over the next week-to-month to detect sneaky edits. Nothing I say on this topic will beat Gwern’ incredibly thorough <a href="https://www.gwern.net/Archiving-URLs">archiving guide</a>. If you pull articles from your own archive, people may counter with “well you can fake anything these days” and not believe that you saw what you saw. <a href="https://www.gwern.net/Timestamping">Timestamp the files</a> as you archive them, so that you can prove when you took the archive, and that you haven’t changed the files since then.</p>
679 <p>Individual competent people are often a much better source than media organisations or institutions, because they aren’t constrained by an institutional narrative. You can sometimes find fellow archivists being smart-alecs on Twitter or in the rare open comment sections on articles, reminding outlets of inconvenient facts that contradict the preferred narrative. These people can be a good source of things you’ve missed, and if they show their working and make correct predictions they make good sources outright. <a href="https://twitter.com/adamscrabble/status/1270465298110582785">Adam Townsend on Twitter</a> (<a href="/blog/images/how-to-follow-an-issue/adam-archive.png">screenshot</a>) teases us with promises of a great archive of COVID-19-related news:</p>
680 <blockquote>
681 <p>Does everyone here know that i keep a daily log about covid-19? I’m the archivist. Everything that you forgot, i didnt. Every crackpot tweet they did, i took screenshots. Every article that has since been “updated” I downloaded in the original form from wayback.</p>
682 Cant bullshit me
683 </blockquote>
684 <p>Unfortunately, the link to his website no longer works. But you can at least scroll his thread for ideas about how to manage your own archive, for the next time you have an issue want to follow properly.</p>
685 <h2 id="how-well-did-i-do">How well did I do?</h2>
686 <p>How well did I follow the whole COVID-19 thing? I give myself a C-minus. I saw someone else <a href="https://putanumonit.com/2020/02/27/seeing-the-smoke/">see the smoke</a> early enough to take some preparatory action, and followed the news obsessively in the early months of the pandemic. I uncritically swallowed the misinformation about how “masks don’t work” and about HCQ’s risk profile, and it took too long for me to realise the bad information didn’t fit and that I should reject it. About a month back, I got into an argument with a friend who couldn’t believe that people weren’t following instructions from “professionals like the CDC”. Pulling out archived tweets showing the <a href="/blog/images/what-i-remember/cdc-masks.png">US Surgeon-General telling everyone to stop buying masks</a> and stating that ‘“professionals like the CDC” burned themselves by lying-to-manage-supply’ was fun but did not go over well. Most of the best material I saw is still in my browser history somewhere, but I only half-remember it and can never find it when I need it. I’ve seen enough to be confident in my beliefs, but when I get into arguments I can only dredge up enough to make a nuisance of myself, but not enough to convince sceptical people. So: C-minus.</p>
687 <h2 id="conclusion">Conclusion</h2>
688 <p>If this all sounds like a lot of work, that’s because it is. If it feels like you could only follow one or two issues <strong>properly</strong> and have any time left over, that’s because it’s true. News media is both in the business of narrative-building, and in direct competition with entertainment media in the <a href="https://www.nngroup.com/articles/attention-economy/">attention economy</a>, so there’s two very powerful incentives to spin everything away from the truth. If even a simple question like “will this drug save lives?” can become so politicised, then I think there are only really two ways to interact with emerging issues:</p>
689 <ol type="1">
690 <li><p>Dip your toe in the water. Get <em>just enough</em> information so that you’re not blindsided by emergencies, accept that most of what you believe is wrong, and stay the hell away from news media otherwise; or</p></li>
691 <li><p>Wade right in. Set up your archive, take notes, and be able to back up what you say. Because nobody will believe you otherwise.</p></li>
692 </ol>
693 <p>Anything in-between is not really worth it.</p>]]></description>
694 <pubDate>Wed, 26 Aug 2020 13:00:00 UT</pubDate>
695 <guid>http://jackkelly.name/blog/archives/2020/08/26/how_to_follow_an_issue/index.html</guid>
696 <dc:creator>Jack Kelly</dc:creator>
697 </item>
698 <item>
699 <title>Abstracting over Applicative, Alternative, Divisible, and Decidable</title>
700 <link>http://jackkelly.name/blog/archives/2020/08/19/abstracting_over_applicative_alternative_divisible_and_decidable/index.html</link>
701 <description><![CDATA[<p>Justin Le recently put out a two-part series on “Enhancing Functor Structures Step-By-Step” (<a href="https://blog.jle.im/entry/enhancing-functor-structures-step-by-step-1.html">Part 1</a>, <a href="https://blog.jle.im/entry/enhancing-functor-structures-step-by-step-2.html">Part 2</a>). In the footnotes of Part 2, he mentions there’s no combined <a href="https://hackage.haskell.org/package/base-4.14.0.0/docs/Control-Applicative.html#t:Applicative"><code>Applicative</code></a>/<a href="https://hackage.haskell.org/package/contravariant-1.5.2/docs/Data-Functor-Contravariant-Divisible.html#t:Divisible"><code>Divisible</code></a> class:</p>
702 <div class="sourceCode" id="cb1"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true"></a><span class="kw">class</span> <span class="dt">DivisibleApplicative</span> f <span class="kw">where</span></span>
703 <span id="cb1-2"><a href="#cb1-2" aria-hidden="true"></a><span class="ot"> conquerpure ::</span> a <span class="ot">-></span> f a</span>
704 <span id="cb1-3"><a href="#cb1-3" aria-hidden="true"></a><span class="ot"> divideAp ::</span> (a <span class="ot">-></span> (b, c)) <span class="ot">-></span> (b <span class="ot">-></span> c <span class="ot">-></span> a) <span class="ot">-></span> f b <span class="ot">-></span> f c <span class="ot">-></span> f a</span></code></pre></div>
705 <p>Let’s see if we can build one.</p>
706 <!--more-->
707 <p>In Ed Kmett’s 2015 talk, <a href="https://www.youtube.com/watch?v=cB8DapKQz-I">Discrimination is Wrong: Improving Productivity</a>, he shows the similarities between <code>Applicative</code>/<a href="https://hackage.haskell.org/package/base-4.14.0.0/docs/Control-Applicative.html#t:Alternative"><code>Alternative</code></a>/<code>Divisible</code>/<a href="https://hackage.haskell.org/package/contravariant-1.5.2/docs/Data-Functor-Contravariant-Divisible.html#t:Decidable"><code>Decidable</code></a> using Day convolution, where <code>*1</code> and <code>*2</code> are two different tensor products:</p>
708 <div class="sourceCode" id="cb2"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true"></a><span class="co">-- Adapted from the slide at 18:10 or so in the video</span></span>
709 <span id="cb2-2"><a href="#cb2-2" aria-hidden="true"></a><span class="kw">data</span> <span class="dt">Day</span> f g a <span class="kw">where</span></span>
710 <span id="cb2-3"><a href="#cb2-3" aria-hidden="true"></a> <span class="dt">Day</span><span class="ot"> ::</span> ((a <span class="op">*</span><span class="dv">1</span> b) <span class="ot">-></span> c) <span class="op">*</span><span class="dv">2</span> f a <span class="op">*</span><span class="dv">2</span> g b <span class="ot">-></span> <span class="dt">Day</span> f g c</span>
711 <span id="cb2-4"><a href="#cb2-4" aria-hidden="true"></a></span>
712 <span id="cb2-5"><a href="#cb2-5" aria-hidden="true"></a><span class="kw">data</span> <span class="dt">ContraDay</span> f g a <span class="kw">where</span></span>
713 <span id="cb2-6"><a href="#cb2-6" aria-hidden="true"></a> <span class="dt">ContraDay</span><span class="ot"> ::</span> (c <span class="ot">-></span> (a <span class="op">*</span><span class="dv">1</span> b)) <span class="op">*</span><span class="dv">2</span> f a <span class="op">*</span><span class="dv">2</span> g b <span class="ot">-></span> <span class="dt">ContraDay</span> f g c</span></code></pre></div>
714 <p>Then:</p>
715 <ul>
716 <li><code>Applicative</code> comes from covariant Day convolution with <code>*1 = (,)</code> and <code>*2 = (,)</code></li>
717 <li><code>Alternative</code> comes from covariant Day convolution with <code>*1 = Either</code> and <code>*2 = (,)</code></li>
718 <li><code>Divisible</code> comes from contravariant Day convolution with <code>*1 = (,)</code> and <code>*2 = (,)</code></li>
719 <li><code>Decidable</code> comes from contravariant Day convolution with <code>*1 = Either</code> and <code>*2 = (,)</code></li>
720 </ul>
721 <p><code>pure</code>, <code>empty</code>, and <code>conquer</code> can be rearranged so that they each mention the unit of their <code>*1</code> tensor, and then lined up to reveal a pattern:</p>
722 <div class="sourceCode" id="cb3"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true"></a><span class="co">-- At 25:00 or so in the video</span></span>
723 <span id="cb3-2"><a href="#cb3-2" aria-hidden="true"></a><span class="ot">pureish ::</span> <span class="dt">Applicative</span> f <span class="ot">=></span> (() <span class="ot">-></span> a) <span class="ot">-></span> f a</span>
724 <span id="cb3-3"><a href="#cb3-3" aria-hidden="true"></a><span class="ot">emptyish ::</span> <span class="dt">Alternative</span> f <span class="ot">=></span> (<span class="dt">Void</span> <span class="ot">-></span> a) <span class="ot">-></span> f a</span>
725 <span id="cb3-4"><a href="#cb3-4" aria-hidden="true"></a><span class="ot">conquerish ::</span> <span class="dt">Divisible</span> f <span class="ot">=></span> (a <span class="ot">-></span> ()) <span class="ot">-></span> f a</span>
726 <span id="cb3-5"><a href="#cb3-5" aria-hidden="true"></a><span class="ot">lose ::</span> <span class="dt">Decidable</span> f <span class="ot">=></span> (a <span class="ot">-></span> <span class="dt">Void</span>) <span class="ot">-></span> f a</span></code></pre></div>
727 <p>And you can do the same with the “<code>liftA2</code>-equivalent” of each class:</p>
728 <div class="sourceCode" id="cb4"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true"></a><span class="co">-- Not in the video</span></span>
729 <span id="cb4-2"><a href="#cb4-2" aria-hidden="true"></a>liftA2 (,)<span class="ot"> ::</span> <span class="dt">Applicative</span> f <span class="ot">=></span> f a <span class="ot">-></span> f b <span class="ot">-></span> f (a, b)</span>
730 <span id="cb4-3"><a href="#cb4-3" aria-hidden="true"></a>\l r <span class="ot">-></span> <span class="dt">Left</span> <span class="op"><$></span> l <span class="op"><|></span> <span class="dt">Right</span> <span class="op"><$></span><span class="ot"> r ::</span> <span class="dt">Alternative</span> f <span class="ot">=></span> f a <span class="ot">-></span> f b <span class="ot">-></span> f (<span class="dt">Either</span> a b)</span>
731 <span id="cb4-4"><a href="#cb4-4" aria-hidden="true"></a><span class="ot">divided ::</span> <span class="dt">Divisible</span> f <span class="ot">=></span> f a <span class="ot">-></span> f b <span class="ot">-></span> f (a, b)</span>
732 <span id="cb4-5"><a href="#cb4-5" aria-hidden="true"></a><span class="ot">chosen ::</span> <span class="dt">Decidable</span> f <span class="ot">=></span> f a <span class="ot">-></span> f b <span class="ot">-></span> f (<span class="dt">Either</span> a b)</span></code></pre></div>
733 <p><a href="https://bartoszmilewski.com/2018/02/17/free-monoidal-functors/">Bartosz Milewski has a 2018 blogpost</a> where he stands up a class <code>Monoidal</code> and claims that it’s equivalent to <code>Applicative</code>:</p>
734 <div class="sourceCode" id="cb5"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true"></a><span class="kw">class</span> <span class="dt">Monoidal</span> f <span class="kw">where</span></span>
735 <span id="cb5-2"><a href="#cb5-2" aria-hidden="true"></a><span class="ot"> unit ::</span> f ()</span>
736 <span id="cb5-3"><a href="#cb5-3" aria-hidden="true"></a><span class="ot"> (>*<) ::</span> f x <span class="ot">-></span> f y <span class="ot">-></span> f (x, y)</span></code></pre></div>
737 <p>That claim doesn’t sound quite right to me: without a <code>Functor</code> superclass constraint, any <code>Divisible</code> can also be written in terms of this <code>Monoidal</code> class. If we choose other biendofunctors inside <code>f</code>, we can abstract over <code>Alternative</code> and <code>Decidable</code> too:</p>
738 <div class="sourceCode" id="cb6"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb6-1"><a href="#cb6-1" aria-hidden="true"></a><span class="ot">{-# LANGUAGE FlexibleContexts, FunctionalDependencies #-}</span></span>
739 <span id="cb6-2"><a href="#cb6-2" aria-hidden="true"></a><span class="kw">import</span> <span class="kw">qualified</span> <span class="dt">Control.Category.Monoidal</span> <span class="kw">as</span> <span class="dt">C</span> <span class="co">-- From `categories`</span></span>
740 <span id="cb6-3"><a href="#cb6-3" aria-hidden="true"></a></span>
741 <span id="cb6-4"><a href="#cb6-4" aria-hidden="true"></a><span class="kw">class</span> <span class="dt">C.Monoidal</span> (<span class="ot">-></span>) p <span class="ot">=></span> <span class="dt">Monoidal</span> f p <span class="op">|</span> f <span class="ot">-></span> p <span class="kw">where</span></span>
742 <span id="cb6-5"><a href="#cb6-5" aria-hidden="true"></a><span class="ot"> unit ::</span> f (<span class="dt">C.Id</span> (<span class="ot">-></span>) p)</span>
743 <span id="cb6-6"><a href="#cb6-6" aria-hidden="true"></a><span class="ot"> mult ::</span> f a <span class="ot">-></span> f b <span class="ot">-></span> f (p a b)</span>
744 <span id="cb6-7"><a href="#cb6-7" aria-hidden="true"></a><span class="ot"> lunit ::</span> f (<span class="dt">C.Id</span> (<span class="ot">-></span>) p) <span class="ot">-></span> f a <span class="ot">-></span> f a</span>
745 <span id="cb6-8"><a href="#cb6-8" aria-hidden="true"></a><span class="ot"> runit ::</span> f a <span class="ot">-></span> f (<span class="dt">C.Id</span> (<span class="ot">-></span>) p) <span class="ot">-></span> f a</span></code></pre></div>
746 <p>You can define <code>newtype</code> wrappers which show that if you have any of <code>Applicative</code>/<code>Alternative</code>/<code>Divisible</code>/<code>Decidable</code>, you can get a <code>Monoidal</code> instance.</p>
747 <p>An interesting application is bidirectional parsing/printing. The following type cannot have <code>Functor</code>/<code>Contravariant</code>/<code>Applicative</code>/<code>Alternative</code>/<code>Divisible</code>/<code>Decidable</code> instances, but can have <a href="https://hackage.haskell.org/package/invariant-0.5.3/docs/Data-Functor-Invariant.html#t:Invariant"><code>Invariant</code></a> and <code>Monoidal</code> instances:</p>
748 <div class="sourceCode" id="cb7"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb7-1"><a href="#cb7-1" aria-hidden="true"></a><span class="co">-- With Functor/Applicative/Alternative instances</span></span>
749 <span id="cb7-2"><a href="#cb7-2" aria-hidden="true"></a><span class="kw">data</span> <span class="dt">Parser</span> a</span>
750 <span id="cb7-3"><a href="#cb7-3" aria-hidden="true"></a></span>
751 <span id="cb7-4"><a href="#cb7-4" aria-hidden="true"></a><span class="co">-- With Contravariant/Divisible/Decidable instances</span></span>
752 <span id="cb7-5"><a href="#cb7-5" aria-hidden="true"></a><span class="kw">data</span> <span class="dt">PrettyPrinter</span> a</span>
753 <span id="cb7-6"><a href="#cb7-6" aria-hidden="true"></a></span>
754 <span id="cb7-7"><a href="#cb7-7" aria-hidden="true"></a><span class="kw">data</span> <span class="dt">Bidirectional</span> a <span class="ot">=</span> <span class="dt">B</span> (<span class="dt">Parser</span> a) (<span class="dt">PrettyPrinter</span> a)</span></code></pre></div>
755 <p>Operations built using <code>Monoidal</code> end up putting <code>(,)</code>s and <code>Either</code>s all over the place, so <a href="https://hackage.haskell.org/package/generics-eot"><code>generics-eot</code></a> turns out to be a pretty neat way of building <code>Bidirectional</code>s for any data type with a <code>Generic</code> instance.</p>
756 <p>I have a library implementing this machinery about 80% ready to publish to Hackage, but I first want to get the core abstraction right. The questions I want to answer:</p>
757 <ol type="1">
758 <li><p>What is this <code>Monoidal</code> class, exactly? I <em>think</em> the instances are lax monoidal functors <code>f</code> from (<strong>Hask</strong>, <code>p</code>, <code>C.Id (->) p</code>) to either (<strong>Hask</strong>, <code>(,)</code>, <code>()</code>) or (<strong>Hask</strong><span class="math inline"><sup><em>o</em><em>p</em></sup></span>, <code>(,)</code>, <code>()</code>). Is <code>Monoidal</code> a good name for this class? Is there a better one?</p></li>
759 <li><p>I don’t especially like the fundep <code>| f -> p</code>, but if I drop it GHC complains. Is there a better way?</p></li>
760 <li><p>For types with a <code>Monoidal</code> instance but no <code>Applicative</code>/<code>Alternative</code>/<code>Divisible</code>/<code>Decidable</code> instance, we can’t <code>fmap</code> nor <code>contramap</code>. The <code>Bidirectional</code> type above would admit an <code>Invariant</code> instance, and I feel like most practical uses of this class will be for types which have <code>Invariant</code> instances. Maybe I should just make <code>Invariant f</code> a superclass of <code>Monoidal f p</code>, which would let me default the <code>lunit</code> and <code>runit</code> methods?</p></li>
761 <li><p>The <code>Monoidal</code> class feels stuck in-between committing to Haskell-as-it-is-used and trying to be quite general. <a href="https://hackage.haskell.org/package/categories-1.0.7/docs/Control-Category-Monoidal.html"><code>Control.Category.Monoidal</code></a> is a little awkward to use because there aren’t many instances, and it’s generalised across other types of arrows so we have to pass in <code>(->)</code> explicitly. The class <a href="https://hackage.haskell.org/package/assoc-1.0.2/docs/Data-Bifunctor-Assoc.html"><code>Assoc</code></a> from the <a href="https://hackage.haskell.org/package/assoc-1.0.2"><code>assoc</code></a> package is almost what we want, but doesn’t have a subclass with unitors. If it did, we could define <code>Monoidal</code> in terms of that, which is probably more practical.</p></li>
762 <li><p>If I break off a unitless superclass from <code>Monoidal</code> (like the classes you find in <a href="https://hackage.haskell.org/package/semigroupoids-5.3.4"><code>semigroupoids</code></a>), I <em>think</em> the result is a lax semi-monoidal functor, which could then abstract over <a href="https://hackage.haskell.org/package/semigroupoids-5.3.4/docs/Data-Functor-Apply.html#t:Apply"><code>Apply</code></a> and <a href="https://hackage.haskell.org/package/semigroupoids-5.3.4/docs/Data-Functor-Alt.html"><code>Alt</code></a> from <code>semigroupoids</code>, as well as <a href="https://hackage.haskell.org/package/functor-combinators-0.3.5.1/docs/Data-Functor-Contravariant-Divise.html#t:Divise"><code>Divise</code></a> and <a href="https://hackage.haskell.org/package/functor-combinators-0.3.5.1/docs/Data-Functor-Contravariant-Decide.html#t:Decide"><code>Decide</code></a> from <a href="https://hackage.haskell.org/package/functor-combinators-0.3.5.1"><code>functor-combinators</code></a>. That seems nifty; maybe I should do that before an initial release?</p></li>
763 </ol>
764 <p>If you have thoughts about this, please share them <a href="https://www.reddit.com/r/haskell/comments/icm66e/abstracting_over_applicative_alternative/">on the reddit thread</a>.</p>
765 <p><strong>Update (2020-08-23):</strong> After some excellent comments on reddit, I have decided not to push a monoidal functors library to <a href="https://hackage.haskell.org/">Hackage</a>, for a few reasons:</p>
766 <ul>
767 <li><p>It appears that there isn’t a lot of other uses for <code>Invariant</code> functors, and requiring them here is a bit of a kludge.</p></li>
768 <li><p>Bidirectional parsing/printing can be done with <a href="https://hackage.haskell.org/package/profunctors-5.5.2/docs/Data-Profunctor.html#t:Profunctor"><code>Profunctor</code></a>s, where you work with a <code>data PrinterParser a b = PrinterParser (PrettyPrinter a) (Parser b)</code>. The <a href="https://hackage.haskell.org/package/product-profunctors-0.11.0.0"><code>product-profunctors</code></a> package provides <a href="https://hackage.haskell.org/package/product-profunctors-0.11.0.0/docs/Data-Profunctor-Product.html#t:ProductProfunctor"><code>ProductProfunctor</code></a> and <a href="https://hackage.haskell.org/package/product-profunctors-0.11.0.0/docs/Data-Profunctor-Product.html#t:SumProfunctor"><code>SumProfunctor</code></a> classes.</p>
769 <ul>
770 <li><p>There is also the work of Xia, Orchard, and Wang on <a href="https://poisson.chat/esop19/composing-bidir-prog-monadically.pdf">composing bidirectional programs monadically</a>. This work uses a profunctor <code>p a b</code>, where <code>p a</code> is also a <code>Monad</code>.</p></li>
771 <li><p>If you have an existing <code>Divisible f</code> and an <code>Applicative g</code>, you can assemble such a <code>Profunctor</code> via <code>Product (Clown f) (Joker g)</code>.</p></li>
772 </ul></li>
773 <li><p>Asad Saeeduddin already has a more mature and more mathematically-thorough <a href="https://github.com/masaeedu/monoidal/">library of monoidal functors</a>. It is not yet on Hackage, but includes <code>Semigroupal</code> functors, support for varying both the input and output tensors and support for oplax (semi-)monoidal functors as well (going e.g., <code>f (a, b) -> (f a, f b)</code>).</p></li>
774 </ul>
775 <p><strong>Update (2020-11-03):</strong> I just found the 2004 <a href="https://www.microsoft.com/en-us/research/wp-content/uploads/2004/01/picklercombinators.pdf">Pickler Combinators</a> functional pearl. Many of the functions they use can be found in the typeclasses we’ve been discussing (e.g. <code>wrap</code> is <code>invmap</code>), but the paper relies heavily on <code>invmap</code>ping partial functions like <code>fromJust</code>. A parser/printer profunctor would probably give us a better way to handle failures.</p>]]></description>
776 <pubDate>Wed, 19 Aug 2020 13:00:00 UT</pubDate>
777 <guid>http://jackkelly.name/blog/archives/2020/08/19/abstracting_over_applicative_alternative_divisible_and_decidable/index.html</guid>
778 <dc:creator>Jack Kelly</dc:creator>
779 </item>
780 <item>
781 <title>Enforcing the Consequent</title>
782 <link>http://jackkelly.name/blog/archives/2020/06/08/enforcing_the_consequent/index.html</link>
783 <description><![CDATA[<p>To successfully do anything at all, you must have some idea of how cause and effect works in the world around you. I’m getting very worried by a pattern I’ve noticed, where large organisations are <em>actively disconnecting themselves from reality</em> because they cannot honestly achieve their goals. This cannot work, but it’s worse than that: the now-blind organisations <em>cannot notice it not working</em>, and risk becoming stuck in their delusions.</p>
784 <p>This phenomenon feels like a special case of <a href="https://www.lesswrong.com/posts/YtvZxRpZjcFNwJecS/the-importance-of-goodhart-s-law">Goodhart’s Law</a> (“when a measure becomes a target, it ceases to be a good measure”) crossed with the fallacy of <a href="https://en.wikipedia.org/wiki/Affirming_the_consequent">Affirming the Consequent</a> (“All men are mortal. Socrates is mortal. Therefore, all men are Socrates.”). The pattern, which I’m calling <strong>enforcing the consequent</strong>, looks like this:</p>
785 <ul>
786 <li>Suppose that you know that <span class="math inline"><em>A</em></span> causes <span class="math inline"><em>B</em></span>.</li>
787 <li>You want more <span class="math inline"><em>A</em></span>, but you don’t know how to get it.</li>
788 <li>Solution: boost <span class="math inline"><em>B</em></span> to high heaven, and hope like hell that you get more <span class="math inline"><em>A</em></span>.</li>
789 </ul>
790 <p>Boosting <span class="math inline"><em>B</em></span> so aggressively often means turning off feedback mechanisms which would signal that more <span class="math inline"><em>B</em></span> is a bad idea. I’m seeing this happen in really large organisations and systems, and that scares me: big systems can make big mistakes. Examples after the jump.</p>
791 <!--more-->
792 <h2 id="higher-education">Higher Education</h2>
793 <p>A university education used to be a near-certain ticket to a good job, and employers could be confident that degree-qualified applicants were of high quality. Giving degrees to more people doesn’t magically make them better or more productive, especially if you have to lower standards to do it. Universities are forced to steadily grow at the best of times (so that there’s enough new professorships to motivate the postdocs (so that there’s enough postdoc places to motivate the PhD students (so that…))), and a rapid expansion of undergraduate places means even more rushed growth.</p>
794 <p>Sustaining this growth is a funding challenge, and universities have addressed this by taking on a <a href="https://theconversation.com/the-slide-of-academic-standards-in-australia-a-cautionary-tale-40464">students-as-customers mindset</a>, adding more international student places, expanding undergraduate offerings, and <a href="https://www.smh.com.au/national/university-standards-on-the-decline-admit-teachers-20081119-6bm1.html">lowering standards</a> (“More than half of staff surveyed did not believe their university offered a better level of education than it did five years ago” - and this report is from 2008!).</p>
795 <p><strong>This is enforcing the consequent.</strong> Having decided that we want more of the sorts of people that used to graduate university, we crank up the number of graduates, close our eyes, and hope that the people who come out are still as good. Degree printer go brrr.</p>
796 <p>This has predictable <a href="https://www.macrobusiness.com.au/2017/10/university-professor-hits-falling-university-standards/">effects on student outcomes</a>: declining full-time employment rates for recent graduates, increasing underemployment for graduates and reduced graduate starting salaries. As the value of a university degree conveys less information about the person holding it, employers (including <a href="https://www.businessinsider.com.au/top-companies-are-hiring-more-candidates-without-a-4-year-degree-2019-4">Apple, Google, Netflix</a>, and the <a href="https://www.ft.com/content/b8c66e50-beda-11e5-9fdb-87b8d15baec2">big four consultancies</a>) move towards other ways of assessing candidates.</p>
797 <h2 id="who">WHO</h2>
798 <p>A larger, more recent, and possibly more dangerous example, and the reason this post is also tagged <code>coronavirus</code>: Youtube’s CEO has indicated they’ll remove anything that contradicts WHO guidelines around SARS-CoV-2 response (<a href="https://www.bbc.com/news/technology-52388586">BBC</a>, <a href="https://www.businessinsider.com.au/youtube-will-ban-anything-against-who-guidance-2020-4">Business Insider</a>). Other tech companies have stated similar intentions. <a href="/blog/archives/2020/04/26/what_i_remember/">Let me remind you</a> that this is the same WHO that dragged its feet on acknowledging human-to-human transmission, on declaring a pandemic, and on saying that COVID-19 is worse than the flu.</p>
799 <p>If the World Health Organisation fails to secure world health, it is <em>perfectly reasonable</em> that people on the ground lose trust in it. If the WHO has become less credible than people pushing snake oil, then it has <em>failed</em>.</p>
800 <p>There’s no point having an organisation like the WHO unless it is trusted on matters of world health. The right way to do this, the long way, would be for the WHO to apologise, correct the processes by which it offered so much dangerously wrong information, and begin consistently offering correct advice about world health. Instead, the response has been to <strong>enforce the consequent</strong>:</p>
801 <ul>
802 <li>The entities that give the best advice will be the most trusted.</li>
803 <li>The WHO needs to give better advice about world health, but fixing that is hard.</li>
804 <li>Solution: the tech companies will make the WHO the most trusted source, by banning everyone who disagrees with it.</li>
805 </ul>
806 <p>Let’s all hope like hell that the WHO magically does better next time, because it’ll be the most trusted voice left in a very empty room.</p>
807 <h2 id="why-it-scares-me">Why it Scares Me</h2>
808 <p>The common thread connecting these examples is the intentional decision to disconnect from an error-correcting mechanism. But once you’ve done that, how do you know when you’re going the wrong way?</p>
809 <p>Lowering standards to pass more students blinds a university’s internal feedback mechanism, and the university can carry on issuing degrees for a long time before reality catches up. The external feedback loop is much longer; it comes from employers and graduate admissions offices judging the quality of a university’s graduates. Students first have to complete their degrees, attempt to enter the workforce, and then the outside world needs to start noticing that graduates aren’t as consistent as they used to be. Only then can <em>any</em> pressure to improve graduate quality begin finding its way back.</p>
810 <p>Large organisations like the WHO digest information and speak truth slowly, if they ever get around to speaking it at all. This is disastrous for an organisation that must act quickly in response to health emergencies. It was contrarian voices who first got their heads around this crisis, and cutting them out of the conversation will get a lot of people killed when the next pandemic rolls around.</p>
811 <p>In both cases, the policy changes amount to putting on the blinders and crossing your fingers, and the consequences of these changes will not appear until years or decades later. That’s a long time for errors to accumulate, and those errors will do a tremendous amount of damage before any error-correction can kick in.</p>
812 <p><strong>Exercise:</strong> look at the news media through this lens, and then try to sleep soundly. Not only is the whole industry a fractal of people enforcing consequents to further their narratives (recent example: <a href="/blog/archives/2020/04/26/what_i_remember/">downplaying SARS-CoV-2</a> was <strong>enforcing the consequent</strong>), the news media is supposed to be an important part of how the public gathers information and makes sense of the world. This information flow has been compromised; how many systems in our society now run on damaged feedback loops?</p>]]></description>
813 <pubDate>Mon, 08 Jun 2020 13:00:00 UT</pubDate>
814 <guid>http://jackkelly.name/blog/archives/2020/06/08/enforcing_the_consequent/index.html</guid>
815 <dc:creator>Jack Kelly</dc:creator>
816 </item>
817 <item>
818 <title>Changing the Distance Metric</title>
819 <link>http://jackkelly.name/blog/archives/2020/05/19/changing_the_distance_metric/index.html</link>
820 <description><![CDATA[<p>Mathematics has this idea of a <em>metric</em>, which generalises the idea of distances that we normally think about. If we take a set <span class="math inline"><em>X</em></span>, we can talk about a <em>metric</em> or <em>distance function</em>, which is a function <span class="math inline"><em>d</em> : <em>X</em> × <em>X</em> → <em>X</em></span>, that means “the distance between <span class="math inline"><em>x</em></span> and <span class="math inline"><em>y</em></span>”, and it has to have the following properties for any <span class="math inline"><em>x</em>, <em>y</em>, <em>z</em> ∈ <em>X</em></span>:</p>
821 <ol type="1">
822 <li><p>The distance between <span class="math inline"><em>x</em></span> and <span class="math inline"><em>y</em></span> is <span class="math inline">0</span> <strong>if and only if</strong> they are the same thing: <span class="math display"><em>d</em>(<em>x</em>, <em>y</em>) = 0 ⇔ <em>x</em> = <em>y</em></span></p></li>
823 <li><p>Symmetry: The distance between <span class="math inline"><em>x</em></span> and <span class="math inline"><em>y</em></span> equals the distance between <span class="math inline"><em>y</em></span> and <span class="math inline"><em>x</em></span>: <span class="math display"><em>d</em>(<em>x</em>, <em>y</em>) = <em>d</em>(<em>y</em>, <em>x</em>)</span></p></li>
824 <li><p>Triangle Inequality: If you’re going between <span class="math inline"><em>x</em></span> and <span class="math inline"><em>y</em></span>, detouring via <span class="math inline"><em>z</em></span> cannot be shorter: <span class="math display"><em>d</em>(<em>x</em>, <em>y</em>) ≤ <em>d</em>(<em>x</em>, <em>z</em>) + <em>d</em>(<em>z</em>, <em>y</em>)</span></p></li>
825 </ol>
826 <!--more-->
827 <p>Euclidian distance, given by the Pythagorean Theorem you might remember from school, is a <em>metric</em> on the set of points in 2D space — <span class="math inline">ℝ<sup>2</sup></span>. There are many others. One interesting example is the <em>discrete metric</em>, which says that the distance between two points is zero if they are the same, and one if they are not.</p>
828 <p>It is obvious that the coronavirus lockdowns have messed with our perception of distance — someone in a different household may as well be on a different world. But it occurred to me the other day that the way we see distance has shifted from something like Euclidian distance to something a lot more like the discrete metric. While people nearby feel far away, people I used to think of as distant aren’t any more remote right now — travelling to visit them is just as impossible, instead of being a more difficult undertaking that I could sit down and plan. One upside to this whole mess is that interstate and overseas friends now feel no farther than local ones, and keeping up with them is paradoxically easier that it was under normal conditions — they’re on the other side of a screen, just like everyone else.</p>
829 <p>The analogy isn’t perfect — timezone differences still make things hard — but it’s interesting to notice and to think about.</p>]]></description>
830 <pubDate>Tue, 19 May 2020 13:00:00 UT</pubDate>
831 <guid>http://jackkelly.name/blog/archives/2020/05/19/changing_the_distance_metric/index.html</guid>
832 <dc:creator>Jack Kelly</dc:creator>
833 </item>
834
835 </channel>
836 </rss>