[HN Gopher] Wikifunctions
___________________________________________________________________
Wikifunctions
Author : edward
Score : 136 points
Date : 2023-12-06 19:04 UTC (3 hours ago)
(HTM) web link (wikimediafoundation.org)
(TXT) w3m dump (wikimediafoundation.org)
| hiAndrewQuinn wrote:
| I think I need an explainer for this. Is this like the number
| sequences library but for pure functions?
| duskwuff wrote:
| The description is a little vague and hand-wavey. Here's a
| concrete example:
|
| A lot of Wikipedia sites have scripts embedded in the wikitext
| which automatically generate or transform information on a
| page, e.g. automatically performing unit conversions to
| generate text like "I would walk 500 miles (804.67 km)",
| performing date math to automatically generate and update a
| person's age based on their birthdate, or querying structured
| data from Wikidata [1] to display in an infobox. One example of
| these scripts is the {{convert}} [2] template on the English
| Wikipedia.
|
| Initially, these scripts were written in MediaWiki template
| logic [3], and were maintained individually on each wiki. This
| quickly proved unmaintainable, and some of those scripts were
| rewritten in Lua using the Scribunto extension [4], but these
| were still per-wiki, and there were frequently issues where
| different wikis would copy scripts from each other and
| introduce their own incompatible features.
|
| The WikiFunctions project is an attempt to centralize
| development of these scripts, much like how Wikimedia Commons
| [5] centralizes hosting of freely licensed images and other
| media.
|
| [1]: https://wikidata.org/
|
| [2]: https://en.wikipedia.org/wiki/Template:Convert
|
| [3]:
| https://www.mediawiki.org/wiki/Help:Extension:ParserFunction...
|
| [4]: https://www.mediawiki.org/wiki/Extension:Scribunto
|
| [5]: https://commons.wikimedia.org/
| hiAndrewQuinn wrote:
| That makes _way_ more sense to me, and my handful of edits
| and additions to Wiktionary 's Finnish lexicon has definitely
| had me wondering how exactly those {{shortcode looking
| things}} actually worked. I'm seriously considering getting
| involved in this project now thanks to your explanation, that
| seems like it could be a very good thing indeed.
| duskwuff wrote:
| For a fairly horrifying example of how parser function
| templates looked, here's an old version of the Convert
| template from the English Wikipedia:
|
| https://en.wikipedia.org/w/index.php?title=Template:Convert
| &...
|
| This isn't even all of the code for the template, either;
| there were a bunch of subtemplates for individual
| operations. And now you probably have a decent idea of why
| Wikimedia wants to move away from that. :)
| nmstoker wrote:
| I've not kept up with the template complexity situation:
| do you have a sense for whether they're succeeding in
| simplifying things yet? (besides the benefit that
| Wikifunctions should bring)
| starkparker wrote:
| They aren't simplifying templates so much as moving them
| to a language better suited for the complexity. There are
| some similarly complex Lua modules that have rats' nests
| of dependencies and piles of undocumented code.
|
| Templates like {{Convert}} are also by and large
| unrelated to Wikifunctions, which are initially and
| primarily more concerned about lower-language natural
| language generation problems, like cross-language
| conjugation, that are relevant to Abstract Wikipedia.
|
| Or, in the Google.org evaluation I mention in a sibling:
| https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Google
| .or...
|
| > Instead of attempting to invent so much from scratch,
| Wikifunctions could have been based on Scribunto. We
| think this is a huge missed opportunity. One of the
| longest-standing requests from the Wikimedia community
| has been to provide a means of reusing code across
| different projects by centralizing common code on a
| global wiki. This is a tractable, well-scoped problem (at
| least in comparison with Wikifunctions as currently
| conceived).
|
| > "V1" of Wikifunctions could have been a new, central
| wiki for common Lua code. Structuring the development
| plan for Wikifunctions in this way would have meant a
| much shorter, surer path to delivering value to
| Wikimedians. It would have satisfied a long-standing
| community request, and (we suspect) earned the team quite
| a lot of goodwill. The team could then innovate on top of
| this initial version by building new graphical interfaces
| for producing and interacting with code, and by extending
| the capabilities of querying Wikidata.
|
| > We are not sure why this was overlooked, particularly
| in light of the project requirements described above. In
| conversations with the Abstract Wikipedia team, team
| members suggested to us that this option was considered
| and rejected because Scribunto is an expert system and
| that Lua is based on English, and that these properties
| made it incompatible with the Wikifunction goal of broad
| accessibility. We think this is a bad argument, for the
| following reasons:
|
| > - Lua has been successfully adopted by many projects
| whose primary language (as in, the language spoken by the
| users and developers) is not English. It is used
| successfully in hundreds of different language editions
| of Wikipedia.
|
| > - Wikifunctions can provide localized, graphical user
| interfaces that generate Lua code behind the scenes (see
| Blockly, for example). Lua can also be used as a
| "transpilation" target for text-based programming
| languages (there is precedent for this on Wikimedia
| wikis). As an intermediate representation, Lua would be
| far more efficient and robust than an ad-hoc function
| composition language.
|
| Which are all great points explicitly disregarded by
| Abstract Wikipedia for Wikifunctions.
| noirscape wrote:
| > We are not sure why this was overlooked, particularly
| in light of the project requirements described above.
|
| I can guess the actual reason for this and it amounts to
| "managerial pride causing NIH". It's no big secret that
| the WMF is so overly ambitious with the projects it works
| on that it would throw away any and all prior art they've
| already made, just to make something that seems new (not
| to mention that many WMF projects kinda just quietly
| strand and are left to rot because of that).
|
| I really wouldn't be suprised if some manager at the WMF
| decided that "no, using Scribunto for this is off the
| table", just because Scribunto is an existing project and
| they wanted the project to have additional complexity (so
| they wouldn't have to consider the viability of the
| project since a "demo" would then be years out).
| jessamyn wrote:
| Thank you so much for this. I'm pretty Wiki-literate and was
| having a hard time understanding this from their initial
| pages.
| starkparker wrote:
| It'd be an even better example if there was an equivalent
| conversion function on Wikifunctions to link to. There
| doesn't appear to be one yet.
|
| The project's response to the funding Google.org Fellows'
| evaluation is a deeper explanation that explicitly compares
| the goals of Wikifunctions to Scribunto (those Lua functions)
| -- and is especially notable in the context of replacing
| templates and Lua modules since Google.org made a bunch of
| recommendations to further decouple Abstract Wikipedia's
| goals from Wikifunctions that Wikimedia thanked them for and
| subsequently ignored: https://meta.wikimedia.org/wiki/Abstrac
| t_Wikipedia/Google.or...
|
| The catalogue of functions useful to Wikimedia is much more
| prosaic than the {{Convert}} template and in line with the
| Google.org examination -- it's more about Abstract
| Wikipedia's goals of natural language generation (NLG):
| https://www.wikifunctions.org/wiki/Wikifunctions:Catalogue
|
| The list of all functions is also similarly lower level;
| language transformations, form validation, sub/string
| operations: https://www.wikifunctions.org/wiki/Special:ListOb
| jectsByType...
| dragonwriter wrote:
| > The description is a little vague and hand-wavey. Here's a
| concrete example:
|
| Much better description of what wikifunctions (the Wikimedia
| blog post seems more about where it fits in a broader
| strategy) is on wikifunctions about page:
|
| https://www.wikifunctions.org/wiki/Wikifunctions:About
| haltist wrote:
| It is a formalization of knowledge with formal expressions. One
| could say it is a formalization of knowledge with
| executable/computational semantics. WikiMedia is actually an AI
| company and that's why they are always asking for donations.
| dark-star wrote:
| Adding to what others have said, it contains functions that can
| be used to do grammatical transformations of nouns/verbs/... as
| in some languages those rules can be quite complicated. So
| having a central function that converts singular->plural for
| example is a huge win for maintainability
| hlandau wrote:
| Tried visiting it. Can't see anything without JavaScript.
|
| This is not OK.
| duskwuff wrote:
| Then there's something wrong with your browser. This is a
| fairly standard Wordpress site and the content is all present
| as perfectly standard HTML.
| hlandau wrote:
| No, I'm referring to each Wikifunction page on the
| Wikifunctions site itself.
| themikesanto wrote:
| How do you propose the functions should run if you don't
| have JavaScript enabled?
| hlandau wrote:
| I don't want to run the functions, I want to view them.
|
| Also the functions execute on the server anyway (plus
| JavaScript isn't even the only supported language), so
| execution should also be possible without a browser
| having JavaScript enabled.
| simonw wrote:
| They are server side functions, not client side.
| gngoo wrote:
| C'mon it's 2023.
| lcnPylGDnU4H9OF wrote:
| 20 years after the term "progressive enhancement" was first
| used in a tech talk[0]. People really ought to have received
| the message by now: javascript as a client-side feature is a
| way to enhance a website rather than to build its basic
| functionality.
|
| 0: https://web.archive.org/web/20141108064903/http://hesketh.
| co...
| hk__2 wrote:
| > javascript as a client-side feature is a way to enhance a
| website rather than to build its basic functionality.
|
| And who decided that, exactly?
| anigbrowl wrote:
| The free market. Much as I miss static HTML, the people
| like widgets.
| speedgoose wrote:
| I also can't see anything when I disable my screen output. Very
| disappointed.
| anigbrowl wrote:
| This is in fact OK. If you hate js that much then just use CURL
| and read it offline. I personally do not like js much either
| but complaining about every instance of it is pointless.
| hlandau wrote:
| How am I supposed to use CURL to read it offline if it
| requires JS?
|
| Won't work in links2 or lynx either.
| anigbrowl wrote:
| Open the document in an editor and parse the script tags,
| find a way to reverse engineer it. It seems like you're
| just complaining for the sake of it.
| solarpunk wrote:
| Gonna dig deeper, but I get the sense Wikipedia is preparing a
| native dataformat for LLM ingestion.
| duskwuff wrote:
| You're probably thinking of something more along the lines of
| Wikidata [1], which is over 10 years old.
|
| [1]: https://www.wikidata.org/wiki/Wikidata:Main_Page
| starkparker wrote:
| It's very much designed for and around Wikidata.
|
| > Wikifunctions will allow easy access to large knowledge
| bases such as Wikidata, but also to binary input and output
| files
|
| https://diff.wikimedia.org/2023/08/07/wikifunctions-is-
| start...
|
| > Using the simple facts housed in Wikidata, you will be able
| to write functions that make calculations, provide a person's
| age, estimate population densities, and more, and integrate
| the results into Wikipedia.
|
| https://www.wikifunctions.org/wiki/Wikifunctions:FAQ
|
| > In the future:
|
| > - It will be possible to call Wikifunctions functions from
| other Wikimedia projects, and integrate their results in the
| output of the page.
|
| > - It will be possible to use data from Wikidata in
| functions.
| zozbot234 wrote:
| Wikidata cannot express the "semantic" content of average
| encyclopedic text, which is an express goal of
| Wikifunctions. So they will have to expand the data model
| quite a bit compared to what Wikidata has today. (This can
| be done quite cleanly though, since the whole point of RDF
| is to be able to express general graphs, and this aspect
| has been made even stronger with RDF* which aligns fairly
| well with the "frame" semantics Wikifunctions plans to use
| for the above purpose.)
| starkparker wrote:
| Wikifunctions are arguably the prerequisite to making
| that effort, since even if you extended Wikidata with
| more semantic connections, you couldn't do anything with
| them.
|
| This is a large part of why Abstract Wikipedia's goal is
| NLG, and almost all of the initial Wikifunctions
| facilitate NLG.
| riku_iki wrote:
| > NLG
|
| what is NLG?..
| haskellandchill wrote:
| Natural Language Generation
| starkparker wrote:
| Previously: https://news.ycombinator.com/item?id=36927695
|
| cf. https://www.wikifunctions.org/wiki/Wikifunctions:About
|
| > Wikifunctions is a Wikimedia project for everyone to
| collaboratively create and maintain a library of code functions
| to support the Wikimedia projects and beyond, for everyone to
| call and re-use in the world's natural and programming languages.
|
| It's a support project of the Abstract Wikipedia initiative to
| model facts with data not specific to any spoken or written
| language, building on and supporting things like Wikidata.
| simonw wrote:
| Here's an example function which shows that they can have
| multiple implementations (click the Details tab) - in this case
| there's an implementation in Python and one in JavaScript:
| https://www.wikifunctions.org/view/en/Z10070
|
| And a much more interesting example which helps illustrate why
| Wikipedia built this: "genitive case of Bangla word in Python":
| https://www.wikifunctions.org/view/en/Z10594
| pantalaimon wrote:
| All the functions are numbered and call each other.
| substring_exists(haystack, needle) is implemented as
| function Z10070( Z10070K1, Z10070K2 ) { return
| Z10070K1.includes(Z10070K2); }
|
| This is atrocious! Might as well come straight out of a
| decompiler.
| starkparker wrote:
| And an explicit goal of the project: https://meta.wikimedia.o
| rg/wiki/Abstract_Wikipedia/Google.or...
|
| > In particular, according to Keet, Grammatical Framework
| notably lacks the flexibility to handle certain aspects of
| Niger-Congo B languages' morphology. The traditional answer
| to this kind of critique would be to say, "Let's just
| organize an effort to work on Grammatical Framework." The
| design philosophy behind Abstract Wikipedia is to make as few
| assumptions about a contributor's facility with English and
| programming experience as possible. Organized efforts around
| Grammatical Framework (or otherwise) are not a bad idea at
| all. They may well solve some problems for some languages.
| However, the contributor pools for under-resourced languages
| are already small. Demanding expertise with specific natural
| languages like English and also specific programming
| paradigms contracts those pools still further.
|
| and
|
| > A solution designed by a small group of Westerners is
| likely to produce a system that replicates the trends of an
| imperialist English-focused Western-thinking industry.
| Existing tools tell a one-voice story; they are built by the
| same people (in a socio-cultural sense) and produce the same
| outcome, which is crafted to the needs of their creators (or
| by the limits of their understanding). Tools are then used to
| build tools, which will be again used to build more tools;
| step by step, every level of architectural decision-making
| limits more and more the space that can be benefitted by
| these efforts.
|
| and
|
| > The evaluation acknowledges little to no value to the
| effort that Wikifunctions is investing in being multilingual
| in terms of natural languages, even for implementations. It
| is unclear as to how a solution that is built on top of (the
| existing Lua-based Scribunto scripting system) would succeed
| in being multilingual, and not just be an English project
| which the other languages may be invited to use, if they
| understand it, but would be mostly blocked from contributing
| to.
| zozbot234 wrote:
| I assume it's intentional, so that source code can be stored
| in a form that's independent of natural language. They can
| just call to a LSP backend for the implementation language to
| rename identifiers in the source code before showing them to
| the user, much like Wikidata does today with its semantic
| content.
| solardev wrote:
| Whoa, I never knew (and accidentally discovered through this)
| that empty strings are valid substrings of other strings.
| lemurien wrote:
| It's a bit like the empty set that is a subset of any other
| set.
| ben0x539 wrote:
| Well, not just _other_ strings.
| anigbrowl wrote:
| I like the idea but the catalog has the look of something that
| will quickly grow unmanageable. I hope it doesn't end up like
| other Wiki projects that die from lack of momentum/direction like
| wikinews. To be clear, the latter still runs but hardly anyone
| uses it.
|
| https://www.wikifunctions.org/wiki/Wikifunctions:Catalogue
|
| Does this have the potential to become some standard reference
| for code proving/transpilation purposes, or is it likely to end
| up like past attempts to unify all programming languages?
| shpx wrote:
| Awful interface. I have to click three times and scroll to see
| code in a tiny font, the name of the function is repeated on the
| page like 7 times.
| slugiscool99 wrote:
| Needs some UI work but this is a really great idea. Hope it gains
| traction
| noirscape wrote:
| Yet another Wikimedia project with way too lofty ambitions...
| this is what Wikimedia donation money gets spent on rather than
| the operation of Wikipedia. For reference, the greater project
| this is a part of has been in development since 2013[0] and seems
| to be an attempt to basically remove the editor from Wikipedia.
|
| For the uninitiated - this is part of an attempt to basically
| turn WikiData[1] into article text. From what I understand, the
| idea behind this is to create a bunch of text rendering functions
| to make it easier to provide automatic translations of rendered
| text.
|
| Don't get me wrong, the idea here is neat, but from what I
| understand the Wikipedia community has been having far greater
| needs not being met rather than some lofty goal to drive even
| more editors out of the site and replacing them with data entry
| drones. It's a bit too often that you hear about some ailing
| extension and the WMF response being "no budget for it", while a
| lot of it gets spend on projects like this.
|
| [0]: https://en.wikipedia.org/wiki/Abstract_Wikipedia
|
| [1]: https://www.wikidata.org/wiki/Wikidata:Main_Page -
| basically, Wikipedia data meant to be in a machine-readable
| format.
| simonw wrote:
| What are some of those greater needs that are not being met?
| noirscape wrote:
| It's been a hot minute since I last looked at the most recent
| list, but the short of it is that the WMF likes to release
| extensions onto Wikipedias that ship with a ton of technical
| bugs, aren't tested beforehand by the Wikipedia community for
| usability (which is ironic given the WMFs obsession with
| making tools as accessible as possible) and when the
| complaints get leveled, the response from the WMF basically
| ends up being "yeah so there's no budget for that". Rinse and
| repeat for a ton of tiny technical issues that affect all
| their extensions on the regular. It's a bit odd to be putting
| so many engineers and money onto lofty projects when the
| maintenance of Wikimedia extensions is in such an extremely
| precarious state already.
|
| The most evergreen example of this is and probably always
| will be the rollout of VisualEditor, their WYSIWG editor. It
| was completely unwanted, was proven to be not ready for
| general use (due to all the problems that plague "combining a
| WYSIWG editor with a code view"), yet forcibly deployed on
| all Wikipedia languages without much room for warning or
| feedback. After that, the WMF promised communities the option
| to opt-out if they didn't want it... before violating it on
| the German Wikipedia because they voted to not roll it out.
| To my knowledge, VisualEditor hasn't improved over the years
| (beyond "rewrite Parsoid into PHP"), but editors have just
| gritted their teeth. That's the extreme example, but the WMFs
| relationship with building features for Wikipedia is...
| pretty much in a vacuum from what users want. A lot of their
| projects seem to be conceived by WMF managers first, without
| consultation to their existing community of volunteers.
|
| Which is I guess fine for moonshot projects like
| Wikifunctions (since they don't involve the volunteers before
| release) but results in a lot of friction when a managers pet
| project is forcibly rolled out on all language Wikipedias.
| Very few of their big extension deployments have gone
| smoothly with the community. I distinctly remember the
| Minerva skin (mobile Wikipedia) needing tons of changes and
| improvements before it was usable as well, after it was
| already rolled out and caused a ton of negative fedback.
| zozbot234 wrote:
| > Yet another Wikimedia project with way too lofty ambitions...
|
| Perhaps, but so what. Wikidata was "a project with way too
| lofty ambitions" back when it started (a.k.a. make the Semantic
| Web something that actually works in a useful way) and it's
| been wildly successful. It now gets more edits per minute than
| the most popular Wikipedia and gets used all over the place by
| Big Tech firms.
| justinpombrio wrote:
| Thinking about this in terms of end end user.
|
| How would you do these things today? They give 3 examples. 2/3
| are answered directly by google (not by a search result, just
| directly on the search page). 3/3 are answered by Wolfram Alpha.
| None of the three are answered directly by DuckDuckGo, though it
| does answer _some_ queries directly, like measurement
| conversions. In cases where WolframAlpha and Google and DDG don
| 't answer, there's probably some random website that tells you,
| in exchange for piles of ads and trackers.
|
| So if Wikipedia gives a good interface to these things, it'll be
| nice to have them all in one trusted place.
|
| Google: distance from Madrid to NY
| https://www.google.com/search?hl=en&q=distance%20from%20madr...
|
| Google: time difference between Madrid and NY
| https://www.google.com/search?hl=en&q=time%20difference%20be...
|
| Wolfram Alpha: distance from Madrid to NY
| https://www.wolframalpha.com/input?i=distance+from+madrid+to...
|
| Wolfram Alpha: time difference between Madrid and NY
| https://www.wolframalpha.com/input?i=time+difference+between...
|
| Wolfram Alpha: volume of a laptop
| https://www.wolframalpha.com/input?i=volume+of+a+laptop
| primitivesuave wrote:
| Terrible UI. I found two example functions (call them `f` and
| `g`), found the dropdown to change the input type of `f` from
| "string" to "function call", and even manage to get `g` suggested
| when I enter it's unique identifier - but cannot set up any
| configuration to compute `f(g(x))` (even after exhaustively
| trying all options).
|
| It seems like it should be a core functionality of such a project
| to easily set up complex computations as function compositions.
| faresahmed wrote:
| Amazing to see RustPython being used in the wild. I wonder why
| they are using it. Is it easier to sandbox? smaller memory
| footprint than CPytohn and PyPy? just because it's cool?
| 13415 wrote:
| I'm kind of disappointed. Based on the name, I thought it's a
| searchable collection of functions, a mixture of OEIS,
| searchonmath, and a classic formula encyclopedia. But it seems to
| be only about little algorithmic snippets without explanation or
| definition?
___________________________________________________________________
(page generated 2023-12-06 23:00 UTC)