Post B6jKZAx7XhmHrxbTpA by legoktm@wikis.world
(DIR) More posts by legoktm@wikis.world
(DIR) Post #B6jBmaL5ZHNUmW8LvU by linear@nya.social
0 likes, 0 repeats
has anyone made a version of the Wikipedia app for android, which uses a downloaded copy of Wikipedia instead of their servers?asking for a myself.
(DIR) Post #B6jBzUUec6NKPqlHu4 by linear@nya.social
0 likes, 0 repeats
i use the wikipedia app in order to access the data stored within the world's greatest collaborative encyclopedia.i see no reason at this point that the Wikimedia Foundation needs to be involved in every search and article lookup i perform on my tablet, and fully offline usage is also a significant benefit in my eyes.
(DIR) Post #B6jCYiBhgquZ1Wgza4 by ann3nova@corteximplant.com
0 likes, 0 repeats
@linear A quick Google search suggested Kiwix.I haven't tested it myself, though.https://github.com/kiwix
(DIR) Post #B6jCZdtd8Cuu0GgNSi by linear@nya.social
0 likes, 0 repeats
@ann3nova@corteximplant.com i have many nitpicks about kiwix and think it impedes itself in its own goals
(DIR) Post #B6jCtwblGrhPpR0Nu4 by linear@nya.social
0 likes, 0 repeats
@ann3nova@corteximplant.com it is also not what i asked for and does not function like what i asked for, it is more of an offline webpage archiving tool. if i wanted that i would instead use one of the other tools i already have set up for that purpose, and that unlike kiwix, work on any webpage and have been accepted as sufficient for evidence in legal cases
(DIR) Post #B6jFB07Lfy5lXo2akS by linear@nya.social
1 likes, 0 repeats
if you are about to suggest something like Kiwix that uses what is effective a scraped web archive of Wikipedia, then you are answering a different question than what i asked.
(DIR) Post #B6jFCVkZYHj5wO6jIW by tamzin@wikis.world
0 likes, 0 repeats
@linear This sounds like something @legoktm might know the answer to.
(DIR) Post #B6jFCW0AcICeilPBbs by legoktm@wikis.world
0 likes, 0 repeats
@tamzin @linear I do! You're looking for @kiwix https://get.kiwix.org/en/solutions/applications/kiwix-reader/
(DIR) Post #B6jFCWCDtTqPK92oOe by linear@nya.social
0 likes, 0 repeats
@legoktm@wikis.world @tamzin@wikis.world @kiwix@mastodon.socialRE: https://nya.social/notes/819e6a71304b583b4df5d849
(DIR) Post #B6jFMq8ScB6y3oCqO0 by linear@nya.social
0 likes, 0 repeats
i want to download the wikipedia database, and then i want my Android tablet to use that instead of contacting wikipedia.if that involves spinning up an entire copy of mediawiki to serve the REST api then that is what i want, not a web archive
(DIR) Post #B6jFSXgM8WyGZ9mOjA by linear@nya.social
1 likes, 0 repeats
a web archive of wikipedia is not wikipedia in the same manner that this picture of a pipe is not a pipe
(DIR) Post #B6jGe6MtcATHHakl7Y by buherator@infosec.place
0 likes, 0 repeats
@linear https://get.kiwix.org/en/solutions/applications/kiwix-reader/ ?
(DIR) Post #B6jGe6YEvzXrqm3ono by linear@nya.social
0 likes, 0 repeats
@buherator@infosec.placeRE: https://nya.social/notes/819e6a71304b583b4df5d849
(DIR) Post #B6jHv0YkZ4h68zvImG by buherator@infosec.place
0 likes, 0 repeats
@linear I'm afraid I don't see the difference between scraping and downloading a copy of something
(DIR) Post #B6jHv0ifyAdMdmZEFU by linear@nya.social
0 likes, 0 repeats
@buherator@infosec.place well in one case i am running of the actual source of truth data, and in the other i am using a pre rendered copy of it.it is the difference between watching a movie and watching a phone camera recording of the movie downloaded from someone on the internet.the fact that the wikimedia provides their own "phone screen recording" version for kiwix does not change what it is.the wikipedia app is a full fledged editor, not just a viewer. i want all of its functionality, and for all of it to work fully offline.i cannot edit the underlying "kiwix database" and get new pages if i wanted to, because there is no such thing, it is a static rendered page.if what i want requires no less than running an entire copy of mediawiki inside the app to serve the REST api to the frontend, then that i what i seek
(DIR) Post #B6jJnlQYcwQb5slTX6 by bvibber@wikis.world
0 likes, 0 repeats
@linear to the best of my knowledge, kiwix is the current gold standard for offline Wikipedia viewing (and yes I believe it's prerendered HTML in a nice package)if you want something more like ... a MediaWiki in a box that renders everything on-device with a local database and can optionally sync with live data from the Wikimedia servers on demand, I'm not aware of any such thing existing right now (and it would be a non-trivial project)
(DIR) Post #B6jJnlhDczktvYYmVE by linear@nya.social
0 likes, 0 repeats
@bvibber@wikis.world my vision, as touched on elsewhere in this thread, is that an instance of mediawiki+restbase is optionally run within the context of the android app, using a downloaded copy of the database, allowing for all features of the app to work, including editing of the local database which can then be re-exported.if not running locally, you should be able to point it at any other running copy.and yes, i am aware of the size that wikipedia is.
(DIR) Post #B6jKZAx7XhmHrxbTpA by legoktm@wikis.world
0 likes, 0 repeats
@linear @kiwix @tamzin sorry, I didn't see that post! I'm not aware of any other offline approaches besides Kiwix.
(DIR) Post #B6jKZBDQZ4p0gXEVF2 by anticomposite@wikis.world
0 likes, 0 repeats
@legoktm @linear @kiwix @tamzin one could, in theory, set up their own MediaWiki instance and import the dumps from https://dumps.wikimedia.org/ to generate their own versions of articles. This would require a significant amount of ongoing maintenance to get approximately the same result, which is why ~no one does it anymore.
(DIR) Post #B6jKqEVjZAW3pOvY1Y by linear@nya.social
0 likes, 0 repeats
@anticomposite@wikis.world @legoktm@wikis.world @kiwix@mastodon.social @tamzin@wikis.world yes. that's what i am asking for.i want the official android app (or a modified version of it) to spawn its own instance of mediawiki+restbase in order to allow all of its functionality to work locally without any external server.yes, i know how large wikipedia is. and i am skeptical that this should actually be that hard or require much in the way of ongoing maintenance once implemented.
(DIR) Post #B6jL3raDbjBgm0WYpU by linear@nya.social
0 likes, 0 repeats
@anticomposite@wikis.world @legoktm@wikis.world @kiwix@mastodon.social @tamzin@wikis.world it should also be self-replicating.i should be able to transfer wikipedia from my phone to a friend's phone in the middle of a desert with no internet connection, including any edits i may have made to it.wikipedia has enough utility that i am willing to devote a significant fraction of my phone/tablet storage space and a bit of programming effort to make sure that nobody i interact with who has a phone or tablet ever has to find themselves without it again if they don't want to, regardless of anything that anyone does to the official copy.
(DIR) Post #B6jLCYS26pu82hq8P2 by disorderlyf@todon.eu
0 likes, 0 repeats
@linear Sorry just saw this comment when I made my own. I've since deleted it
(DIR) Post #B6jLWHf6F02dlOML44 by f4grx@chaos.social
0 likes, 0 repeats
@linear the db is too big.
(DIR) Post #B6jLWHr9WBgOMlzxqq by linear@nya.social
0 likes, 0 repeats
@f4grx@chaos.social less than 50 gigabytes hardly seems "too big" when there are gacha games on my tablet larger than that which haven't been opened in a year
(DIR) Post #B6jMz3oEvJPb9Vv6fo by linear@nya.social
0 likes, 1 repeats
@f4grx@chaos.social if i were trying to run this on a 20 year old laptop or a raspberry pi or a $5 VPS, maybe i'd have an issuebut the thing about modern android devices is that they have much higher specs than that. most people have a cell phone of some sort now, and most cellular providers subsidize devices and give them hardware more powerful than they could otherwise afford. even the cheapest android phone i can get from my carrier ("free" if you don't cancel your plan within 3 years), Samsung Galaxy A16 5G, has 128G of storage and should be capable of holding a copy of wikipedia's database if that's what the phone's owner wants
(DIR) Post #B6jOOiqzYfzldw0Y1w by f4grx@chaos.social
0 likes, 0 repeats
@linear Kiwix files are directly available from https://dumps.wikimedia.org/ ; these are not scrapes, but official exports.The main database for the english wikipedia alone, without images, is a 23GB bz2 compressed xml file:https://dumps.wikimedia.org/enwiki/20260501/enwiki-20260501-pages-articles.xml.bz2Uncompressed, it's gigantic. I've attempted to parse it into an index and individual compressed pages, you end up with millions of files, it's not easy to handle.
(DIR) Post #B6jOOj6acgTKQJJ0LI by linear@nya.social
0 likes, 0 repeats
@f4grx@chaos.social the fact that wikipedia chooses to make available of their own accord the equivalent of taking a cell phone recording of a feature film, does not change that this is what they are publishing.a static web archive is not the same thing as the underlying data that produced it.i know full well how large those archives are, this is not my first time exploring this problem and i am as i write this trying to spin up a working copy of english wikipedia on my server using them, and evaluating how i can compress and package this environment to work within the context of an android app.which i am sure will be annoying, but probably less annoying than some of the other things i've had to make android apps do as an embedded software engineer.
(DIR) Post #B6jOcqin6RwI6AuMro by linear@nya.social
0 likes, 0 repeats
@f4grx@chaos.social the answer to the question i am asking allows for me to make an edit to my local copy, transfer the entire app and database to my friend who is sitting next to me in the middle of a desert with no internet connection available, and have them open and view the article i just edited on their phone.i currently choose to refuse any answer that says this is impossible
(DIR) Post #B6jOxnHl4b9tWVFcZs by f4grx@chaos.social
0 likes, 0 repeats
@linear It is definitely doable. My own exploration was to have the thing on a sd card attached to an esp32 with a screen, like the defunct wikireader devices. I'm not settled on the final archive format yet and I lost interest for now. I'll probably get back to it at some point.
(DIR) Post #B6jPFZUH73xZwRaB5E by f4grx@chaos.social
0 likes, 0 repeats
@linear it's definitely doable, just possibly not yet available off the shelf.
(DIR) Post #B6jpeyUlruAbYBaXj6 by tryst@imu.li
0 likes, 0 repeats
@linear @f4grx this is in my longer term goals as a wikipedia+offline edits that can be merged with future wikipedia update and other people’s offline edits.the groundwork for that is taking a while tho. (i do keep the wikitext dumps around in the meantime, but rendering them with anything other than mediawiki is a project in and of itself).