ChatVault - store your whatsapp messages reddit / selfhosted ---------------------------------------------------------------------- NeoJackOfBlades · 2023-11-14 22:35 I think this is a pretty niche demand and probably another topic for r/DataHoarder but anyway, here I am. I created this application to basically have a way to store my WhatsApp messages away from the Google/Meta servers. Or at least not depend so much on Google backup. Whatsapp has a very limited export functionality, which any user can use through the app's own interface. Once these messages and media have been exported, you can place them in a folder monitored by ChatVault, send them to an email monitored by ChatVault or upload them via the interface. Once ingested by chatvault, it will record the chat media on disk and save the messages in a database in a structured way. These messages can be accessed in a front end similar to a chat application. It's still under development, some things need to be improved (mainly the UI), it's still far from ideal, it's true, the way Whatsapp allows us to export messages is quite bad, which makes the entire process of exporting and ingesting it into chatvault quite coupled but it can still be useful for those who want to store their messages independently, just like I wanted. [https://github.com/vitormarcal/chatvault](https://github.com/vitormar cal/chatvault) ---------------------------------------------------------------------- Haliphone · 2023-11-14 23:55 Superb! I'll take a look ---------------------------------------------------------------------- Plain-Tangerine3715 · 2023-11-15 00:02 This is pretty interesting. I've been trying to figure out some way to export the whatsapp messages from an old phone to a new phone to have them forever. The old phone's just been sitting there waiting. Are you using this to sync occasionally? How does that flow work if you are still using whats app regularly? ---------------------------------------------------------------------- NeoJackOfBlades · 2023-11-15 00:41 In my case, I regularly export each chat from time to time, after exporting I delete the WhatsApp chat, and use WhatsApp normally until I eventually export and clean it again. I do this to avoid duplication of records. For my use case it's already good, but depending on the purpose of use, it's an inconvenience having to clean the original chat to avoid data duplication. Therefore, I am thinking about developing a way to deduplicate messages soon, so that if I import messages that have already been imported, the base will not be duplicated. You can also import it into a chat under another name. I intend to implement a way to merge chats, so I will be able to import a draft that can later be merged to a definitive chat, and then I will also have deduplication. ---------------------------------------------------------------------- f0rc3u2 · 2023-11-15 07:26 Looks very promising! Thanks :) ---------------------------------------------------------------------- f0rc3u2 · 2023-11-15 07:26 Yes, that would be very important for me as well ---------------------------------------------------------------------- keixver · 2023-11-15 07:38 Awesome work! ---------------------------------------------------------------------- katroome · 2023-11-15 10:00 I worked on a similar project for a long time... I don't really have code that I can share right now, but perhaps I can offer some hints. Instead of exporting chats from the app (since the function is too limited) I decrypted the backup database that WhatsApp creates. This was originally possible only on rooted Android phones, but it's now possible if the user has chosen to encrypt the backup (and the encryption key is known). Once the database is decrypted, it is just a matter of extracting messages and media information from it, and rebuild the chat. One of the problems here (and one of the reasons why I could not keep spending time on this project) is that Meta changed the structure of the database multiple times over the last years. Here you can see an example of a chat I recreated from the database. The display format is HTML, trying to replicate the WhatsApp original look. ​ ​ Let me know if you want more info. I will be happy to share! ---------------------------------------------------------------------- NeoJackOfBlades · 2023-11-15 11:49 >I worked on a similar project for a long time...I don't really have code that I can share right now, but perhaps I can offer some hints. > >Instead of exporting chats from the app (since the function is too limited) I decrypted the backup database that WhatsApp creates. This was originally possible only on rooted Android phones, but it's now possible if the user has chosen to encrypt the backup (and the encryption key is known). > >Once the database is decrypted, it is just a matter of extracting messages and media information from it, and rebuild the chat. One of the problems here (and one of the reasons why I could not keep spending time on this project) is that Meta changed the structure of the database multiple times over the last years. > >Here you can see an example of a chat I recreated from the database. The display format is HTML, trying to replicate the WhatsApp original look. > >Let me know if you want more info. I will be happy to share! > >EDIT: It seems I cannot post a picture in the comment. Is that a limitations of this community? I didn't know you could access these messages without root these days, good to know! I was already aware of these frequent structure changes, so I went with the simplest as a proof of concept. Anyway, I plan to study a little more to find out how the Whatsapp base works, I don't rule out an experimental import of this type. Thanks for the information! :D ---------------------------------------------------------------------- katroome · 2023-11-15 11:52 >Thanks for the information! :D I am very happy to help someone who has my same fixation on saving old conversations (and memories) :D :D ​ I will share a screenshot of the chat I recreated via private message, if I can. Please don' hesitate asking for info, I will be happy to share as much as I can! ---------------------------------------------------------------------- katroome · 2023-11-15 11:56 In case someone else is interested, to decrypt the database I used this: https://github.com/ElDavoo/WhatsApp-Crypt14-Crypt15-Decrypter ---------------------------------------------------------------------- programmer-ke · 2023-11-15 13:12 Good find on my side, I've been looking for some way of archiving whatsapp messages, I'll take a look ---------------------------------------------------------------------- olivercer · 2023-11-15 14:11 **THANK YOU!!!** I have 15+ GB of Whatsapp stuff and it is unmanageable. My best friend has 25 GB if I remember correctly. This stuff are chats and media from about 10 years ago until today. ChatVault is the archiving solution I've been dreaming for YEARS, so I (we) can finally offload most of the old chats from our mobile devices, while still keeping them accessible via a frond end. I will follow this closely. Please also consider accepting donations - I will definitely donate if ChatVault fits my use, this is a game changer! ---------------------------------------------------------------------- olivercer · 2023-11-15 14:13 This would be very appreciated! ---------------------------------------------------------------------- SnooPets20 · 2023-11-15 14:33 I have a WhatsApp backup zip file. This won't work, right? Since I think it's encrypted. ---------------------------------------------------------------------- AKDub1 · 2023-11-15 14:34 This is great! I will definitely try this out when I get the chance and follow development. For the last year or so I had been thinking of a similar thing and planning to write a post on r/datahoarder about it, but decided not to because as someone who knows nothing about software development I would be basically just asking people to create an app I want for free. Here are some random thoughts I had had for that app: Import from multiple sources WhatsApp/Signal/Telegram/etc - either from databases,backups, exports, or just parsing text files for if you are able to compose a chat history in any other way. Optional chat merge - merge chats from different sources into a single chat history instead of into a separate chat isothere Address book/Contact - so that you can manage things clearly when merging ---------------------------------------------------------------------- kon_dev · 2023-11-15 15:39 If anyone searches for a comparable alternative for Signal, I can recommend https://github.com/bepaald/signalbackup-tools. You can export a backup in Signal Android and decrypt it and export HTML with that cli. It even offers search via Javascript. ---------------------------------------------------------------------- harriershmarrier · 2023-11-15 15:44 How does the importing work? I have a whatsapp folder containing logs.zip, Databases, Backups, Media. I put the whole folder into the import folder of chatvault but execute disk import does not seem to do anything ---------------------------------------------------------------------- siddharthal · 2023-11-15 17:18 Couldn't get it to work. Tried running it in docker. Getting this error :: Spring Boot :: (v3.1.2) 2023-11-15T17:10:53.882Z INFO 1 --- [ main] dev.marcal.chatvault.Boot$Companion : Starting Boot.Companion v0.0.1-SNAPSHOT using Java 17.0.9 with PID 1 (/app/chatvault.jar started by root in /app) 2023-11-15T17:10:53.976Z INFO 1 --- [ main] dev.marcal.chatvault.Boot$Companion : No active profile set, falling back to 1 default profile: "default" [Too many errors, abort] ---------------------------------------------------------------------- NeoJackOfBlades · 2023-11-15 18:51 For now, the application cannot deal with the physical base of WhatsApp, the application can import based on the txt of messages and media that WhatsApp generates when exported through the application interface ---------------------------------------------------------------------- NeoJackOfBlades · 2023-11-15 18:55 For now, you enter the Whatsapp interface, go to export chat, choose whether you want to send it by email or on disk. With these files you can configure import via email, import through the interface with the zip file or place them in a specific folder, so that you can trigger the disk scan through the chatvault interface. ---------------------------------------------------------------------- NeoJackOfBlades · 2023-11-15 19:14 I just cleared all images and containers to make sure I wasn't working with any cache and that everything went fine. This message is expected because we are not defining any profiles: No active profile set, falling back to 1 default profile: "default". Then I would have to see what error happened after that. The only properties required are the database connection properties. That being said, you can run compose.yml in the project root. It will build an image locally. Or replace the \`build: ./\` line with the \`image: [ghcr.io/vitormarc al/chatvault:latest\`](https://ghcr.io/vitormarcal/chatvault:latest) \`docker-compose -f compose.yml\` I tested these two ways here and they continue to work. Have you added database information? ---------------------------------------------------------------------- NeoJackOfBlades · 2023-11-15 19:18 Yes, I think it will end up being relatively easy to port to other types of chats as well. I focused more on Whatsapp because unfortunately it's what everyone in my region uses, so most of my conversations are there. But eventually I'm going to do something for Telegram too, so I need to make the solution I have now for Whatsapp a little more solid. Thanks for the suggestions! ---------------------------------------------------------------------- Particular-Meet-5945 · 2023-11-15 20:37 Hey, I’ll love to use (and donate) too. Thanks for building this ---------------------------------------------------------------------- siddharthal · 2023-11-16 04:30 I tried with this setup only. This was my docker setup. chatvault: image: ghcr.io/vitormarcal/chatvault:latest restart: unless-stopped environment: - SPRING_DATASOURCE_URL=jdbc:postgresql://postgres:5432/chatvault - SPRING_DATASOURCE_USERNAME=${POSTGRES_USER} - SPRING_DATASOURCE_PASSWORD=${POSTGRES_PASSWORD} ports: - 8106:8080 volumes: - '~/chatvault:/opt/chatvault' - '~/chatvault/config:/config' depends_on: - postgres ---------------------------------------------------------------------- ThaCrrAaZyyYo0ne1 · 2023-11-16 05:51 Awesome project! Sorry for the noob question but how do I export the chat? I mean, when I hit export chat button it only gives me option to send it via email (and them attaches a .txt and the .jpgs - not a zip file) or share it with telegram, no option to save in disk. ---------------------------------------------------------------------- ovizii · 2023-11-16 09:18 I'm also looking forward to a new way to import as like you, I manually backup the complete WA folder from my phone and thus have never lost any messages as well as been reliant on the Google Drive backups. ---------------------------------------------------------------------- MRCP_PC · 2023-11-16 14:18 This is an interesting project, I'll keep an eye on this for sure. Does anyone know any similar projects that are more general not only restricted to WhatsApp? ---------------------------------------------------------------------- urinka · 2023-11-16 19:55 hii tnx its look very promising when i install it with the docker compose ime getting an error `root@docker3:~/docker/chatvault# docker compose up -d` `[+] Building 0.6s (2/2) FINISHED docker:default` `=> [chatvault internal] load build definition from Dockerfile 0.2s` `=> => transferring dockerfile: 2B 0.0s` `=> [chatvault internal] load .dockerignore 0.1s` `=> => transferring context: 2B 0.0s` `failed to solve: failed to read dockerfile: open /var/lib/docker/tmp/buildkit-mount2870723879/Dockerfile: no such file or directory` `root@docker3:~/docker/chatvault# docker pull ghcr.io/vitormarcal/chatvault:latest` `latest: Pulling from vitormarcal/chatvault` `Digest: sha256:4409d7edc18b3fe95b3cca0bce8355732f848f11c6fcb9e4969924 dd96586c25` `Status: Image is up to date for ghcr.io/vitormarcal/chatvault:latest` [`ghcr.io/vitormarcal/chatvault:latest`](https://ghcr.io/vitormarcal/c hatvault:latest) ​ this is from a new docker instance that i created for testing this using the docker script form the proxmox helper am i doing something wrong ? ---------------------------------------------------------------------- teslawhytho · 2023-11-18 04:26 Please don’t take this as criticism, this is a great idea and I fully plan on contributing to the codebase. With that said I spent a few hours trying to get it to work. No luck. Docker, no. Docker compose. No. I took the code and built / run manually. That worked but then I couldn’t import a chat. I tested with one line with no attachments. From just that one line, here are the problems so far: - it doesn’t seem like WhatsApp has a standard way of exporting the text file. Your text file and my text file are different. In the US the format is [datetime] name msg. In your file it’s different and so it breaks the moment it hits the [. - unfortunately not accounting for locale. US stupidly uses mm/dd/yy. You have hardcoded the formatter for dd/mm/yy. Maybe you need to have a locale selection in the UI before import. Without that, no US messages are coming in. - it doesn’t account for 6/6/23. It’s expecting 06/06/2023. Again formatter and padding can fix that. - ui creates an entry in chats table for every attempt regardless of if a message was imported. - exceptions in other languages. - missing tests for the stuff above Again none of that is supposed to be criticism. This is a great idea and I fully intend to help out with it. Good job! ---------------------------------------------------------------------- NeoJackOfBlades · 2023-11-18 22:01 Hello, criticism is certainly welcome! If you can open an issue on github it will be easier for me to follow, as I may not see these comments. About the message date, you are right! Until I divulge the project, I was the only user so I didn't know it could have multiple types of message date formats, then I developed for a specific format (which isn't even uncommon). Someone had already warned me about this and so I started to develop something that could format the date message in a rigth way. It's almost ready, but unfortunately, there are dates that end up being ambiguous like 01/01/2023 and it's not possible to infer the correct format, so I'll probably have to create an environment variable for that, which I really didn't want. If you look on github I opened a bug issue for this (although maybe it's not really a bug but rather an improvement, because it works but with a specific format) and in the Github's Projects part it's already under development. Regarding the duplication of data, I mentioned it in a comment here on this post, but perhaps I should have made it clearer. Anyway, as I said in the post, the project is still far from ideal, despite it working very well for my use case (every import I do is always new messages). Anyway, I know this is a very important point, so I created a way of deduplication considering the last message in the database as a cutoff parameter. This is already in the latest version of the docker image. Regarding docker, more information about the error is welcome, you weren't the first person to talk about it but I couldn't replicate the problem. I tested it on Fedora and the Ubuntu server, I built it locally, I pulled it from the Registry, did docker system prune --volumes -a, and it still worked as expected. ---------------------------------------------------------------------- NeoJackOfBlades · 2023-11-18 22:36 I'm sorry about that, is your Operating System Unix or Windows? x86 or arm? I tested it on Ubuntu server and fedora and it is correct, I will test it on Windows soon. I'm trying to imagine what it could still be. ---------------------------------------------------------------------- NeoJackOfBlades · 2023-11-18 22:37 I'm sorry about that,I'm trying to imagine what it could still be. Is your Operating System Unix or Windows? x86 or arm? I tested it on Ubuntu server and fedora and it is correct, I will test it on Windows soon. ---------------------------------------------------------------------- siddharthal · 2023-11-19 02:40 It was ARM ---------------------------------------------------------------------- urinka · 2023-11-20 10:36 hii tnx for the reply i am using proxmox as main hypervisor and lxc container to contain my dockers i am using Proxmox helper scripts to create docker container so the main system (the hypervisor) is base on debian and the lxc container also base on debian ---------------------------------------------------------------------- RatchetSan · 2023-11-21 14:21 I had the same problem. You need to pull all the files from the github page and then edit the docker-compose file to your liking or just start it up. ---------------------------------------------------------------------- RatchetSan · 2023-11-21 14:23 This is definitely what i was looking for the whole time. Thank you for this project. I have a question though. After setting it up on docker, i tried to import a chat onto it, but nothing happens. I only get a "file was imported" and a bit of loading, but its still empty. Any advice? ---------------------------------------------------------------------- NeoJackOfBlades · 2023-11-21 15:10 Hi! The UI still needs some work. Refresh the page, if you imported successfully it will be listed on the page. If it was not imported, there may have been a problem analyzing the messages, most likely in the date format, which unfortunately can vary from device to device. There must be a log on the server stating this. Then you can put an unambiguous date in the first message of the file (12/25/2023 is unambiguous because you can know what month it is and what day it is and 05/01/2023 is ambiguous because you can't know it) or define a format default date format chatvault.msgparser.dateformat (e.g. dd/MM/yyyy HH:mm) If it still doesn't work, there may be a problem with the regex that identifies the messages. If you could open an issue and send me an example of your file so I can analyze it and include it in the regex, that would help. In this case you can obviously remove any confidential data, or leave an example of a few messages modifying the content if you prefer. ---------------------------------------------------------------------- urinka · 2023-11-24 22:48 tnx that help now after git clone the reposetory its go farther now ime getting this error ​ `=> ERROR [chatvault backend_builder 6/6] RUN ./gradlew clean build 39.5s` `------` `> [chatvault backend_builder 6/6] RUN ./gradlew clean build:` `0.717 Downloading https://services.gradle.org/distributions/gradle-8.2.1-bin.zip` `2.085 ............10%............20%............30%.............40%.. ..........50%............60%............70%.............80%........... .90%............100%` `8.595` `8.596 Welcome to Gradle 8.2.1!` `8.596` `8.597 Here are the highlights of this release:` `8.597 - Kotlin DSL: new reference documentation, assignment syntax by default` `8.598 - Kotlin DSL is now the default with Gradle init` `8.599 - Improved suggestions to resolve errors in console output` `8.599 - Reduced sync memory consumption` `8.600` `8.600 For more details see https://docs.gradle.org/8.2.1/release- notes.html` `8.600` `8.794 Starting a Gradle Daemon (subsequent builds will be faster)` `38.69` `38.70 FAILURE: Build failed with an exception.` `38.70` `38.70 * What went wrong:` `38.70 A problem occurred configuring root project 'chatvault'.` `38.70 > java.util.concurrent.ExecutionException: org.gradle.api.GradleException: Failed to create Jar file /root/.gradle/caches/jars-9/8fa220ef21385a3519e9105d711e7fff/kotlin- compiler-embeddable-1.8.22.jar.` `38.70` `38.70 * Try:` `38.70 > Run with --stacktrace option to get the stack trace.` `38.70 > Run with --info or --debug option to get more log output.` `38.71 > Run with --scan to get full insights.` `38.71 > Get more help at https://help.gradle.org.` `38.71` `38.71 BUILD FAILED in 38s` any suggestions ??