I made an open-source, self-hostable synced narration platform for ebooks reddit / selfhosted ---------------------------------------------------------------------- scrollin_thru · 2023-12-23 20:10 Hi, I made a thing! This is by far the most work I've ever sunk into a side project; I've been working on this thing for over two years, and I'm super proud of it, even though there's still a lot more to do! Storyteller is a self-hosted platform for ebooks with synced narration. This is basically self-hosted WhisperSync, for anyone familiar with that Amazon product. It's currently made up of two self-hostable backend systems and a mobile app for reading and listening to the books it produces. Technically it uses an open spec, EPUB 3's "Media Overlay", for syncing the narration, but very few ebook apps actually support Media Overlays, and even fewer work well and have nice interfaces. The mobile app is available on the Apple App Store as "Storyteller Reader", and I plan to release it for Android as well early next year. Anyway, I hope someone finds this interesting or useful! ---------------------------------------------------------------------- JimmyRecard · 2023-12-23 21:10 Oh that's interesting. I was just think about how with the advances in natural sounding TTS something like like that should come about. Is there a demo? How can I judge if the TTS quality is past the uncanny valley? The only thing that gives me pause regarding this is that I'm absolutely in love with KOReader (aftermarket open source eink reader software) so if there was some way for me to sync books and progress that'd be absolutely amazing. ---------------------------------------------------------------------- scrollin_thru · 2023-12-23 21:19 There's no TTS, actually! Storyteller takes two inputs: A textual ebook and an audiobook, and produces a textual ebook with synced narration. So it uses the audio from the narration in the audiobook; it doesn't generate its own! A demo is probably still a good idea, though; I'll look into that! There are some screenshots on the App Store page if you just want to get a sense of how the reading/listening experience works. Do you know how KOReader stores/syncs progress? Storyteller doesn't have progress syncing yet, but it's on my list of post-launch features to add! ---------------------------------------------------------------------- e3m88 · 2023-12-23 21:26 I used syncabook in the past with good results, definitely I will try this project. ---------------------------------------------------------------------- scrollin_thru · 2023-12-23 21:35 Awesome, I can't wait to hear what you think! Storyteller sort of does the "opposite" of syncabook; rather than using TTS to generate speech from text, and then aligning on the audio, Storyteller uses Whisper for transcription to generate text from audio, and then aligns on the text. I also gave up and built my own reading apps, because (as noted in the syncabook README) most of the existing ones are either unmaintained or have significant usability issues. I think that this is for the best, in the long run, though it did delay this release by the better part of a year! ---------------------------------------------------------------------- Theon · 2023-12-23 21:43 Way cool! This is exactly the kind of project whose benefits are obvious, I wish existed, and I'd assume 99% won't actually exist because everybody just kind of accepts the way things are. I appreciate the TTS-less approach too (I don't want to listen to robot narration, thank you). ---------------------------------------------------------------------- scrollin_thru · 2023-12-23 21:52 Thanks!! It’s been a lot of work getting here, I’m really glad this is appealing to folks other than myself! ---------------------------------------------------------------------- Schuhsohle · 2023-12-23 21:59 Looks great. Do you have an link for the iOS app? I can’t seem to find it in the App Store or is it not available in Germany? ---------------------------------------------------------------------- JimmyRecard · 2023-12-23 22:01 Oh, apologies, I may have misunderstood the docs, I gave them just a brief glance So the UX is essentially that I, as the user, have to bring an DRM unencumbered .epub and a DRM unencumbered audiobook (in which formats?) and a tool would produce an open-standard compliant .epub with an embedded audiobook? Then any compliant app (including the iOS/Androids you're creating) can display the media-enabled .epub either purely textually (like any .epub), or play it as a standard audiobook, or both at the same time. Is that about right? If I understand what your tool is about, is there a chance that you might bring TTS-based audiobook generation in the future, maybe when your tool is a bit more mature? I don't exactly know what the state of the open-source TTS is nowadays, but if it is even within an order of magnitude of the capability of a commercial TTS solution, being able to generate a full audiobook would be a game changer. Moreover, what's the state of multilingual support? Being able to both read and listen to a book in a foreign language you're trying to learn would be a massive boon for any language learners who don't have the benefit of learning a language based on immersion. ---------------------------------------------------------------------- JimmyRecard · 2023-12-23 22:09 Unfortunately, while human-read audiobook will always be the best choice, there is a strong case for creating TTS audiobooks, especially in the non-profit context. Many folks are visually impaired. Many speak small languages where audiobook production is non-existent, so it's TTS or nothing. Many people are slow readers and thus don't enjoy written books. Many drive or perform other work where they cannot read visually, but can listen to an audiobook. Nobody wants to see VAs out of work, but TTS has the potential to allow many more people to enjoy the written word. ---------------------------------------------------------------------- anidal · 2023-12-23 22:12 This is incredible work. Read through your documentation on how the algorithm works and its pretty impressive. Thanks for building this and sharing. ---------------------------------------------------------------------- [deleted] · 2023-12-23 22:15 [deleted] ---------------------------------------------------------------------- ang3l12 · 2023-12-23 22:25 Ooooooh!!!!! Guess I have a new project to spin up. I just got done setting up Audiobookshelf, but I do miss having audible+kindle syncing. ---------------------------------------------------------------------- Moderntweety · 2023-12-23 22:27 This is something I have been looking for will def look around to set this up ---------------------------------------------------------------------- scrollin_thru · 2023-12-23 22:27 It should be available in all countries, and I should definitely include a link to it on the docs! Here’s a link: https://apps.apple.com/us/app/storyteller- reader/id6474467720 ---------------------------------------------------------------------- scrollin_thru · 2023-12-23 22:39 You’ve got the UX right! Audiobooks can be provided as a zip archive of MP3s or as an M4B/MP4. Finding DRM-free audiobooks is pretty straightforward (libro.fm has every book I’ve ever looked for), but DRM-free epubs is a lot more challenging. TTS and multilingual support are both excellent ideas and I’m planning on looking into both! Whisper, the transcription model that Storyteller uses, has support for languages other than English, though it’s not quite as good (it’s probably good enough, though!). There’s some pretty incredible AI TTS stuff happening now, though most of it is proprietary. I will definitely look into it, though! Thank you for these ideas :) ---------------------------------------------------------------------- scrollin_thru · 2023-12-23 22:42 Thank you so much, I really appreciate it! ---------------------------------------------------------------------- scrollin_thru · 2023-12-23 22:45 Please do, and please don’t hesitate to reach out if you run into any issues! ---------------------------------------------------------------------- nonlinear_nyc · 2023-12-23 22:48 Ooh I find it very useful. I'm a usability/product designer and I volunteer for open source projects. Let me know if you need any help. DM me if that's the case. ---------------------------------------------------------------------- scrollin_thru · 2023-12-23 22:48 I’m so glad to hear that! Please let me know if you run into any issues or have any feedback! ---------------------------------------------------------------------- nonlinear_nyc · 2023-12-23 22:49 I agree. Reader apps are a graveyard of bad usability. You'd think community would care about it, like tv show community does with beautiful open source projects, but it's mostly unusable tools with broken promises. I never understood that. ---------------------------------------------------------------------- nosyrbllewe · 2023-12-23 23:05 Thank you for making this. I have always wondered why ebooks with synced audio were never really a thing, despite EPUB 3 supporting it. I will be sure to check it out. ---------------------------------------------------------------------- scrollin_thru · 2023-12-23 23:09 That’s very generous!! Thank you so much for offering! I may very well reach out after the holidays; I would love to do some design work on the admin site. If you have a chance to try the project out before then, I’d love to hear what you think! ---------------------------------------------------------------------- scrollin_thru · 2023-12-23 23:18 I hope you like it! ---------------------------------------------------------------------- pseudosinusoid · 2023-12-23 23:22 You should integrate this with Audiobookshelf, that would be amazing… ---------------------------------------------------------------------- nonlinear_nyc · 2023-12-23 23:29 You're welcome. Usability and open source is *hard* so I'm willing to bridge the gap how I can. I sent you a link for a scheduling... I'll try to find some of the ideas I had for ebooks so far so(I understand you want help on the admin part of it, but we talk). ---------------------------------------------------------------------- PPsmalls · 2023-12-24 01:48 This looks really great. I'd been tooling around with a similar side project that fell to the wayside. My use case is keeping progress in sync between KOReader and another service like audiobookshelf or Plex. I had used whisper to generate the transcription upfront and then wrote a KOReader lua plugin to push the status to my server which would then update any configured audiobook services. At the time "whisper.cpp" could only do sentence level transcriptions and since the transcription has slight differences from the written text it was a bit of a challenge. I'd dabbled with various full text searches (SQLite / meilisearch / typesense) to index individual sentences but the result wasn't very robust. It seems like you've figured this out. I'm going to spend some of my holiday reviving my plugin to try accessing the media overlay metadata in the storyteller generated ebook and then call the audiobook service directly from my Kobo reader. Storyteller looks very useful, thanks! ---------------------------------------------------------------------- carcus5 · 2023-12-24 02:00 Maybe something like https://github.com/rhasspy/piper would be worth looking into ---------------------------------------------------------------------- Thought_Crash · 2023-12-24 02:44 I would love it if an open source TTS engine can just ask for API keys and you can just give it Azure, Polly, Eleven Labs, etc. keys to make them work. Saves from paying a middle man company. Also, where are the open source TTS engines that use AI voices from Coqui, Piper, etc? ---------------------------------------------------------------------- _risho_ · 2023-12-24 02:49 It's up and running for me, but when i load the web ui it asks me to log in, but i havent made a user yet so it fails to authenticate. ---------------------------------------------------------------------- scrollin_thru · 2023-12-24 03:03 Shoot! What happens if you specifically go to `/init`? It should be redirecting you there automatically ---------------------------------------------------------------------- scrollin_thru · 2023-12-24 03:08 Thanks! Yeah, I don't know if you took a look at [this section of the docs](https://smoores.gitlab.io/storyteller/docs/how-it-works/the- algorithm#step-3-produce-the-synced-book), but the vast majority of the backend work on this project involved figuring out the best heuristics for fuzzy matching the actual text against Whisper's transcriptions. If you manage to get something working, please let me know! Also don't hesitate to reach out if there's anything I can do to make this easier for you; off the top of my head, it might make sense to add an entrypoint to the API container that allows you to just sync a book ad hoc, if you don't need the whole service and are just looking for the synchronization output. ---------------------------------------------------------------------- _risho_ · 2023-12-24 03:16 it brought me to the page to make the admin account but when i tried it says this: Application error: a server-side exception has occurred (see the server logs for more information). Digest: 3933982100 https://pastebin.com/mDD72pqU this is the docker logs from web and api. ---------------------------------------------------------------------- scrollin_thru · 2023-12-24 03:21 Got it, thanks for sharing the logs! I think this is the primary issue: `Error: connect ECONNREFUSED` [`127.0.0.1:8010`](https://127.0.0.1:8010) This looks like you have the web UI configured to find the API at [`127.0.0.1:8010`](https://127.0.0.1:8010), but the API is actually running on a different port (or isn't available on the localhost that the web UI is running on). From the logs you shared, it looks like the API is actually running on port 8000, rather than 8010. Can you share the commands that you used to start up the two services? ---------------------------------------------------------------------- _risho_ · 2023-12-24 03:25 docker run -v /stuff/mounts/StorytellerData:/data -p 8010:8000 registry.gitlab.com/smoores/storyteller/api:latest docker run -e STORYTELLER_API_HOST=http://localhost:8010 -p 8011:8001 registry.gitlab.com/smoores/storyteller/web:latest ---------------------------------------------------------------------- _risho_ · 2023-12-24 03:30 huzzah, changing it from localhost to my lan IP did in fact fix it. thanks! ---------------------------------------------------------------------- scrollin_thru · 2023-12-24 03:31 Thank *you* for debugging my docs! I'll update tonight. Glad it's working!! ---------------------------------------------------------------------- _risho_ · 2023-12-24 03:53 hmm, it appears that whenever I click on an element inside of the webui it automatically sends me back to the login screen. I tried deleting the images and remaking both of the containers, but it still just logs me out whenever I try to do anything. Any idea what I might be doing wrong? https://pastebin.com/jzVTrGk5 api logs ▲ Next.js 13.5.4 - Local: http://b79730a62754:8001 - Network: http://172.17.0.10:8001 ✓ Ready in 366ms Error: 'sharp' is required to be installed in standalone mode for the image optimization to function correctly. Read more at: https://nextjs.org/docs/messages/sharp-missing-in-production Error: 'sharp' is required to be installed in standalone mode for the image optimization to function correctly. Read more at: https://nextjs.org/docs/messages/sharp-missing-in-production and all that is in the webui logs. ---------------------------------------------------------------------- scrollin_thru · 2023-12-24 03:58 Huh! Odd. I just tested this locally and am not having this issue. It sounds like the auth cookie isn't being persisted for some reason; are you able to check your browser storage and see if there's an `st_token` cookie being saved for the localhost domain? Which browser are you using? ---------------------------------------------------------------------- _risho_ · 2023-12-24 04:05 https://i.imgur.com/pNIKicn.png https://i.imgur.com/vz4XmC2.png if i'm doing that correctly it doesn't appear to have a cookie. i'm using firefox on ubuntu 23.10. same thing appears to happen with chromium and brave though. I'm going to be heading off to bed here in a bit, but i could try it on macos or windows tomorrow to see if maybe it's just a weird linux thing. ---------------------------------------------------------------------- scrollin_thru · 2023-12-24 04:08 Weird, I'm testing on Firefox on Fedora. There should definitely be a cookie there! I'll poke around tomorrow; in the meantime, if you have a chance, I just updated the docs to include setting up a proper network for the containers to communicate over. Maybe give that a shot, in case there's some weird networking thing I'm missing somehow? ---------------------------------------------------------------------- e3m88 · 2023-12-24 04:17 `INFO: 172.17.0.1:39292 - "GET /books HTTP/1.1" 200 OK` `INFO: 172.17.0.1:39306 - "OPTIONS /books/epub HTTP/1.1" 400 Bad Request` I can't even upload a book: `INFO: 172.17.0.1:39306 - "OPTIONS /books/epub HTTP/1.1" 400 Bad Request` ---------------------------------------------------------------------- _risho_ · 2023-12-24 04:40 hmm, i tried it with your new commands though it still seems to have the same issue. I tried it on my phone using firefox as well and it still logs me out whenever I click on an element and has similar logs saying okay and then forbidden. it is still possible that maybe i'm just making a mistake somewhere. I changed the api port to 8000:8000 to see if that would help. added the logs with the stuff before the auth errors since maybe there is something interesting in that part. https://pastebin.com/RJ5eELLq ---------------------------------------------------------------------- Ecredes · 2023-12-24 05:08 This is very cool. I'm going to try it out tomorrow when I get a chance. Does this produce a new epub3 with imbedded audio? Or does the audiobook remain separate file? ---------------------------------------------------------------------- scrollin_thru · 2023-12-24 05:11 Thank you! Yes, it produces a new epub with the audio embedded ---------------------------------------------------------------------- Ecredes · 2023-12-24 05:13 Are these files readable/playably by most epub readers? Or can they be played by ffmpeg? Is there's a certain player backend you're using in your app? ---------------------------------------------------------------------- scrollin_thru · 2023-12-24 05:41 Shoot, this is a CORS issue, yet another thing I forgot to document! The API server needs to be run with an environment variable allowing requests from your web interface for file uploads. The docker run command should be: docker run \ -it \ -v ~/Documents/Storyteller:/data \ -e STORYTELLER_ALLOWED_ORIGINS=http://localhost:8001 \ -p 8000:8000 \ --name storyteller-api \ --network storyteller \ registry.gitlab.com/smoores/storyteller/api:latest Assuming that your web interface is running at port 8001. Note that this needs to be *the URL that you type into your browser* in order for requests to be accepted by the API. Thank you again for helping me find these issues, and sorry again for the friction! I hope this works! ---------------------------------------------------------------------- scrollin_thru · 2023-12-24 05:44 There's nothing special about the audio files themselves; they're just the MP3 or MP4 files that you upload initially! Any epub reader that has support for Media Overlays can play them, though at the moment that is unfortunately a fairly small number of readers, as best I can tell. Hopefully that will change and more folks will start supporting the spec! ---------------------------------------------------------------------- scrollin_thru · 2023-12-24 06:33 If you have a chance to hop into the Gitter chat tomorrow, I would definitely like to try to work this out! ---------------------------------------------------------------------- e3m88 · 2023-12-24 07:17 I'll try tomorrow but I get `401 Unauthorized` when trying to upload a book, then I need to login again, I can't even access the settings or any other button, it asks for login. ``` 2023-12-24 02:15:24 api-1 | INFO: 172.31.0.1:36352 - "GET /books HTTP/1.1" 200 OK 2023-12-24 02:15:30 api-1 | INFO: 172.31.0.1:36366 - "POST /books/epub HTTP/1.1" 401 Unauthorized 2023-12-24 02:15:44 api-1 | INFO: 172.31.0.1:36368 - "GET /books HTTP/1.1" 401 Unauthorized 2023-12-24 02:16:04 api-1 | INFO: 172.31.0.1:36370 - "GET /books HTTP/1.1" 401 Unauthorized ``` ---------------------------------------------------------------------- Defiant-Ad-5513 · 2023-12-24 09:27 Piper is open source and uses the wyoming protocol you just need an interface for it to be a rest api. ---------------------------------------------------------------------- proskater_83 · 2023-12-24 11:32 This is my dream feature. You just made my Christmas ---------------------------------------------------------------------- scrollin_thru · 2023-12-24 16:24 Got it, this sounds like the same issue that u/_risho_ is having, where cookies aren't being stored for localhost for some reason. I will do my best to figure out why this isn't working today! ---------------------------------------------------------------------- scrollin_thru · 2023-12-24 16:26 I'm so glad to hear that!! ---------------------------------------------------------------------- scrollin_thru · 2023-12-24 16:28 Do you see anything in the browser console after you log in? Maybe there's some errors there indicating why the browser is rejecting the cookie! ---------------------------------------------------------------------- scrollin_thru · 2023-12-24 17:00 Ok, I \_think\_ I may know what the issue was; I was incorrectly setting `Secure` on the cookie always, which would mean that it wasn't sent for requests on \`http://localhost\` (because it's not over HTTPS). I can't explain why this *was* working for me, but I just pushed a change to make the web server more deliberate about choosing the Secure attribute when the login request is made over HTTPS, rather than always. Hopefully if you pull the latest web image that will work for you! Tagging u/_risho_ as well (or, at least, trying to? This didn't work last time I tried to tag you; Reddit seems to be processing the underscores into italics before it processes the `u/` :/ ) ---------------------------------------------------------------------- scrollin_thru · 2023-12-24 17:02 Reddit is very bad at tagging you, `_risho_`! I think this comment might be relevant to what you're seeing, though: https://www.reddit.co m/r/selfhosted/comments/18pdvtv/comment/kerbmpv/?utm\_source=share&utm \_medium=web2x&context=3 ---------------------------------------------------------------------- RAWDOGRONALD · 2023-12-24 17:04 This looks awesome! Would love to try it out but there don't seem to be any arm64 images, any plans to add this in the future? ---------------------------------------------------------------------- scrollin_thru · 2023-12-24 17:07 Oh, thank you for calling that out! I will definitely add that to my list of things to do. I'll let you know when it's available! ---------------------------------------------------------------------- e3m88 · 2023-12-24 17:46 Finally, it seems to be working, thanks. I'm trying to process a book I'll report back when it finishes. ---------------------------------------------------------------------- scrollin_thru · 2023-12-24 17:47 Phew, what a relief! Thanks again for bearing with me while we figured that out ---------------------------------------------------------------------- Aramaki87 · 2023-12-26 19:22 Audiobookshelf is slow af. I would not recommend it. I can recommend Kavita and Podgrab. ---------------------------------------------------------------------- pseudosinusoid · 2023-12-26 21:51 Care to elaborate? I’ve never had a problem with it. The mobile app is very nice. AFAICT neither Kavita nor Podgrab support audiobooks so I don’t know what planet you’re on. ---------------------------------------------------------------------- Aramaki87 · 2023-12-28 19:46 To keep it short: More files in a folder take ridiculously longer to load. At about 900 files I got Socket Errors because it takes over a minute to load. The errors could also come from my reverse proxy but the source problem is the performance. A lot of loading. I don’t know of any released Audiobookshelf app for ios. I only use podgrab to download and sort Podcasts. After that I use jellyfin to listen to them. That works pretty good. Smooth and fast. ---------------------------------------------------------------------- New-Championship7579 · 2023-12-29 21:36 I'd give your reverse proxy settings a peek, I've got around 1000 books in my library and Audiobookshelf has been working fine for me with no performance issues. Or not, if you're happy with your existing setup, whatever works for you ---------------------------------------------------------------------- New-Championship7579 · 2023-12-31 00:23 I've wanted something like this for a long time (I actually followed the Audiobookshelf feature request thread for this very thing to your project, then found this post after the fact), so I'm super excited to see this! First impression, the forced alignment seems to be pretty good, and synchronization between audio and text was working pretty well when I tried it out. I was excited to see you being open to integrations with other services like audiobookshelf because I cannot *stand* listening to audiobooks at 1x, so lack of playback speed control is pretty much a dealbreaker for me. I imagine many people have their own opinions about what feature constitutes a dealbreaker for them, so being able to integrate with 3rd party services should be a huge win in that respect. What I want more than almost anything is to be able to listen to a book in audiobookshelf and then later open the ebook in an e-reader and have it pick up where I left off. Storyteller is well- positioned to be not just a great e-reader but also a great synchronization framework for other applications to talk through. Overall, this is great work! I'm so pleased to see work being done on this to create a viable alternative to Amazon's whispersync. ---------------------------------------------------------------------- scrollin_thru · 2023-12-31 02:46 Thanks, I’m so glad you’re excited about it! For what it’s worth, I also plan on adding more audio features (speed control, sleep timer, etc) to Storyteller, but I’m also a big fan of giving people control and freedom, and Storyteller will always rely on open specifications and protocols that allow it to integrate with other systems. > Storyteller is well-positioned to be not just a great e-reader but also a great synchronization framework for other applications to talk through. I hope this is true! It should be pretty straightforward to add backend support to converting audio timestamps to epub locations (right now that logic only exists in the mobile apps), if we get to a point where other systems want to integrate with Storyteller’s syncing system!