How I wish self-hosted photo apps (Memories, Immich, Photoprism) worked reddit / selfhosted ---------------------------------------------------------------------- Accomplished-Lack721 · 2023-11-13 15:35 Like a lot of people on this sub, I've migrated away from using Google Docs and Photos (most of the time) in favor of self-hosted services. For photo use, I'm really impressed by everything that's gone in to Immich, Photoprism and Memories, with the latter coming the closest to my preferred way for such an app to work. But I still find some difficulty with the workflow and design philosophy with of these apps when it comes to two things. 1) Working with large existing libraries, like one exported from Google Photos using takeout, while maintaining the organization of the original library. 2) Updating metadata. On point 1 -- I've got a few hangups about this. The biggest is how duplicates are handled. For instance, let's say in Google Photos, you had your favorite photo from your 30th birthday both in an album called "2009 favorites" and another "30th birthday pics." When (for instance) Photoprism is scanning those photos for its library, it'll import the first one it finds, and ignore the second, without any indication in the UI that happened (but it will be in the logs). Then, later, when you go looking in the "30th birthday pics" photos in Photoprism's "folders "view and it's not there, it's not clear to the user why. This is particularly confusing since the file IS in both places on the disk, and you'd think a "folders" view would represent the filesystem as-is. Yet just ignoring the fact that you have duplicates, as Memories does, and leaving it you to manage them separately, also doesn't seem ideal. On point 2 - Of the above, only Memories writes changes to EXIF. This is mostly great for my purposes since I keep good backups to undo any unwanted changes, but Immich and Photoprism have good reason to be cautious about messing with original files. It seems to me there's a halfway measure that would work well with both. Here's what I'd love for file imports/organization: * Never try to adopt an existing file structure (like a bunch of folders from a Google Takeout) and simply use it in place. If the app is being pointed to one, use it for imports only, and copy everything over to the app's own folders/file organization. * Organize that *new* file structure however the developers like, without touching the originals. * During the import, offer to convert all the folders from the *original* file system (ie the Google Takeout folders) to *albums*, managed in the app's own database. If the user declines, import the photos in a flat structure without assigning them to albums. * During import, the app may notice a photo appearing in more than one of the original system's folders. If so, only copy it to the new file structure once, but assign it to *both* corresponding albums. This way, that birthday photo appears in both in the newly created "2009 favorites" and "30th birthday pics" albums without actually being in the new file structure twice. * If you're regarding anything other than exact file matches as duplicates, let the user see somewhere in your UI which files have been detected as duplicates and override the detection if necessary. * Don't expose a "folder view" to the user under normal circumstances. Optionally have it available as an advanced feature, disabled by default. The user should manage things as albums, and the app takes care of the underlying folder structure on its own, however, the developers find makes sense. And for metadata: * When importing files, check for EXIF, json, xmp and any other sidecar files. In your advanced settings, have an option letting the user specify which to prioritize if the existing files have incompatible metadata in multiple places (including an option for a prompt). Import than metadata into your database. * When updating metadata via the app, only write those changes to the database (or, optionally, the databased and a human-readable sidecar file, like Photoprism does). Do not, by default, write it directly back into the EXIF. * **Important**: If the database's metadata is different than what's currently in the EXIF, because the user has updated it in your app, give the user an **option** to write it to the EXIF as a batch operation. Warn the user these changes are irreversible before proceeding. Have a flag in the UI that shows the user when the EXIF data and database data are not the same, and allow the user to filter their view by it. * **Important**: When exporting a photo from your app, if the EXIF metadata and database data aren't the same, **by default** write the database data to the EXIF of the exported photo. Do not change the original photo stored in the app's file structure -- only make this change to the exported version. Have an option to disable this. That way, under normal circumstances, when someone exports a photo from your app, and imports it into a photo editor or other library, it'll carry over the changes they made in your app unless otherwise specified ... without risking the original files that remain untouched for safety. * Optional: Provide a command-line script to interpret your human- readable sidecar files and apply them to photos' EXIF data in batches using something like EXIFtool (I wish Photoprism had this already and I'm surprised I haven't found anything generated by a user to do it). ---------------------------------------------------------------------- autogyrophilia · 2023-11-13 16:51 What you want it's called a folder and a text editor. Apps have to do that to remain consistent . Otherwise the work gets duplicated exponentially for handling all types of work. ---------------------------------------------------------------------- Accomplished-Lack721 · 2023-11-13 16:52 Which part are you saying apps have to do to remain consistent? I layed out quite a few asks and suggested steps for how they'd work, so I'm not clear what part you're addressing. ---------------------------------------------------------------------- autogyrophilia · 2023-11-13 17:01 Every configurable option it's a bifurcation that needs to be changed. It means you have to go and change so much more with each update. I sincerely mean that a file share it's a more appropriate tool for what you want. Most of us like our photo organizers to organize ---------------------------------------------------------------------- Accomplished-Lack721 · 2023-11-13 17:13 I think you may be missing some of what I'm describing above. It's definitely not what can be done with a simple folder share. I *also* want my photo organizer web app to organize. And they all do, with different philosophies about how. I'm suggesting yet another organizational scheme that I think meets multiple needs at once. I'm suggesting a system that I think does two things well: * If you have an pre-existing file structure with photos sorted into folders -- as you might if you exported your photos from Google with Takeout -- import than into an *albums* setup. Recognize duplicates and don't double-import them into the new underlying file system (which would be hidden from the user typically), but still have them show up in multiple albums. This would, for instance, preserve the albums you had in a previous system like Google Photos if you had images that appeared in multiple albums. * Allow updates to metadata that are made in the photo management app to also be reflected in the exif of exported files, so that when you open them in something else (like Photoshop) you see the changes you made. But don't touch the original files by default, and only do so after warning the user of the consequences. What I'm looking for is *not* a straight view of the file system, which I could get with just a share. It's full-on management of a library, with albums and a cautious but user-friendly way of handling metadata that still allows exported files to have the updates a user reasonable expects they already made once opened in another app. And I'm still looking for all the other benefits of a web-based photo app, like the ability to share albums or images with other users. Of the apps I've used, Memories comes the closest to what I'm seeking, in that it can update EXIF directly, and that it allows a view of the original file structure. If it had an option to auto-convert pre- existing folders to albums and then hide its folders view, it would be even closer. So far, I'm unaware of any app that does it quite the way I'm describing, but I don't see any reason why one *couldn't* if the developers thought it was a good idea. Alas, I don't have anything like the skill needed to do it myself -- if I did, I'd be plugging away at it. ---------------------------------------------------------------------- lespasapp · 2023-11-14 02:05 There are continuous discussion on what's the best way to organize your photo collections. It's fair to say that none of them stands out as the predominant winner. My way (and of course Les Pas is implemented exactly as this is) looks like the following: Shoot as much as I can with my phone and digital camera. After the trip or event or simply a period of time like a month or so, revisit the photos, choose the best of them, then edit and put them into a new album in Les Pas. Then publish (means share in Les Pas's term) the album to others if I think they care. For those photos deem to be mediocre, they stay in the camera roll (which is nicely backup on the server) or in their respective folder organized in date. And eventually I will move them to an external hard drive and they stay offline. I have very bad experience in the past of hard drive broke down after short period of time because they are kept spinning all days. I currently have only 5000 photos in 30 albums online, and they come out of those tens of thousands which are now sleeping in the offline drive. I also have albums which won't end in a specific day, like the one documenting the growth of my kids, and the one collecting recipes footage. Whenever I shot a good one or found a piece of interesting stuff on the Internet, I add it to my collections respectively. Each member of my family follows these procedure. Whenever we need someone else help, we can publish the album as Joint Album, so that others can contribute by adding photos into it. If I need to show my works to someone who doesn't have account setup on my server, I can make a photo blog out of an existing album and Nextcloud would be happy to share it to the outside world. I believe organize photo in albums and the cleansing job in step 2 is a effective way to enhance my memory of those moments. I can locate a specific photo more quickly then using date/location/AI search among thousands of photos, even though Les Pas provide ways to do some of these search. It will take some extra time to do it, but you will get reward in the long run. ---------------------------------------------------------------------- botterway · 2023-11-14 07:42 If you haven't looked already, my app Damselfly might work. It uses the existing folder structure, displays all files (duplicates or otherwise) and has metadata editing and updating. You can run it alongside other apps, so you could use Damselfly *and* Photoprism, for example, and get the best of both. Album support is not really there, but it focuses on folders mainly (because that's my use case). I have it running on a Synology, with a 5tb collection of 670k photos, and it's fast. http://github.com/webreaper/Damselfly ---------------------------------------------------------------------- Accomplished-Lack721 · 2023-11-14 17:52 Thanks for the heads up on this project. It looks like it might work very well for some people who basically want a web app as a view right into a filestystem for dealing with folders. Unfortunately, it doesn't really meet the needs I'm laying out. The use case I'm describing is still one where the web app abstracts away the file system and uses albums. It just lays out a (smart, I think) way of recognizing and interpreting the organization in a pre-existing library, like one created from a Google Photos takeout, when bringing photos into its own system -- accounting for duplicates in albums without doubling them up on disks. Direct editing of EXIF is handy. Memories does that too, and it's part of why it's what I'm using. But my ideal situation would be one where the app only writes metadata changes to its own database *initially*, but then (optionally) applies it to EXIF when exporting/downloading files without touching the original files. And it would also give the user an option to apply metadata to EXIF for the original files, but only after first prompting with warnings. It seems your design goals are pretty different than any of that -- which isn't a criticism, as I'm sure it works well for the way a lot of people like to work (just not me). ---------------------------------------------------------------------- botterway · 2023-11-14 17:54 Feel free to raise issues describing these. I could always implement them. In particular, only writing metadata to the DB is something that would be trivial. ---------------------------------------------------------------------- jrasm91 · 2023-11-17 19:42 To be fair, the amount of text you had to write to explain your ideal solutions (at least to me) hints at the fact that photo management software is very complicated. Specifically, there are lots of use cases and choices to make about the design and implementation, all with their tradeoffs and complexities. Supporting all of them is quite hard to do, so it is no surprise to me this is the state of things right now. ​ (disclosure - core Immich maintainer) ---------------------------------------------------------------------- Accomplished-Lack721 · 2023-11-17 19:46 Absolutely! And I'm very grateful for the work you and the maintainers of the other projects like Memories put in. I'm very impressed with Immich, and while it doesn't work exactly the way it would if I were King of the World and could just make my priorities everyone else's priorities, it's a great piece of software. And I've been finding that between the immich cli and immich-go project, I can accomplish at least some of what I'd want for transitioning a pre-existing library. I've also seen the discussions on your github about using the album-flag-style logic for working with external libraries, and I'm interested to see where that goes. Thanks for all you do. ---------------------------------------------------------------------- jrasm91 · 2023-11-17 20:07 Thanks! Yeah, each release we're getting better. I think about it like this - you can have a bunch of amazing features if you're willing to make some compromises. Hopefully, with some more time, we can get rid of those to begin with, but Immich has only been around for 18 months, so it's still figuring things out. ---------------------------------------------------------------------- Accomplished-Lack721 · 2023-11-17 20:22 While we've got each other's "ears" - is there any known way I just haven't come across yet to apply metadata from your database to images files directly, even though Immich doesn't do it itself? Some script using exiftool a user cobbled together, or a way of exporting out database entries into a format some outside tool could use for that in batches? ---------------------------------------------------------------------- jrasm91 · 2023-11-17 20:43 Exif tool is probably the way to go. A few quick notes though: \- Writing exif can be destructive, especially when you try to write data to RAW files, it may corrupt them and there are lots of potential problems with certain file types, tags, etc. \- One major reason why everyone has so many duplicate copies of pictures is because when you update the original you essentially have a new file, indistinguishable from the original, this is why sidecars exist and why we prefer to use this approach instead \- There is actually a PR opened by a community member to add some edit support directly through the UI ([https://github.com/immich- app/immich/pull/5066](https://github.com/immich-app/immich/pull/5066)) \- Once we have a system in place for "writing", we plan to make it possible to automatically keep the database and sidecar file "in sync". Ie, writing facial data, tags, location, created date, etc. to the "original" via a sidecar \- In the future, I'm sure we could add the option to write to the original, it's just not really recommended because of the above- mentioned issues, but if people really want it, I assume it can be added afterwards. As far as a script goes now, nothing exists that I am aware of. Right now not that much data exists in Immich that doesn't come from the file directly. We have a few descriptions, albums, and facial recognition data. To put those back into exif you would need to query the database or API, and craft an exiftool write command. ---------------------------------------------------------------------- Accomplished-Lack721 · 2023-11-17 21:52 Writing to sidecars first (or only) makes sense. Where I hit a snag is when it's done the way Photoprism does, with its own human-readabke sidecars that (AFIAK) aren't in a format other software knows how to interpret. I think it would be particularly helpful to have either those sidecars available for download when a photo is downloaded from the UI, or to have EXIF info optionally written into those downloaded files, while leaving the versions in the system untouched. That way, info updated in Immich could be carried over to other apps when a file is exported/downloaded out, without risking the integrity of the original file. ---------------------------------------------------------------------- jrasm91 · 2023-11-18 00:35 Yup, this is pretty much the goal and the plan. ---------------------------------------------------------------------- Accomplished-Lack721 · 2023-11-18 00:44 Awesome. And thanks for taking the time to reply here. I know you folks get requests and suggestions in multiple places from lots of people with competing priorities, and I'm sure that's only gotten moreso as the project has grown. ---------------------------------------------------------------------- jrasm91 · 2023-11-18 00:49 I know, right. I used to respond to a lot more of the questions and feature requests, but it just keeps growing and my free time \_does not\_ haha