[HN Gopher] Cleaning up my 200GB iCloud with some JavaScript
___________________________________________________________________
Cleaning up my 200GB iCloud with some JavaScript
Author : amin
Score : 275 points
Date : 2024-01-04 06:36 UTC (1 days ago)
(HTM) web link (andykong.org)
(TXT) w3m dump (andykong.org)
| atlas_hugged wrote:
| Am I reading this right?
|
| If this is widespread, this could be seen as apple bloating
| figures to push people to upgrade, which could lead to a lawsuit,
| no?
|
| IANAL
| abhinavk wrote:
| If you edit a photo/video in file rather than saving as a new
| file, iOS retains the original file so that you can
| undo/revert. It doesn't show the original anywhere in Photos or
| iCloud gallery. That might be the reason.
| Joeri wrote:
| You can also edit with third party software, and they can
| embed whatever metadata they want to allow those edits to be
| modified, so there might not actually be an upper bound on
| how much larger you can make a photo or video by editing.
| mojo74 wrote:
| Funny, this was may take on the article too. I suppose one
| could download x amount of files from one cloud storage and
| upload it to another to see if there is any obvious
| discrepancy. Perhaps Apple are still using their old file
| system format for icloud? It's been around for a good few years
| now and perhaps they simply haven't bothered to ever change the
| type of storage format they use. Maybe those with older
| accounts have their images / videos on the 'older' drives?
| Nextgrid wrote:
| I don't believe iCloud is a filesystem, it's not block-level.
| It's object-level, like S3, with a bunch of OS-side jank to
| make it _look_ like a conventional directory and filesystem
| (which often fails miserably and dangerously).
|
| It's similar levels of reliability to an FTP account mounted
| with curlftpfs[1] - except the latter at least fails in
| understandable ways and can be debugged.
|
| [1]: https://news.ycombinator.com/item?id=9224
| mojo74 wrote:
| 'Failing miserably and dangerously' is the new 'Move fast
| and break things'
| realusername wrote:
| That's exactly my mental model of it as well, some janky
| home made s3 fixed with tape. I really advise anybody to
| save their data somewhere else as well before in inevitably
| stops working.
| isodev wrote:
| > this could be seen as apple bloating figures
|
| No, the OP doesn't take into account that adding media to
| Photos is not a simple "copy and store a file". The Photos app
| (like all apps) has its custom Photos Library file format. So
| when one ads a picture or a video, the Photos app analyses it
| and stores all kinds of metadata that's needed for the Photos
| app to work, including edit history and other bits. This is
| what eventually gets synced to iCloud.
| Havoc wrote:
| Good timing..I've been getting the same warning mails (though
| 50gb). Was planning to go through photos but checking vids first
| makes sense
| sgt wrote:
| Been getting the same warning emails (200GB in my case). So I
| tried this now, but it doesn't seem to highlight anything after
| pasting in the gist into Chrome. Does it work for you?
| Havoc wrote:
| >after pasting in the gist into Chrome.
|
| No idea what this means so can't tell you whether it works
| for me
| sgt wrote:
| The original article had a JavaScript snippet you post into
| the browser to do the highlighting for you.
| 369548684892826 wrote:
| If you have a Windows machine lying around, the iCloud app
| lets you download all your media out of iCloud Photos. Once
| it's all on Windows it's a lot easier to find the largest
| files!
| sgt wrote:
| You can also do that with Photos (leaving "optimize
| storage" off) and downloading all to your Mac, I believe. I
| haven't actually done yet that since it is quite a large
| download.
| cianmm wrote:
| Interesting article. I have my own problem, that caused me to
| have to upgrade my iCloud plan, and thus may not be a high
| priority fix for Apple.
|
| If you shoot RAW+JPEG (not a super-rare thing to do, for photo
| enthusiasts) then Apple Photos links the two images. Which is
| useful, rather than having a bunch of kinda-duplicates littering
| your library you can easily toggle between RAW and JPEG.
|
| But this combining, along with the file system design described
| in this article, makes it impossible (as far as I can tell,
| anyway) to easily separate them and delete the RAWs. So years
| later, I have HUGE RAW files that I'll never touch that I can't
| delete, because I want to keep the much smaller JPEGS.
|
| Any method that I've found to clean them up (exporting the
| originals, deleting them from the library, and then re-importing
| the JPEGs only seems easiest) will lose all of the years of
| metadata that I've built up in the library.
|
| So I have to upgrade.
| ylk wrote:
| Possibly stupid question: why can't the metadata be exported
| and imported? Is there other metadata aside from the exif data?
| Or does Apple not export all of it? And in case you're talking
| about additional features like face recognition, doesn't the
| app do that again once you imported the jpeg?
| Ayesh wrote:
| Im not in Apple ecosystem at all, so I don't have an answer
| to your question.
|
| But in RAW/JPG world, the metadata and edits is already a
| solved problem with Sidecar files + EXIF data. Sure, EXIF
| fields are kinda messy but I'm sure it's better than Apple
| has rolled by their own.
| icebergonfire wrote:
| Not in your situation at this point, but as a photo nerd I will
| eventually be so I took to Google.
|
| > Any method that I've found to clean them up (exporting the
| originals, deleting them from the library, and then re-
| importing the JPEGs only seems easiest) will lose all of the
| years of metadata that I've built up in the library.
|
| Apparently when you File/Export Unmodified Originals it will
| export the RAW+HEIC and a separate sidecar file containing the
| metadata. You can then move the RAW file away and import the
| HEIC file, which will autoimport the sidecar metadata file too.
|
| You lose edits though, although it seems you can "copy edits"
| somehow. Surely a technically inclined person can AppleScript
| their way through this...
|
| Yet it seems needlessly cumbersome and should be a built-in
| function in Photos.app, it's clearly not prioritized because it
| helps funnel people into higher iCloud tiers.
| maherbeg wrote:
| ugh yeah, currently in the same boat and haven't taken on the
| work to deal with this hassle. definitely looking for a
| solution!
| saagarjha wrote:
| I kind of hate opaque cloud solutions like this, because their
| storage usage is inscrutable and their interfaces are basically
| never optimized to help you clear things out. Whether that's
| intentional and malicious or just not a priority, I don't really
| care, I just want to be in control of the stuff I'm using.
|
| I have a very similar issue with iMessage in iCloud, and unlike
| photos, there's no web interface. You have to interact with it on
| your device. Thankfully I have a Mac so I can load up the chat
| database from there and see what's using the space, but cleaning
| it up has been a nightmare: I have a script to find the big
| attachments, but if I delete them then some agent that manages
| the database goes into a loop for like five minutes _per
| deletion_. Then the change gets synced to iCloud (at least, I
| assume). So unless I fully reverse what the deletion process does
| and whether it is possible to do a batch operation I'm basically
| at the mercy of the front end they provide to clean things up,
| which as I mentioned earlier is absolutely not designed to make
| it easy to do this.
| Nextgrid wrote:
| The opaqueness is definitely a problem. My iMessage database
| has a corrupted spot somewhere in the message history; if I
| accidentally make the Messages app read that spot (via
| searching or scrolling back through old conversations) it'll
| silently start maxing out the CPU and burning energy for no
| reason. There is no error message, nothing obviously wrong-
| looking, and as far as I know everything works.
| farconics wrote:
| Sounds like a nightmare. Do you know what could have caused
| the database corruption?
| Nextgrid wrote:
| Couple years of macOS/iOS reinstalls along with the typical
| shittiness of iCloud and the iMessage clients. I would be
| more surprised if it _wasn 't_ corrupt.
| o-o- wrote:
| Oh, so that's what's happening? You made me realise I have
| the same exact issue!
| Nextgrid wrote:
| Well I can't for sure prove it's indeed DB corruption, but
| I don't have many other theories.
|
| It's unlikely to be a "static" bug affecting everyone
| because it would've been caught in QA and/or user reports
| and fixed already (this is not a new issue, been happening
| for a year at least).
|
| Therefore my hypothesis is that it's dependent on corrupt
| persistent data that a relative minority of users have
| which is not easy for Apple to detect/replicate.
| css wrote:
| Hey, this sounds like an interesting problem. I am always
| looking for edge cases to test, if you have time would you
| mind checking if https://github.com/ReagentX/imessage-
| exporter works for you and if it crashes in that spot?
| the_gastropod wrote:
| Holy crap. I've been looking for ways to archive my
| imessages somehow. imessage-exporter looks like exactly
| what I wanted. What a cool project!
| css wrote:
| Cheers! It was fun to build.
| AnonC wrote:
| Search on the messages app is broken for me on iOS and macOS.
| It really can't find things that are there. So I have to scroll
| and look around. The sizes reported are also inaccurate. While
| iCloud will report some large number, when I dig into the
| largest messages section it doesn't show anything close to that
| size (with just a few items in that list).
|
| Since the macOS Messages app is some Catalyst abomination, it
| won't even support expected keyboard shortcuts for navigation,
| selection and deletion. Having to do everything using the
| trackpad is a slow process.
|
| It's frustrating that messages is so bad.
| jahnu wrote:
| Why are message apps like this? Tons of effort for almost
| useless stuff like stickers but super basic media management.
|
| Even in open source apps where in theory we can contribute
| functionality like Signal it's difficult. E.g. it's
| impractical to delete lots of media in groups in the iOS app.
| I added a PR last year to help this but still waiting for a
| merge even though it has been approved :(
|
| Edit: PR for the curious
|
| https://github.com/signalapp/Signal-iOS/pull/5685
| AnonC wrote:
| Though Telegram's chats are not end-to-end encrypted by
| default (and E2EE is not available for group chats), its
| storage management on device and on the cloud are quite
| good.
| cosentiyes wrote:
| I have the same problem on iMessage. iCloud reports 7+gb of
| message storage being consumed, but each of my devices and my
| top conversations only report 1-2gb of content. I haven't found
| a way to force sync the missing content so I'm stuck paying the
| $1/month plan.
| eertami wrote:
| > I haven't found a way to force sync the missing content so
| I'm stuck paying the $1/month plan.
|
| And this is why they (and many companies) behave this way,
| there's no incentive to fix a problem that you will just pay
| them more money to fix. Fixing the bug means reduced revenue.
| Complaining about the problem sadly won't help, you can only
| stop using the app and stop buying their products.
| judge2020 wrote:
| Not iCloud but Google makes it easy to drill down into how your
| storage quota is being used - https://one.google.com/storage
| more_corn wrote:
| The opaqueness is malicious. Apple offers a free iCloud account
| but no method of cleaning up space. Normal usage of a phone
| with all optional features disabled will eventually fill up the
| free tier space.
|
| The user's options are quit the ecosystem or pay up. Apple's
| lock-in monopoly means they get to shove their customers into
| another monthly fee.
|
| This is a classic example of the harm done to consumers by
| allowing anticompetitive practices. (The consumer has no choice
| but to pay)
|
| When the US government gets off their butts and investigates
| Apple for antitrust, this will be one of the key findings. The
| fine should be some multiple of the amount of money they have
| illegally extracted from their customers.
| noahjk wrote:
| > no web interface
|
| Same with HomeKit Secure Video (HKSV). The only way to access
| recordings is through the Home app. I'm surprised nobody has
| figured out a way to browse them outside of the app.
| dthakur wrote:
| Compression?
| Jleagle wrote:
| Must be bad compression if the files are getting larger?
| Snow_Falls wrote:
| Maybe adding a lot of metadata? I know AWS block storage adds
| an additional 4kb metadata file for everything you upload,
| maybe icloud doing something similar?
| cnicolaou wrote:
| Surprisingly, I've received a similar storage notice from Apple
| before the holidays and I decided to download all photos/videos
| into my own media server instead of hosting it on iCloud. There
| is not an easy, straight-forward way to download your archive
| from iCloud. I am slowly getting there using multiple machines
| and devices.
|
| The issue with the recent changes with Apple is that they
| increase the prices for no good reason. We're always going to
| take photos/videos and their sizes keep increasing with modern
| tech and capabilities.
| sgloutnikov wrote:
| What about the Photos app on MacOS if you have one? I keep a
| local copy of my iCloud Photo Library and sync it through the
| MacOS Photos app.
| chrischen wrote:
| Photos app for offline storage is also woefully neglected by
| Apple. They don't really design it for use with an external
| drive (you are expected to locate your library on a internal
| storage or permanently attached storage). The reason is that
| the photo library constantly gets corrupted on external
| drives, forcing a long rebuild/repair process.
|
| Often the photos app doesn't even detect the iphone even when
| plugged in, and it's a serious bug that Apple has neglected
| for years.
| geekifier wrote:
| Everything Apple does seems to be designed to drive
| hardware sales. Why support external drives, when you can
| be up-sold for larger internal storage (at a huge markup)?
| The "Photo Library" could simply be a database file with
| references to photo locations, alas that might confuse
| Mac/iOS user with "files" vs "photos".
| Nextgrid wrote:
| > What about the Photos app on MacOS if you have one?
|
| If you ever try a large-scale import/export into macOS Photos
| be prepared for 100% CPU, endless spinner cursors and the
| process ending up killed because it ran out of memory.
| msh wrote:
| There is an easy way to get your Data. It's just kind of
| hidden.
|
| It's works the same way as Google Takeout. To get to it do
| this:
|
| Sign in to your Apple ID account page at appleid.apple.com on a
| Mac, iPhone, iPad or PC. Go to "Data & Privacy" and select
| "Manage Your Data and Privacy." On the following page, go to
| "Get a copy of your data" and select "Get started."
| asdaq1312512 wrote:
| privacy.apple.com is a neat shortcut, it's the first option
| there.
| mmh0000 wrote:
| >> There is not an easy, straight-forward way to download your
| archive from iCloud.
|
| Let me revolutionize your life:
|
| https://github.com/icloud-photos-downloader/icloud_photos_do...
| otterpro wrote:
| Thanks! That tool is what I was looking for. I think you
| saved me hours of frustration.
| leipert wrote:
| AFAIK, it doesn't support shared iCloud libraries but
| https://github.com/steilerDev/icloud-photos-sync does.
|
| Edit: Seems like it has gotten support since I've last
| looked: https://github.com/icloud-photos-
| downloader/icloud_photos_do...
| Mister_Snuggles wrote:
| > There is not an easy, straight-forward way to download your
| archive from iCloud.
|
| I use PhotoSync[0] to copy photos from my iPhone to a NAS. It's
| an excellent program.
|
| It will even download photos from iCloud as needed and can do
| format conversions. I run it every few days to push new photos
| to my NAS so that I've always got a local copy (these also get
| backed up to Backblaze B2 nightly). The format conversion lets
| me keep HEIC+JPG pairs of my photos, so I have the original and
| something that's more readily usable.
|
| What I really want is something that will do the same thing,
| but with iCloud Drive. I keep a bunch of stuff in there and it
| bothers me that I don't have a reasonable way to back it up.
| Apple's recommended methods[1] leave a lot to be desired.
|
| [0] https://www.photosync-app.com/home
|
| [1] https://support.apple.com/en-us/HT204055
| landswipe wrote:
| It's called a bug 'turning a blind eye'.. ie. a scam.
| wouldbecouldbe wrote:
| It would make sense if they save the video in a few formats for
| being able to load them fast in the Apple apps, most video hosts
| do the same.
| darkstar_16 wrote:
| But that shouldn't count in the user's storage quota.
| mft_ wrote:
| Okay, so it could be a bug, but it is possible that iCloud is
| secretly storing more than one version of the file in some cases?
| (After all, Apple does similar things with other media files.)
|
| The example given at the end is interesting:
|
| > So iCloud says the video is 128MB, I download it and the video
| is actually 48MB, and my free storage increases by ~170MB when I
| deleted it. Interesting!
|
| This suggests that iCloud isn't simply misrepresenting the size
| of the example file, as then you'd expect that deleting the 128MB
| file would clear ~128MB of iCloud space. Instead, the deletion
| clears _roughly_ the space it reports (128MB) _plus_ the space of
| the downloaded version (48MB): 128MB + 48MB = 176 MB - which
| might be close enough, allowing for rounding errors, as iCloud
| reports the free space (from the article 's example) to the
| nearest 10 MB.
| Nextgrid wrote:
| That's probably indeed the case, but it doesn't excuse it.
|
| When you buy a hard drive or USB stick, you get a certain
| amount of GBs to use as you please. If you put a 1GB file on it
| your free space decreases by 1GB (yes this is filesystem
| dependent and you might lose a few KBs for metadata, but the
| choice of filesystem is up to you and not mandated by the
| storage decide). It doesn't matter that the NAND controller
| probably used a few megabytes of the overprovisioned area to
| store its block mapping tables, or maybe even duplicated your
| data for its convenience - you were never charged for that
| overprovisioned area.
|
| Here, you are sold a storage device (that you access over HTTP
| instead of SATA/PCIe), but when you write a 1GB file, they
| duplicate/convert/etc it for _their_ convenience yet still
| charge _you_ to store those duplicates you haven 't asked for.
| That's new and unexpected.
| Dalewyn wrote:
| >That's new and unexpected.
|
| For you (and me) perhaps, but not for most people[1] who
| _appreciate_ having the file system abstracted away.
|
| [1]:
| https://news.slashdot.org/story/21/09/27/2032200/students-
| do...
| tcfhgj wrote:
| Don't get the complaints, you can use different messaging
| apps, that don't do that to your oCloud
| Machado117 wrote:
| On iPhone, the edits to pictures and videos are just metadata
| and the original file is kept. I just tested recording a video,
| cutting it in half and downloading the file from iCloud. The
| cut file it's smaller than the iCloud reported size however if
| I choose to download the unmodified original the size matches
| the iCloud reported size.
| isodev wrote:
| I think the confusion here is also rooted at the fact that
| photos and videos in the Photos library are not simply files
| copied to storage. Photos also stores metadata about edits,
| renditions and thumbnails and other data needed for various
| functions of the Photos app.
|
| So when you're syncing photos to iCloud, it's not just the
| individual files that get synced but it's the "Photos Library"
| managed container of the Photos app.
|
| If you add individual files directly in Finder or the Files app
| then their size matches exactly both in iCloud and on the local
| file system.
| frostix wrote:
| Differential backups or any sort of versioning seemed like one
| of the most obvious culprits (that and or total redundant
| storage to preserve the file) but the issue with all of this is
| it's entirely opaque.
|
| Ultimately you're increasingly tethered to some service for
| your storage that you pay for periodically based on total
| storage yet you have little-to-no information how to best
| optimize that storage if you want to operate in a fixed cost
| bracket or lower storage/cost ratio. So as a consumer, do I
| just wave my hands and keep throwing more and more money at the
| problem, especially now that devices are increasingly pushing
| everything, including storage, as a subscription service to
| meet my actual functional needs (that realistically could be
| met by local storage options if manufacturers didn't have a
| vested interest in pushing me towards service based storage
| solutions)?
|
| The modern business strategy in technology is simply hiding
| behind complexity. The cost is too complex for you to
| understand, it gives too much information away about our
| internals to competitors, and so on. Yet somehow these metrics
| are derived to assure the business is operating above cost
| because when the rubber meets the road it must be done, yet
| when the consumer wants to understand it's suddenly too
| complex. The problem is that tech in many cases is growing to
| scales that really is too complex and business managers know
| this, so it's often a valid excuse to hide behind. Conveniently
| that's where they focus on investment and padding margins
| though.
| Dalewyn wrote:
| >So as a consumer, do I just wave my hands and keep throwing
| more and more money at the problem,
|
| Yes.
|
| I can go and buy 1TB of Microsoft OneDrive or 2TB of Google
| Drive for less than a Franklin a year, and most people won't
| even need 1TB let alone 2TB. Both Microsoft and Google also
| offer 100GB plans for a Jackson a year, which is what I
| purchase myself. The average person can get by paying a
| Washington per month to Apple for 50GB of iCloud.
|
| The amount of money I would save from managing photos myself
| locally isn't worth the time spent nor the money spent on the
| hardware.
|
| EDIT:
|
| For the downvoters, consider this: If I were to manage all
| this myself, I would need at least three storage mediums with
| one being a different form factor to satisfy the 3-2-1 backup
| scheme. I would also need to procure arrangements for that
| third backup copy in the 3-2-1 scheme. And I would need to
| spend time managing it all.
|
| That is going to cost me more than a Franklin per year. Life
| is short, my time is precious, and my money is ultimately
| expendable.
| chrismeller wrote:
| This is the same argument behind paying for a streaming
| service. Could I "find" everything I want to watch
| somewhere else and maintain it myself on a Plex server?
| Sure, but the cost-benefit analysis just doesn't make sense
| to me.
|
| Particularly the older I get the more I value my finite
| free time. Throwing $20 at something to remove a problem
| that would take me hours (not to mention a large startup
| cost) to do myself is just an obvious choice.
| ebiester wrote:
| Now, consider a world where you are making $42,000 a year.
| That's 20 dollars an hour and a very common wage. How do
| they handle the same when there are so many competing
| Franklins for them?
| naltroc wrote:
| bump this, and the implicit vendor lockin that this
| ideology creates
| a_vanderbilt wrote:
| Hackers don't understand something that regular people more
| readily do: It works, it's cheap, and I'm paying for the
| convenience. We can all have our ideals about how tech
| should be, but these choices are driven by practicality.
| mysteria wrote:
| As someone who has a Proxmox cluster at home (storage on
| RAIDZ, hot backups with PBS, cold backups on external HDDs)
| I literally recommend cloud storage for most people that
| ask me about backup solutions as it's _simply not worth it_
| for the average user. Those people already have all their
| data in the cloud anyways and share it on Facebook et al
| and they don 't really care about the privacy side.
|
| Remember it's not just buying the equipment, it's
| maintaining and understand it as well (e.g. I have to be
| familar with how ZFS works, how to restore a failed node,
| write some scripts, etc.). And with every backup solution
| you also need to be familar with the restoration process
| and test it occassionally to make sure it actually works as
| expected.
| plagiarist wrote:
| Another danger of doing it yourself I have found is that
| if you give a dev a Proxmox, they are going to play
| around making toy Kubernetes setups instead of
| implementing the backup system they intended to make.
| nytesky wrote:
| Yeah I have always backed up to external drives and had
| cloud storage. But now I learn there isn't rot, so I have
| to either build a RAID and have refresh software that
| rewrites and validates, or just buys a new drive every 3
| years and backup all over again. And that's a simple
| setup.
| chaxor wrote:
| "640KB is more than anyone will ever need"
|
| It's a little absurd to think people don't need more than
| 2TB - especially on HN. Gamers will likely have 2TB in
| games alone, videographers often have many TBs of videos
| and photos from weddings and events in their life, many
| that care about health may have a few TB in genomic data
| mirrored on their computers to analyze, etc.
|
| I would imagine it's hard to find people that _wouldn 't_
| have TBs of data, if they were allowed to do so. The reason
| many people don't have TBs of data is they're limited by
| these exact companies you're claiming 'solve the problem'
| by offering limited storage.
|
| It is notable however, that having better tools to
| organize, deduplicate, and compress data would be helpful
| to reduce some of the size of data that many people have.
| Over the years I've noticed my family will have multiple
| tar.gz archives, zip archives, etc, which (after
| extraction/unencryption) will share 20% files here, 10%
| files there, a 4kb jpg that's the same as a 100MB PNG here
| and there, etc. So yes, those 10TB archives may end up
| being 5TB if someone spent the time to really comb over,
| understand, make good decisions, and organize that data.
| But I have not yet seen anything that can scratch that
| surface yet, other than perhaps
| https://github.com/jjuliano/aifiles - but I won't use it
| until it's local only and has guarantees not to destroy
| data without explicit permission. An overlay filesystem
| that shows compression/deduplication with LLM capability
| like aifiles is probably the best option here.
|
| However, I wouldn't imagine that most people's life data is
| less than 2TB even with all of this - it's mostly imposed
| as an artificial constraint by these companies.
| vel0city wrote:
| > Gamers will likely have 2TB in games alone
|
| If I were to install all the games on my steam account it
| would be many hundreds of terabytes of total storage. In
| the end I have about a terabyte of games on my computer.
| And of that 0 bytes are in my cloud storage.
|
| I've been an amateur photographer for over 15 years. I
| tend to curate the photos I keep, largely because I don't
| need 20+ pictures of the same scene. Its more of a burden
| to casually flip through my photos if the majority of
| them are near duplicates. In the end my total collection
| is only several hundred gigs.
|
| Most people aren't videographers.
|
| Most people in my family have far less than even 50 gigs
| of actual data they care about. They maybe take a dozen
| compressed photos a week, maybe 30 minutes of videos a
| month. A lot of my friends take even fewer photos and
| pictures.
| nirvdrum wrote:
| > For the downvoters, consider this: If I were to manage
| all this myself, I would need at least three storage
| mediums with one being a different form factor to satisfy
| the 3-2-1 backup scheme. I would also need to procure
| arrangements for that third backup copy in the 3-2-1
| scheme. And I would need to spend time managing it all.
|
| > That is going to cost me more than a Franklin per year.
| Life is short, my time is precious, and my money is
| ultimately expendable.
|
| When you're looking at cloud services, you need to perform
| your own off-site backup. Apple, Google, Microsoft, etc.
| will maintain copies that they'll restore in the event of a
| hardware failure. But, if your account gets compromised or
| a buggy sync or bad API event happens, your data is gone.
| They're not going to go restore it from tape for you. This
| is a big part of why I do have an in-home NAS. Maybe you
| have everything sync'd with a laptop and that has you
| covered, but Apple's expanded storage options are
| outlandishly expensive so I doubt many with the 2TB+ plans
| are able to do that. (Yes, you could use external storage,
| but that's also rather inconvenient for a Photos.app
| library.)
|
| We could both get what we want if these storage operations
| weren't wrapped up in proprietary APIs. If I use iCloud I
| get a seamless experience on macOS, but no access at all on
| Linux. If I use Dropbox I get access on Linux, but little
| more than photo sync on an iPhone. Given the decades of
| precedent with filesystems and I/O APIs, I suspect we could
| have an abstraction layer and an implementation layer that
| would allow for interoperability. Anyone that wants to pay
| for iCloud are free to do so, others could use their
| preferred storage engine. But, allowing access into the
| walled garden is far less profitable.
|
| For most people, storage needs are going to increase over
| time (more + higher resolution photos & videos, larger
| apps, document storage, etc.). 6TB for a family is not
| unreasonable and that's what? Three Franklins and three
| Jacksons per year + whatever for an external drive for your
| offsite backups. What comes after the 6TB option? Storage
| costs have decreased drastically over time, unless you're
| using a proprietary service; consumers are not benefiting
| at all from those gains in efficiency.
| Terretta wrote:
| > What comes after the 6TB option?
|
| Well, 12TB.
|
| And if you are head of household and share storage, you
| can combine storage plans. Mine currently shows "2.3TB of
| 14TB used".
| nirvdrum wrote:
| Thanks. I overlooked the 12 TB. I'm not sure doubling the
| capacity and doubling the cost is really ideal for many,
| but it's nice to know it's there.
| ValentineC wrote:
| > _Three Franklins and three Jacksons per year + whatever
| for an external drive for your offsite backups._
|
| I know you're just following the theme set by the parent
| commenter, but there are a bunch of us folk on HN who
| aren't US residents, and have no idea how much those
| presidents mean in terms of currency.
| dblangford wrote:
| A Franklin is more or less two Turings and a Jackson is
| just less than a Turner.
| nirvdrum wrote:
| I'm sorry. I use the currency and had to think about what
| the values were. I was trying to follow the theme by the
| previous author, but I can definitely see how that'd be
| hard for others to follow. It's $360 USD (Franklin = $100
| USD, Jackson = $20 USD).
| astura wrote:
| In addition to being confusing for non-Americans, it also
| confusing for Americans because $100 bills in American
| slang are "Benjamins," not "Franklins." It's most notable
| use is in the song "It's All About the Benjamins"
| Dalewyn wrote:
| I've heard it both ways myself; I like Franklin better
| since the others are all last/family names too.
| Dalewyn wrote:
| >6TB for a family is not unreasonable and that's what?
|
| Microsoft in particular has a 6TB for $100/year family
| plan, sharable with up to 5 other family members for a
| total of 6 persons each with 1TB. Google's plans can all
| also be shared with up to 5 other family members, though
| their bytes-per-dollar can't compete with that particular
| Microsoft family plan.
|
| Basically: Local storage with personal management needs
| to be _very_ easy, cheap, and carefree (which it isn 't)
| to compete practically with cloud storage.
|
| The only exception is if one's needs are niche and
| specific. I actually have a Synology NAS at home that I
| keep most of my data on, but that's because my data is
| mostly "bottle of rum" and "Linux ISO" in nature and thus
| not something I can throw on cloud storage in the first
| place.
| Terretta wrote:
| Presidents on US currency: - $100,000:
| Wilson - $1,000: Cleveland - $500: McKinley
| - $100: Franklin* - $50: Grant - $20: Jackson
| - $10: Hamilton* - $5: Lincoln - $2: Jefferson
| - $1: Washington * not a president
|
| > _I can go and buy 1TB of Microsoft OneDrive or 2TB of
| Google Drive for less than $100 a year, and most people won
| 't even need 1TB let alone 2TB. Both Microsoft and Google
| also offer 100GB plans for a $20 a year, which is what I
| purchase myself. The average person can get by paying $1
| per month to Apple for 50GB of iCloud._
|
| While we're at it, iCloud+ offers these monthly storage
| plans now: United States: 50GB: $1
| 200GB: $3 2TB: $10 6TB: $30 12TB: $60
|
| See everywhere in the world here:
|
| https://support.apple.com/en-us/HT201238
| nobodyandproud wrote:
| Partly you're getting downvoted because you're not
| accounting for the value of the content to the user. Even
| if the chance of a cloud provider cancelling an account or
| deleting content is 0.0001%, the value of a single photo to
| Microsoft will be magnitudes less than to an individual.
|
| I imagine the other reason is because they're not mutually
| exclusive: For instance, Synology makes it easy to have
| both an in-home NAS and cloud sync.
| briffle wrote:
| You still need to manage backups. You're trusting
| everything to the vendor. I run a docker image weekly that
| pulls my google photos, copies to my USB drive in my pi,
| and also copies to backblaze b2.
| LUmBULtERA wrote:
| That's an interesting bug/problem(?) he ran into.. Though, on the
| topic of reducing storage usage, in the Photos iOS app, you can
| remove photo duplicates under "Albums" at the very bottom -- if
| you have duplicates that it recognizes, there will be a line item
| for this, otherwise the line item will disappear.
| captn3m0 wrote:
| The thing I detest the most is the weird tier that jumps from
| 200GB to 2TB. There's no way to pay for incremental storage for
| 500GB or 1TB, which would be perfect for many families.
|
| In today's age where storage is a commodity, this ought to be
| priced per GB used.
| baz00 wrote:
| You can chuck as many 50GB increments on your 200GB if you need
| it.
| Alifatisk wrote:
| Really? How? I can only jump from 50 GB to 200 GB when I look
| at the available plans
| baz00 wrote:
| Actually I may be wrong there. I was on 200Gb with Apply
| Family but have stuck 50GB on top of that for PS0.99 month.
|
| I can only swap that out for a 200GB increment. This is
| weird because I'm sure you used to be able to do that.
| Alifatisk wrote:
| Oh okey, I knew that sounded too good to be true.
| captn3m0 wrote:
| I didn't even get an option for 200GB increment.
| Alifatisk wrote:
| That's weird, so you can only go from 5 GB -> 50 GB -> 2
| TB?
| captn3m0 wrote:
| Sorry, I meant I never got any additional increment
| options, even when I was at 200GB.
|
| So I jumped 5, 50, 200, 2000.
| Alifatisk wrote:
| Right, it's the same here. The poster clarified what they
| actually experienced.
| treesknees wrote:
| You can stack the storage of your iCloud subscription on
| top of your Apple One subscription, once.
|
| >After you subscribe to Apple One, you can buy more
| iCloud storage if you need more. With both Apple One and
| an iCloud+ plan, you can have up to 14TB of total iCloud
| storage.
|
| https://support.apple.com/en-us/108104
| chaostheory wrote:
| That way they can subsidize the free tier
| fermentation wrote:
| The free tier is 5gb which nearly grows on trees these days
| InsomniacL wrote:
| I'm pretty sure that iCloud stores the full version and downloads
| a lower quality optimised version to your phone.
|
| If you open iPhone settings and browse to: >
| Apple ID > iCloud > Photos
|
| There is an option to 'Optimise iPhone Storage' which is enabled
| by default. This states: > If your phone is low
| on space, full resolution photos and videos are automatically
| replaced with smaller device sized versions. Full-Resolution
| versions can be downloaded from iCloud at any time.
|
| This seems perfectly reasonable to me.
| Nextgrid wrote:
| Downloads should still return the original file though, so that
| can't be used to explain the storage size discrepancy.
| jonathanlydall wrote:
| There are two kinds of download options on the iCloud
| website, the "full quality" (e.g. HEIC file for photos) or a
| "quick" / "small" download which is a JPG.
| sliq wrote:
| Can confirm, have some ideas: Apple uses different file formats
| "internally" and also different resolutions, maybe saving them
| all in parallel as "one" video, which might cause this issue. So
| the total space per image is actually the sum of original .mov,
| optimized .mp4 and thumbnail image.
|
| Also, (automated) version history might have caused this! Maybe
| (automatic) image optimiztion saved old and new versions of
| everything.
| sccxy wrote:
| iCloud backup is real mess.
|
| Free 5GB is not enough for backing up iPhone system data any
| more.
|
| Most insane is that if you remove some apps from backup, then it
| iCloud usage goes from 4.5GB to 4.6GB.
| sandreas wrote:
| That's one of the reasons I self-host. I just don't trust the
| cloud providers regarding clarity and transparency... even if my
| self-hosted solution was far less reliable, less secure and less
| performant (I imagine it's not ;), I probably wouldn't change.
|
| I personally use immich[1], a very complete solution with iOS /
| Android App, Server-Component and Sync / Backup option.
|
| [1]: https://immich.app/
| Alifatisk wrote:
| It looks interesting but the warning at the top tells me I
| should wait for it to become more stable?
| rscrawfo wrote:
| I would if you want to use it as backup. But if you just use
| it to display photos and have another process for sync it's
| absolutely fantastic!
|
| To me it's better than any other interface I've tried.
| Commercial or home lab.
| sandreas wrote:
| Yeah, that's what I thought... but using it for 6 months now
| they have been pretty reliable regarding upgrades and new
| features.
|
| The features I appreciate the most are: -
| Auto-Synchronisation on Android AND iOS (this is hard to find
| in any other app) - Photo sharing (Accounts can have
| partners to share all their photos with - ideal for me and my
| wife) - Deduplication - The Web Interface
|
| The feature I miss the most is tags[1].
|
| Once the app stopped working but there was a clear message on
| the repository / homepage that the server has to be upgraded.
| Since it is docker based, it was very easy to upgrade without
| losing any data. Same applies for backup...
|
| So if you ask me, there is no need to worry - but I would not
| use it as only option to store my photos and it does not
| replace an (off-site)-backup.
|
| [1]: https://github.com/immich-
| app/immich/discussions/1651#discus...
| brlewis wrote:
| I think Apple just hasn't realized yet what a great sales
| opportunity a UI for quota management is. There's no better
| place to sell more storage. That's what Google does. See
| https://one.google.com/storage and
| https://photos.google.com/quotamanagement which work well for
| finding stuff to delete, in the hopes that eventually you'll
| tire of deleting things and go ahead and buy more storage.
| alephnan wrote:
| Don't you remember how painful it was to import/export an mp3
| file into an IPod?
|
| It was especially painful on a Windows PC.
|
| The biggest selling point of other mp3 players for me was the
| ability to transparently copy and paste files into the
| filesystem.
| Alifatisk wrote:
| I've tried iCloud, Onedrive, Google Photos. Out of all these
| three, I stuck with Google Photos on my Iphone.
|
| iClouds bad pricing and storage issues led me to switch to Google
| Photos, It's way better. The lack of native support on iOS can be
| a little cumbersome though.
|
| The only thing I wish these cloud providers offered was a way to
| deduplicate photos / videos. That would make my life so much
| easier.
| Liftyee wrote:
| > The only thing I wish these cloud providers offered was a way
| to deduplicate photos / videos.
|
| Same here ... but of course that would make it easier for
| customers to not have to pay more, so sadly don't think it's
| ever happening.
| Alifatisk wrote:
| What makes it worse is even if you run a deduplication app on
| the folder your syncing to Google Photos, the synchronization
| is one way only, so the changes does not remove the
| duplicated items on Google Photos.
| ezfe wrote:
| iCloud Photos has deduplication
| nytesky wrote:
| Yeah I ran it and shrunk my library by half. It was kinda
| scary since I had 100000 photos, I couldn't really inspect
| the process
| treesknees wrote:
| I've used an app called PhotoSweeper[1]. It can access your
| Photos library directly. You can manually review them side-
| by-side or let it run automatically. You can also set
| thresholds of % similarity to reduce false removals. I
| believe it can also move the duplicates into a separate
| album for review later.
|
| https://apps.apple.com/us/app/photosweeper/id463362050?mt=1
| 2
| lovegrenoble wrote:
| It's better to self-host
| asdaq1312512 wrote:
| Photos.app doesn't show file size... so I thought, why not build
| a little Photos.app extension or a separate app that queries for
| large files?
|
| Turns out that the API doesn't expose "file size", at least I
| didn't find a straight-forward way.
|
| I _think_ that all "photos" or "videos" are just a _view_ of the
| underlying "photo or video object". If you crop a video, the
| full-size video will remain. Only if you export the video, it
| will be cropped and the smaller file size will manifest.
|
| I guess that's why the file sizes differ.
|
| [Edit: someone created an AppleScript to query file sizes - I
| didn't test it, yet:
| https://discussions.apple.com/docs/DOC-250000422 ]
| naltroc wrote:
| this is the best plausible reason for why the file size
| differs. I've also noticed some unepxected media restoration
| (thought I cropped/edited a video, it is still there in full
| length and resolution).
|
| This also helps explain why my iPhone storage always seems to
| be at its limits, despite my obsessive management.
|
| Anyway, we all know foss unix ftw :P
| GeekyBear wrote:
| > thought I cropped/edited a video, it is still there in full
| length and resolution
|
| It's been possible to create a clip from a video file that
| merely changes what parts of the video are displayed without
| effecting the data in the original since the Classic Mac OS
| days.
|
| If you want to completely remove unwanted portions of a video
| to reduce the size without a loss of quality, there are many
| options. LosslessCut is a cross platform option that is both
| free and open source.
|
| https://github.com/mifi/lossless-cut
| alexwlchan wrote:
| > I think that all "photos" or "videos" are just a view of the
| underlying "photo or video object". If you crop a video, the
| full-size video will remain. Only if you export the video, it
| will be cropped and the smaller file size will manifest.
|
| Yup, the Photos app keeps the unmodified original file, and
| then any edits/crops are stored separately. You can always
| revert to the original file and redo your edits. So they might
| be storing multiple copies of the same image, with and without
| edits.
|
| Which API were you looking at for "file size"?
|
| I was able to get the size data from Photos.app with the
| PhotoKit API [1]. I've only tested it with my library of ~26k
| items, but it was useful for getting an indicator of the
| biggest items. (Although I didn't think to check whether
| exporting a 1GB video caused my iCloud usage to drop by 1GB.)
|
| [1]: https://alexwlchan.net/2023/finding-big-photos/
| asdaq1312512 wrote:
| Ahh, I did consider PHAsset.fetchAssets but my understanding
| was that the method will download the file if not present
| locally - which wouldn't be acceptable for an app, I guess.
|
| Do you know more? The introduction says "Retrieve asset
| metadata or request full asset content.", but I can't find
| clarification when it actually accesses full content.
| Groxx wrote:
| Since you're willing to write since code apparently: have you
| checked the internal databases? They're just sqlite, moderately
| understandable at a glance last time I looked.
| w-ll wrote:
| Side note, been a while sense i looked, is there a sqlite db
| for the contacts table?
| sbr464 wrote:
| Yes
| asdaq1312512 wrote:
| Looks promising - will look into it!
| asdaq1312512 wrote:
| So it appears you can conveniently query the DB for
| filename (+directory), UUID, and (original) file size. But
| I can't find a good way of opening a Photo by filename or
| UUID.
|
| After all, the file might not be present locally, so
| opening it should go through the Photos.app. But once you
| call AppleScript to open the file, you might as well use
| the AppleScript to comb through the database like the
| script I linked to earlier.
| iscrewyou wrote:
| You might know this already but the Photos
| catalog/photiolibrary is just a folder. You can actually right
| click and "show package contents". From there, you can use
| something like Daisy Disk to display all the files based on
| their size. A simple drag and drop has worked for me. Once you
| find the large file, you can search for the name in the photos
| app. I've deleted some quite large videos this way to clear up
| some space.
| znpy wrote:
| I'm being sarcastic here, but the solution could be quite simple:
| iCloud is billing you (both in terms of storage size and money)
| for both your data and the redundant copies.
|
| Other companies would build the redundancy costs into the final
| pricing, but Apple is known to "think different".
| Havoc wrote:
| Just discovered you can see file sizes for some items under >
| icloud > recommended for you
|
| Finds duplicates, screenshots and big vids. Had a random vid that
| was 1.8gb for reasons unknown
|
| Doesn't always seem to be a accessible though
| aledalgrande wrote:
| In terms of iCloud cleanup, has anyone found a way to clean up
| iMessage attachments > x years old, greater than n MB or from
| certain contacts? It would clean up so much space for me. Apple
| only clears > 1 year old from the settings.
| tugberkk wrote:
| Apart from the iCloud fiasco, I very much like these kind of
| programming. We generally lost the fun of it try to make
| enterprise/corporate programs, we forgot to use them for personal
| tasks.
| FaridIO wrote:
| Additionally, if you take a 4MB photo and add it to a shared
| album with 5 people, those 5 people all have to use up 4MB from
| their storage. Instead of reusing the same image and having
| pointers to it, it looks like it's "copied".
| carleton wrote:
| That doesn't make sense to me. Apple says here that photos in a
| shared album do not count against your iCloud storage.
|
| https://support.apple.com/en-us/HT202786
| arthurmorgan wrote:
| Shared Albums don't count against your iCloud Storage.
| seffignoz wrote:
| Thanks for the idea/solution!
|
| I converted this into a TamperMonkey/Greasemonkey script. Also
| added a feature to "hide" all elements that do not match the
| threshold.
|
| https://github.com/seffignoz/icloudcleanup
| NDizzle wrote:
| With the iCloud desktop app for windows I'm able to sort by
| size...
| crossroadsguy wrote:
| There's will be a time when there will be many open (or
| relatively more open) devices like FairPhone et cetera; and OSes
| that do not track users and suffocate you with dark patterns and
| hostility all around - like Graphene and Lineage; and hopefully
| they'd all be able to join their hands together and not
| reimplement the wheels of N types for the 1000th time all
| incompatible with each other. Hope is that there will be services
| and softwares on top of that spec and API of these OSes that will
| be the user's choice to pick - for contact, for calendar, SMS,
| IM, call , media, sync, backup, device account management etc.
|
| And such devices will be widely available, also in non-first
| world countries, with proper OEM warranties and support. Hell, a
| local manufacturer can just build for that spec.
|
| I know it's like a wishful dream. But if this happens, and when
| this happens, I hope a lot of us will be able to breathe free and
| hopefully would be candidly able to shit upon legacies of those
| so called fucking visionaries, who were barely not subhuman and
| were just rather pathetic jerks, who ensured such shit-show of
| walled gardens and opaquely implemented utterly inferior systems
| where users go and get stuck in one of the two houses of the
| duopoly to experience a lot of shitty things including, but not
| limited to, some variation of Stockholm syndrome and clear and
| conscious apologism.
|
| Why? Because neither of the two is acceptably good and they have
| become so big and they have closed it down so much that nobody
| else can even make a dent even if they try. And they try!
|
| So yeah, until then I will rant and feel shitty about both my
| iPhone 14 (as they call it - my "daily driver") and Pixel 5a (my
| bread and butter phone; form factor wise less shitty one though).
| sneak wrote:
| Apple interested in maximizing recurring subscription revenue?
|
| You don't say. I'm sure this is just a harmless bug and not the
| source of $20M extra revenue per year across their hundreds of
| millions of active iCloud users.
| tamiral wrote:
| there is something weird happening on iCloud, my phone storage is
| off even though the total size of data isnt as big as it says, i
| throw a lot of .zip/rar files on icloud and it seems like the
| data on icloud is more than it is on my HDD....
| 867-5309 wrote:
| some possible reasons to investigate:
|
| 1. older videos have a larger file size because they are encoded
| less efficiently (h264 vs h265)
|
| 2. gigabytes vs gibibytes (like how a 500GB drive formats to ~465
| GiB)
|
| 3. rounding errors
|
| 4. Apple fucking its customers
|
| 5. combo of the above
| kossTKR wrote:
| Tangent but does anyone else find iCloud sync random / slow?
|
| The smart thing about having both an iPhone and a Macbook should
| potentially be; snap a bunch of photos, instantly have them
| available on the computer - but no. Apple apparently chooses a
| random time depending on 100 factors to upload the photos in the
| next 30 minutes to 7 days.
|
| So you often have to airdrop a bunch of photos files completely
| invalidating the purpose of the sync function.
| SirMaster wrote:
| Can't you just use the Apple Shortcuts app to do a lot of this?
|
| Pretty sure that can enumerate and query things like the filesize
| of the photos and videos in your iCloud.
| tamimio wrote:
| > Videos anonymized by slow internet loading
|
| I chuckled!
|
| I believe the size difference has to do with the encoding, can't
| tell for a fact since you didn't show that part, but maybe it
| gets re-encoded when you download it hence the difference.
| baxtr wrote:
| Slightly off topic: anyone know how to backup iCloud data (mainly
| photos and videos, but also notes) to an outside platform?
|
| I'm an Apple fan, but still don't like the idea of having all my
| digital life on iCloud only.
| acheron wrote:
| Disclaimer I haven't tried this yet, but I found this project
| recently that supposedly downloads all iCloud photos (e.g. to a
| NAS for local backup). https://github.com/icloud-photos-
| downloader/icloud_photos_do...
| fckgw wrote:
| I've been using a version of this for Unraid for over a year.
| Downloads everything from my iCloud storage to my home
| server.
|
| You need to re-authenticate every 90 days due to 2fa
| restrictions but it works great.
| BigBalli wrote:
| I use Image Capture with my phone plugged in to save them on an
| external hard drive.
| AnonC wrote:
| This is not for incremental backups but for all data tied to
| your Apple ID -- Apple provides a data download option to get
| your contacts, photos, videos and a whole lot of other data.
| You can visit this KB [1] for details and instructions. There
| are some exclusions, like data that's end-to-end encrypted.
|
| [1]: https://support.apple.com/en-us/102208
| BigBalli wrote:
| Was the video with size mismatch edited? Apple keeps both
| versions (same goes for photos, they include image and MOV for
| live photos).
| very_good_man wrote:
| Apple needs to face criminal penalties for the dark patterns with
| which they prey upon their users.
| spockz wrote:
| I've been on a similar hunt. My devices are littered with copies
| of videos that were shared via iMessage, WhatsApp, and other
| means. Each of these apps then store the copy locally which then
| gets backed up in iCloud.
|
| Unfortunately I haven't found a way to automatically delete all
| the videos/photos that I send which still have the original in
| the photo stream. It would be awesome to be able to automate
| that.
| treesknees wrote:
| This isn't perfect, nor automated, but you can search the
| Messages app for photos/videos and delete them all. Assuming
| you've saved them already, this should clear up some iCloud
| space.
|
| https://www.igeeksblog.com/how-to-use-smart-search-filters-i...
| https://www.igeeksblog.com/delete-multiple-imessage-photos-a...
| godzillabrennus wrote:
| It seems like a giant class action lawsuit with a juicy target if
| Apple is overbilling customers on usage for videos in iCloud. I
| know lots of people paying for more storage in iCloud would
| appreciate the reprieve of lower costs.
| sroussey wrote:
| I need a good way to cleanup a gmail account...
| raz32dust wrote:
| I am so tired of these patterns. Google Photos also does not
| allow any easy or good way to clean up junk. Why would they? They
| want you to run out of space so you can pay for their cloud
| storage. They also have low incentive to keep photo/video storage
| space-efficient for the same reason.
| shreezus wrote:
| On this topic...anyone know the best way to export an entire
| iCloud Photo Library at full resolution?
| simplezeal wrote:
| If you have a Macbook, you can sync iCloud library at full
| resolution using Photos app and then export.
| meinheld111 wrote:
| There is an export service that returns a archive
___________________________________________________________________
(page generated 2024-01-05 23:00 UTC)