[HN Gopher] Supabase Storage v3: Resumable Uploads with support ...
___________________________________________________________________
Supabase Storage v3: Resumable Uploads with support for 50GB files
Author : creativedg
Score : 174 points
Date : 2023-04-12 13:03 UTC (9 hours ago)
(HTM) web link (supabase.com)
(TXT) w3m dump (supabase.com)
| Whiteshadow12 wrote:
| This team ships, every few years, there is a company HN loves,
| few years ago it was CloudFlare/Stripe, Supabase feels to be in
| the same class. Linear as well.
| paulgb wrote:
| I'd add Tailscale and Fly.io to the category, too. Well-written
| technical blogs are truly the way to HN's heart.
| traceroute66 wrote:
| > Fly.io ... well-written technical blogs
|
| I'm not sure I would class fly.io as well-written _technical_
| blogs. Well-written yes, technical no.
|
| I _used to_ think the fly.io ones were good, but having read
| a good few of them I 've found they follow the same
| repetitive structure: - Very light on
| detail (fly.io blogs are carefully written to *sound* like
| they're giving you detail, but in reality it's all a 50,000ft
| view, and a lightweight one at that. Given the length of
| their average blog post, they could and should do better.).
| - Constantly going off-tangent, whether randomly talking
| about sandwich fillings, food-types, or just a paragraph with
| the author's rant about how they dislike a particular
| technology. The first couple of times its cute, but after
| that, not so much ...
| mrkurt wrote:
| Would you mind sharing specifics about the first bullet?
| What's the last post you read you thought could have gone
| deeper? We definitely want to go deep, some of our posts
| are better than others.
| kiwicopple wrote:
| fwiw Kurt, I love reading Fly.io posts. your "globally
| distributed postgres" post especially[0], which had
| plenty of depth and no sandwhiches
|
| [0] https://fly.io/blog/globally-distributed-postgres/
| 0zemp1c wrote:
| tbh you seem too focused on making HN happy, makes me
| wonder about your focus
| mrkurt wrote:
| I appreciate it. I do struggle with focus. Thank you for
| giving me a nudge.
| joshmanders wrote:
| In his defense, HN is his target audience so making them
| happy is important too.
| no_wizard wrote:
| Supabase and Linear do deserve the hype IMO and Cloudflare
| still does.
|
| Stripe has gotten a bit more iffy for me, with some of their
| poor practices around frozen accounts
| Whiteshadow12 wrote:
| I get the iffyness, I suspect there is probably more to it
| then the public sees, for example it could be their
| underlying layer or something else.
| waboremo wrote:
| Cloudflare and Stripe have just reached that stage where
| there isn't much to hype about. They're now dealing with
| problems that are gnarly, as opposed to "we just launched a
| storage service!" Not to diminish Supabase, Fly, etc. and
| their tremendous work, it's just a completely different lane
| of expectations.
| no_wizard wrote:
| To me its more than just release new features and
| improvements (though Cloudflare could do well improving the
| speed around this a bit, a nice cadence around services and
| a public-ish roadmap would be ideal IMO).
|
| More importantly, I think Cloudflare, Supabase and Linear
| are innovating _and_ really nice to use, generally
| speaking[0]
|
| [0]: they all have edge cases and what not, and sometimes
| failures, don't get me wrong
| shados wrote:
| Yeah, supabase is great. I think if they could get the auth
| story a little more polished (it "works", but its definitely
| weak/buggy in some aspects compared to alternatives, and hard
| to use), and made some advanced use cases with Prisma easier to
| do (or better documented) since that's such a common pairing,
| they'd be even easier to recommend.
|
| Right now I still use Supabase because I think they're a solid
| "Postgresql as a service" offering, but they're just SO CLOSE
| to being so much more than that. They have all the pieces, they
| just don't all work quite the way folks expect. They'll get
| there, I'm sure.
| tough wrote:
| Supabase Launch Weeks are so friggin cool fr.
| vamega wrote:
| Looking through the supabase storage docs, I wasn't able to
| identify if storing to local disk was an option? I know it
| launched with S3 as a supported backend, but I'm curious to know
| if this ever expanded.
|
| Or perhaps supporting Minio as a backend, which I could run on
| the same machine.
| fenospro wrote:
| Hey, Supabase Storage engineer here
|
| Yes, we support S3, Minio and any S3 compatible storage
| servers. As well as the File System (local disk) - This is for
| the self hosted version.
|
| When using Supabsase managed version we are storing your files
| in a S3 bucket managed by us
| fenospro wrote:
| In the future we might add the option of "bring your own
| bucket" to Supabase Storage and we'll upload the assets
| there.
|
| The bucket could be located in a minio server of your choice,
| for example.
| fenospro wrote:
| Is this what you were looking for?
| sickcodebruh wrote:
| Supabase looks so good and keeps getting better. I started
| playing with it a couple months ago and could see myself loving
| it. One of my biggest problems right now is inertia: I'm
| comfortable with my existing tools (RDS, S3, lambda) and thinking
| about building a new product while also adding new tools is a
| tough pill to swallow. But the product sure makes a strong case!
|
| Resumable uploads is a big deal for what I'm working on. But I'm
| also blocked by another issue, maybe a Supabase engineer on the
| thread can weight in. I'm using ffmpeg on AWS lambda to transcode
| media files after successful upload to S3. Lambda makes it easy
| to add ffmpeg binaries via lambda layers. Supabase functions
| don't seem to provide any such option. Is there a way to solve
| this within Supabase as it exists now?
|
| I'm also curious about function timeouts. If I have a file
| conversion that takes a couple minutes, is a Supabase function
| the right way to go?
| subpar wrote:
| I was trying to do the same thing (ffmpeg to do file
| transformations on upload) and couldn't figure it out with
| Supabase functions. After poking around in the Deno discord for
| awhile, it seems that ffmpeg.wasm for Deno just isn't much of a
| thing yet. I was able to get it working with Vercel functions,
| but it was a total nightmare and the runtime made it a non-
| starter. I landed where you started: just do it in Lambda!
| kiwicopple wrote:
| > _Is there a way to solve this within Supabase as it exists
| now?_
|
| No, not right now. However we do have a planned path to this.
| The 3 features we will release are:
|
| 1. Custom execution limits/timeouts for Edge Functions.
|
| 2. WASM support in Edge Functions
|
| 3. Storage Webhooks, which can trigger an Edge Functions
|
| Features 1 & 2 will be possible with the Edge Runtime we
| released yesterday[0]. Feature 3 is already possible using our
| Postgres Database Webhooks[1], and we will also provide
| additional support for this directly in the storage server
|
| [0] Edge Runtime discussion:
| https://news.ycombinator.com/item?id=35525222
|
| [1] Postgres Database Webhooks:
| https://supabase.com/docs/guides/database/webhooks
| debarshri wrote:
| That is the same inertia we see while selling to early stage
| company.
|
| I also see quite some people build out in supabase and migrate
| to AWS, especially if they want SOC2 or some compliance
| certifications. The Compliance automation tools work really
| well with known cloud providers.
| burggraf wrote:
| Just as a note, Supabase received SOC2 Type 2 compliance last
| week (we've had Type 1 for a long time already) and now we're
| working on HIPAA.
| inian wrote:
| Hello HN, Supabase Engineer here.
|
| This release brings resumable uploads to Storage. If you users
| are uploading large files or uploading from a mobile network,
| this improves the user experience by continuing the upload from
| exactly where it was interrupted. As a side benefit, you also get
| progress events exposed in a neat way.
|
| And we continue to use more of Postgres. We use it as a database
| to store object metadata, RLS policies for Authorization[1], the
| SKIP Locked feature for building a queue [2] and advisory locks
| for implementing a distributed locking mechanism [3].
|
| Will be hanging around to answer any questions.
|
| [1] https://supabase.com/docs/guides/storage/access-control
|
| [2] https://www.2ndquadrant.com/en/blog/what-is-select-skip-
| lock...
|
| [3] https://supabase.com/blog/storage-v3-resumable-
| uploads#enter...
| moralestapia wrote:
| >you also get progress events exposed in a neat way
|
| Yes! :D
| ecuaflo wrote:
| S3
| kiwicopple wrote:
| hey hn, supabase ceo here
|
| This release introduces a few new features to Supabase Storage:
| Resumable Uploads , Quality Filters, Next.js support, and WebP
| support.
|
| As a reminder, Supabase Storage is for file storage, not to be
| confused with Postgres Storage. Resumable Uploads is the biggest
| update because it means that you can build more resilient apps:
| your users can continue uploading a file if their internet drops
| or if they accidentally close a browser tab
|
| This implementation uses TUS[0], which is an open source
| protocol. We opted for this over s3's protocol to support the
| open source ecosystem. This means you can use several existing
| libraries and frameworks (like Uppy.js[1]) for multi-part
| uploads.
|
| It also has some neat technical details, using Postgres Advisory
| Locks to solve concurrency issues.
|
| The Storage team will be in the comments to cover any technical
| questions.
|
| [0] TUS: https://tus.io/
|
| [1] Uppy: https://uppy.io/docs/tus/
| moomoo11 wrote:
| Hi. I was checking out the pricing for supabase database.
|
| It seems like it may be more costly than firestore, although
| you are offering postgres as the underlying data store.
|
| I'm wondering if I'm reading that right.
|
| When it says 500 connections, is that 500 clients connected at
| one time? Just trying to compare the pricing vs firebase.
|
| Thank you
| kiwicopple wrote:
| It's very hard to compare Supabase + Firebase pricing because
| it depends on which features you're using. For example, API
| requests are free on Supabase and paid on Firebase.
|
| The 500 connections you quoted is for the "Realtime" feature
| (which provides multiplayer functionality). You're correct,
| this is 500 clients connected simultaneously
| rockwotj wrote:
| I was just going to ask if you'd considered tus for this :)
|
| Kudos on the launch - great addition
| kiwicopple wrote:
| credit to the Fabrizio and Inian on this one. I'm personally
| impressed how fast they were able to implement this since we
| shipped a major update to Storage last Launch Week.
|
| From what I understand, the TUS protocol wasn't necessarily
| simple, but the TUS community has made it a lot easier with
| their node server which they recently converted to
| Typescript: https://github.com/tus/tus-node-server
| inian wrote:
| Yeah the Typescript port gave a good starting point.
| However there were some missing pieces which we had to
| build into our server - https://github.com/tus/tus-node-
| server#extensions
|
| We plan to contribute these back to the Node TUS server
| after launch week.
| mplemay wrote:
| Thanks for working on such an awesome project and releasing
| such useful features!
|
| Yet, what stops us from using supabase is not the set of
| features, but the state of the current apis. Last night I was
| evaluating the python sdk and some of the examples were broken,
| baseline features like RLS are unimplemented
| (https://github.com/supabase-community/supabase-py/issues/58),
| and progress seems slow (it seems to have been release two
| years ago and it is still in alpha). Even though supporting
| many platforms is boring (i.e. no flashy feature announcements)
| and costly (many FTEs are required for maintenance let alone
| adding features), are there any plans to bring feature parity
| more of the sdks to allow more developers to leverage this
| awesome platform?
| kiwicopple wrote:
| client libs are one of our next focuses. we've been
| discussing internally how to do this. So far we have
| concentrated on the JS/Dart libs because that makes up a
| large portion of our users. Over the last 3 months the
| community has added (and documented) libraries for Swift[0]
| and C#[0].
|
| We _hope_ to continue making this a community-driven
| endeavor, so if anyone reading this would like to become a
| maintainer for one of the libraries (especially Python),
| please reach out. If we can 't find maintainers we'll find a
| polyglot who can work on this full-time.
|
| [0] Swift:
| https://supabase.com/docs/reference/swift/introduction
|
| [1] C#:
| https://supabase.com/docs/reference/csharp/introduction
| worldmerge wrote:
| Will you consider supporting C++? I'd like to use the
| supabase APIs, specifically the real-time APIs, from my
| ESP32s and other C++ projects.
| kiwicopple wrote:
| We don't have a lot of C++ experience in the team, but
| I'll share your feedback with the Realtime team - perhaps
| we can find someone in the community to help
| bachmeier wrote:
| > So far we have concentrated on the JS/Dart libs because
| that makes up a large portion of our users.
|
| I once had a computer magazine publisher tell me they only
| published Windows content because none of their readers
| used Mac or Linux.
| kiwicopple wrote:
| sounds like they were just getting started and knew the
| efficacy of focus :)
| dperfect wrote:
| Awesome to see supabase constantly improving. Been using
| supabase for the past few weeks and have really enjoyed it!
|
| I was a bit surprised, however, that there's not currently a
| good way to reference storage objects from my postgres tables.
| I found that the recommended way is to store the object's path
| (as a string) in the database. While that works, it isn't
| optimal as I'd like to enforce consistency between the object
| and the table referencing it.
|
| I've tried referencing the id of the corresponding row in the
| storage.objects table, but (1) apparently the schema supabase
| uses to manage storage.objects may change, and (2) it still
| requires separate (non-atomic) operations - or additional
| triggers - for keeping things in sync. Using buckets
| (corresponding to tables) and folders with ids is another way
| to work around it, but still feels suboptimal.
|
| Not 100% sure what the best solution would look like, but
| ideally the supabase client could emulate storage operations
| for objects "attached" to a given table record, and supabase
| (the backend piece) could implement them as atomic operations
| (e.g., uploading the actual storage asset, storing the
| necessary metadata, and updating my table row to reference the
| newly-created storage object; exposing a helper function to
| return the URLs for any storage objects attached to a record;
| etc).
|
| Anyway, just a suggestion. Keep up the great work!
| kiwicopple wrote:
| this is a similar issue we face with Auth. We want to give
| direct access to the data/tables, and at the same time we
| need some flexibility to alter the tables on rare occasions.
|
| We've dabbled with the idea of offering versioned views which
| have a "set" interface (eg `storage_v1`, `storage_v2`). We're
| still debating all the ways that we might do it.
|
| All that to say - we're aware that this experience can be
| improved and we're working on it.
| jonplackett wrote:
| This is great - the caps on image size / mime type per bucket
| is super useful
| fenospro wrote:
| Yes, I agree!! Very very useful!
| joshspankit wrote:
| Any chance we could choose to use B2 instead of S3?
|
| Even if Supabase keeps the same pricing (though why would you)
| the folks at BackBlaze have had my respect for a long time and
| I'm happier when I can stay out of the AWS universe.
| kiwicopple wrote:
| yes, it's planned. I believe the only blocker we saw last
| time we looked was a 1-hour downtime-window every week (not
| that always went offline, but that they _could_ use that time
| for maintenance). Do you know if that 's still the case?
___________________________________________________________________
(page generated 2023-04-12 23:02 UTC)