Supabase Storage v3: Resumable Uploads with support for 50GB files

Supabase Storage v3 has been released with Resumable Uploads feature which enables users to continue uploading a file from where they left off, even if they lose internet connectivity or accidentally close the browser tab while uploading. This release also allows users on the Pro tier or higher to upload files as large as 50GB. Postgres Advisory locks were used to solve concurrency problems and to make it easy to upload to Storage from the client directly without requiring validation from an intermediary server. The TUS open protocol was used for resumable file uploads and tus-node-server was implemented with some additional features such as limiting the upload to files of a certain size, running multiple instances of TUS, and expiring upload URLs after a certain amount of time. With just a few lines of code, developers can support parallel, resumable uploads of multiple files, with progress events. Advisory locks were used to mediate concurrent operations to the same object in S3.

Deep Dive into Resumable Uploads

Resumable uploads were implemented using the TUS open protocol, which provides options for resuming uploads after connection interruptions or other issues. Unlike the HTTP protocol, TUS allows for the uploading of individual segments of a larger file. To support this, Supabase Storage v3 uses tus-node-server to manage the uploads and stores the files in S3. Some additional features such as limiting the upload to files of a certain size, running multiple instances of TUS, and expiring upload URLs after a certain amount of time were also added to tus-node-server. Advisory locks were used to mediate concurrent operations to the same object in S3.

Conclusion

Supabase Storage v3 with Resumable Uploads feature enables users to easily upload large files and continue from where they left off, making it easier to manage and transfer files. The TUS open protocol and tus-node-server were used to support this feature, with additional features added to tus-node-server to enable more flexibility in file uploads. Advisory locks were used to mediate concurrent operations to the same object in S3, making it a key part of the Supabase Stack to solve difficult concurrency problems.