Upload files in chunk

[Nadiyar Mansouri]

Hi ranchers

I'm trying to figure out how can I upload a file into chunks and upload it. Let's ignore the client side for now, currently I'm just trying to respond to an API and send them a file, but in managed chunks instead of sending it all at once.

So let's say I have a 1G file and I want to break it into 100mb chunks and send them.

How can I achieve this? I'm currently using Javalin to handle requests but if I need to use another framework it would be ok.

thanks

[Ron McLeod]

Does the API specify how you are expected to break the contents in to chunks so that it can properly be reassembled on the server side?

[Nadiyar Mansouri]

Does the API specify how you are expected to break the contents

No, anything works, I'm writing this from scratch so we can design the api as needed.

and just to clarify, it's not a production project, I'm just trying to learn.

[Ron McLeod]

Ok.  

These are some of the kinds of questions I would ask myself if I was doing this:
  - Why break the contents in to chunks?
  - Will one or more chunks be uploaded at the same time, or will they be uploaded sequentially?
  - Will each chunk be identified somehow?
  - How will the server side know if it has received all the chunks, and how would it reassemble them?
  - If required, how would the server indicate if a chunk was not properly received and needs to be resent again?
  - If a chunk needs to be resent, would the client upload the entire chunk, or continue from some point?

[Nadiyar Mansouri]

Why break the contents in to chunks?  

The end goal is going to be creating a mechanism in which if upload interrupted, either by an error or manually, it can be continued from where its stopped.

Will one or more chunks be uploaded at the same time, or will they be uploaded sequentially?

it's sequentially, one at a time

Will each chunk be identified somehow?

Preferably yes. we can store each chunk temporary with names like 0_chunk, 1_chunk and ...


for the last three questions, I assume we can think about them later. after figuring out how to just upload a file chunk by chunk :)

[Tim Holloway]

There is HTTP protocol support for downloading data in chunks ("chunking"). Offhand, I cannot recall about support for the other direction.

The major thing to keep in mind is that HTTP is strictly request/response, so you cannot simply connect to the server and start uploading chunks. Each chunk would be a separate connection (request/response).

The other thing to take note of is that TCP can operate asynchronously. That means that your chunks may arrive out of sequence. To handle this, each chunk should contain a sequence number so that they may be identified and assembled in the proper order on the server. You'll also want that if you are concerned about interruptions that lead to lost chunks. TCP itself carries packet sequence numbers for the same reasons, but those are too low-level to be of use for manual chunking.

[Ron McLeod]

If I were doing this, I would first define what I need the transfer mechanism to do, and then think about it could work.

For example, the transfer protocol could work like this:

1. Create Upload Session
Request the server to create a new upload session.

POST /contentUpload
{
"name" : "daily-sales-20250529.csv",
"bytes" : 19594204
"requestedChunkBytes" : 5000000
"checksum" : "c4de1f0c46e01576810740c9242097cbab1f8bd0137a63d94f2362e7bdf682bd"
}

201 CREATED
{
"sessionId" : "c7e0b1c3-8f2d-4c9e-b6e6-5d7f2f7a4b12"
"chunkBytes" : 5000000
}


2. Upload Chunks
Could be uploaded concurrently or sequentially.

PUT /contentUpload/c7e0b1c3-8f2d-4c9e-b6e6-5d7f2f7a4b12/1
<5000000 bytes of content>

200 OK

PUT /contentUpload/c7e0b1c3-8f2d-4c9e-b6e6-5d7f2f7a4b12/2
<5000000 bytes of content>

200 OK

PUT /contentUpload/c7e0b1c3-8f2d-4c9e-b6e6-5d7f2f7a4b12/3
<5000000 bytes of content>

500 INTERNAL SERVER ERROR

PUT /contentUpload/c7e0b1c3-8f2d-4c9e-b6e6-5d7f2f7a4b12/4
<9188408 bytes of content>

200 OK


3. Check Status
See if any chunks didn't make it.

GET /contentUpload/c7e0b1c3-8f2d-4c9e-b6e6-5d7f2f7a4b12

200 OK
{
"chunks" : [1, 2, 4]
}

4. Re-upload Any Missing Chunks

PUT /contentUpload/c7e0b1c3-8f2d-4c9e-b6e6-5d7f2f7a4b12/3
<5000000 bytes of content>

200 OK

5.Finalize Upload Session
Server assembles the chunks and verifies the checksum.

PUT /contentUpload/c7e0b1c3-8f2d-4c9e-b6e6-5d7f2f7a4b12/save

200 OK

Cancel Upload Session
If you want to backout of the upload.

DELETE
/contentUpload/c7e0b1c3-8f2d-4c9e-b6e6-5d7f2f7a4b12

[Rob Spoor]

I was thinking about something similar. I'd probably turn the save action into a POST though, as I'd let it clean-up the chunks if successful, thereby making it non-idempotent: at the end the upload session wouldn't exist anymore.
I'd also let it fail with a 400 or 409 if not all chunks were uploaded successfully, or if there was a checksum mismatch.

[Niko Wild]

Hallo schaue dir mal  die Lib OkHttp an.
Für dein vorhaben wäre ein Multipart Request doch am einfachsten.

Translation:
Hello, take a look at the OkHttp library.
For your project, a multipart request would probably be the easiest solution.

NOTE: We are an English only web site.

[Rob Spoor]

A multipart request won't help here. That allows you to send multiple parts in a single request, where each part can be a file or text. But if you're uploading all chunks in one request, it's nonsense to use multipart - just upload the file as a single blob.

[Niko Wild]

Then you apparently haven't looked at the OkHttp library. It also offers a chunk upload. Take a look here.


https://sourcebae.com/blog/how-to-decide-whether-to-use-chunked-upload-in-okhttp-or-not/

[Rob Spoor]

What you link to is not about multipart requests, it's about chunked data transfers. For both the client and the server it's still one request, they just don't know the size up-front, and the data is best handled in a streaming way.

[Niko Wild]

Then scroll down a bit to

Practical Guide – Implementing Chunked Upload in OkHttp
Step-by-Step Implementation of Chunked Upload

[Niko Wild]

How big do you want a chunk to be? Or do you simply want to split a large file into many smaller files and then send them individually.

[Rob Spoor]

Niko Wild wrote:Then scroll down a bit to

Practical Guide – Implementing Chunked Upload in OkHttp
Step-by-Step Implementation of Chunked Upload

The entire word "multipart" is not found on the page you linked to.
The word "multi" is found once: "multiple small chunks". That's not the same as multipart.
The word "address" is found once: the address at the bottom left.

So while chunking may or may not be possible option, multipart requests, mostly through multipart/form-data, is not a (good) option. It only adds more work to split up the file into chunks manually, only to send them again in the same (chunked or not) request.

[Niko Wild]

I think we wait for an answer from the therd creator

[Ron McLeod]

Niko Wild wrote:I think we wait for an answer from the therd creator

The OP (thread creator) wants to be able to resume a failed upload.  The feature in the link that you shared does not provide such a capability.

Nadiyar Mansouri wrote:The end goal is going to be creating a mechanism in which if upload interrupted, either by an error or manually, it can be continued from where its stopped.