AstroZero Document

Appearance

OBS multipart upload ​

Maximum size of a customized interface request body5MB. When the size of the uploaded file exceeds5MB, you need to upload a complete file through multipart upload.

The multipart upload capability is directly inherited from the object storage service.OBSFor details about the implementation process, see.OBS Official Website Documents

Multipart upload involves the following steps:

  1. Initializing a Multipart Upload Task (create-multipart-upload)
  2. Uploading parts one by one or in parallel (upload-part)
  3. Merge Segments (complete) or cancel the multipart upload task (abort)

Developing a Multipart Upload Script and Encapsulating the Customized Interface ​

1. Create a script for initializing a multipart upload task. ​

Before using multipart upload to transfer data, you must notify OBS to initialize a multipart upload task. This operation will return a globally unique ID created on the OBS server.upload_id), which is used to identify the multipart upload task. You can perform operations based on the unique ID, for example, canceling a multipart upload task, listing multipart upload tasks, and listing uploaded parts.

Creating a Scriptcreate_multipart_upload

ts
/*
 *Copyright (c) Huawei Technologies Co., Ltd. 2012-2022. All rights reserved.
 * 
 */

import * as obs from 'objectstorage';

//The object name is used as the input parameter.
export class Input {
    @action.param({ type: "String", required: true, description: "filename" })
    object: string;
}

export class Output {
    @action.param({ type: "String", required: true })
    uploadId: string;
}

export class Calculator {
    @action.method({ input: "Input", output: "Output", description: "do a operation" })
    run(input: Input): Output {
      	//File name restrictions can be added here. This section is only an example and is used to show the main process of multipart upload.
        let cli = obs.newClient(obs.StoreType.PROXY, "sound__proxy_obs");
        let uploadId = cli.createMultipartUpload(input.object, { acl: "private", contentType: "video/quicktime" });
        let output = new Output();
        output.uploadId = uploadId
        return output;
    }
}
/*
 *Copyright (c) Huawei Technologies Co., Ltd. 2012-2022. All rights reserved.
 * 
 */

import * as obs from 'objectstorage';

//The object name is used as the input parameter.
export class Input {
    @action.param({ type: "String", required: true, description: "filename" })
    object: string;
}

export class Output {
    @action.param({ type: "String", required: true })
    uploadId: string;
}

export class Calculator {
    @action.method({ input: "Input", output: "Output", description: "do a operation" })
    run(input: Input): Output {
      	//File name restrictions can be added here. This section is only an example and is used to show the main process of multipart upload.
        let cli = obs.newClient(obs.StoreType.PROXY, "sound__proxy_obs");
        let uploadId = cli.createMultipartUpload(input.object, { acl: "private", contentType: "video/quicktime" });
        let output = new Output();
        output.uploadId = uploadId
        return output;
    }
}

Input parameter

  • object: name of the object to be uploaded.

Output parameter:

  • uploadId: upload task ID, which needs to be used in subsequent multipart upload and upload completion APIs.

Creating a public interfacecreatemultipartuploadEncapsulate the script.

createmultipartupload

Example request

bash
POST /service/m__quickjs/1.0.0/fileupload/createmultipartupload
Content-Type: application/json
Access-Token: xxx

{
	"object":"xxxxx"
}
POST /service/m__quickjs/1.0.0/fileupload/createmultipartupload
Content-Type: application/json
Access-Token: xxx

{
	"object":"xxxxx"
}

Input Result

{
    "uploadId":"xxxxx"
}
{
    "uploadId":"xxxxx"
}

2. Create a multipart upload script. ​

After a multipart upload task is initialized, you can specify the object name anduploadIdTo upload data in parts. Each uploaded part has a part number, that is, a part number, ranging from 1 to 10000. For the same uploadId, the segment ID uniquely identifies the data segment and the relative position of the data segment in the entire object. If you upload new data with the same part number, the existing data with the part number on OBS will be overwritten. Except the last segment, the size of the other segments ranges from 100 KB to 5 GB. The last segment size ranges from 0 GB to 5 GB. Each part does not need to be uploaded in sequence. You can even upload it on different processes and machines. OBS sorts the parts by part number to form the final object.

ts
/*
 *Copyright (c) Huawei Technologies Co., Ltd. 2012-2022. All rights reserved.
 * 
 */

import * as obs from 'objectstorage';

export class Input {
    @action.param({ type: "String", required: true, description: "object name" })
    object: string;
    @action.param({ type: "String", required: true, description: "upload id" })
    uploadId: string;
    @action.param({ type: "Number", required: true, description: "part number" })
    partNum: number;
    @action.param({ type: "Blob", required: true, description: "data" })
    $data: any;
}

export class Output {
    @action.param({ type: "Any", required: true })
    part: obs.CompletedPart
}

export class Calculator {
    @action.method({ input: "Input", output: "Output", description: "do a operation" })
    run(input: Input): Output {
        let cli = obs.newClient(obs.StoreType.PROXY, "sound__proxy_obs");
        let cp = cli.uploadPart(input.object, input.uploadId, input.partNum, input.$data)
        let output = new Output();
        output.part = cp;
        return output;
    }
}
/*
 *Copyright (c) Huawei Technologies Co., Ltd. 2012-2022. All rights reserved.
 * 
 */

import * as obs from 'objectstorage';

export class Input {
    @action.param({ type: "String", required: true, description: "object name" })
    object: string;
    @action.param({ type: "String", required: true, description: "upload id" })
    uploadId: string;
    @action.param({ type: "Number", required: true, description: "part number" })
    partNum: number;
    @action.param({ type: "Blob", required: true, description: "data" })
    $data: any;
}

export class Output {
    @action.param({ type: "Any", required: true })
    part: obs.CompletedPart
}

export class Calculator {
    @action.method({ input: "Input", output: "Output", description: "do a operation" })
    run(input: Input): Output {
        let cli = obs.newClient(obs.StoreType.PROXY, "sound__proxy_obs");
        let cp = cli.uploadPart(input.object, input.uploadId, input.partNum, input.$data)
        let output = new Output();
        output.part = cp;
        return output;
    }
}

Creating a Scriptmultipart_upload_part.

Input parameter

  • object: object name, which must be the same as the object name transferred during upload task creation.
  • uploadId: upload task ID, which is generated in step 1.
  • partNum: number of the uploaded part. The value is an integer ranging from 1 to 10000. Each segment must have a different code
  • $data: segment content of the object file, which is obtained from the request body.

The output parameter is a structure.

  • part

    • eTag: label for uploading a part.
    • partNumber: part number.
ts
Note:
1. Except for the part last uploaded, the sizes of other parts must be greater than 100 KB. However, the part upload API does not verify the size of the uploaded part immediately because it does not know whether the part is the last one. The part size is verified only when the part merge API is called.
2. OBS returns the ETag (MD5 value of the part data) received by the server to the user.
3. To prevent errors during data transmission, you can set the MD5 value and add it to the Content-MD5 request header. The OBS server will compare the MD5 value of the uploaded data with that calculated by SDK to ensure data integrity.
4. You can use put_properties.md5 to set the MD5 value of the uploaded data. The MD5 value is provided for the OBS server to verify data integrity.
5. Part numbers range from 1 to 10000. If the value is out of the range, OBS returns 400 Bad Request.
6. The minimum part size supported by a bucket of OBS 3.0 is 100 KB, and that supported by a bucket of OBS 2.0 is 5 MB. Perform multipart upload on a bucket of OBS 3.0.
Note:
1. Except for the part last uploaded, the sizes of other parts must be greater than 100 KB. However, the part upload API does not verify the size of the uploaded part immediately because it does not know whether the part is the last one. The part size is verified only when the part merge API is called.
2. OBS returns the ETag (MD5 value of the part data) received by the server to the user.
3. To prevent errors during data transmission, you can set the MD5 value and add it to the Content-MD5 request header. The OBS server will compare the MD5 value of the uploaded data with that calculated by SDK to ensure data integrity.
4. You can use put_properties.md5 to set the MD5 value of the uploaded data. The MD5 value is provided for the OBS server to verify data integrity.
5. Part numbers range from 1 to 10000. If the value is out of the range, OBS returns 400 Bad Request.
6. The minimum part size supported by a bucket of OBS 3.0 is 100 KB, and that supported by a bucket of OBS 2.0 is 5 MB. Perform multipart upload on a bucket of OBS 3.0.

Creating a Customized Interface Encapsulation and Upload Script

custom_api_upload_with_binary Example request

bash
POST /service/m__quickjs/1.0.0/upload/binary?object={$object_name}&uploadId={$upload_id}&partNum={$partNum}
Access-Token: xxx

[binary-data]
POST /service/m__quickjs/1.0.0/upload/binary?object={$object_name}&uploadId={$upload_id}&partNum={$partNum}
Access-Token: xxx

[binary-data]

Note that the method of the custom interface must be defined as POST and the content type must bebinary-data. Therefore, the other parameters need to bequeryParameters are passed in.

3. Create a script for uploading a multi part file. ​

After all parts are uploaded, you need to call the part combination API to generate the final object on the OBS server. When performing this operation, you need to provide all valid segment lists (including segment numbers and segment ETag values). After receiving the submitted part list, OBS verifies the validity of each part one by one. After all segments are verified, OBS combines the segments into a final object.

Creating a Scriptcompete_multipart_upload.

ts
/*
 *Copyright (c) Huawei Technologies Co., Ltd. 2012-2022. All rights reserved.
 * 
 */

import * as obs from 'objectstorage';

export class Input {
    @action.param({ type: "String", required: true, description: "object name" })
    object: string;
    @action.param({ type: "String", required: true, description: "upload id" })
    uploadId: string;
    @action.param({ type: "Any", required: true, description: "data" })
    completeParts: obs.CompletedPart[];
}

export class Output {
    @action.param({ type: "String", required: true, description: "address" })
    endpoint: string;
}

export class Calculator {
    @action.method({ input: "Input", output: "Output", description: "do a operation" })
    run(input: Input): Output {
        let cli = obs.newClient(obs.StoreType.PROXY, "sound__proxy_obs");
        //Invoke completeUpload to merge the final files. If completeUpload is invoked successfully, the files are uploaded successfully.
        cli.completeUpload(input.object, input.uploadId, input.completeParts, { async: true });
        let output = new Output();
        output.endpoint = cli.getEndpoint() + input.object;
        return output;
    }
}
/*
 *Copyright (c) Huawei Technologies Co., Ltd. 2012-2022. All rights reserved.
 * 
 */

import * as obs from 'objectstorage';

export class Input {
    @action.param({ type: "String", required: true, description: "object name" })
    object: string;
    @action.param({ type: "String", required: true, description: "upload id" })
    uploadId: string;
    @action.param({ type: "Any", required: true, description: "data" })
    completeParts: obs.CompletedPart[];
}

export class Output {
    @action.param({ type: "String", required: true, description: "address" })
    endpoint: string;
}

export class Calculator {
    @action.method({ input: "Input", output: "Output", description: "do a operation" })
    run(input: Input): Output {
        let cli = obs.newClient(obs.StoreType.PROXY, "sound__proxy_obs");
        //Invoke completeUpload to merge the final files. If completeUpload is invoked successfully, the files are uploaded successfully.
        cli.completeUpload(input.object, input.uploadId, input.completeParts, { async: true });
        let output = new Output();
        output.endpoint = cli.getEndpoint() + input.object;
        return output;
    }
}

Input parameter

  • object: object name, which must be the same as the preceding two steps.
  • uploadId: upload task ID, which is obtained from step 1.
  • completeParts: segment information array, which is obtained from step 2 and combined into an array.

Output parameters can be customized. The complete path of the uploaded file to OBS is displayed.

Creating a Customized Interface Encapsulation Script

Example request

bash
POST /service/sound__fight/1.0.0/fileupload/completeupload
Access-Token: xxx
Content-Type: application/json

{
  "object":"xxx/xxx/xxx.jpg",
  "uploadId":"xxx",
	"completeParts": [
		{
			"etag": "xxx",
			"partNum": 1
		},
		{
			"etag": "xxx",
			"partNum": 2
		}
	]
}
POST /service/sound__fight/1.0.0/fileupload/completeupload
Access-Token: xxx
Content-Type: application/json

{
  "object":"xxx/xxx/xxx.jpg",
  "uploadId":"xxx",
	"completeParts": [
		{
			"etag": "xxx",
			"partNum": 1
		},
		{
			"etag": "xxx",
			"partNum": 2
		}
	]
}