4. 文件上传

在NOS中用户的基本操作单元是对象,亦可以理解为文件,NOS C# SDK提供了丰富的上传接口,可以通过以下的方式上传文件:

  • 内存内容上传
  • 流式上传
  • 单块上传
  • 分块上传

流式上传、单块上传最大为100M;分块上传除最后一块外,分块不能小于16k,每一个分块不能大于100M,最多能上传10000个分块,即分块上传的文件最大支持1T。

4.1. 内存内容上传

您可以使用AmazonS3Client.PutObject上传一个一个Stream中的内容,具体实现如下:

/// <summary>
/// 内存内容上传
/// </summary>
/// <param name="bucket">桶名</param>
/// <param name="key">对象名</param>
/// <param name="content">想要上传的内容</param>
public void PutObjectByContent(string bucket,string key,string content){
    PutObjectRequest request = new PutObjectRequest(){
        BucketName=bucket,
        Key=key,
        ContentBody=content,
    };
    try{
        PutObjectResponse resp = s3Client.PutObject(request);
    } catch(AmazonS3Exception ex){
        Console.WriteLine("Failed with HTTPStatus: {0}; \nErrorCode: {1}; \nErrorMessage: {2}; \nRequestID:{3};",
        ex.StatusCode, ex.ErrorCode, ex.Message, ex.RequestId);
    } catch (Exception ex)
    {
        Console.WriteLine("Failed with error info: {0}", ex.Message);
    }

}

Attention

  1. 上传的流内容不超过100M

4.2. 流式上传

您可以使用AmazonS3Client.PutObject上传一个一个Stream中的内容,具体实现如下:

//需要把想要上传的对象放入Stream中
public void PutObjectByStream(string bucket,string key,Stream stream){
    PutObjectRequest putObjectRequest = new PutObjectRequest(){
        BucketName = bucket,
        Key = key,
        InputStream = stream
    };
    try
    {
        PutObjectResponse resp = s3Client.PutObject(putObjectRequest);
    }
    catch (AmazonS3Exception ex)
    {
        Console.WriteLine("Failed with HTTPStatus: {0}; \nErrorCode: {1}; \nErrorMessage: {2}; \nRequestID:{3};",
        ex.StatusCode, ex.ErrorCode, ex.Message, ex.RequestId);
    }
    catch (Exception ex)
    {
        Console.WriteLine("Failed with error info: {0}", ex.Message);
    }

}

Attention

  1. 上传的流内容不超过100M

4.3. 单块上传

您可以使用AmazonS3Client.PutObject上传文件内容,具体实现如下:

public void PutObjectByFilePath(string bucket, string key, string filePath)
{
    PutObjectRequest putObjectRequest = new PutObjectRequest()
    {
        BucketName = bucket,
        Key = key,
        FilePath = filePath
    };
    try
    {
        PutObjectResponse resp = s3Client.PutObject(putObjectRequest);
    }
    catch (AmazonS3Exception ex)
    {
        Console.WriteLine("Failed with HTTPStatus: {0}; \nErrorCode: {1}; \nErrorMessage: {2}; \nRequestID:{3};",
        ex.StatusCode, ex.ErrorCode, ex.Message, ex.RequestId);
    }
    catch (Exception ex)
    {
        Console.WriteLine("Failed with error info: {0}", ex.Message);
    }

}

Attention

  1. 上传的文件内容不超过100M

4.4. 分块上传

除了通过putObject接口上传文件到NOS之外,NOS还提供了另外一种上传模式-分块上传,用户可以在如下应用场景内(但不限于此),使用分块上传模式,如:

  • 需支持断点上传
  • 上传超过100M的文件
  • 网络条件较差,经常和NOS服务器断开连接
  • 上传文件之前无法确定文件的大小

分块上传一般流程如下所示:

  • 初始化一个分块上传任务(InitiateMultipartUpload)
  • 上传分块(UploadPart)
  • 完成分块上传(CompleteMultipartUpload)或者取消分块上传(AbortMultipartUpload)

4.4.1. 初始化分块上传

您可以使用AmazonS3Client.InitiateMultipartUpload初始化分块上传,具体实现如下:

public string InitMultipart(string bucketName,string key){
    InitiateMultipartUploadRequest request = new InitiateMultipartUploadRequest(){
        BucketName = bucketName,
        Key = key
    };

    try
    {
        InitiateMultipartUploadResponse resp =  s3Client.InitiateMultipartUpload(request);
        return resp.UploadId;
    }
    catch (AmazonS3Exception ex)
    {
        Console.WriteLine("Failed with HTTPStatus: {0}; \nErrorCode: {1}; \nErrorMessage: {2}; \nRequestID:{3};",
        ex.StatusCode, ex.ErrorCode, ex.Message, ex.RequestId);
    }
    catch (Exception ex)
    {
        Console.WriteLine("Failed with error info: {0}", ex.Message);
    }
    return null;
}

4.4.2. 上传分块

您可以使用AmazonS3Client.UploadPart上传分块,具体实现如下:

public List<PartETag> UploadParts(String bucket, String key, String fileToUpload,
                                      String uploadId, long partSize)
{
    // 计算片个数
    var fi = new FileInfo(fileToUpload);
    var fileSize = fi.Length;

    // 开始分片上传
    var partETags = new List<PartETag>();
    try
    {
        long filePosition = 0;
        for (var i = 1; filePosition < fileSize; i++)
        {
            partSize = Math.Min(partSize, (fileSize - filePosition));
            var request = new UploadPartRequest()
            {
                BucketName = bucket,
                Key = key,
                UploadId = uploadId,
                PartNumber = i,
                FilePosition = filePosition,
                FilePath = fileToUpload,
            };
            //如果是需要支持客户端加密,那么需要加入这段代码,因为加密最后一块需要对齐
            if ((partSize + filePosition) == fileSize){
                request.IsLastPart = true;
            } else {
                request.PartSize = partSize;
            }
            var result = s3Client.UploadPart(request);
            partETags.Add(new PartETag(result.PartNumber,result.ETag));
            filePosition += partSize;
            Console.WriteLine("uploadPart PartNumber : {0}, ETag : {1}",result.PartNumber,result.ETag);
        }
        Console.WriteLine("Put multipart upload succeeded");
    }
    catch (AmazonS3Exception ex)
    {
        Console.WriteLine("Failed with HTTPStatus: {0}; \nErrorCode: {1}; \nErrorMessage: {2}; \nRequestID:{3};",
            ex.StatusCode, ex.ErrorCode, ex.Message, ex.RequestId);
    }
    catch (Exception ex)
    {
        Console.WriteLine("Failed with error info: {0}", ex.Message);
    }

    return partETags;
}

由于加密算法使用的是CBC,所以不能实现并行分块上传

4.4.3. 完成分块上传

您可以使用AmazonS3Client.CompleteMultipartUpload完成上传分块,具体实现如下:

public void CompleteUploadPart(String bucket, String key,
                        String uploadId, List<PartETag> partETags)
{
    try
    {
        var completeMultipartUploadRequest = new CompleteMultipartUploadRequest(){
            BucketName=bucket,
            Key=key,
            UploadId=uploadId,
            PartETags=partETags,

        };


        s3Client.CompleteMultipartUpload(completeMultipartUploadRequest);
        Console.WriteLine("Complete Upload Part succeeded");
    }
    catch (AmazonS3Exception ex)
    {
        Console.WriteLine("Failed with HTTPStatus: {0}; \nErrorCode: {1}; \nErrorMessage: {2}; \nRequestID:{3};",
            ex.StatusCode, ex.ErrorCode, ex.Message, ex.RequestId);
        s3Client.AbortMultipartUpload(new AbortMultipartUploadRequest(){
            BucketName=bucket,
            Key=key,
            UploadId=uploadId
        });
    }
    catch (Exception ex)
    {
        Console.WriteLine("Failed with error info: {0}", ex.Message);
        s3Client.AbortMultipartUpload(new AbortMultipartUploadRequest()
        {
            BucketName = bucket,
            Key = key,
            UploadId = uploadId
        });
    }
}

4.4.4. 分块上传示例

下面通过一个完整的示例说明了如何进行分片上传操作,可以参考下面代码:

 public void MultipartUpload(string bucketName, string key, string filePath,long partSize){
    InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest()
    {
        BucketName = bucketName,
        Key = key
    };

    InitiateMultipartUploadResponse initResp = s3Client.InitiateMultipartUpload(initRequest);
    // 计算片个数
    var fi = new FileInfo(filePath);
    var fileSize = fi.Length;

    // 开始分片上传
    var partETags = new List<PartETag>();
    try
    {
        long filePosition = 0;
        for (var i = 1; filePosition < fileSize; i++)
        {
            partSize = Math.Min(partSize, (fileSize - filePosition));
            var request = new UploadPartRequest()
            {
                BucketName = bucketName,
                Key = key,
                UploadId = initResp.UploadId,
                PartNumber = i,
                FilePosition = filePosition,
                FilePath = filePath,
                PartSize = partSize
            };
            var result = s3Client.UploadPart(request);
            partETags.Add(new PartETag(result.PartNumber, result.ETag));
            filePosition += partSize;
            Console.WriteLine("uploadPart PartNumber : {0}, ETag : {1}", result.PartNumber, result.ETag);
        }
        Console.WriteLine("Put multipart upload succeeded");
        var completeMultipartUploadRequest = new CompleteMultipartUploadRequest()
        {
            BucketName = bucketName,
            Key = key,
            UploadId = initResp.UploadId,
            PartETags = partETags,

        };
        s3Client.CompleteMultipartUpload(completeMultipartUploadRequest);

    }
    catch (AmazonS3Exception ex)
    {
        Console.WriteLine("Failed with HTTPStatus: {0}; \nErrorCode: {1}; \nErrorMessage: {2}; \nRequestID:{3};",
            ex.StatusCode, ex.ErrorCode, ex.Message, ex.RequestId);
        s3Client.AbortMultipartUpload(new AbortMultipartUploadRequest()
        {
            BucketName = bucketName,
            Key = key,
            UploadId = initResp.UploadId
        });
    }
    catch (Exception ex)
    {
        Console.WriteLine("Failed with error info: {0}", ex.Message);
        s3Client.AbortMultipartUpload(new AbortMultipartUploadRequest()
        {
            BucketName = bucketName,
            Key = key,
            UploadId = initResp.UploadId
        });
    }

}

Attention

  1. 上面程序一共分为三个步骤:1. initiate 2. uploadPart 3. complete
  2. UploadPart 方法要求除最后一个Part以外,其他的Part大小都要大于或等于16K。但是Upload Part接口并不会立即校验上传Part的大小(因为不知道是否为最后一块);只有当Complete Multipart Upload的时候才会校验。
  3. Part号码的范围是1~10000。如果超出这个范围,NOS将返回InvalidArgument的错误码。
  4. 分块上传除最后一块外,分块不能小于16k,每一个分块不能大于100M。
  5. 每次上传Part时都要把流定位到此次上传块开头所对应的位置。
  6. 分片上传任务初始化或上传部分分片后,可以使用abortMultipartUpload接口中止分片上传事件。当分片上传事件被中止后,就不能再使用这个Upload ID做任何操作,已经上传的分片数据也会被删除。
  7. 每次上传Part之后,NOS的返回结果会包含一个 PartETag 对象,它是上传块的ETag与块编号(PartNumber)的组合。在后续完成分片上传的步骤中会用到它,因此我们需要将其保存起来,然后在第三步complete的时候使用,具体操作参考上面代码。

4.4.5. 取消分块上传

您可以使用AmazonS3Client.AbortMultipartUpload取消上传事件,具体实现如下:

public void AbortMultipartUpload(string bucketName,string key,string uploadId){
    try
    {
        var abortRequest = new AbortMultipartUploadRequest(){
            BucketName=bucketName,
            Key=key,
            UploadId=uploadId
        };
        s3Client.AbortMultipartUpload(abortRequest);

        Console.WriteLine("Abort Multipart Upload succeeded");
    }
    catch (AmazonS3Exception ex)
    {
        Console.WriteLine("Failed with HTTPStatus: {0}; \nErrorCode: {1}; \nErrorMessage: {2}; \nRequestID:{3};",
            ex.StatusCode, ex.ErrorCode, ex.Message, ex.RequestId);
    }
    catch (Exception ex)
    {
        Console.WriteLine("Failed with error info: {0}", ex.Message);
    }
}

4.4.6. 查看已经上传的分片

查看已经上传的分片可以罗列出指定Upload ID(InitiateMultipartUpload时获取)所属的所有已经上传成功的分片,您可以通过AmazonS3Client.ListParts接口获取已经上传的分片,可以参考以下代码:

public void ListParts(string bucketName,string key,string uploadId){
    var listPartRequest = new ListPartsRequest()
    {
        BucketName = bucketName,
        Key = key,
        UploadId = uploadId,
        MaxParts=5
    };
    try{
        var resp = s3Client.ListParts(listPartRequest);
        foreach(var part in resp.Parts){
            Console.WriteLine("part : {0}",part.ToString());
        }
    }
    catch(AmazonS3Exception ex){
        Console.WriteLine("Failed with HTTPStatus: {0}; \nErrorCode: {1}; \nErrorMessage: {2}; \nRequestID:{3};",
            ex.StatusCode, ex.ErrorCode, ex.Message, ex.RequestId);
    }
    catch{
        Console.WriteLine("Failed with error info: {0}", ex.Message);
    }

}

查看已经上传的分片可以指定以下参数:

参数 描述
MaxParts 响应中的limit个数 类型:整型
PartMumberMarker 分块号的界限,只有更大的分块号会被列出来。 类型:字符串

4.4.7. 查看当前正在进行的分片上传任务

查看正在进行的分片上传任务可以罗列出正在进行,还未完成的分片上传任务,您可以通过AmazonS3Client.ListMultipartUploads接口当前的上传任务,可以参考以下代码:

public void ListMultipartUploads(string bucket){
    try{
        var result = s3Client.ListMultipartUploads(bucket);
        foreach (var mu in result.MultipartUploads)
        {
            Console.WriteLine("Key:{0}, UploadId:{1}", mu.Key, mu.UploadId);
        }
    }
    catch (AmazonS3Exception ex)
    {
        Console.WriteLine("Failed with HTTPStatus: {0}; \nErrorCode: {1}; \nErrorMessage: {2}; \nRequestID:{3};",
            ex.StatusCode, ex.ErrorCode, ex.Message, ex.RequestId);
    }
    catch (Exception ex)
    {
        Console.WriteLine("Failed with error info: {0}", ex.Message);
    }
}

上述代码中使用的options参数如下:

参数值 描述
KeyMarker 指定某一uploads key,只有大于该KeyMarker的才会被列出
MaxUploads 最多返回MaxUploads条记录,默认1000

4.5. 设置文件元信息

文件元数据(object meta),是上传到NOS的文件属性描述信息,C# SDK只支持设置用户自定义元数据。文件元信息可以在各种上传方式(流式上传、单块上传、分块上传)时进行设置。

4.5.1. 设定自定义http header

下面的源代码实现了上传文件时设置Http header:

public void PutObjectWithMeta(string bucketName,string key){
    var request = new PutObjectRequest()
    {
        BucketName = bucketName,
        Key = key,
        ContentBody="content",
    };
    request.Metadata.Add("x-nos-meta-my-metaMyKey1", "MyValue1");
    request.Metadata.Add("x-nos-meta-my-metaMyKey2", "MyValue2");

    try{
        s3Client.PutObject(request);
    }
    catch(AmazonS3Exception ex)
    {
        Console.WriteLine("Failed with HTTPStatus: {0}; \nErrorCode: {1}; \nErrorMessage: {2}; \nRequestID:{3};",
        ex.StatusCode, ex.ErrorCode, ex.Message, ex.RequestId);
    }
    catch (Exception ex)
    {
        Console.WriteLine("Failed with error info: {0}", ex.Message);
    }
}

Attention

  1. 各种上传方式(包括流式上传、单块上传、分块上传)都可以设置元数据信息。流式上传设置元数据的方式与单块上传相同
  2. 文件的元信息可以通过AmazonS3Client.GetObjectMetadata获取;
  3. user meta的名称以”x-amz-meta-“开头,例如设置为:’x-amz-meta-name’,读取名字为x-amz-meta-name的参数即可

4.6. 获取文件元信息

下面的源代码实现了获取对象的自定义原信息:

 public void GetObjectMeta(string bucketName,string key){
    var request = new GetObjectMetadataRequest()
    {
        BucketName = bucketName,
        Key = key,
    };

    try
    {
        var resp = s3Client.GetObjectMetadata(request);
        foreach(var mKey in resp.Metadata.Keys){
            Console.WriteLine("key : {0} , value : {1}",mKey,resp.Metadata[mKey]);
        }
    }
    catch (AmazonS3Exception ex)
    {
        Console.WriteLine("Failed with HTTPStatus: {0}; \nErrorCode: {1}; \nErrorMessage: {2}; \nRequestID:{3};",
        ex.StatusCode, ex.ErrorCode, ex.Message, ex.RequestId);
    }
    catch (Exception ex)
    {
        Console.WriteLine("Failed with error info: {0}", ex.Message);
    }
}