Skip to content

Commit

Permalink
Merge pull request #165 from EmmaZhu/dev
Browse files Browse the repository at this point in the history
Upgrade to Azure Storage Client Library 9.4.2 and fix bugs
  • Loading branch information
vinjiang authored Mar 4, 2019
2 parents 014a24a + 609a9f1 commit ecfadcb
Show file tree
Hide file tree
Showing 46 changed files with 677 additions and 183 deletions.
6 changes: 6 additions & 0 deletions BreakingChanges.txt
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
Tracking Breaking Changes since 0.9.0

- Changed dependency from azure storage client library 9.3.2 to azure storage blob client library 9.4.2 and azure storage file client library 9.4.2
To upgrade to DataMovement Library 0.10.0 from previous version, please uninstall azure storage client library package first to avoid compiling error
- For DataMovement Library on .Net Framework, the minimum required .Net Framework version changed from 4.5 to 4.5.2

Tracking Breaking Changes since 0.8.1

- Change to overwrite destination's metadata with source's metadata in async copying instead of keeping destination's metadata
Expand Down
17 changes: 11 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Microsoft Azure Storage Data Movement Library (0.9.0)
# Microsoft Azure Storage Data Movement Library (0.10.1)

The Microsoft Azure Storage Data Movement Library designed for high-performance uploading, downloading and copying Azure Storage Blob and File. This library is based on the core data movement framework that powers [AzCopy](https://azure.microsoft.com/documentation/articles/storage-use-azcopy/).

Expand Down Expand Up @@ -30,7 +30,7 @@ For the best development experience, we recommend that developers use the offici

## Target Frameworks

- .NET Framework 4.5 or above
- .NET Framework 4.5.2 or above
- Netstandard2.0

## Requirements
Expand Down Expand Up @@ -60,12 +60,17 @@ within your project you can also have them installed by the .NET package manager

## Dependencies

### Azure Storage Client Library
### Azure Storage Blob Client Library

This version depends on Azure Storage Client Library
This version depends on Azure Storage Blob Client Library

- [WindowsAzure.Storage](https://www.nuget.org/packages/WindowsAzure.Storage/)
- [Microsoft.Azure.Storage.Blob](https://www.nuget.org/packages/Microsoft.Azure.Storage.Blob/)

### Azure Storage File Client Library

This version depends on Azure Storage File Client Library

- [Microsoft.Azure.Storage.File](https://www.nuget.org/packages/Microsoft.Azure.Storage.File/)


## Code Samples
Expand Down Expand Up @@ -101,7 +106,7 @@ Once you setup the storage blob context, you can start to use `WindowsAzure.Stor
```csharp
// Setup the number of the concurrent operations
TransferManager.Configurations.ParallelOperations = 64;
// Setup the transfer context and track the upoload progress
// Setup the transfer context and track the upload progress
SingleTransferContext context = new SingleTransferContext();
context.ProgressHandler = new Progress<TransferStatus>((progress) =>
{
Expand Down
10 changes: 10 additions & 0 deletions changelog.txt
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
2019.02.18 Version 0.10.1
* All Services on both .Net Framework and .Net Core
- Changed dependency from azure storage client library 9.3.2 to azure storage blob client library 9.4.2 and azure storage file client library 9.4.2
- Fixed issue of config to disable content MD5 checking in download doesn't take effect during resuming
- Fixed issue of hang when uploading from a non-fixed sized stream
* Blob Service on both .Net Framework and .Net Core
- Fixed issue of block id may not be correct when changing BlockSize during uploading from a non-fixed sized stream to block blob
* All Services on .Net Framework
- Minimum required .Net Framework version is changed from 4.5 to 4.5.2

2018.10.25 Version 0.9.0
* All Services on both .Net Framework and .Net Core
- Upgrade azure storage client library to 9.3.2
Expand Down
25 changes: 17 additions & 8 deletions lib/DataMovement.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,11 @@
<AppDesignerFolder>Properties</AppDesignerFolder>
<RootNamespace>Microsoft.WindowsAzure.Storage.DataMovement</RootNamespace>
<AssemblyName>Microsoft.WindowsAzure.Storage.DataMovement</AssemblyName>
<TargetFrameworkVersion>v4.5</TargetFrameworkVersion>
<TargetFrameworkVersion>v4.5.2</TargetFrameworkVersion>
<FileAlignment>512</FileAlignment>
<SolutionDir Condition="$(SolutionDir) == '' Or $(SolutionDir) == '*Undefined*'">..\</SolutionDir>
<TargetFrameworkProfile />
<DefineConstants>CODE_ACCESS_SECURITY;BINARY_SERIALIZATION;REQUEST_EVENT_ARGS_EXPOSES_REQUEST;$(DefineConstants)</DefineConstants>
<DefineConstants>CODE_ACCESS_SECURITY;BINARY_SERIALIZATION;$(DefineConstants)</DefineConstants>
<RestorePackages>true</RestorePackages>
<WINDOWSDESKTOP />
</PropertyGroup>
Expand Down Expand Up @@ -54,21 +54,27 @@
<Reference Include="Microsoft.Azure.KeyVault.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
<HintPath>..\packages\Microsoft.Azure.KeyVault.Core.1.0.0\lib\net40\Microsoft.Azure.KeyVault.Core.dll</HintPath>
</Reference>
<Reference Include="Microsoft.Azure.Storage.Blob, Version=9.4.2.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
<HintPath>..\packages\Microsoft.Azure.Storage.Blob.9.4.2\lib\net452\Microsoft.Azure.Storage.Blob.dll</HintPath>
</Reference>
<Reference Include="Microsoft.Azure.Storage.Common, Version=9.4.2.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
<HintPath>..\packages\Microsoft.Azure.Storage.Common.9.4.2\lib\net452\Microsoft.Azure.Storage.Common.dll</HintPath>
</Reference>
<Reference Include="Microsoft.Azure.Storage.File, Version=9.4.2.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
<HintPath>..\packages\Microsoft.Azure.Storage.File.9.4.2\lib\net452\Microsoft.Azure.Storage.File.dll</HintPath>
</Reference>
<Reference Include="Microsoft.WindowsAzure.Configuration, Version=1.8.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
<HintPath>..\packages\Microsoft.WindowsAzure.ConfigurationManager.1.8.0.0\lib\net35-full\Microsoft.WindowsAzure.Configuration.dll</HintPath>
<Private>True</Private>
</Reference>
<Reference Include="Microsoft.WindowsAzure.Storage, Version=9.3.2.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
<HintPath>..\packages\WindowsAzure.Storage.9.3.2\lib\net45\Microsoft.WindowsAzure.Storage.dll</HintPath>
<Private>True</Private>
</Reference>
<Reference Include="Newtonsoft.Json, Version=10.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed, processorArchitecture=MSIL">
<HintPath>..\packages\Newtonsoft.Json.10.0.2\lib\net45\Newtonsoft.Json.dll</HintPath>
</Reference>
<Reference Include="System" />
<Reference Include="System.Core" />
<Reference Include="Microsoft.CSharp" />
<Reference Include="System.Data" />
<Reference Include="System.Net.Http" />
<Reference Include="System.Runtime.Serialization" />
<Reference Include="System.XML" />
<Reference Include="WindowsBase" />
Expand All @@ -79,6 +85,7 @@
</Compile>
<Compile Include="AssemblyInfo.cs" />
<Compile Include="ChunkedMemoryStream.cs" />
<Compile Include="Extensions\StorageCopyState.cs" />
<Compile Include="FileNativeMethods.cs" />
<Compile Include="LongPathFile.cs" />
<Compile Include="LongPathFileStream.cs" />
Expand Down Expand Up @@ -210,7 +217,9 @@
</EmbeddedResource>
</ItemGroup>
<ItemGroup>
<None Include="packages.config" />
<None Include="packages.config">
<SubType>Designer</SubType>
</None>
</ItemGroup>
<Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />
<Import Project="$(SolutionDir)\.nuget\NuGet.targets" Condition="Exists('$(SolutionDir)\.nuget\NuGet.targets')" />
Expand All @@ -220,4 +229,4 @@
</PropertyGroup>
<Error Condition="!Exists('$(SolutionDir)\.nuget\NuGet.targets')" Text="$([System.String]::Format('$(ErrorText)', '$(SolutionDir)\.nuget\NuGet.targets'))" />
</Target>
</Project>
</Project>
132 changes: 132 additions & 0 deletions lib/Extensions/StorageCopyState.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
//------------------------------------------------------------------------------
// <copyright file="StorageCopyState.cs" company="Microsoft">
// Copyright (c) Microsoft Corporation
// </copyright>
//------------------------------------------------------------------------------

namespace Microsoft.WindowsAzure.Storage.DataMovement.Extensions
{
using System;

internal enum StorageCopyStatus
{
//
// Summary:
// The copy status is invalid.
Invalid = 0,
//
// Summary:
// The copy operation is pending.
Pending = 1,
//
// Summary:
// The copy operation succeeded.
Success = 2,
//
// Summary:
// The copy operation has been aborted.
Aborted = 3,
//
// Summary:
// The copy operation encountered an error.
Failed = 4
}

internal class StorageCopyState
{
public StorageCopyState(Microsoft.WindowsAzure.Storage.Blob.CopyState blobCopyState)
{
this.CopyId = blobCopyState.CopyId;
this.SetStatus(blobCopyState.Status);
this.Source = blobCopyState.Source;
this.BytesCopied = blobCopyState.BytesCopied;
this.TotalBytes = blobCopyState.TotalBytes;
this.StatusDescription = blobCopyState.StatusDescription;
}

public StorageCopyState(Microsoft.WindowsAzure.Storage.File.CopyState fileCopyState)
{
this.CopyId = fileCopyState.CopyId;
this.SetStatus(fileCopyState.Status);
this.Source = fileCopyState.Source;
this.BytesCopied = fileCopyState.BytesCopied;
this.TotalBytes = fileCopyState.TotalBytes;
this.StatusDescription = fileCopyState.StatusDescription;
}

private void SetStatus(Microsoft.WindowsAzure.Storage.Blob.CopyStatus blobCopyStatus)
{
switch (blobCopyStatus)
{
case Blob.CopyStatus.Invalid:
this.Status = StorageCopyStatus.Invalid;
break;
case Blob.CopyStatus.Pending:
this.Status = StorageCopyStatus.Pending;
break;
case Blob.CopyStatus.Success:
this.Status = StorageCopyStatus.Success;
break;
case Blob.CopyStatus.Aborted:
this.Status = StorageCopyStatus.Aborted;
break;
case Blob.CopyStatus.Failed:
this.Status = StorageCopyStatus.Failed;
break;
default:
this.Status = StorageCopyStatus.Invalid;
break;
}
}

private void SetStatus(Microsoft.WindowsAzure.Storage.File.CopyStatus fileCopyStatus)
{
switch (fileCopyStatus)
{
case File.CopyStatus.Invalid:
this.Status = StorageCopyStatus.Invalid;
break;
case File.CopyStatus.Pending:
this.Status = StorageCopyStatus.Pending;
break;
case File.CopyStatus.Success:
this.Status = StorageCopyStatus.Success;
break;
case File.CopyStatus.Aborted:
this.Status = StorageCopyStatus.Aborted;
break;
case File.CopyStatus.Failed:
this.Status = StorageCopyStatus.Failed;
break;
default:
this.Status = StorageCopyStatus.Invalid;
break;
}
}

//
// Summary:
// Gets the ID of the copy operation.
public string CopyId { get; private set; }
//
// Summary:
// Gets the status of the copy operation.
public StorageCopyStatus Status { get; private set; }
//
// Summary:
// Gets the source URI of a copy operation.
public Uri Source { get; private set; }
//
// Summary:
// Gets the number of bytes copied in the operation so far.
public long? BytesCopied { get; private set; }
//
// Summary:
// Gets the total number of bytes in the source of the copy.
public long? TotalBytes { get; private set; }
//
// Summary:
// Gets the description of the current status, if any.
public string StatusDescription { get; private set; }
}
}
2 changes: 1 addition & 1 deletion lib/Resources.Designer.cs

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ namespace Microsoft.WindowsAzure.Storage.DataMovement.TransferControllers
using System.Threading.Tasks;
using Microsoft.WindowsAzure.Storage.Blob;
using Microsoft.WindowsAzure.Storage.Blob.Protocol;
using Microsoft.WindowsAzure.Storage.DataMovement.Extensions;
using Microsoft.WindowsAzure.Storage.File;

internal abstract class AsyncCopyController : TransferControllerBase
Expand Down Expand Up @@ -497,7 +498,7 @@ private bool HandleStartCopyResult(StorageException se)
if (null != se.RequestInformation
&& BlobErrorCodeStrings.PendingCopyOperation == se.RequestInformation.ErrorCode)
{
CopyState copyState = this.FetchCopyStateAsync().Result;
StorageCopyState copyState = this.FetchCopyStateAsync().Result;

if (null == copyState)
{
Expand Down Expand Up @@ -564,7 +565,7 @@ private async Task GetCopyStateAsync()
this.hasWork = false;
this.StartCallbackHandler();

CopyState copyState = null;
StorageCopyState copyState = null;

try
{
Expand Down Expand Up @@ -593,7 +594,7 @@ private async Task GetCopyStateAsync()
await this.HandleFetchCopyStateResultAsync(copyState);
}

private async Task HandleFetchCopyStateResultAsync(CopyState copyState)
private async Task HandleFetchCopyStateResultAsync(StorageCopyState copyState)
{
if (null == copyState)
{
Expand All @@ -617,7 +618,7 @@ private async Task HandleFetchCopyStateResultAsync(CopyState copyState)
Resources.MismatchFoundBetweenLocalAndServerCopyIdsException);
}

if (CopyStatus.Success == copyState.Status)
if (StorageCopyStatus.Success == copyState.Status)
{
this.UpdateTransferProgress(copyState);

Expand All @@ -632,7 +633,7 @@ private async Task HandleFetchCopyStateResultAsync(CopyState copyState)

this.SetFinished();
}
else if (CopyStatus.Pending == copyState.Status)
else if (StorageCopyStatus.Pending == copyState.Status)
{
this.UpdateTransferProgress(copyState);

Expand All @@ -657,7 +658,7 @@ private async Task HandleFetchCopyStateResultAsync(CopyState copyState)
}
}

private void UpdateTransferProgress(CopyState copyState)
private void UpdateTransferProgress(StorageCopyState copyState)
{
if (null != copyState &&
copyState.TotalBytes.HasValue)
Expand Down Expand Up @@ -782,7 +783,7 @@ await sourceLocation.AzureFile.FetchAttributesAsync(
protected abstract Task DoFetchDestAttributesAsync();
protected abstract Task<string> DoStartCopyAsync();
protected abstract void DoHandleGetDestinationException(StorageException se);
protected abstract Task<CopyState> FetchCopyStateAsync();
protected abstract Task<StorageCopyState> FetchCopyStateAsync();
protected abstract Task SetAttributesAsync(SetAttributesCallbackAsync setAttributes);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ namespace Microsoft.WindowsAzure.Storage.DataMovement.TransferControllers
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;
using Microsoft.WindowsAzure.Storage.DataMovement;
using Microsoft.WindowsAzure.Storage.DataMovement.Extensions;

/// <summary>
/// Blob asynchronous copy.
Expand Down Expand Up @@ -121,7 +122,7 @@ protected override Task<string> DoStartCopyAsync()
if (BlobType.BlockBlob == this.destBlob.BlobType)
{
return (this.destBlob as CloudBlockBlob).StartCopyAsync(
this.SourceFile.GenerateCopySourceFile(),
this.SourceFile.GenerateCopySourceUri(),
null,
destAccessCondition,
Utils.GenerateBlobRequestOptions(this.destLocation.BlobRequestOptions),
Expand Down Expand Up @@ -168,15 +169,15 @@ protected override void DoHandleGetDestinationException(StorageException se)
}
}

protected override async Task<CopyState> FetchCopyStateAsync()
protected override async Task<StorageCopyState> FetchCopyStateAsync()
{
await this.destBlob.FetchAttributesAsync(
Utils.GenerateConditionWithCustomerCondition(this.destLocation.AccessCondition),
Utils.GenerateBlobRequestOptions(this.destLocation.BlobRequestOptions),
Utils.GenerateOperationContext(this.TransferContext),
this.CancellationToken);

return this.destBlob.CopyState;
return new StorageCopyState(this.destBlob.CopyState);
}

protected override async Task SetAttributesAsync(SetAttributesCallbackAsync setCustomAttributes)
Expand Down
Loading

0 comments on commit ecfadcb

Please sign in to comment.