Home > Azure > Exception While uploading Block Blob : Azure blob storage invalid content .StorageClientException – The specified block list is invalid

Exception While uploading Block Blob : Azure blob storage invalid content .StorageClientException – The specified block list is invalid

Hi,

Windows Azure Blob is the simplest way to store text or binary data with windows azure. With the new Version of Azure, there are 2 types of Blob one is Block Blob and other is Page Blob. I suggest to go through this link before further going further in my blog .
In digest,
Block Blob –
Block blob is made of block each having unique blockID.Maximum block size is 4MB
Page Blob –
Page blob is collection of pages of size 512 bytes.

Block Blob provides stream of data. Whereas Page blob provides data in page wise.
We prefer Page blob for frequent Insert, update and read i.e. random read/ write. Whereas Block Blob is for large size where we have only read operation.

When we upload huge data into Blob, it takes too much time to upload. With the latest version of Azure SDK, where we can break the file into multiple block and start uploading parallel into ATS. Each and Every Block uploaded into Azure is associated with Block ID.

int blockSize = 4*1024;
using (MemoryStream ms = new MemoryStream(data)) // Read the Data from FileStream and copy into MemoryStream
{
ms.Position = 0;
// block counter
var blockId = 0;
// open file
while (ms.Position < ms.Length)
{
// calculate buffer size (blockSize in KB)
var bufferSize = blockSize * 1024 < ms.Length - ms.Position ? blockSize * 1024 : ms.Length - ms.Position;
var buffer = new byte[bufferSize];
// read data to buffer
ms.Read(buffer, 0, buffer.Length);
// save data to memory stream and put to storage
using (var mstream = new MemoryStream(buffer,true))
{
// set stream position to start
mstream.Position = 0;
// convert block id to Base64 Encoded string
var blockIdBase64 = Convert.ToBase64String(Encoding.UTF8.GetBytes(blockId.ToString(CultureInfo.InvariantCulture)));
blob.PutBlock(blockIdBase64, mstream, null);
blockIdList.Add(blockIdBase64);
// increase block id
blockId++;
}

}
blob.PutBlockList(blockIdList);
}

In the above code, we are reading chunks of data from file and upload the data to block blob with ID.Here is code for that
blob.PutBlock(blockIdBase64, mstream, null);
blockIdList.Add(blockIdBase64);

Once all the blocks are uploaded dont get forget about calling
blob.PutBlockList(blockIdList);
Otherwise you blob is uncommitted transaction. When you run the above code, after uploading certain number of block, system throws exception “The specified block list is invalid”

@ var blockIdBase64 = Convert.ToBase64String(Encoding.UTF8.GetBytes(blockId.ToString(CultureInfo.InvariantCulture)));

When I troubleshoot for hours, I got know that BlockID length are not same , If the BlockId starts from 0 then blockID will have variable length like 2 digit or 3 digit and soon, BlockID length varies. Since BlockID’s length are varying , ATS unable to upload block. To fix the Issue, Just replace

var blockIdBase64 = Convert.ToBase64String(Encoding.UTF8.GetBytes(blockId.ToString(CultureInfo.InvariantCulture)));
to

var blockIdBase64 = Convert.ToBase64String(Encoding.UTF8.GetBytes(blockId.ToString(CultureInfo.InvariantCulture).PadLeft(32, ‘0’)));

Now I see BlockIds are of Fixed Length.

I hope this will help someone who are facing same issue where is pretty tuff to identify the issue.

-Mahender

Advertisements
Categories: Azure
  1. Shweta
    October 7, 2013 at 10:59 am

    Hi Mahender,

    Thanks for your blog.
    Even I’m facing the same issue, but was unable to fix it with the solution you provided. Please check this code and let me know where I’m going wrong.
    I want to use a .net thread pool for the upload operation.

    Upload(imgFile)
    {
    FileInfo file = new FileInfo(imgFile);
    byte[] data = File.ReadAllBytes(imgFile);

    IEnumerable mySet = GetFileBlocks(data);
    foreach (var item in mySet)
    {
    blockIdList.Add(item.Id);
    }

    //Creating the array of handles.
    WaitHandle[] waitHandles = new WaitHandle[blockIdList.Count];
    MethodParameters threadParams = new MethodParameters();
    threadParams.cloudBlobBlock = blob;

    ThreadPool.SetMaxThreads(6, 5);

    for (int i = 0; i <blockIdList.Count; i++)
    {
    waitHandles[i] = new AutoResetEvent(false);
    threadParams.threadID = mySet.ElementAt(i).Id;
    threadParams.dataSet = mySet.ElementAt(i);
    threadParams.waitHandler = waitHandles[i] as AutoResetEvent;
    Interlocked.Increment(ref threadCount);
    ThreadPool.QueueUserWorkItem(new WaitCallback(ThreadProc), threadParams);
    }

    while (Interlocked.Read(ref threadCount) != 0)
    {
    Thread.Sleep(500);
    }

    blob.PutBlockList(blockIdList);
    }

    public void ThreadProc(object data)
    {
    try
    {
    MethodParameters controller = (MethodParameters)data;
    string index = controller.threadID;
    controller.cloudBlobBlock.PutBlock(controller.threadID, new MemoryStream(controller.dataSet.Content, true), null);

    // Decrement the count of blocked threads.
    Interlocked.Decrement(ref threadCount);

    //lets inform the main thread that this method has finished the execution.
    AutoResetEvent waitHandle = (AutoResetEvent)controller.waitHandler;
    waitHandle.Set();
    }
    catch (Exception ex)
    {

    }
    }

    private IEnumerable GetFileBlocks(byte[] fileContent)
    {
    try
    {
    HashSet hashSet = new HashSet();
    if (fileContent.Length == 0)
    return new HashSet();
    int blockId = 0;
    int uploaded = 0;
    int currentBlockSize = BlockLength;
    while (currentBlockSize == BlockLength)
    {
    if ((uploaded + currentBlockSize) > fileContent.Length)
    currentBlockSize = fileContent.Length – uploaded;
    byte[] chunk = new byte[currentBlockSize];
    Array.Copy(fileContent, uploaded, chunk, 0, currentBlockSize);
    hashSet.Add(new FileBlock()
    {
    Content = chunk,
    //Id = Convert.ToBase64String(System.BitConverter.GetBytes(blockId))
    Id= Convert.ToBase64String(Encoding.UTF8.GetBytes(blockId.ToString(CultureInfo.InvariantCulture).PadLeft(32, ‘0’)))
    });
    uploaded += currentBlockSize;
    blockId++;
    }

    return hashSet;
    }
    catch (Exception ex)
    {
    Console.WriteLine(“File Blocks creation FAILED!!!!”);
    throw ex;
    }

    }

    [Serializable]
    private class MethodParameters
    {
    public FileBlock dataSet { get; set; }
    public CloudBlockBlob cloudBlobBlock { get; set; }
    public string threadID { get; set; }
    public AutoResetEvent waitHandler { get; set; }
    }

    Thanks in advance.

    Regards,
    Shweta

  2. October 7, 2013 at 5:37 pm

    Can you please check Blob Name and see whether CloudBlockBlob is according to naming convention. Looks like you upload part of code. I need additional code for fixing ur code problem. My guess is either you don’t have permission or Blob is not according to Naming convention. Please try to upload block blob without any thread and check whether it is working or not. for me to debug the code, take time and also need additional code (References) which is missing . Hash Set is of generic type.. how can u create non generic type HashSet.

    • Shweta
      October 8, 2013 at 12:07 pm

      Thanks for your reply Mahender. I’am able to fix the issue. There was an issue in the upload method where I need to create the object threadParams within the foreach loop. This is because in a multithreading environment, if the same object is used it is replacing the data with the latest value.

      int index = 0;
      foreach (var i in mySet)
      {
      MethodParameters threadParams = new MethodParameters();
      threadParams.cloudBlobBlock = blob;
      threadParams.threadID = i.Id;
      threadParams.dataSet = mySet.ElementAt(index);
      Interlocked.Increment(ref threadCount);
      ThreadPool.QueueUserWorkItem(new WaitCallback(Put), (object)threadParams);
      index++;
      }

      Thanks,
      Shweta

  3. October 8, 2013 at 12:25 pm

    Great Shweta…

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: