Azure Blob Storage Best to Upload With Stream or Byte Array

• 
• Updated engagement January 07, 2019

• 94.onek
• 6

In this blog, nosotros will learn how to upload, download, or delete a file stream in Azure blob storage using C# and .NET.

Problem

I have a database tabular array with 15 columns. I of these columns is a Blob/Binary Column that we used to store the PDF binary data. Day by day, the storage and the size of the database grew, and at present, the Oracle/SQL Server file size is shut to one.5 TB with x Million records. The size is growing 200GB each year with 1 1000000 records.

Solution

I decided to move the Hulk cavalcade (PDF File Stream) to Azure Blob Storage.

How

Fetch the blob column from the Oracle/SQL Server and upload record by tape with GUID reference. Delight follow the below steps to be more specific.

And then, the concept here is, the hulk/binary column is in byte[] format. When nosotros upload a file to a blob storage, we need to catechumen and save the binary/file stream information with a unique proper name. For that, nosotros will be using the boosted column (GUID) every bit mentioned below. The GUID column volition act as a Master and Foreign key relationship betwixt Oracle/SQL Server databases and the Blob Storage.

Results

After moving the Blob/Binary cavalcade to Azure Hulk Storage, I deleted that column from the tabular array. Information technology saved u.s. 1.4 TB space.

How did I do that?

Step 1

Create a blob storage account in Azure. Here, I'thousand not going over blob storage. If you're not familiar with Azure Blob Storage, get the Understanding of Hulk Storage in Azure

My focus is, how to upload, download, and delete file streams from a blob storage.

Finalize the database activity before moving to C#

Create a column Varchar (50) in your tabular array. I created the 16^th column and named information technology BlobStorageRefKey. Now, update the column with System_GUID that nosotros are going to use to refer between Blob storage and Oracle/SQL Server table.

Here is the SQL query.

1. Update TableName set  BlobStorageRefKey = Sys_GUID();

C#.Net Project.

Footstep 2

Create a web application using Visual Studio.

Step iii

Create a connection or app setting for Azure Blob Storage connection.

Ex,

1. <appSettings>
2.    <addkey="AzureConnection" value="DefaultEndpointsProtocol=https;AccountName= etc….. />
3. </appSettings>

Pace iv

We are going to read the blob column from database and upload with the BlobStorageRefKey column every bit reference to blob storage system.

FYI

When yous have a large number of records in the database, if you fetch all records at a fourth dimension, you may end off with "Out of Memory" upshot.

I recommend fetching 5000 records at a time, processing them, and fetching the adjacent 5000 and so on.

Statistics

Reading 5000 records (Blob Column) from the database and uploading information technology to the Azure Hulk Storage took i.8 minutes for me. I executed from the server similar Azure Virtual Machine/client remote machines. If you practise it from your local machine, it may take upwardly to 10 minutes for every 5000 records.

First of all, how exercise you lot identify the first 5000 and side by side 5000…. When you lot accept one million records, hither is the solution.

Fetch the total number of records (count (BlobStorageRefKey)) from the tabular array and pass a List< BlobStorageRefKey> as an input to the below method (Parameter: Locations) and apply the below method to split the count for every 5000.

8. publicstatic IEnumerable < List < T >> SplitList < T > (List < T > locations,int  nSize = 5000) {
9. for  ( var  i = 0; i < locations.Count; i += nSize) {
10.         yieldreturn locations.GetRange(i, Math.Min(nSize, locations.Count - i));
11.     }
12. }

The above method will give you the List of List count and your first alphabetize in your startingIndex and the list final index in your EndIndex so, utilize these 2 indexes to fetch the records from the primary table using beneath query.

Ex

If you have 100K records in the database, the in a higher place methods result like Listing<List<ColumnName>> so, you volition have two foreach() loops. The first foreach() loop volition give you the first List<T> from List<List<T>> which 5000 and once you are done, your next List<T> will also take 5000 which are adjacent available 5000 for yous so then along...

The below method will assist yous to pull the Blob/Binary Column with BlobReferenceKey to a list of object/class.

5. publicstatic List < ClassName > RetrieveBlobFromDatabase(string strConnection,int  startIndex, int  endIndex, ILog log) {
6. endeavour  {
7.         log.Info("Database records fetching first Time: "  + DateTime.Now);
8.         Panel.WriteLine("Database records fetching first Time: "  + DateTime.Now);
9. var  lstClassName = new  List < ClassName which has Binary and Cord Properties > ();
10. var  con = new  OracleConnection(strConnection);
11.         con.Open();
12. var  commandText = "SELECT PDFColumn, BlobReferenceKeyFROM Maintable WHERE ID between startIndex and EndIndex" ;
14. var  cmd = new  OracleCommand {
15.             CommandText = commandText,
16.                 Connectedness = con
17.         };
18. var  dr = cmd.ExecuteReader();
19. if  (dr.HasRows) {
20. while  (dr.Read()) {
21. var  objInfo = new  ClassName {
22.                     PDF = (byte []) dr[ "PDFColumn" ],
23.                         BLOBSTORAGEKEY = dr["BlobReferenceKey" ].ToString()
24.                 };
25.                 lstClassName.Add(objInfo);
26.             }
27.         }
28.         con.Shut();
29.         log.Info("Database records fetching End Time: "  + DateTime.At present);
30.         Panel.WriteLine("Database records fetching End Time: "  + DateTime.Now);
31. render  lstClassName;
32.     }catch  (Exception e) {
33.         log.Error(due east);
34. throw ;
35.     }
36. }

The lstClassName contains your hulk/binary cavalcade information and BLOBSTORAGEKEY is a GUID associated with that row. At present we are going to upload this binary information to blob storage in Azure.

UPLOAD

Step 5

The below method will help you upload your binary data to the Azure Hulk Storage account.

Practise foreach to the above lstClassName and pass outset property to byte[] parameter and 2^nd belongings to filename.

1. foreach( var  objResult in  lstClassName) {
2.     UploadBinaryAsync(objResult.PDF, objResult.BLOBSTORAGEKEY).Wait();
3. }
10. privatestaticasync Task UploadBinaryAsync(byte [] bytes, string fileName) {
11. try  {
12. var  cloudStorageAccount = CloudStorageAccount.Parse(ConfigurationManager.AppSettings[ "AzureConnection" ]);
14. var  cloudBlobClient = cloudStorageAccount.CreateCloudBlobClient();
16. var  cloudBlobContainer = cloudBlobClient.GetContainerReference( "Your Container Name" );
18. var  cloudBlockBlob = cloudBlobContainer.GetBlockBlobReference(fileName);
19.         cloudBlockBlob.Properties.ContentType = System.Net.Mime.MediaTypeNames.Application.Pdf;
20. if  (!cloudBlockBlob.Exists()) {
21.             using(Stream stream =new  MemoryStream(bytes)) {
22.                 look cloudBlockBlob.UploadFromStreamAsync(stream);
23.             }
24.         }
25.     }catch  (Exception e) {
26.         Console.WriteLine("Exception on UploadBinaryAsync : "  + fileName);
27.         log.Error(e);
28.     }
29. }

Download a file from blob Storage

To download a hulk from the Azure Hulk Storage, you can employ your Oracle/SQL Server table column blobReferenceKey as reference to download that from Azure.

6. publicbyte[] DownloadFileFromBlob(string blobReferenceKey) {
7. var  storageAccount = CloudStorageAccount.Parse(ConfigurationManager.ConnectionStrings[ "blobConnection" ].ToString());
8. var  blobClient = storageAccount.CreateCloudBlobClient();
10. var  container = blobClient.GetContainerReference( "Your Container Proper name" );
12. var  blockBlob = container.GetBlockBlobReference(blobReferenceKey);
14.     using(var  ms = new  MemoryStream()) {
15. if  (blockBlob.Exists()) {
16.             blockBlob.DownloadToStream(ms);
17.         }
18. return  ms.ToArray();
19.     }
20. }

If you are using a MVC application form this method from FileResult looks like below.

1. [HttpGet]
2. public  FileResult DownloadSds(string blobStorageReference) {
3. var  resultInfo = DownloadFileFromBlob(blobStorageReference);
4. render  File(resultInfo, System.Net.Mime.MediaTypeNames.Awarding.Pdf, fileName + ".pdf" );
5. }

Delete a Blob from Blob Storage

8. privatestaticvoid DeleteBlob(cord blobReferenceKey, ILog log) {
9. try  {
10. var  cloudStorageAccount = CloudStorageAccount.Parse(ConfigurationManager.AppSettings[ "AzureConnection" ]);
11. var  cloudBlobClient = cloudStorageAccount.CreateCloudBlobClient();
12. var  cloudBlobContainer = cloudBlobClient.GetContainerReference( "your container name" );
13. var  blobReference = cloudBlobContainer.GetBlobReference(blobReferenceKey);
14. if  (blobReference.Exists()) {
15.             log.Info(blobRef);
16.             blobReference.DeleteIfExistsAsync();
17.         }
18.     }catch  (Exception east) {
19.         Console.WriteLine(e);
20. throw ;
21.     }
22. }

Finally, this is how it looks in the Azure Hulk Storage.

arnettbizin1998.blogspot.com

Source: https://www.c-sharpcorner.com/blogs/uploaddownloaddelete-a-file-file-stream-in-azure-blob-storage-c-sharpnet

Share this post