Azure Batch equivalent of FILE.AppendAllText?

[etl2016]

Hi,

Could you please let me know the Azure Batch Service (Storage Account blob processing) equivalent of C# FILE.AppendAllText?

Basically, I have an input file. I need to read it line by line with Stream Reader or its Azure Batch Service equivalent. Do transformation. Write the output file to Blob Storage Account. And this should happen line by line in Azure

thank you

 

[Skydiver]

My quick read of Azure Batch is that it deals with entire files at a time, instead of incrementally streaming the file contents.

Azure Batch runs large parallel jobs in the cloud - Azure Batch

Learn about using the Azure Batch service for large-scale parallel and HPC workloads.

docs.microsoft.com

So are you actually asking about how to read a file line by line and then write it out line by line within an Azure Batch Task? (e.g. right after step 4 and right before step 5 in the linked text/image)

Or are you asking about how to upload and Azure Batch (step 1) line by line, and then take the download (step 6) and write to the blob storage account?

 

[etl2016]

thank you, yes.

For instance, lets take non-Azure standard C# processing scenario. Lets say I have 10 row input file, my process takes row 1, does the transformation creates output file with 1 row. The, repeats the process for row 2. And, in here, it uses FILE.AppendAllText to enhance the output file to hold 1+1=2 rows. And this continues until the last row is processed and the output file holds 9+1=10 rows.

I am looking for Azure Batch Service equivalent of FILE.AppendAllText, so that either #1. output file is constructed on the processing VM node and then it is pushed onto Storage Account's blob as a single blob or #2.row-by-row it is pushed and appended to the blob in the Blob container. Any option that works and is performant is okay

thank you once again

 

[Skydiver]

Did you look at this?

Persist job and task output to a data store - Azure Batch

Learn how to persist output data from Azure Batch tasks and jobs to Azure Storage or other stores.

docs.microsoft.com

 

[Skydiver]

This sample looks like it may have aspects of what you are looking to do:

azure-batch-samples/CSharp/TextSearch at master · Azure-Samples/azure-batch-samples

Azure Batch and HPC Code Samples. Contribute to Azure-Samples/azure-batch-samples development by creating an account on GitHub.

github.com

 

[etl2016]

thank you, once again.

I ran option 1 of the PersistOutput solution, and as advised in the FileConventionsExample's ReadMe file, the project was published under framework 4.6.1 and the .exe along with associated dll's were uploaded to Azure Batch Service portal using application-package feature. When I ran the solution from Visual Studio 2017, the command line (under Azure portal's pool/job/task/properties ) generated automatically by the program is expecting .exe just under the dynamically constructed "job" temp folder in the vm node's applications folder. So, whatever Debug.zip or Release.zip or Publish.zip folder i upload to application-package area, it displaces the command line expected path by 1 subfolder. Is there a work-around to resolve this problem (without modifying the Github code, as it would have been tested successfully before reaching Github repository or am missing something?)

thank you

 

[Skydiver]

Hmmm... Part of me feels that you went up to the "bin" directory, right clicked on the "Debug" directory, and "Send to compressed file", but the expectation was for you to go into "bin/Debug" directory, select all, and then right click "Send to compressed file". I'm just speculating, though. I've never used Azure Batch so I have no practical experience. I've only been skimming the docs above to give you suggestions about hat to look into.

 

[etl2016]

I added the missing sub-directory in the path (code change), and it started working.

thank you.

 

[Skydiver]

I suggest sending a pull request. They will either accept it indicating that there truly was a big in the code, or they will reject it and hopefully explain why the subdirectory is not needed.