Azure Batch equivalent of FILE.AppendAllText?

etl2016

Member
Joined
Jun 29, 2016
Messages
12
Programming Experience
3-5
Hi,

Could you please let me know the Azure Batch Service (Storage Account blob processing) equivalent of C# FILE.AppendAllText?

Basically, I have an input file. I need to read it line by line with Stream Reader or its Azure Batch Service equivalent. Do transformation. Write the output file to Blob Storage Account. And this should happen line by line in Azure

thank you
 

Skydiver

Staff member
Joined
Apr 6, 2019
Messages
1,406
Location
Virginia Beach, VA
Programming Experience
10+
My quick read of Azure Batch is that it deals with entire files at a time, instead of incrementally streaming the file contents.

So are you actually asking about how to read a file line by line and then write it out line by line within an Azure Batch Task? (e.g. right after step 4 and right before step 5 in the linked text/image)

Or are you asking about how to upload and Azure Batch (step 1) line by line, and then take the download (step 6) and write to the blob storage account?
 

etl2016

Member
Joined
Jun 29, 2016
Messages
12
Programming Experience
3-5
thank you, yes.

For instance, lets take non-Azure standard C# processing scenario. Lets say I have 10 row input file, my process takes row 1, does the transformation creates output file with 1 row. The, repeats the process for row 2. And, in here, it uses FILE.AppendAllText to enhance the output file to hold 1+1=2 rows. And this continues until the last row is processed and the output file holds 9+1=10 rows.

I am looking for Azure Batch Service equivalent of FILE.AppendAllText, so that either #1. output file is constructed on the processing VM node and then it is pushed onto Storage Account's blob as a single blob or #2.row-by-row it is pushed and appended to the blob in the Blob container. Any option that works and is performant is okay

thank you once again
 

Skydiver

Staff member
Joined
Apr 6, 2019
Messages
1,406
Location
Virginia Beach, VA
Programming Experience
10+

Skydiver

Staff member
Joined
Apr 6, 2019
Messages
1,406
Location
Virginia Beach, VA
Programming Experience
10+
This sample looks like it may have aspects of what you are looking to do:
 

etl2016

Member
Joined
Jun 29, 2016
Messages
12
Programming Experience
3-5
thank you, once again.

I ran option 1 of the PersistOutput solution, and as advised in the FileConventionsExample's ReadMe file, the project was published under framework 4.6.1 and the .exe along with associated dll's were uploaded to Azure Batch Service portal using application-package feature. When I ran the solution from Visual Studio 2017, the command line (under Azure portal's pool/job/task/properties ) generated automatically by the program is expecting .exe just under the dynamically constructed "job" temp folder in the vm node's applications folder. So, whatever Debug.zip or Release.zip or Publish.zip folder i upload to application-package area, it displaces the command line expected path by 1 subfolder. Is there a work-around to resolve this problem (without modifying the Github code, as it would have been tested successfully before reaching Github repository or am missing something?)

thank you
 

Skydiver

Staff member
Joined
Apr 6, 2019
Messages
1,406
Location
Virginia Beach, VA
Programming Experience
10+
Hmmm... Part of me feels that you went up to the "bin" directory, right clicked on the "Debug" directory, and "Send to compressed file", but the expectation was for you to go into "bin/Debug" directory, select all, and then right click "Send to compressed file". I'm just speculating, though. I've never used Azure Batch so I have no practical experience. I've only been skimming the docs above to give you suggestions about hat to look into.
 

etl2016

Member
Joined
Jun 29, 2016
Messages
12
Programming Experience
3-5
I added the missing sub-directory in the path (code change), and it started working.

thank you.
 

Skydiver

Staff member
Joined
Apr 6, 2019
Messages
1,406
Location
Virginia Beach, VA
Programming Experience
10+
I suggest sending a pull request. They will either accept it indicating that there truly was a big in the code, or they will reject it and hopefully explain why the subdirectory is not needed.
 
Top Bottom