How to make process very fast on websocket's response ?

[Hardik Dhankecha]

Hi,
I am connecting to Polygon.io using websocket for fetch live market data. We are getting nearly 70,000 records on every minute, and after that I am storing those records according to particular that symbols. Like we get AAPL data then we will store that data into AAPL file, and so on. Same process is for 10,000 symbols, means we need to handle 10,000 files at a time. So when I apply store data into file logic on response then socket response become slower and we are getting message like connection close.
I am using c# for development.
And here I have write my few lines of code for explanation.

public static void WriteRecord(string data)
{
List<LiveAMData> ld = JsonConvert.DeserializeObject<List<LiveAMData>>(data);
foreach (var item in ld)
{
    var fileName = @"D:\MINUTESANDSECONDS\AAPL_QuoteTicks.txt";
    fileLines = System.IO.File.ReadAllLines(fileName).AsParallel().Skip(1).ToList();
    var timeInTicks = Convert.ToInt64(item.t) * TimeSpan.TicksPerMillisecond;
    var result = new DateTime(1970, 1, 1, 0, 0, 0, 0, DateTimeKind.Utc).AddTicks(timeInTicks);
    DateTime newDT = TimeZoneInfo.ConvertTimeFromUtc(result, targetZone);
    fileLines.Add(item.sym +"," +newDT.ToString("dd/MM/yyyy")+","+item.ax +","+item.bx+","+item.ap+","+item.bp+","+item.@as+","+item.bs);
    System.IO.File.WriteAllLines(fileName, fileLines);
}
}

So how to make this process very fast ? I am expecting that we need to make fast process same as we are receiving response in websocket.

 

[JohnH]

Now you for each item read file, add one item, and write file.
Instead you can read file once, loop to add each item, write file once.

 

[Hardik Dhankecha]

Now you for each item read file, add one item, and write file.
Instead you can read file once, loop to add each item, write file once.

Sorry but I didn't get it. So please can you explain it ?

 

[JohnH]

read file
loop items
   add item
end loop
write file

 

[Hardik Dhankecha]

read file
loop items
   add item
end loop
write file

ok. But in this case data is coming from websocket and every time records will be different. So we need to store all symbol records according particular to it's separate file. Like if we get record for AAPL symbol then it will be store in AAPL file. second time if we get record for IBM then it will be store in IBM file. same for other symbols.
So please can you share me your suggestions for this condition ?

 

[JohnH]

If data contains items that should be written to different files then maybe you can group data for each file first.

Since file IO is very slow, depending on amount of data you could also keep some data in memory and only do file IO at regular intervals, for example based on time or current count of items in memory.