How to quickly read large files(.CSV 100 million lines)?

[patrick]

Hello~~

I need your help.


- Development environment : C# Winform (VS2019), .NET Framework 4.6.1
- The csv file contains data of 100 million lines.
====> It took over an hour to read a CSV file containing data of 100 million lines(=Big Data).
====> It should not take more than an hour.

Is there any way to shorten the time?

Please reply

 

[Skydiver]

Show us your current code for reading the CSV. I am guessing you are doing things inefficiently based on past code you have shown us for doing other things.

What are you doing with the data? Do you really need all the contents of the CSV in memory all at the same time, or do you need to act on each row of data at a time and not need that row anymore?

What version of .NET are you using? New versions support the use of spans which dramatically cut down on having to allocate memory and copy data around. These small savings multiply into huge performance gains.

 

[patrick]

Show us your current code for reading the CSV. I am guessing you are doing things inefficiently based on past code you have shown us for doing other things.

What are you doing with the data? Do you really need all the contents of the CSV in memory all at the same time, or do you need to act on each row of data at a time and not need that row anymore?

What version of .NET are you using? New versions support the use of spans which dramatically cut down on having to allocate memory and copy data around. These small savings multiply into huge performance gains.

Your Answer) Do you really need all the contents of the CSV in memory all at the same time?
===> I will draw a Chart (Graph) in C#.NET

Your Answer) do you need to act on each row of data at a time and not need that row anymore?
===> I didn't understand what it meant.

Your Answer) New versions support the use of spans which dramatically cut down on having to allocate memory and copy data around.
===> VS2019, C#.NET, Winform, I can check the .NET version on Thursday at 9am.

New versions support the use of spans which dramatically cut down on having to allocate memory and copy data around.
===> Please teach me. Please...

 

[Skydiver]

Do you really need all the contents of the CSV in memory all at the same time?
===> I will draw a Chart (Graph) in C#.NET

You need just a row from the CSV just long enough to add the point into the chart and then discard the row.

You still have not shown us your code.

 

[Skydiver]

New versions support the use of spans which dramatically cut down on having to allocate memory and copy data around.
===> Please teach me. Please...

C# - All About Span: Exploring a New .NET Mainstay

docs.microsoft.com

 

[patrick]

You need just a row from the CSV just long enough to add the point into the chart and then discard the row.

You still have not shown us your code.

- Chart : DevExpress Chart

C# - All About Span: Exploring a New .NET Mainstay

docs.microsoft.com

====>

GitHub - nreco/csv: Fast C# CSV parser

Fast C# CSV parser. Contribute to nreco/csv development by creating an account on GitHub.

github.com

Is this the same as your example?

 

[Skydiver]

Show us your code. How are you loading data into DevExpress Chart?

 

[Skydiver]

GitHub - nreco/csv: Fast C# CSV parser

Fast C# CSV parser. Contribute to nreco/csv development by creating an account on GitHub.

github.com

Is this the same as your example?

No.

You asked to read about spans. I provided a link about spans.

 

[patrick]

No.

You asked to read about spans. I provided a link about spans.

Thank you for answer.

 

[Skydiver]

You still haven't shared your code for reading the CSV file data into a chart. I have a suspicion that you are using an inefficient way of loading in the data into your chart.