memory

[patrick]

It is fast because it is loaded into memory at once. processing speed is faster.
So I try to load it into memory all at once.
Is there any way to solve the memory problem in this case?

All of that will have to be loaded into memory to use the LINQ OrderBy().
==> CSV File Size : 998,139KB, 1,200,000 Row, 43 Column

Your Answer) All of that will have to be loaded into memory
==> What should I do?

 

[jmcilhinney]

Please stop expecting us to provide you with a magic solution. If you want to do X and someone tells you that, in order to do X, you have to do Y then you have to choices: do X or don't do Y. That's it, that's all. There is no magic here. Either load all the data into memory or don't sort it using LINQ. You already know this so there's no point to your question.

 

[cjard]

All of that will have to be loaded into memory to use the LINQ OrderBy().

Or, maybe not. Just load the data you want to sort, and a unique identifier. Sort the data, then retrieve the other data

Or, hand all this problem off to a database and then it's not your problem how it's sorted, stored, loaded. You can rest assured that some very clever people at Microsoft, postgre, oracle, wherever, have spent many hours working very hard on making their database the best, fastest, lightest, whateverest data retrieval system

 

[Skydiver]

And if you don't trust those companies' databases, I also offered the idea of you implementing external sorting which is a well-known CS technique that unfortunately not as popular anymore because of the gobs of memory modern systems now provide.

 

[Skydiver]

You know that we will be better able to help you if you don't keep dodging the question "what are you trying to do?" And please stop saying "doing a code migration". Tell us what the entire program/application does. Why are you porting the code? Why are you abandoing the step of importing the CSV into SQLite?