Handling out of memory exception

[OldBoy]

Hi guys
I am beginner in C# programming. Have a small program running perfectly. It handles csv data files, sizes are increasing. At one point I got "out of memory exception..." problem. I read somewhere that C# handles only about 2GB memory, but with some codes (tricks) it can be increased. (maybe I misunderstood it)
I would like to know if I can have C# handling a higher level of RAM, which I do have in my PC.
Thanks for any help

 

[Skydiver]

I read somewhere that C# handles only about 2GB memory

Untrue. I'm currently trying to convince a team to fix their code because they are holding on to 85GB of memory -- on our 48GB server that they share with 100+ other apps.

I think you are mistaking the default array indexing limit with the memory limit.

 

[Skydiver]

Anyway, do you really need all the rows of the CSV in memory all at the same time? If not, then process each row of the .CSV one at a time. (This was the basic issue of that team's code where they were holding everything in memory despite them only actually working on just one row at a time. The only reason why they were keeping everything in memory was so that later they could call Count(). *sigh*)

 

[cjard]

Amen; if you need more than 2Gb of memory to process a CSV you might need a different way to process your CSV..

 

[OldBoy]

Anyway, do you really need all the rows of the CSV in memory all at the same time? If not, then process each row of the .CSV one at a time. (This was the basic issue of that team's code where they were holding everything in memory despite them only actually working on just one row at a time. The only reason why they were keeping everything in memory was so that later they could call Count(). *sigh*)

Thanks, this will be the solution.
I guess, my program really reads all the file into RAM, but I need to compare only two lines at a time: always the first line (index 0), and then each line, one by one. This is my code: ( do not laugh, I am a real beginner)

fbe = File.ReadAllLines(fnevbe, Encoding.UTF8);
            for (int i = 0; i < fbe.Length - 1; i++)
            {
                osszeg = 0;
                string[] sv = fbe[0].Split(';');
                string[] sv2 = fbe[i + 1].Split(';');

                for (int j = 0; j < fbe.Length; j++)
                {
                    string szam1string = sv[j + 1];
                    szam1 = Convert.ToInt32(szam1string);
                    string sorszamstring = sv2[0];
                    sorszam = Convert.ToInt32(sorszamstring);
                    string szam2string = sv2[j + 1];
                    szam2 = Convert.ToInt32(szam2string);

                      if (szam2 > szam1)                                     
                    {
                        osszeg = osszeg + 1;                               
                        if (osszeg > maxosszeg)
                        {
                            maxosszeg = osszeg;                            
                            maxsor = sorszam;
                            sorindex = i + 1;
                        }
                    }
                }
            }

I should really solve that my program reads it only lline by line, and forget the previous data.

 

[Skydiver]

Instead of File.ReadAllLines() followed by the for loop, you could use:

bool headerRow = true;
foreach(var line in File.ReadLines(...))
{
    if (headerRow)
    {
        // parse header row
    }
    else
    {
        // parse data row
    }
    headerRow = false;
}

Also consider using the TextFieldParser.

C# TextFieldParser Examples: Read CSV - Dot Net Perls

Use TextFieldParser to read CSV files. Use the Microsoft.VisualBasic.FileIO namespace.

www.dotnetperls.com

 

[OldBoy]

Instead of File.ReadAllLines() followed by the for loop, you could use:

bool headerRow = true;
foreach(var line in File.ReadLines(...))
{
    if (headerRow)
    {
        // parse header row
    }
    else
    {
        // parse data row
    }
    headerRow = false;
}

Also consider using the TextFieldParser.

C# TextFieldParser Examples: Read CSV - Dot Net Perls

Use TextFieldParser to read CSV files. Use the Microsoft.VisualBasic.FileIO namespace.

www.dotnetperls.com

Thanks! I will try to convert my program into such format. Hope I succeed.

 

[cjard]

Also consider using the TextFieldParser.

Or a decent CSV library

CsvHelper is fairly simple and popular

You'd have a CSV:

Szam,Osseg
1,2
3,4

You'd have a class:

class Whatever{
  public int Szam {get;set;}
  public int Osseg {get;set;}
}

You'd read the file:

using (var reader = new StreamReader("path\\to\\file.csv"))
using (var csv = new CsvReader(reader, CultureInfo.InvariantCulture)){   

  Whatever w = null;
  foreach(var r in csv.GetRecords<Whatever>())
    if(w == null) {
      //first row
      w= r;
      continue;
    }

    if(r.Szam > w.Szam) ...
 }

All this shooing and converting of data you're doing; that's what the CSV library is for. If your file doesn't have a header row, read all of Getting Started | CsvHelper

 

[OldBoy]

Thanks for both of you guys, these are good tips.
First great result is: I can alread open and handle the 2 GB CSV file!
I have started already handling data inside. In your posts there are many new details for me that I have to study and try.
I am sure, slowly I can convert my program into the new format.
I will show up if stuck, and, if not, will surely let you know of the final good result.

 

[OldBoy]

Or a decent CSV library

CsvHelper is fairly simple and popular

You'd have a CSV:

Szam,Osseg
1,2
3,4

You'd have a class:

class Whatever{
  public int Szam {get;set;}
  public int Osseg {get;set;}
}

You'd read the file:

using (var reader = new StreamReader("path\\to\\file.csv"))
using (var csv = new CsvReader(reader, CultureInfo.InvariantCulture)){   

  Whatever w = null;
  foreach(var r in csv.GetRecords<Whatever>())
    if(w == null) {
      //first row
      w= r;
      continue;
    }

    if(r.Szam > w.Szam) ...
 }

All this shooing and converting of data you're doing; that's what the CSV library is for. If your file doesn't have a header row, read all of Getting Started | CsvHelper

Error: r.Szam does not exist in this context (w.Szam is OK)
Why?

 

[Skydiver]

Show us your code.

Since @cjard eschews the C# coding convention of using Allman indents, and rather prefers to use OTBS like in Java, he is missing an opening curly brace in his demo code on line 5. That causes line 12 to be outside of the foreach scope and the compiler complains.

 

[OldBoy]

Show us your code.

Since @cjard eschews the C# coding convention of using Allman indents, and rather prefers to use OTBS like in Java, he is missing an opening curly brace in his demo code on line 5. That causes line 12 to be outside of the foreach scope and the compiler complains.

OK OK, just wanted to see how it works. It is running, after I installed csvhelper.
I really need time to experiment with these solutions new to me.

 

[cjard]

Any time that you're writing code, if you think "gee, this is laborious/hard/repetitive/boring/menial" have a look around to see if something exists to take that pain away; you're almost certain to have not been the first person to think so

Parsing strings into CSVs is a good done-to-death example

 

[Skydiver]

Very true, except if you are in school. In which case, you are expected to do it the hard laborious way or best risk a low grade for going around the parameters of the problem, or at worse face expulsion for plagiarism.