Resolved CSVHelper throws error as "No header record found"

[Ankit]

Hi, Below is the code I am trying to read a csv and trying to copy the file in a tabular storage, but I receive the error as "No header record found".
Below is sample csv which i am trying to read

Sample CSv to be read:

PartitionKey;Time;RowKey;State;RPM;Distance;RespirationConfidence;HeartBPM
te123;2020-11-06T13:33:37.593Z;10;1;8;20946;26;815
te123;2020-11-06T13:33:37.593Z;4;2;79944;8;36635;6
te123;2020-11-06T13:33:37.593Z;3;3;80042;9;8774;5
te123;2020-11-06T13:33:37.593Z;1;4;0;06642;6925;37
te123;2020-11-06T13:33:37.593Z;6;5;04740;74753;94628;21
te123;2020-11-06T13:33:37.593Z;7;6;6;2;14;629
te123;2020-11-06T13:33:37.593Z;9;7;126;86296;9157;05
te123;2020-11-06T13:33:37.593Z;5;8;5;3;7775;08
te123;2020-11-06T13:33:37.593Z;2;9;44363;65;70;229
te123;2020-11-06T13:33:37.593Z;8;10;02;24666;2;2

    public Type DataType
        {
            get
            {
                switch( Type.ToUpper() )
                {
                    case "STRING":
                        return typeof(string);

                    case "INT":
                        return typeof( int );

                    case "BOOL":
                    case "BOOLEAN":
                        return typeof( bool );

                    case "FLOAT":
                    case "SINGLE":
                    case "DOUBLE":
                        return typeof( double );

                    case "DATETIME":
                        return typeof( DateTime );

                    default:
                        throw new NotSupportedException( $"CSVColumn data type '{Type}' not supported" );
                }
            }
        }
// This is the method where the stream is passed and the error is received at line "csv.ReadHeader()"
private IEnumerable<Dictionary<string, EntityProperty>> ReadCSV(Stream source, IEnumerable<TableField> cols)
        {
            var size = source.Length;
            source.Position = 0;
            using (TextReader reader = new StreamReader(source, Encoding.UTF8))
            {
                var cache = new TypeConverterCache();
                reader.ReadLine();
                cache.AddConverter<float>(new CSVSingleConverter());
                cache.AddConverter<double>(new CSVDoubleConverter());
                var csv = new CsvReader(reader,
                    new CsvHelper.Configuration.CsvConfiguration(global::System.Globalization.CultureInfo.InvariantCulture)
                    {
                        Delimiter = ";",
                        HasHeaderRecord = true,
                        CultureInfo = global::System.Globalization.CultureInfo.InvariantCulture,
                        TypeConverterCache = cache
                    });
                csv.Read();
                csv.ReadHeader();

                var map = (
                        from col in cols
                        from src in col.Sources()
                        let index = csv.GetFieldIndex(src, isTryGet: true)
                        where index != -1
                        select new { col.Name, Index = index, Type = col.DataType }).ToList();

                while (csv.Read())
                {
                    yield return map.ToDictionary(
                        col => col.Name,
                        col => EntityProperty.CreateEntityPropertyFromObject(csv.GetField(col.Type, col.Index)));
                }

            }

        }
// This is the method from where the stream is being returned to ReadCsv() method above
public async Task<Stream> ReadStream(string containerName, string digestFileName, string fileName, string connectionString)
        {
            string data = string.Empty;
            string fileExtension = Path.GetExtension(fileName);
            var contents = await DownloadBlob(containerName, digestFileName, connectionString);             
              
          
            return contents;
        }

// method where blob is read as stream
public async Task<Stream> DownloadBlob(string containerName, string fileName, string connectionString)
        {
          
            Microsoft.Azure.Storage.CloudStorageAccount storageAccount = Microsoft.Azure.Storage.CloudStorageAccount.Parse(connectionString);
            CloudBlobClient serviceClient = storageAccount.CreateCloudBlobClient();
            CloudBlobContainer container = serviceClient.GetContainerReference(containerName);
            CloudBlockBlob blob = container.GetBlockBlobReference(fileName);
            if (!blob.Exists())
            {
                throw new Exception($"Unable to upload data in table store for document");
            }
          
            return await blob.OpenReadAsync();
          
        }

 

[JohnH]

Cross-post: CsvHelper Error "No header record found" on reading a csv stream

Again the problem seems related to "DownloadBlob".

 

[Ankit]

Cross-post: CsvHelper Error "No header record found" on reading a csv stream

Again the problem seems related to "DownloadBlob".

Well it doesn't seems to be this time, If you see last time I was reading and passing MemoryStream and that worked fine with my readCSV() method. Now if there are large csv files then reading in Memory is not an ideal solution and that is why we are moving to read the Stream using OpenRead() method which returns a object of stream. and in stream I can see the length of contents.

 

[Skydiver]

So you have the length of the stream, did you look at the first few bytes to make sure that the header line is really the first line, and not an empty line?

 

[Skydiver]

Also cross posted in CsvHelper's issues list.

 

[Ankit]

So you have the length of the stream, did you look at the first few bytes to make sure that the header line is really the first line, and not an empty line?

How do I check that? how do I verify the bytes, I have a debugger for reader and in reader I see length and position

 

[JohnH]

using (TextReader reader = new StreamReader(source, Encoding.UTF8)) {
var cache = new TypeConverterCache();

Put a breakpoint on line 2 here and enter ?reader.ReadLine() in Immediate window.

 

[Ankit]

Put a breakpoint on line 2 here and enter ?reader.ReadLine() in Immediate window.

reader.ReadLine() after var cache = new TypeConverterCache(); gives value as null

 

[JohnH]

StreamReader.ReadLine Method (System.IO)

Reads a line of characters from the current stream and returns the data as a string.

docs.microsoft.com

The next line from the input stream, or null if the end of the input stream is reached.

So it is the same problem as your other thread. Likely the stream you have has Position set to end, try setting streams .Position = 0 before you read from it. What does ReadLine return if you do this?

 

[Ankit]

StreamReader.ReadLine Method (System.IO)

Reads a line of characters from the current stream and returns the data as a string.

docs.microsoft.com

So it is the same problem as your other thread. Likely the stream you have has Position set to end, try setting streams .Position = 0 before you read from it. What does ReadLine return if you do this?

streams .Position = 0 gives me null object reference not set. I put this at the beginning of the ReadCsv() method. Other discussion in other forums suggest that the stream is not supported in Csvhelper. Only memory and FileStream is supported. I cant ignore that but would like to know if someone has read Stream using CsvHelper

 

[JohnH]

No, it doesn't. Maybe some place elsewhere in code, but not at that line. Set it in first thing in ReadCSV, then do the test in post 7 again.

 

[Skydiver]

This is in tangent to the thread above. In an old thread in this forum, a poster was asking why a particular stream operation was failing, and after reproducing the problem and then digging in the source code, I found an issue with on of the HTTP network streams where there is a corner case of the needing a line break at a particular location in the data stream, and the caller trying to read a number of bytes (zero bytes as I recall), would confuse the internal data structures of the network stream implementation. MS acknowledged the bug, but declined to fix it because it was such a rare corner case.

Given Microsoft engineers tendency to copy code if they are crossing divisions or businesses -- knee jerk reaction to the antitrust suit and the threat of breaking up the company -- it is possible the same bug may have made it into the Azure data blob streams implementation.

That is just pure speculation on my part with low probability. I think it a higher probability that our OP is mismanaging the stream, specially given that last post where he says that the reference to the stream is null when he is trying to reset the stream position.

Also, most of the network stream implementations, do not support seeking. So if CsvHelper needs seek support (also given that some suggest that CsvHelper only supports FileStream and MemoryStream), then talking directly to a network stream will lead to failure. You will either need to go the easy route with reading everything into a MemoryStream first (which is what you were trying to avoid), or implement your own stream wrapper around the network stream to support buffering and seeking.

 

[JohnH]

StreamReader throws ArgumentNullException if stream is null, and that didn't happen before, so why would it now?

 

[Ankit]

StreamReader throws ArgumentNullException if stream is null, and that didn't happen before, so why would it now?

streams .Position = 0 was set as the first thing in the ReadCsv() method and i get object reference not set to an instance. though i see the valid position and length in the quick watch for the stream

 

[Skydiver]

Sounds odd to to me. Are you sure you are testing the same code that you are compiling?

As an aside, please post your updated code.