Resolved CSVHelper throws error as "No header record found"

[Ankit]

Hi, Below is the code I am trying to read a csv and trying to copy the file in a tabular storage, but I receive the error as "No header record found".
Below is sample csv which i am trying to read

Sample CSv to be read:

PartitionKey;Time;RowKey;State;RPM;Distance;RespirationConfidence;HeartBPM
te123;2020-11-06T13:33:37.593Z;10;1;8;20946;26;815
te123;2020-11-06T13:33:37.593Z;4;2;79944;8;36635;6
te123;2020-11-06T13:33:37.593Z;3;3;80042;9;8774;5
te123;2020-11-06T13:33:37.593Z;1;4;0;06642;6925;37
te123;2020-11-06T13:33:37.593Z;6;5;04740;74753;94628;21
te123;2020-11-06T13:33:37.593Z;7;6;6;2;14;629
te123;2020-11-06T13:33:37.593Z;9;7;126;86296;9157;05
te123;2020-11-06T13:33:37.593Z;5;8;5;3;7775;08
te123;2020-11-06T13:33:37.593Z;2;9;44363;65;70;229
te123;2020-11-06T13:33:37.593Z;8;10;02;24666;2;2

    public Type DataType
        {
            get
            {
                switch( Type.ToUpper() )
                {
                    case "STRING":
                        return typeof(string);

                    case "INT":
                        return typeof( int );

                    case "BOOL":
                    case "BOOLEAN":
                        return typeof( bool );

                    case "FLOAT":
                    case "SINGLE":
                    case "DOUBLE":
                        return typeof( double );

                    case "DATETIME":
                        return typeof( DateTime );

                    default:
                        throw new NotSupportedException( $"CSVColumn data type '{Type}' not supported" );
                }
            }
        }
// This is the method where the stream is passed and the error is received at line "csv.ReadHeader()"
private IEnumerable<Dictionary<string, EntityProperty>> ReadCSV(Stream source, IEnumerable<TableField> cols)
        {
            var size = source.Length;
            source.Position = 0;
            using (TextReader reader = new StreamReader(source, Encoding.UTF8))
            {
                var cache = new TypeConverterCache();
                reader.ReadLine();
                cache.AddConverter<float>(new CSVSingleConverter());
                cache.AddConverter<double>(new CSVDoubleConverter());
                var csv = new CsvReader(reader,
                    new CsvHelper.Configuration.CsvConfiguration(global::System.Globalization.CultureInfo.InvariantCulture)
                    {
                        Delimiter = ";",
                        HasHeaderRecord = true,
                        CultureInfo = global::System.Globalization.CultureInfo.InvariantCulture,
                        TypeConverterCache = cache
                    });
                csv.Read();
                csv.ReadHeader();

                var map = (
                        from col in cols
                        from src in col.Sources()
                        let index = csv.GetFieldIndex(src, isTryGet: true)
                        where index != -1
                        select new { col.Name, Index = index, Type = col.DataType }).ToList();

                while (csv.Read())
                {
                    yield return map.ToDictionary(
                        col => col.Name,
                        col => EntityProperty.CreateEntityPropertyFromObject(csv.GetField(col.Type, col.Index)));
                }

            }

        }
// This is the method from where the stream is being returned to ReadCsv() method above
public async Task<Stream> ReadStream(string containerName, string digestFileName, string fileName, string connectionString)
        {
            string data = string.Empty;
            string fileExtension = Path.GetExtension(fileName);
            var contents = await DownloadBlob(containerName, digestFileName, connectionString);             
              
          
            return contents;
        }

// method where blob is read as stream
public async Task<Stream> DownloadBlob(string containerName, string fileName, string connectionString)
        {
          
            Microsoft.Azure.Storage.CloudStorageAccount storageAccount = Microsoft.Azure.Storage.CloudStorageAccount.Parse(connectionString);
            CloudBlobClient serviceClient = storageAccount.CreateCloudBlobClient();
            CloudBlobContainer container = serviceClient.GetContainerReference(containerName);
            CloudBlockBlob blob = container.GetBlockBlobReference(fileName);
            if (!blob.Exists())
            {
                throw new Exception($"Unable to upload data in table store for document");
            }
          
            return await blob.OpenReadAsync();
          
        }

 

[Ankit]

Sounds odd to to me. Are you sure you are testing the same code that you are compiling?

As an aside, please post your updated code.

updated the code with latest readCsv() method. If someone with azure can test this from their end, reading blob as i did with DownloadBlob method and passing the stream to be read by csvHelper .

 

[JohnH]

updated the code with latest readCsv() method

It is not a good idea to change the previous code that has been discussed for some time in this thread, it could make the following replies look really weird and wrong. Instead you should make a new post with the new code, so we can follow the progress.

Anyway, line 38 in updated post 1 should not be there, it will break your code. You should do as described in post 7 to test what the first line returns now. After the test you must stop the program, since reading a line from the stream will break the code that follows. The value you should see in IM is the header string. If that is the result you are on the right path and can proceed. Fix the code, do the test and please confirm.

 

[Skydiver]

updated the code with latest readCsv() method.

No! Don't do that on this forum (and most other forums which are threaded). It maybe the accepted practice to update the question on StackOverflow (because there responses are not threaded), but not here.

 

[Ankit]

Finally, the issue is resolved now. It was a StreamReader which was setting the position of the stream to the end and thats why it was returning null and for some reason I am still unable to set Position to 0 but now manipulated the function and removed StreamReader before calling it in ReadCsv()

 

[Skydiver]

Glad you resolved it. I hope your taking time to update all your other crossposts so that other people aren't wasting their time still trying to answer your question.

Hopefully someday you'll also explain why having the StreamReader worked when you were using a MemoryStream or a FileStream, but not the Azure blob stream.

 

[Ankit]

Glad you resolved it. I hope your taking time to update all your other crossposts so that other people aren't wasting their time still trying to answer your question.

Hopefully someday you'll also explain why having the StreamReader worked when you were using a MemoryStream or a FileStream, but not the Azure blob stream.

yes other posts are updated. MemoryStream and FileStream always worked and initially I was working with MemoryStream but blob stream was taken as to avoid memory exception for large size blobs

 

[Skydiver]

Yes, and you were also the stream reader with the file stream and memory streams? If so, why did it use to work?