Streamreader skips lines

[muro]

Dear friends,

i have got this code where the Streamreader sometimes reads the file fine and sometimes skips a few lines and i have absoloutely no idea why it is doing this.

            FtpWebRequest request = (FtpWebRequest)WebRequest.Create("ftp://myfile.txt");
            request.Method = WebRequestMethods.Ftp.DownloadFile;
            request.Credentials = new NetworkCredential("user, "pw");
            FtpWebResponse response = (FtpWebResponse)request.GetResponse();
            Stream responseStream = response.GetResponseStream();
            StreamReader reader = new StreamReader(responseStream);

            while (!reader.EndOfStream)
            {
                MessageBox.Show(reader.ReadLine());
            }
            response.Close();
            reader.Close();

 

[Skydiver]

If "myfile.txt" is in Unix file format, the end of line markers are different than on Windows.

 

[muro]

But why does it sometimes work and sometimes not? That´s the part that makes me angry.

 

[muro]

Should i just use another method and download the file and then read the file? Because actually it works very good. This is what i got:

   using (WebClient webcl = new WebClient())
            {
                webcl.DownloadFileAsync(new Uri("http://file.txt/"), datei);
                webcl.DownloadFileCompleted += Webcl_DownloadFileCompleted;
            }
        }
        }

  private void Webcl_DownloadFileCompleted(object sender, AsyncCompletedEventArgs e)
        {
           if (e.Error == null)
            {
                foreach (string line in File.ReadLines(datei))
                {
                    ortstrassenTableAdapter.Insert(Ortid, line);
               }
               orteTableAdapter.UpdateStrassendatei(Strassendatei, Ortid);

            }
            else
            {
                MessageBox.Show("Ein Fehler ist aufgetreten!" + Environment.NewLine + e.Error.Message, "", MessageBoxButtons.OK, MessageBoxIcon.Error);
            }
   
        //}

 

[Skydiver]

Compare the contents of the file downloaded via FTP versus the file downloaded by HTTP. Does `fc.exe /b ftpdownload.txt httpdownload.txt` report that the files are exactly the same? I am guessing that they are different due to the web server doing some file processing on your behalf to setup consistent line endings.

 

[Skydiver]

Aha! UseBinary defaults to true. If you set it to false as part of the FTP download, you should get the correct line endings on Windows:

FtpWebRequest.UseBinary

 

[NoUserHere]

Your code doesn't like me.



Why use FTP if you are not remotely getting files from stream to disk? If you want to use a stream and read to a memory slot, use a webclient and you can handle the complete file after the downloadComepleted event confirms the file completed downloading, just as you've done above. Then read it into memory providing the file is not humongous.

 

[NoUserHere]

If "myfile.txt" is in Unix file format, the end of line markers are different than on Windows.

There should be a little program in the perl folder on unix to fix this. @Skydiver Windows vs. Unix Line Endings And this one is from our buddies at coding horrors. It covers newline The Great Newline Schism

Although, it appears the stream was being disposed and so it could no longer read in the reader. If the OP is blindly using this code in a Try Catch without handling the error, it will appear as though the file is cutting off mid-way. Sorry I don't have more time to play with this today. I need to delve into some ASP work projects.

 

[muro]

Yes that is happening to me too @Sheepings. I guess I will stick with the webclient method i posted above. Thanksd you so much all.

 

[JohnH]

If "myfile.txt" is in Unix file format, the end of line markers are different than on Windows.

StreamReader.ReadLine reads any line ending, see Remarks: StreamReader.ReadLine Method (System.IO) | Microsoft Docs

 

[NoUserHere]

Yes that is happening to me too Sheepings. - It's important to know why.

This will give you something to play with ::

            FtpWebRequest request = (FtpWebRequest)WebRequest.Create("ftp://127.0.0.1/CFile.txt");
            request.Method = WebRequestMethods.Ftp.DownloadFile;
            request.Credentials = new NetworkCredential("admin", "admin");
            using (FtpWebResponse response = (FtpWebResponse)request.GetResponse())
            using (Stream responseStream = response.GetResponseStream())
            {
                var FileName = "SampleFile.txt";
                using (Stream fileStream = File.Create(Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Desktop), FileName)))
                {
                    responseStream.CopyTo(fileStream);
                }
            }

Ponder this. Why does it not throw that exception when running that? Pay attention to what i changed.

No more error. But i concur, you're better with the webclient method.

 

[Skydiver]

If you look under the source code for the WebClient it uses WebRequests anyway. The WebClient just gives a us a much easier interface to think and reason with.

Frankly, I'm confused at to why the underlying network stream would be disposed in the code from post #1. The response and responseReader are not closed until after the while loop. The responseReader has a reference to the responseStream so the garbage collector shouldn't be trying to dispose of the stream, the response object, nor the reader while the loop is still running. I guess I'll just have to try to setup a repro case and try debugging.

 

[NoUserHere]

XAMPP has a FTP server you can install Skydiver. You just need to add your user and group in its backend and then login.

 

[Skydiver]

Okay using XAMPP instead of IIS. Testing with a 7KB text file with 40 lines.

Hmmm... No exception with a console application...

Spoiler

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Text;
using System.Threading.Tasks;

namespace TestFtp
{
    class Program
    {
        static void Main(string[] args)
        {
            FtpWebRequest request = (FtpWebRequest)WebRequest.Create("ftp://localhost/x64.txt");
            request.Method = WebRequestMethods.Ftp.DownloadFile;
            request.Credentials = new NetworkCredential("user", "password");
            FtpWebResponse response = (FtpWebResponse)request.GetResponse();
            Stream responseStream = response.GetResponseStream();
            StreamReader reader = new StreamReader(responseStream);

            while (!reader.EndOfStream)
            {
                Console.WriteLine(reader.ReadLine());
            }
            response.Close();
            reader.Close();
        }
    }
}

Going to try WinForms next... No exception either.

Spoiler

using System;
using System.Collections.Generic;
using System.Drawing;
using System.IO;
using System.Linq;
using System.Net;
using System.Threading.Tasks;
using System.Windows.Forms;

namespace TestFtpWinForms
{
    class TestForm : Form
    {
        public TestForm()
        {
            Text = "Test Form";
            var button = new Button() { Text = "Download", Location = new Point(20, 20) };
            button.Click += (o, e) => Download();
            Controls.Add(button);
        }

        void Download()
        {
            FtpWebRequest request = (FtpWebRequest)WebRequest.Create("ftp://localhost/x64.txt");
            request.Method = WebRequestMethods.Ftp.DownloadFile;
            request.Credentials = new NetworkCredential("user", "password");
            FtpWebResponse response = (FtpWebResponse)request.GetResponse();
            Stream responseStream = response.GetResponseStream();
            StreamReader reader = new StreamReader(responseStream);

            while (!reader.EndOfStream)
            {
                MessageBox.Show(reader.ReadLine());
            }
            response.Close();
            reader.Close();
        }
    }

    static class Program
    {
        [STAThread]
        static void Main()
        {
            Application.EnableVisualStyles();
            Application.SetCompatibleTextRenderingDefault(false);
            Application.Run(new TestForm());
        }
    }
}

I tried running the console and winforms apps about 7 times each. No exceptions thrown. Do I need a bigger test file? I also tried a test file with Linux line endings with the same result.

 

[Skydiver]

Aha! I got it to reproduce the problem. The file needs to not end on a blank line. This won't cause an exception:

Test 1
Test 2
Test 3
~this line is actually blank~

The "~this line is actually blank~" should be a blank line. The online forum editor won't let me leave a blank line there.


But this will

Test 1
Test 2
Test 3

Now time to do some debugging...