speed up

[Asdshka]

help make the program faster. at least 10 times faster

public static void Main(string[] args)
{
    Task firstTask = Task.Factory.StartNew(() => InizializeCycle(Path.Combine(DesktopPath, "1.txt"), Path.Combine(DesktopPath, "01.txt")));
    Console.WriteLine($"Start Task: {firstTask.Id}");
    firstTask.Wait();
   
    Task.WaitAll(firstTask);
    Console.WriteLine("All threads complete");
 
    Console.Read();
}
 
static readonly string CurrDir = Environment.CurrentDirectory, UserName = Environment.UserName;
static readonly string DesktopPath = Environment.GetFolderPath(Environment.SpecialFolder.Desktop);
 
const int BUFFERSTREAM = 10 * 1024 * 1024;
 
static void InizializeCycle(string inputFile, string OutputFile)
{
    if (File.Exists(inputFile))
    {
        try
        {
            using StreamReader sr = new StreamReader(new BufferedStream(File.OpenRead(inputFile), BUFFERSTREAM));
            using StreamWriter sw = new StreamWriter(new BufferedStream(File.OpenWrite(OutputFile), BUFFERSTREAM));
 
            string line = "", response = "";
 
            while ((line = sr.ReadLine()) != null)
            {
                Uri url = new Uri($"...", UriKind.RelativeOrAbsolute);
                response = new WebClient().DownloadString(url);
                if (response.Contains("..."))
                {
                    Console.WriteLine($"+{line}");
                    sw.WriteLine($"{line}");
                }
                else
                {
 
                }
            }
        }
        catch (Exception ex)
        {
            Console.WriteLine($"Error: {ex}\r\n");
            File.AppendAllText("Error_Read.txt", $"{ex}\r\n");
        }
    }
}

 

[JohnH]

What is size of input file and how many lines are there?
Does the output lines need to be in same order as input?
Is the web request for same or different servers? Is there a limit for concurrent requests?

 

[Asdshka]

What is size of input file and how many lines are there?
Does the output lines need to be in same order as input?
Is the web request for same or different servers? Is there a limit for concurrent requests?

The size of the source file is 15-20 megabytes.
The source file has 1 million lines of 10 characters.
The order of the lines doesn't matter. Moreover, many lines as a result will not be written to the output file.
Request for one server. I don't know if there is a limit. Perhaps there is.

If you can help me, then I'm willing to pay you.

 

[Skydiver]

What is the real URI being formed on line 31? As it stands now, you are always using the URI ... which should always fail anyways when you try to download from it.

What response are you looking for in line 33? Are you really looking for just ...?

 

[Skydiver]

If the request is always for just one server, then just request once and then search the response multiple times, unless the request is for one server but different kinds of requests.

This is why you need to go into more details about your lines 31 and 33. What are you really requesting, and what are you searching for?

 

[Skydiver]

Also, as it stands now, there is really no need to start a task on a thread pool thread. Thread pool threads have lower priority than other threads. Just do your work on the main thread since you are writing a console program.

 

[Asdshka]

What is the real URI being formed on line 31? As it stands now, you are always using the URI ... which should always fail anyways when you try to download from it.

What response are you looking for in line 33? Are you really looking for just ...?

Take any site for a query and look for any text.

 

[Skydiver]

So that line read on line 29 is part of the query? If not, then move the query outside the loop.

 

[Asdshka]

Yes "line" is part of the request

 

[Asdshka]

The program itself cannot be accelerated, but the result can be accelerated. Can you make this code run in 100 parallel threads?

string s;
string response;
StreamReader f = new StreamReader(@"C:\Users\Admin\Desktop\1.txt");

while ((s = f.ReadLine()) != null)
{
    Uri url = new Uri($"https://url.com/?CTN={s}");
    response = new WebClient().DownloadString(url);

    if (response.Contains("..."))
    {
        StreamWriter file = new StreamWriter(@"C:\Users\Admin\Desktop\01.txt", true);
        file.WriteLine(s);
        file.Close();
    }
    else { }
}
f.Close();

 

[Skydiver]

Only if you had 100 cores... You can have multiple threads running (usually a multiple of the number of cores), and have some of them stalled while waiting for web responses.

The least code writing approach would be to use Parallel.ForEach().

 

[Skydiver]

Some general advice:
WebClient is obsolete. Use a single instance of the more efficient HttpClient.

Opening a file is slow. If possible open the output just once.

 

[Asdshka]

WebClient is obsolete. Use a single instance of the more efficient HttpClient.

Show me an example of how such a request can be made here:

string s;
string response;
StreamReader f = new StreamReader(@"C:\Users\Admin\Desktop\1.txt");

while ((s = f.ReadLine()) != null)
{
    Uri url = new Uri($"https://url.com/?CTN={s}");
    response = new WebClient().DownloadString(url);

    if (response.Contains("..."))
    {
        StreamWriter file = new StreamWriter(@"C:\Users\Admin\Desktop\01.txt", true);
        file.WriteLine(s);
        file.Close();
    }
    else { }
}
f.Close();

 

[Skydiver]

See documentation. If you can't figure it out, follow-up with a specific question.

HttpClient Class (System.Net.Http)

Provides a class for sending HTTP requests and receiving HTTP responses from a resource identified by a URI.

learn.microsoft.com

 

[Skydiver]

As an example of using Parallel.ForEachAsync() along with HttpClient can be found here:

Parallel.ForEachAsync in .NET 6

Great tweet from Oleg Kyrylchuk (follow him!) showing how cool ...

www.hanselman.com