Question C# Async code doesn't work on Visual Studio Release mode, works perfectly well on debug mode (with or without attaching the debugger)

[Rakesh__]

I am using Microsoft Visual Studio Community 2022 (64-bit) - Current Version 17.3.1

.Net version 6.0

The below code, hangs in the middle when run from release mode.

This code pings each ip address 4 times asynchronously, and the entire function is made to run on 5k IP addresses using task.Add and Task.WhenAll.

Works perfectly fine in debug mode, but stops in the middle when run on release mode.

I also tries to remove the for loop in the IcmpPing class to just ping once, but still won't run in the release mode.

If I run for smaller number of IP addresses, like 100-800. it runs with no issues, but when I increase the number to above 1000, it starts to run and completes 50-60% of the ip's and just hangs or stops/pauses after that,
without any errors

using Microsoft.VisualBasic;
using System.Net.NetworkInformation;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using System.Threading.Tasks;
using System.Diagnostics;

namespace PingMinimal
{
    internal class Program
    {
        static async Task Main(string[] args)
        {
            Console.WriteLine("Started!");
            Stopwatch stopwatch = new Stopwatch();
            stopwatch.Start();


            string jsonFile = File.ReadAllText(@"C:\Users\inputFile.json");
            // Json FIle contains a dictionary of length 10k keys, and has the format
            // {"DE08W2534": ["10.218.131.72", true], "de08w2574": ["172.31.232.204", true],.......}

            JObject jsonObj = JsonConvert.DeserializeObject<JObject>(jsonFile);
            IEnumerable<JProperty> first5000Values = jsonObj.Children<JProperty>().Take(5000);

            List<Task> tasks = new List<Task>();

            foreach (JProperty pair in first5000Values)
            {
                tasks.Add(IcmpPing.PingIP(pair.Value[0].ToString()));
            };

            await Task.WhenAll(tasks);

            stopwatch.Stop();
            Console.WriteLine("Total Elapsed Time {0}", stopwatch.Elapsed);
            Console.WriteLine("Completed!");
        }
    }

    internal class IcmpPing
    {
        public static async Task PingIP(string ip)
        {
            using (Ping pingInstance = new Ping())
            {
                for (int i = 0; i < 4; i++)
                {
                    try
                    {
                        PingReply reply = await pingInstance.SendPingAsync(ip);
                    }
                    catch
                    {

                    }

                }
            }
        }
    }
}

 

[Skydiver]

I'm assuming that you have permission from your network admin to flood the network with all those pings.

Just to verify, the debug build works correctly with 5000 IP addresses, it is only the release build that cannot handle 5000 IP addresses?

 

[cjard]

So is your hope to launch 5000 pings near simultaneously?

 

[Rakesh__]

I'm assuming that you have permission from your network admin to flood the network with all those pings.

Just to verify, the debug build works correctly with 5000 IP addresses, it is only the release build that cannot handle 5000 IP addresses?

Well now I switched to .net 7 now the debug can handle 10k IPs and release can’t do more than 5k, for .net 6 still the release can’t handle more that 1k. But debug can handle near 5k. And I have all the permissions, I wrote the same code in python and I use subprocess module with async and distribute the work to multiple core and I am able to ping 45k IPs well under 10 minutes, But I wanted bring this time even down, so coded it in C#.

 

[Rakesh__]

So is your hope to launch 5000 pings near simultaneously?

Yes, but even more, I did the same code in python and it can handle 45k ips smoothly,

 

[Skydiver]

From what I am seeing while trying to reproduce the problem, it looks like you are running into the too many context switches between all the threads. On release builds, I'm seeing something close to 200 threads, but on debug builds its only about 100 threads. Strangely, if I replace the Ping calls with just a Task.Delay(), no problem at all, so I don't think the issue is with async/await or the (lack of a) synchronization context.

Spoiler

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Net.NetworkInformation;
using System.Threading.Tasks;

class Program
{
    const int Max = 5000;

    static async Task Main()
    {
        Console.WriteLine("Started!");
        Stopwatch stopwatch = new Stopwatch();
        stopwatch.Start();
        var tasks = new List<Task>();

        for(int i = 0; i < Max; i++)
            tasks.Add(DoWork(i.ToString()));

        await Task.WhenAll(tasks);
        stopwatch.Stop();
        Console.WriteLine("Total Elapsed Time {0}", stopwatch.Elapsed);
        Console.WriteLine("Completed!");
    }

    static async Task DoWork(string address)
    {
        using (var ping = new Ping())
        {
            for(int i = 0; i < 4; i++)
            {
                try
                {
                    // await ping.SendPingAsync("localhost");
                    await Task.Delay(2000);
                }
                catch(Exception ex)
                {
                    Trace.TraceError(ex.Message);
                }
            }
        }
    }
}

I recommend using TPL DataFlow so that you can control the maximum level of parallelism. Normally a good limit for I/O bound is twice the number of cores, but you might be able to get away with up to 100 since debug seems to handle that much.

Dataflow (Task Parallel Library) - .NET

Learn how to use dataflow components in the Task Parallel Library (TPL) to improve the robustness of concurrency-enabled applications.

learn.microsoft.com

Or if you don't want to use TPL DataFlow, try hacking something together with Parallel.ForEach() and see if you can throttle the number of threads with the options available there.

I am assuming that you are getting more throughput with .NET 7 because Microsoft did make concerted effort to improve performance, including the in the networking stack.

 

[Rakesh__]

From what I am seeing while trying to reproduce the problem, it looks like you are running into the too many context switches between all the threads. On release builds, I'm seeing something close to 200 threads, but on debug builds its only about 100 threads. Strangely, if I replace the Ping calls with just a Task.Delay(), no problem at all, so I don't think the issue is with async/await or the (lack of a) synchronization context.

Spoiler

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Net.NetworkInformation;
using System.Threading.Tasks;

class Program
{
    const int Max = 5000;

    static async Task Main()
    {
        Console.WriteLine("Started!");
        Stopwatch stopwatch = new Stopwatch();
        stopwatch.Start();
        var tasks = new List<Task>();

        for(int i = 0; i < Max; i++)
            tasks.Add(DoWork(i.ToString()));

        await Task.WhenAll(tasks);
        stopwatch.Stop();
        Console.WriteLine("Total Elapsed Time {0}", stopwatch.Elapsed);
        Console.WriteLine("Completed!");
    }

    static async Task DoWork(string address)
    {
        using (var ping = new Ping())
        {
            for(int i = 0; i < 4; i++)
            {
                try
                {
                    // await ping.SendPingAsync("localhost");
                    await Task.Delay(2000);
                }
                catch(Exception ex)
                {
                    Trace.TraceError(ex.Message);
                }
            }
        }
    }
}

I recommend using TPL DataFlow so that you can control the maximum level of parallelism. Normally a good limit for I/O bound is twice the number of cores, but you might be able to get away with up to 100 since debug seems to handle that much.

Dataflow (Task Parallel Library) - .NET

Learn how to use dataflow components in the Task Parallel Library (TPL) to improve the robustness of concurrency-enabled applications.

learn.microsoft.com

Or if you don't want to use TPL DataFlow, try hacking something together with Parallel.ForEach() and see if you can throttle the number of threads with the options available there.

I am assuming that you are getting more throughput with .NET 7 because Microsoft did make concerted effort to improve performance, including the in the networking stack.

Well I am trying to avoid creating threads in the first place, In python async never creates threads, is there a way to just creates the async task queue that event loop checks constantly for the completed tasks in a single thread?

 

[Skydiver]

Umm, Windows console apps do not have an event loop. Perhaps you were thinking NodeJS?

If you write that same code in WinForms or WPF, the default synchronization context there will not use other threads for the async/await. Alternatively, you can write a custom synchronization context that forces only a single thread and register that with the framework in your console app.

 

[Skydiver]

See:

Await, SynchronizationContext, and Console Apps

When I discuss the new async language features of C# and Visual Basic, one of the attributes I ascribe to the await keyword is that it “tries to bring you back to where you were.” For example, if you use await on the UI thread of your WPF application,

devblogs.microsoft.com