Question Why not errors written to DB?

raysefo

Well-known member
Joined
Feb 22, 2019
Messages
119
Programming Experience
10+
@Skydiver and @Sheepings thank you for your feedback. You are right, as I said many times I am not experienced with rest API. I googled a lot and trying to follow step by step tutorials. There was this all in one article (rest API) and mostly referred to this one.
 

Skydiver

Well-known member
Joined
Apr 6, 2019
Messages
291
Location
Virginia Beach, VA
Programming Experience
10+
Here's my suggestion: Create a version of your project where your Controllers essentially just return success without actually do any work. Run JMeter to see if you can get past the 1000 user limit. If you can, then you know that it's the rest of your code that is causing the issue. If it doesn't, then just like the link provided to you by Nang Yu suggests: there is an issue with the load balancer thinking that your application can't handle that number of connections.
 

raysefo

Well-known member
Joined
Feb 22, 2019
Messages
119
Programming Experience
10+
Here's my suggestion: Create a version of your project where your Controllers essentially just return success without actually do any work.
@Skydiver thank you, you mean controller just returns 200 right? No handlers, no repositories, no unit of work etc.
 

Skydiver

Well-known member
Joined
Apr 6, 2019
Messages
291
Location
Virginia Beach, VA
Programming Experience
10+
Yes. Figure out whether it is your infrastructure that is causing the problem, or if it is your code.

Chances are that it's your code because as I've said, my web services easily supports that number of users/transactions, but then, I also know how my F5 load balancer and my IIS machines are configured in my environment. It looks like you don't know how your environment is configured.
 

Sheepings

Senior Programmer
Joined
Sep 5, 2018
Messages
277
Location
UK
Programming Experience
10+
Yea, I think I suggested the same as Skydiver at some point before. Strip out everything, and leave only the guts. Then once you know it works, you will know what to change when putting it all back together again.
You are right, as I said many times I am not experienced with rest API. I googled a lot and trying to follow step by step tutorials.
May I suggest reading this : Asynchronous programming in C# It is an excellent piece with really easy to follow analogies on how and when to use await and it is very informative re- asynchronous programming. Grab a cup of tea and dig in. It's only a ten minute read.
 

raysefo

Well-known member
Joined
Feb 22, 2019
Messages
119
Programming Experience
10+
but then, I also know how my F5 load balancer and my IIS machines are configured in my environment. It looks like you don't know how your environment is configured.
My server is in the cloud. I asked the admin if the server is behind the LB or any other configuration issues.
 

Skydiver

Well-known member
Joined
Apr 6, 2019
Messages
291
Location
Virginia Beach, VA
Programming Experience
10+
Something to keep in mind: Most cloud based hosting worth their salt have automatic DOS (and DDOS) attack detection and prevention. If the system sees a pattern or burst of multiple connection attempts, it will automatically throttle the incoming connections by stopping or denying the other connection attempts. (Yes, from one point of view, this can be seen as the DOS attack has been successful because some people were denied service, but it does keep the system running.) If your JMeter test is only emanating requests from a single IP, then it looks even more suspiciously like a naive DOS attack.

In a way, a load test is a DOS attack if the purpose of the load test is to find the breaking point, as opposed to a load test meant to discover how the system performs when under that expected/designed for load.

A side question is: Is that your webservice even supposed to handle that many users in a such a small amount of time? Is it in the acceptance criteria for your web service, or just something that some one arbitrarily picked? If it is in the acceptance criteria, presumably someone had a logical reason for picking that number that is based in reality, rather than just for bragging rights. For example, on one system we have, we planned to support 90,000 users logged on and running simultaneously, but realistically not all 90,000 will all hit the logon page all at 8:00 AM Eastern. The users were actually spreadout across multiple timezones, and based on our research of AD logon logs, people within each timezone tended to be spreadout over a 3 hour period from 7-10AM. So our load testing simulated this type of behavior, instead of the originally conceived idea of 90,0000 people all logging in over a 15 minute period.
 

raysefo

Well-known member
Joined
Feb 22, 2019
Messages
119
Programming Experience
10+
@Skydiver thank you. Hopefully, it is due to attack detection and prevention of the host :)
If your JMeter test is only emanating requests from a single IP, then it looks even more suspiciously like a naive DOS attack.
Yes, I am testing from a single IP, which is my development laptop.
Is that your webservice even supposed to handle that many users in a such a small amount of time? Is it in the acceptance criteria for your web service, or just something that some one arbitrarily picked?
It is my web service but it does not have to handle such a small amount of time. (Not an acceptance criteria) Actually, I think concurrent 350 users are more than enough but who knows there may be more than that in future. So I need to know if I can handle more users.
 

Sheepings

Senior Programmer
Joined
Sep 5, 2018
Messages
277
Location
UK
Programming Experience
10+
I highly doubt that, and I say that because I work with this stuff almost daily. Worth a note @Skydiver; OP is creating connections but the connection in the pool are not being released; hence becoming full. So connections are entering the system. You have made a good point re what could be seen as a denial of service by lambasting the server with connections form the same IP. But this would not cause CloudFlare to hold any resources from sending packets back to the application or stopping it from closing a connection. Note, in between connections, a connection to the server was refused error would likely occur. If this was a CloudFlare issue, OP would not be able to access his network at all and would likely need it to be unblocked VIA an administration request. @raysefo, I suggest that you contact your network administration and ensure that they white-list your IP which you are testing from. This is something you should have known to do from the get-go. But its obviously not the cause of your issue in my opinion. That's not to say that the problem may not be something else server-side. Knowing more about your configuration might be helpful to narrowing causes, but I'd firstly do as said on post #22 before anything else.
 
Last edited:

Skydiver

Well-known member
Joined
Apr 6, 2019
Messages
291
Location
Virginia Beach, VA
Programming Experience
10+
It looks like the OP is having two different kinds of connection issues: There's the database connection issue where he is exhausting his database connection pool. Definitely suspicious there.
Then there's his IIS connection issue (e.g "Connection_Dropped_List_Full"), where IIS has stopped sending requests to his application. Although the primary reason why IIS won't send anymore connections to his app is because it is busy (likely because of some bug that is also causing the database connection exhaustion) the link that Nan Yu provided in the OP's ASP.NET Forums thread (see post #17) also suggests another potential reason for the connection dropped error: some load balancers will allow the initial connection and then shut it down before IIS can get around to servicing it because the load balancer thinks that IIS is unhealthy or busy.

My gut feel though is that it's very likely the OP's code, but it's worth confirming that the infrastructure + MVC is working correctly first, so that the OP can focus on just the code, instead of trying to figure out if it's the environment or if it's the code.
 

Sheepings

Senior Programmer
Joined
Sep 5, 2018
Messages
277
Location
UK
Programming Experience
10+
I have already read that post and the replies including the post by Benjamin Perkins. I am reaching a conclusion but I want to be sure before posting. As the logs do point to the OP's code as the source of the problem. But after enough research into this, I'm now on the fence on this one, and I do believe there is something else happening. @raysefo Are you using a http client, and If you have written one, can you share the settings for it? I believe something is closing your connections outside of your API calls. My codesmell radar is going nuts at the minute.

I don't think its something you've inadvertently wrote, but likely something you've missed/probably pasted inadvertently or a setting you failed to set properly. The only way to resolve that would be to trace where the issues reside. I provide a link to do that below. I also think you might have more than three concurrent issues. Clearly it seems that your connections are being closed outside the API calls which is causing your error log to overflow and that's causing its own issue in itself. Leaving us with two issues to fix if we can trace where the breakage is happening. Also, have you enabled failed request tracing rules in IIS? I believe this will help narrow down problems and may even identify what it causing these interruptions. See link below.

Edit : Clarity added - Apologies for the poorly worded reply. I'm having a rush day, and limited time to spare :)
 
Last edited:

Sheepings

Senior Programmer
Joined
Sep 5, 2018
Messages
277
Location
UK
Programming Experience
10+

raysefo

Well-known member
Joined
Feb 22, 2019
Messages
119
Programming Experience
10+
Thank you @Sheepings.
Are you using a http client, and If you have written one, can you share the settings for it?
Yes, I am using http client in order to call 3rd party rest API. Here is how I am calling 3rd party rest API in my code:
C#:
namespace BusinessService.Utility
{
    public class Utilities
    {
        private static readonly HttpClient _httpClient = new HttpClient();

        //some code here
        
        public static async Task<HttpResponseMessage> CallRazer(GameRequest gameRequest, string url)
        {
            try
            {
                FormUrlEncodedContent content = null;

                if (url == "Product/")
                {
                    try
                    {
                        //Transform GameRequest into ProductDTO
                        var config =
                            new MapperConfiguration(cfg => { cfg.CreateMap<GameRequest, ProductRequestDto>(); });
                        var iMapper = config.CreateMapper();
                        var productRequest = iMapper.Map<GameRequest, ProductRequestDto>(gameRequest);

                        //Convert request
                        var keyValues = productRequest.ToKeyValue();
                        content = new FormUrlEncodedContent(keyValues);

                        //Call 3rd party API
                        var response = await _httpClient.PostAsync("https://test.com/" + url, content);
                        return response;
                    }
                    catch (Exception e)
                    {
                        Console.WriteLine(e);
                        throw;
                    }
                    finally
                    {
                        content.Dispose();
                    }
                }
                else
                {
                    try
                    {
                        //Convert request
                        var keyValues = gameRequest.ToKeyValue();
                        content = new FormUrlEncodedContent(keyValues);

                        //Call 3rd party API
                        var response = await _httpClient.PostAsync("https://test.com/" + url, content);
                        return response;
                    }
                    catch (Exception e)
                    {
                        Console.WriteLine(e);
                        throw;
                    }
                    finally
                    {
                        content.Dispose();
                    }
                }
            }
            catch (Exception e)
            {
                Console.WriteLine(e);
                throw;
            }
        }

        
    }

    public static class ObjectExtensions
    {
        public static IDictionary<string, string> ToKeyValue(this object metaToken)
        {
            if (metaToken == null)
            {
                return null;
            }

            JToken token = metaToken as JToken;
            if (token == null)
            {
                return ToKeyValue(JObject.FromObject(metaToken));
            }

            if (token.HasValues)
            {
                var contentData = new Dictionary<string, string>();
                
                foreach (var child in token.Children().ToList())
                {
                    var childContent = child.ToKeyValue();
                    if (childContent != null)
                    {
                        contentData = contentData.Concat(childContent)
                            .ToDictionary(k => k.Key, v => v.Value);
                    }
                }

                return contentData;
            }

            var jValue = token as JValue;
            if (jValue?.Value == null)
            {
                return null;
            }

            var value = jValue?.Type == JTokenType.Date
                ? jValue?.ToString("o", CultureInfo.InvariantCulture)
                : jValue?.ToString(CultureInfo.InvariantCulture);

            return new Dictionary<string, string> {{token.Path, value}};
        }
    }
}
And here is how I call CallRazer:
C#:
namespace BusinessService
{
    public class GameServices : IGameServices, IDisposable
    {
        private readonly UnitOfWork _unitOfWork;

        public GameServices(UnitOfWork unitOfWork)
        {
            _unitOfWork = unitOfWork;
        }

        public async Task<HttpResponseMessage> GamePurchase(RequestDto requestDto)
        {
            HttpResponseMessage response = null;
            using (_unitOfWork)
            {
                response = await CallRazerService(requestDto);
                return response;
            }
        }

        private async Task<HttpResponseMessage> CallRazerService(RequestDto requestDto)
        {
            HttpResponseMessage response = null;

            //Transform DTO into GameRequest for calling Razer Initiate
            var config = new MapperConfiguration(cfg =>
            {
                cfg.CreateMap<RequestDto, GameRequest>();
                cfg.CreateMap<GameRequest, GameConfirmRequest>();
                cfg.CreateMap<GameConfirmResponse, GameConfirmResponseDto>();
                cfg.CreateMap<Coupon, CouponDto>();
                cfg.CreateMap<GameRequest, GameRequestDto>();
            });
            var iMapper = config.CreateMapper();
            var gameRequest = iMapper.Map<RequestDto, GameRequest>(requestDto);

            //Unique reference ID
            gameRequest.referenceId = Guid.NewGuid();

            var gameRequestDto = iMapper.Map<GameRequest, GameRequestDto>(gameRequest);

            //Create signature
            gameRequest = Utilities.CreateSignature(gameRequestDto, RequestType.Initiate);

            //Set service
            gameRequest.service = "RAZER";

            //Add initiation request into database
            _unitOfWork.GameRepository.Insert(gameRequest);

            #region Call Razer initiate/confirm

            //Call Razer for initiation
            response = await Utilities.CallRazer(gameRequest, "purchaseinitiation");

            //Read response
            var htmlResponse = await response.Content.ReadAsStringAsync();
            var gameResponse = JsonConvert.DeserializeObject<GameResponse>(htmlResponse);

            //Adding initiation response into database
            _unitOfWork.GameResponseRepository.Insert(gameResponse);

            if (gameResponse.initiationResultCode == "00")
            {
                gameRequestDto = iMapper.Map<GameRequest, GameRequestDto>(gameRequest);
                gameRequestDto.validatedToken = gameResponse.validatedToken;
                //Create signature
                var gameConfirmRequest = Utilities.CreateSignature(gameRequestDto, RequestType.Confirm);

                //Transform DTO into GameRequest for calling Razer Initiate
                var gameConfirmRequests = iMapper.Map<GameRequest, GameConfirmRequest>(gameConfirmRequest);

                //Add confirm request into database
                _unitOfWork.GameConfirmRequestRepository.Insert(gameConfirmRequests);

                //Call Razer for confirm
                response = await Utilities.CallRazer(gameConfirmRequest, "purchaseconfirmation");

                //Read response
                htmlResponse = await response.Content.ReadAsStringAsync();
                var gameConfirmResponse = JsonConvert.DeserializeObject<GameConfirmResponse>(htmlResponse);

                //Set service
                gameConfirmResponse.service = "RAZER";
                //Add confirm response into database
                _unitOfWork.GameConfirmResponseRepository.Insert(gameConfirmResponse);
            }

            #endregion

            await _unitOfWork.SaveAsync();

            return response;
        }

        
        public void Dispose()
        {
            _unitOfWork.Dispose();
        }
    }
}
 

Sheepings

Senior Programmer
Joined
Sep 5, 2018
Messages
277
Location
UK
Programming Experience
10+

Skydiver

Well-known member
Joined
Apr 6, 2019
Messages
291
Location
Virginia Beach, VA
Programming Experience
10+
That HttpClient on post #33 is an outbound HTTP client connection. The errors logged by IIS in HTTPERR logs are for inbound connections.
 

raysefo

Well-known member
Joined
Feb 22, 2019
Messages
119
Programming Experience
10+
I made a load test with 2500 users and there are no errors. It is magical right :) But at the very beginning before the load test, I just made a request with Postman in order to see if the service running correctly, I got this error.

C#:
Message --- The remote name could not be resolved: 'test.com'
--- End Inner Exception ---
Message --- An error occurred while sending the request.
I asked the server admin if they did something, they said they did NOT do anything. And all of a sudden after a couple of minutes there was no such en error either :) Anyway, this was not a complete load test because the whole code didn't covered. (If portion did not run) Hopefully, a complete load test will be done tomorrow.

C#:
private async Task<HttpResponseMessage> CallRazerService(RequestDto requestDto)
        {
            HttpResponseMessage response = null;

            //Transform DTO into GameRequest for calling Razer Initiate
            var config = new MapperConfiguration(cfg =>
            {
                cfg.CreateMap<RequestDto, GameRequest>();
                cfg.CreateMap<GameRequest, GameConfirmRequest>();
                cfg.CreateMap<GameConfirmResponse, GameConfirmResponseDto>();
                cfg.CreateMap<Coupon, CouponDto>();
                cfg.CreateMap<GameRequest, GameRequestDto>();
            });
            var iMapper = config.CreateMapper();
            var gameRequest = iMapper.Map<RequestDto, GameRequest>(requestDto);

            //Unique reference ID
            gameRequest.referenceId = Guid.NewGuid();

            var gameRequestDto = iMapper.Map<GameRequest, GameRequestDto>(gameRequest);

            //Create signature
            gameRequest = Utilities.CreateSignature(gameRequestDto, RequestType.Initiate);

            //Set service
            gameRequest.service = "RAZER";

            //Add initiation request into database
            _unitOfWork.GameRepository.Insert(gameRequest);

            #region Call Razer initiate/confirm

            //Call Razer for initiation
            response = await Utilities.CallRazer(gameRequest, "purchaseinitiation");

            //Read response
            var htmlResponse = await response.Content.ReadAsStringAsync();
            var gameResponse = JsonConvert.DeserializeObject<GameResponse>(htmlResponse);

            //Adding initiation response into database
            _unitOfWork.GameResponseRepository.Insert(gameResponse);

            if (gameResponse.initiationResultCode == "00")
            {
                gameRequestDto = iMapper.Map<GameRequest, GameRequestDto>(gameRequest);
                gameRequestDto.validatedToken = gameResponse.validatedToken;
                //Create signature
                var gameConfirmRequest = Utilities.CreateSignature(gameRequestDto, RequestType.Confirm);

                //Transform DTO into GameRequest for calling Razer Initiate
                var gameConfirmRequests = iMapper.Map<GameRequest, GameConfirmRequest>(gameConfirmRequest);

                //Add confirm request into database
                _unitOfWork.GameConfirmRequestRepository.Insert(gameConfirmRequests);

                //Call Razer for confirm
                response = await Utilities.CallRazer(gameConfirmRequest, "purchaseconfirmation");

                //Read response
                htmlResponse = await response.Content.ReadAsStringAsync();
                var gameConfirmResponse = JsonConvert.DeserializeObject<GameConfirmResponse>(htmlResponse);

                //Set service
                gameConfirmResponse.service = "RAZER";
                //Add confirm response into database
                _unitOfWork.GameConfirmResponseRepository.Insert(gameConfirmResponse);
            }

            #endregion

            await _unitOfWork.SaveAsync();

            return response;
        }
 

Skydiver

Well-known member
Joined
Apr 6, 2019
Messages
291
Location
Virginia Beach, VA
Programming Experience
10+
I hate those "Fixed by magic" issues, especially when the network or machine admin claims they didn't change anything (but the session logs show that they were logged on to the machine for some reason).
 

raysefo

Well-known member
Joined
Feb 22, 2019
Messages
119
Programming Experience
10+
Is there anything that may cause those errors in the if block?
 

Skydiver

Well-known member
Joined
Apr 6, 2019
Messages
291
Location
Virginia Beach, VA
Programming Experience
10+
"The remote name could not be resolved: 'test.com'" would be indicative on a DNS issue.
 
Top Bottom