What is the best way to perform several api requests?

[W_Lopes]

Hello developers.

I would like to understand how you would solve the problem below:

An application that makes a large number of requests to an external API, this external API in recent days has been showing several request errors, so I needed to set up a structure that keeps trying several times to get it right (maximum of 5 attempts).
I believe that this is not the best way, and the request errors happen because I am sending the requests without having an interval.
What is the best practice for creating an API request framework that needs to send multiple requests to an endpoint without overwhelming it?

 

[cjard]

Insert a delay that means you're respecting the rate limits of the remote end

If you use HangFire to trigger your fetches, take a look at Concurrency & Rate Limiting — Hangfire Documentation

 

[Skydiver]

And if you do decide to use Hangfire, for the love of God make sure you are running on your own server instead of a web farm or web garden where multiple web apps share a single server.

 

[W_Lopes]

Insert a delay that means you're respecting the rate limits of the remote end

If you use HangFire to trigger your fetches, take a look at Concurrency & Rate Limiting — Hangfire Documentation

I'm using a structure like this :

while (numberOfTentatives < _maxTentatives) // 5 attempts
        {
            try
            {
                response = client.Get<Table_Price[]>(request);
                return response;
            }
            catch (HttpRequestException ex)
            {
                if (ex.StatusCode == HttpStatusCode.Unauthorized)
                {
                    token = $"Bearer {GetToken(_authRequest)}";
                }
                lastExpection = ex;
            }
            catch (Exception ex)
            {
                lastExpection = ex;
            }
            finally
            {
                numberOfTentatives++;
                timeOut += 100;
            }
            Thread.Sleep(timeOut);
        }

 

[Skydiver]

Well designed REST APIs will return a value in the response header telling you how long you should wait before trying again.

RFC 9110: HTTP Semantics

The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document describes the overall architecture of HTTP, establishes common terminology, and defines aspects of the protocol that are shared...

www.rfc-editor.org

 

[gulshan212]

Hello this is Gulshan Negi
Well, in C#, there are two primary methods for performing multiple API requests:
1. Sequential API calls: Multiple API requests can be made easily in this manner. In order, each request is made one after the other. If you need to make a lot of API requests, this is the most reliable method, but it can be slow.
2. Parallel API calls: Multiple API requests can be made more quickly this way. In parallel, each request is made simultaneously. This can be a lot quicker than successive Programming interface calls, yet it is more complicated to execute.
Thanks

 

[Skydiver]

@gulshan212 : And how does that address the heart of the OP's issue where he is trying to handle the error condition of retrying an API request?

 

[gulshan212]

Well, I have not too much idea about it, well above answer I wrote on behalf of my search.
Thanks

 

[nickntg]

There's some info missing, for example are the requests being made concurrently or serially? This will make a difference to the possible approach on solving the problem.

If the requests are made serially, I would simply suggest waiting for a period of time using exponential backoff if an error occurs. If on the other hand you're making the requests concurrently as part of a service or something like that, I'd have a look at Polly.