Question sending an HTTP request/POST to a website

nofu1

Member
Joined
Oct 30, 2020
Messages
23
Programming Experience
Beginner
I am still a beginner in c#. I have a console application. In javascript, we can do the following document.getElementById("myText").value = "Mary; which is used to set an HTML element. Is this possible in a console application that sending an HTTP requests to a website or should the application be using a post method. Please see code so far below. There is a textbox with the tag name testmedownelement at the website(www.~~~~~~~~~~~~~~~~~~~~~~.com).

This is the javascript code: document.getElementsByName('testmedownelement').value = "XXXXXX", I am looking for a C# equivalent since I can't use java script in my console application

C#:
            WebRequest request = WebRequest.Create("www.~~~~~~~~~~~~~~~~~~~~~~.com");
        

           request.Method = "GET";
            request.ContentType = "application/html";
            
          

            // Get the response.

            HttpWebResponse response = (HttpWebResponse)request.GetResponse();

            //Display Status

            Console.WriteLine(response.StatusDescription);

            // Get the stream containing content returned by the server.

            Stream dataStream = response.GetResponseStream();

            // Open the stream using a StreamReader for easy access.

            StreamReader reader = new StreamReader(dataStream);

            // Read the content.

            string responseFromServer = reader.ReadToEnd();

            // Display the content.

            Console.WriteLine(responseFromServer);

            // Cleanup the streams and the response.

            reader.Close();

            dataStream.Close();

            response.Close();

            Console.ReadKey();
 

nofu1

Member
Joined
Oct 30, 2020
Messages
23
Programming Experience
Beginner
Using VS2019 by itself won't fix things. You still need to build your code targeting .NET Framework 4.8. What VS2019 does is allow you to pick that target.

As an aside, you could also build your code with just the .NET Framework 4.8 SDK on the command line, but that will require you to use VS Code, or some other text editor. Since you are still on the learning curve, keep things simple and just install VS2019. You learn the language first, then later learn about how things are actually done by the IDE for you.

(Yes, we C# folks are less strict than the C and C++ folks that insist that you first learn how the compiler and linker works before for learn how to write code.)

thanks for the clarification. The issue I was referring to though was in regards to the 401 authorization error I was getting in relation to my console application
 

Skydiver

Staff member
Joined
Apr 6, 2019
Messages
2,586
Location
Chesapeake, VA
Programming Experience
10+
The TLS fix allows you to talk to the website. Now that you can talk to it, you need to pass in credentials that is authorized to use the web service you are trying to talk to. That is not a technology problem, but rather as social/security problem. Have you talked to the website owners to find out what credential you need to pass in?
 

nofu1

Member
Joined
Oct 30, 2020
Messages
23
Programming Experience
Beginner
The TLS fix allows you to talk to the website. Now that you can talk to it, you need to pass in credentials that is authorized to use the web service you are trying to talk to. That is not a technology problem, but rather as social/security problem. Have you talked to the website owners to find out what credential you need to pass in?
I know the username and password of the website and the associated url information, should I be asking for other stuff as well
 

Skydiver

Staff member
Joined
Apr 6, 2019
Messages
2,586
Location
Chesapeake, VA
Programming Experience
10+
So where are you passing in the credentials in your request?
 

nofu1

Member
Joined
Oct 30, 2020
Messages
23
Programming Experience
Beginner
So where are you passing in the credentials in your request?


In the following section of the code below

the networkcredential takes the username and password
// credentialCache.Add(new System.Uri(url), "Basic", new NetworkCredential(~~~~, ~~~~~~~~~~~~));

C#:
private static CredentialCache GetCredential()
{
    ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 | SecurityProtocolType.Tls | (SecurityProtocolType)3072;
    string url = @"https://~~~~~~~~~~~~~~~~~~~~~~.com/~~~~~";
    CredentialCache credentialCache = new CredentialCache();
    credentialCache.Add(new System.Uri(url), "Basic", new NetworkCredential(~~~~, ~~~~~~~~~~~~));
    return credentialCache;
}

public static void testcall (string url)
{

    WebRequest request = WebRequest.Create(url);
    request.PreAuthenticate = true;
    request.Credentials = GetCredential();

    ((System.Net.HttpWebRequest)request).Referer = url;
    ((System.Net.HttpWebRequest)request).Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8";
    ((System.Net.HttpWebRequest)request).UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0";

    // Set the Method property of the request to POST.
    request.Method = "POST";
    // Create POST data and convert it to a byte array.
    string postData = "param1=XXXXXX";
    ASCIIEncoding encoding = new ASCIIEncoding();

    byte[] byteArray = encoding.GetBytes(postData);
    // Set the ContentType property of the WebRequest.
    request.ContentType = "application/x-www-form-urlencoded";
    // Set the ContentLength property of the WebRequest.
    request.ContentLength = byteArray.Length;
    // Get the request stream.
    Stream dataStream = request.GetRequestStream();
    // Write the data to the request stream.
    dataStream.Write(byteArray, 0, byteArray.Length);
    // Close the Stream object.
    dataStream.Close();



    // Get the response.
    WebResponse response = request.GetResponse();
    // Display the status.
    Console.WriteLine(((HttpWebResponse)response).StatusDescription);
    // Get the stream containing content returned by the server.
    dataStream = response.GetResponseStream();
    // Open the stream using a StreamReader for easy access.
    StreamReader reader = new StreamReader(dataStream);
    // Read the content.
    string responseFromServer = reader.ReadToEnd();
    // Display the content.
    Console.WriteLine(responseFromServer);
    // Clean up the streams.
    reader.Close();
    dataStream.Close();
    response.Close();
}
 

Skydiver

Staff member
Joined
Apr 6, 2019
Messages
2,586
Location
Chesapeake, VA
Programming Experience
10+
It looks like the website needs more than just the user name and password. Does it also need headers or cookies that contains some other secrets? What did the website owner tell you about how to authenticate against their website using a direct web request instead of going through the GUI?
 

nofu1

Member
Joined
Oct 30, 2020
Messages
23
Programming Experience
Beginner
It looks like the website needs more than just the user name and password. Does it also need headers or cookies that contains some other secrets? What did the website owner tell you about how to authenticate against their website using a direct web request instead of going through the GUI?

it is just username and password or is there anyway to verify this by studying the site.
 

Sheepings

Retired Programmer
Staff member
Joined
Sep 5, 2018
Messages
1,971
Location
UK
Programming Experience
10+
The TLS fix allows you to talk to the website.
Skydiver, it's important to note that the below is not required to sent in your requests if you are using the latest .net versions.
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 | SecurityProtocolType.Tls | (SecurityProtocolType)3072; is not required in 4.8.
Since the error the OP is getting is of that of authentication of credentials, may also be linked to a TLS issue. Irradiating irrelevant code is a start, and running the latest version of .net framework is advised above :
What he needs to do first is download the VS2019 Community Edition (it's free) then he can open his project in it and change it to target .Net 4.8
If the OP doesn't try, they will never know if the line of code I quoted is the issue or not. But as I said, Microsoft took that out in v4.6 I think it was, and it is no longer needed as later versions of .net determine the protocol for you.
 

nofu1

Member
Joined
Oct 30, 2020
Messages
23
Programming Experience
Beginner
Skydiver, it's important to note that the below is not required to sent in your requests if you are using the latest .net versions.

Since the error the OP is getting is of that of authentication of credentials, may also be linked to a TLS issue. Irradiating irrelevant code is a start, and running the latest version of .net framework is advised above :

If the OP doesn't try, they will never know if the line of code I quoted is the issue or not. But as I said, Microsoft took that out in v4.6 I think it was, and it is no longer needed as later versions of .net determine the protocol for you.
Hi I just installed visual studio 2019 and likewise the framework 4.5 wasn't able to integrate with visual studio 2019 automatically after several attempts. But most importantly, I am still getting the remote server unauthorized 401 error in visual studio 2019 as well
 

Sheepings

Retired Programmer
Staff member
Joined
Sep 5, 2018
Messages
1,971
Location
UK
Programming Experience
10+
You need a recap on what I told you.
I have a console app and I can use 4.8.

There is an option to install other versions.

Move up to 4.8, and remove the line I told you to remove. Then tell us if you get a new error or not?
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 | SecurityProtocolType.Tls | (SecurityProtocolType)3072; is not required in 4.8.
And your problem is indicative of a problem with either TLS or your credentials.

You will need someone else to help from here. I have server deployments to attend to.

Look at the screenshot, install if missing any version
 

Skydiver

Staff member
Joined
Apr 6, 2019
Messages
2,586
Location
Chesapeake, VA
Programming Experience
10+
As @Sheepings said above, TLS 1.2 is enabled by default on 4.7 and up. You are targeting 4.5. As I said above, TLS 1.2 still won't fix your 401 error because that is a different problem. What is wrong with talking to the website owner and finding out what you need to do to authenticate? Or are you illegally page scraping their site.
 

nofu1

Member
Joined
Oct 30, 2020
Messages
23
Programming Experience
Beginner
As @Sheepings said above, TLS 1.2 is enabled by default on 4.7 and up. You are targeting 4.5. As I said above, TLS 1.2 still won't fix your 401 error because that is a different problem. What is wrong with talking to the website owner and finding out what you need to do to authenticate? Or are you illegally page scraping their site.

They said the username and password were the only credentials needed. The username and password were tested manual and worked fine. Please let not make any false assumptions. I only mentioned the framework as clarity after installation.
 
Last edited:

Skydiver

Staff member
Joined
Apr 6, 2019
Messages
2,586
Location
Chesapeake, VA
Programming Experience
10+
In the original post, you mentioned that you had this working in JavaScript, but the only code that you showed was you setting value on the DOM. You never actually showed how you post the form back to the server so that the server gets the value that you set. Did you just do a plain old HTTP POST, or did some headers or cookies need to be set?

So the question is: at the point when you are changing the DOM, did the user have to manually log on to the web site? And if so, did the server send back some kind of session cookie or token to represent that session? And if so, how our you simulating that same kind of logon and getting back a cookie using your HTTP web request code?

Use Fiddler (or whatever web proxy tool you prefer -- maybe the F12 developer tools will be enough). Watch the traffic that is sent when you are using your browser. Compare it to the traffic that is sent when you are using your console application.
 

Sheepings

Retired Programmer
Staff member
Joined
Sep 5, 2018
Messages
1,971
Location
UK
Programming Experience
10+
As I said above, TLS 1.2 still won't fix your 401 error because that is a different problem.
That's not necessarily correct. It is possible to get a 401 error if the security of the request is not correct. On DIC I answered a question not so long ago regarding TLS, and after the OP of that thread removed line for TLS/updated .Net, their issue was relinquished, and hence; why I suggested moving to 4.8 .Net. There are a range of issues that can cause a 401 when using TLS : tls 401 error - Google Search

Since user is using TLS, what does the server use? Server side cert or client side cert? If this is a two-way TLS, you have a lot bigger problems. I also don't like the direction this topic is going. If there was a legitimate API to connect to, there would have been no reason to do anything with the DOM using JS. Which means, that this website or server you are trying to connect to is not meant to be connected to in the manner you are attempting.
 

nofu1

Member
Joined
Oct 30, 2020
Messages
23
Programming Experience
Beginner
That's not necessarily correct. It is possible to get a 401 error if the security of the request is not correct. On DIC I answered a question not so long ago regarding TLS, and after the OP of that thread removed line for TLS/updated .Net, their issue was relinquished, and hence; why I suggested moving to 4.8 .Net. There are a range of issues that can cause a 401 when using TLS : tls 401 error - Google Search

Since user is using TLS, what does the server use? Server side cert or client side cert? If this is a two-way TLS, you have a lot bigger problems. I also don't like the direction this topic is going. If there was a legitimate API to connect to, there would have been no reason to do anything with the DOM using JS. Which means, that this website or server you are trying to connect to is not meant to be connected to in the manner you are attempting.
I hope you know there are mapping websites that allows you to determine the associated lat/long based on an input such as an address (DOM JS), furthermore these websites also gives you the option to make API calls as well utilizing their configuration to determine the associated lat/long. Also, sometimes people tend to mistakenly look in the wrong direction required to solve an API problem. and there are a range of issues that can cause 401 as well.
 

Sheepings

Retired Programmer
Staff member
Joined
Sep 5, 2018
Messages
1,971
Location
UK
Programming Experience
10+
Sorry you lost me when you started talking about latitude etc... What does any of that have to do with what I said?
Also, sometimes people tend to mistakenly look in the wrong direction required to solve an API problem.
Really? Like what?

If I want to get data from a server that doesn't accept data requests, I make an API to deal with that problem. Anyway, I don't see the relevance of what point you are trying to establish here?
and there are a range of issues that can cause 401 as well.
Well yea, I kinda pointed that out in the link I posted... anyway, nothing in that reply brings you any closer to resolving your problem. Maybe if you gave us a link to what you're trying to connect too, we might be able to come up with some other suggestions. Have you since installed the latest VS community edition along with Dotnet4.8 and removed that line I mentioned previously?
 

Sheepings

Retired Programmer
Staff member
Joined
Sep 5, 2018
Messages
1,971
Location
UK
Programming Experience
10+
Your problem is, you are trying to connect to a server in a way that is likely not allowed by its owners. Otherwise, You'd be doing jack shit with the DOM and you would be looking to the owners API for answers.

Can you show me there API? Or their documentation for the API you are using?

If they have an API, why were you mauling the DOM?

And if the owner of the server wanted you to have access to their resource, they would provide you with credentials and an example to use.

You have not shown us this API, or provided any links to either of its documentation or a URI. This makes it a little harder for us to provide any quality assistance to you.
 

nofu1

Member
Joined
Oct 30, 2020
Messages
23
Programming Experience
Beginner
I have scrapped trying to create a console app, I rather have them just send me a .csv file with the data. but thanks everyone for the effort.
 

Sheepings

Retired Programmer
Staff member
Joined
Sep 5, 2018
Messages
1,971
Location
UK
Programming Experience
10+
Why are you reluctant to share with us the information I requested?

You are limiting the amount of support we can give by retaining answers to questions we have asked you.
 
Top Bottom