scrape data from website with login

[nkat]

Hello!
My goal is to scrape some data from a webpage 192.168.1.21/app/admin/directories.asp?id=username
The website that controls it needs login before querying directories. This is the problem I’m trying to solve. For that, I’ve read this post, and it addresses a lot of my questions, but does not solve the problem completely.

Important note, the 192.168.1.21 provides a user with 3 ways to login:

1. a button “recognize me automatically” that runs JavaScript function loginNTLM()
2. two fields, “username” and “password” that prompt a user to use his Active Directory credentials
3. a field “superuser password” that requires just a password to login. The password is known to me

Following guidelines from the post I installed Fiddler, captured the traffic while doing #3 and got the following string in the TextView tab of the POST request " login_maintenance=%24MAINTENANCE%5Croot&pwd_maintenance=BadDog22&url_redirect=%2Fwatchdoc%2Fadmin%2Fdefault.asp%3Fs%3DDEFAULT "
Then the post suggests a dev to replicate the request in his code. I took an example from the post and edited in the following way

using System.Net;

var cookieContainer = new CookieContainer();

HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("http://192.168.1.21/app/admin");
request.CookieContainer = cookieContainer;
//set the user agent and accept header values, to simulate a real web browser
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";


//SET AUTOMATIC DECOMPRESSION
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;

Console.WriteLine("FIRST RESPONSE");
Console.WriteLine();
using (WebResponse response = request.GetResponse())
{
    using (StreamReader sr = new StreamReader(response.GetResponseStream()))
    {
        Console.WriteLine(sr.ReadToEnd());
    }
}

request = (HttpWebRequest)HttpWebRequest.Create("http://192.168.1.21/app/admin/directories.asp?s=DEFAULT");
//set the cookie container object
request.CookieContainer = cookieContainer;
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";

//set method POST and content type application/x-www-form-urlencoded
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";

//SET AUTOMATIC DECOMPRESSION
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;

//insert your username and password
string data = string.Format("username={0}&password={1}", "root", "BadDog22");
byte[] bytes = System.Text.Encoding.UTF8.GetBytes(data);

request.ContentLength = bytes.Length;

using (Stream dataStream = request.GetRequestStream())
{
    dataStream.Write(bytes, 0, bytes.Length);
    dataStream.Close();
}

Console.WriteLine("LOGIN RESPONSE");
Console.WriteLine();
using (WebResponse response = request.GetResponse())
{
    using (StreamReader sr = new StreamReader(response.GetResponseStream()))
    {
        Console.WriteLine(sr.ReadToEnd());
    }
}

but in both responses I got back the same login page. Just can not get through it.
Would someone please suggest a solution?

 

[nkat]

just in case anyone else comes across. This code got me the page I was looking for

var baseAddress = new Uri("http://192.168.1.21");
using (var handler = new HttpClientHandler { UseCookies = false })
using (var client = new HttpClient(handler) { BaseAddress = baseAddress })
{
    var message = new HttpRequestMessage(HttpMethod.Get, "/app/admin/directories.asp?act=test&id=META&login=admroot&s=DEFAULT");
    message.Headers.Add("Cookie", "t%5FDEFAULT%5Fadmin=969082094");
    var result = await client.SendAsync(message);
    //result.EnsureSuccessStatusCode();
    var resultString = await result.Content.ReadAsStringAsync();
    Console.WriteLine(resultString);
    Console.ReadKey();
}

I also found this 1- Introduction to HttpClient useful