Verify URL automate

[ssridhar296]

Hi Friends

I would like to automate the process of verifying the urls. I have more than 300 urls where i need to check there health.

Either it can be good valid url or the page might have malware alert. So i want to track this on daily basis. It would be really helps if someone unblock me.

Thanks alot

 

[Skydiver]

You can check whether the URL is accessible by doing pings and HTTP GETs, but you won't be able to easily check if there is malware at the site.

The scale of the malware detection problem is huge. For that part you'll need to lean on your AV software, assuming that the AV software supports blocking malicious sites, and it has an API for you to have it test sites.

 

[ssridhar296]

You can check whether the URL is accessible by doing pings and HTTP GETs, but you won't be able to easily check if there is malware at the site.

The scale of the malware detection problem is huge. For that part you'll need to lean on your AV software, assuming that the AV software supports blocking malicious sites, and it has an API for you to have it test sites.

I would like to have a process to identify good and bad(where we get a red page with a warning). For such we won't get response. So is there any way to caputre the url response. I am new to C# i would like to implement this in SSIS. So verifying huge number of urls on daily basis would be really though.

 

[Skydiver]

The red page warning does not come from the web page itself. Think about it, would a malicious site actively identify itself as as being malicious when their main goal is to try to trick you? The red page you are seeing is a function of your web browser recognizing that either the URL is a known bad actor, or that the content being served up by the page is malicious.

If you can get access to the list of known bad actors, then you are partway there. But you also have to detect if one your URLs redirects to a bad URL.

As for recognizing malicious contents, that is why I said you'll need to defer to an AV package because they have the expertise, as well as the resources to stay up-to-date on the latest malware tricks and how to detect them.

Anyway, if you are a beginner, and you are looking at using SSIS, you don't even need C#. With Powershell, you could just use Invoke-WebRequest. But if you really want to do this in C#, Invoke-WebRequest just calls the .NET Framework's HttpClient class to get the web contents.

 

[NoUserHere]

Little more info needed.

I would like to automate the process of verifying the urls. I have more than 300 urls where i need to check there health.

What are you building?
Where do the links come from?
Is there a request made by your apps user to request pages, or are these pages you are storing in a database for user selection?
Who has access to change the pages of the links being served up through your app?

I am not aware of an AV package to recommend for this per the suggestion above. But if you have a bunch of 'signatures' to check against, you could crawl the source of the pages before they are served up by your app. Html agility pack can be used for this.

This topic seems very odd. Why would there be a malware alert on pages you are serving up?

 

[ssridhar296]

Take this as an example, when i hit the url i got this page. I don't get response. For the valid urls i get response. Is there a way we can track that response page. So i can identify which are good and which are blocking(with no response).

 

[JohnH]

Have you checked if there is an API available for developers? Google often has that for its many services.

 

[Skydiver]

Looks like the Google API is super easy to use. Just build out some JSON content with all the URLs you want to check (up to 500 at a time), and then get back Santa's Naughty List.

Safe Browsing Lookup API (v4) | Safe Browsing APIs (v4) | Google for Developers

developers.google.com

Beware that the API is only for non-commercial use. If you are going to be using Google to do site verification for profit, Google has a different API for you to use: Web Risk