Answered How download dynamic HTML of a site?

Lolzer

New member
Joined
Nov 22, 2020
Messages
4
Programming Experience
Beginner
Hello everyone, If anything, I apologize in advance for my bad English. Recently I faced a problem such as the lack of a part of an HTML document. I tried to upload HTML from Twitter.com, but saw that the part I needed was missing. During the development of the program I found out that there is an additional JS script loaded, which has the data I need (most likely). If anything, I'm looking for a data-testid element that is in Google's developer tools (F12 > Elements), but is not in my HTML file. I don't want to use any libraries ( Selenium, HTMLAgillytyPack and other, They don't help anyway ). Can anyone help me?
 
The first question is why are you uploading HTML from twitter.com to some other location. It feels like you are violating the Twitter Terms of Service by trying to do some screen scraping.
 
The first question is why are you uploading HTML from twitter.com to some other location. It feels like you are violating the Twitter Terms of Service by trying to do some screen scraping.
What difference does it make what I break? My goal is to do.
 
The first question is why are you uploading HTML from twitter.com to some other location. It feels like you are violating the Twitter Terms of Service by trying to do some screen scraping.
I'm sorry for such a harsh answer, but does it really matter?
 
Because:
1. Twitter had an API which you should be using to extract data from Twitter; and
2. This feels like an XY Problem.
 
Yes it does, sadly. We also prefer not to help people violate other companies polices. There is a reason why API's come with restrictions, and when they do, they often come with a policy which states any such scraping of data not available VIA the API is a violation of their terms of use. You too should respect that. It also sounds like you are trying to access data served up by the server which is likely protected. Twitter and Instagram and the likes have good protection which makes scraping some information difficult. Scrapping Twitter is difficult enough and without using external help from other libraries, you won't succeed.
 
Back
Top Bottom