HtmlAgilityPack Issue

inkedGFX

Well-known member
Joined
Feb 2, 2013
Messages
142
Programming Experience
Beginner
I am scraping data from a series of web pages, each page has 50 links for other pages , i load these pages one at a time and collect data from each page, each page has up to 60 fields of data that needs to be collected...this is working fine. My issue is that random fields will not be collected on random pages. I load a HtmlListCollection of all the fields and then do a foreach loop on the list to collect the data for each field....the issue is not all the fields are being collected...it could be 1 in 1000 pages that doesn't collect correctly or it could be a series of pages in a row...I never know which fields aren't getting collected until after the program is finished running...it usually takes 3 hrs to complete.

could this be a performance issue on my pc or maybe a connection problem to the internet?

any ideas as to what you may think is the problem would be greatly appreciated!

Thank You
-InkedGFX
 
Top Bottom