Processing Large Xml Files

inkedGFX

Well-known member
Joined
Feb 2, 2013
Messages
142
Programming Experience
Beginner
I am processing large xml files and need an efficient way to load the xml into memory and save the updates....what I am doing is getting data from web pages each page has 30 or more fields , once I have all the fields the program loads the xml with XDocument load....it then saves all the fields ..then goes to the next web page and gets the fields and then loads the xml again and saves it.

is there a way to load the xml once and then continue to save the new fields to it?

below is the code to load and save the xml

 XDocument spiderXml = XDocument.Load(xmlPath);
                            XElement Listing =  new XElement("Listing",          
                                                              new XElement("Position", pos),
                                                              new XElement("ID", splitID[1]),
                                                              new XElement("Title", carTitleFinal),
                                                              new XElement("ListingImages",
                                                                  new XElement("Img0", imageListOne[0]),
                                                                  new XElement("Img1", imageListOne[1]),
                                                                  new XElement("Img2", imageListOne[2])),
                                                              new XElement("Dealer",
                                                                  new XElement("DealerName", dName),
                                                                  new XElement("DealerAddress", dealerAddress),
                                                                  new XElement("DealerPhoneNumber", dPhoneNumber),
                                                                  new XElement("DealerImageLink", dealerImageFinal),
                                                                  new XElement("DealerLink", DealerLink)),
                                                              new XElement("Description", descFinal),
                                                              new XElement("StockNumber", stockNumber),
                                                              new XElement("Price", price),
                                                                new XElement("VehicleLocation",
                                                                  new XElement("Country", country),
                                                                  new XElement("CityState", cityState),
                                                                  new XElement("Zipcode", zip)),
                                                                new XElement("VehicleBasics",
                                                                  new XElement("TrimLevel", trim),
                                                                  new XElement("Odometer", odometer),
                                                                  new XElement("Year", year),
                                                                  new XElement("Vin", vin),
                                                                  new XElement("Style", style),
                                                                  new XElement("Convertible", convertible)),
                                                                new XElement("ExteriorDetails",
                                                                  new XElement("ExteriorColor", exteriorColor),
                                                                  new XElement("ExteriorRestorationHistory", exteriorRestorationHistory),
                                                                  new XElement("ExteriorCondition", exteriorCondition)),
                                                                new XElement("InteriorDetails",
                                                                  new XElement("InteriorColor", interiorColor),
                                                                  new XElement("SeatMaterial", interiorSeatMaterial),
                                                                  new XElement("InteriorCondition", interiorCondition)),
                                                                new XElement("EngineDetails",
                                                                  new XElement("EngineConfig", engineConfig),
                                                                  new XElement("EngineSize", engineSize),
                                                                  new XElement("EngineModifications", engineMods),
                                                                  new XElement("EngineHistory", engineHistory),
                                                                  new XElement("EngineCondition", engineCondition)),
                                                                new XElement("TransmissionDetails",
                                                                  new XElement("TransmissionType", transType),
                                                                  new XElement("DriveTrain", driveTrain)));
                                                              
                            
                            spiderXml.Root.Add(Listing);
                            spiderXml.Save(xmlPath);
                        }


Thank You for any help
-InkedGFX
 

wim sturkenboom

Well-known member
Joined
Aug 6, 2014
Messages
85
Location
Roodepoort, South Africa
Programming Experience
10+
If all fields are defined, you can consider the use of xml serialisation and deserialisation.

You define classes like Dealer, VehicleLocation, EngineDetails etc; for anything that is like an array (e.g. the images) I would use a List<> (List<string> for the images). You combine those into the class Listing.

Assuming you can have multiple listings in one xml, create a class for that contains a member List<Listing>.
 
Top Bottom