How To Extract Dynamic Ajax Content From A Web Page

My requirement is to extract the required content from a web page. The page has a section which is being populated using ajax. When i view in page source it is not showing the cont

Solution 1:

In C# you can use HTMLAgilityPack to craw data, but if you use webBrowser.DocumentText, you can't load ajax content from webpage to get xpath. So after webBrowser control loaded webpage completely. In Document_Complete method you add some codes below:

HtmlAgilityPack.HtmlDocumentdoc=newHtmlAgilityPack.HtmlDocument();
this.webBrowser1.Document;
IHTMLDocument2currentDoc=(IHTMLDocument2)this.webBrowser1.Document.DomDocument;

doc.LoadHtml(currentDoc.activeElement.innerHTML);

Solution 2:

Use Firebug under Firefox. Under NET tab you will see the extra content loaded.