Webrequest Returning More Source Than Browser > View Source

I am writing a Selenium utility in C# to help with automated testing. I may be missing something completely obvious here, but why is it that if I send an HttpWebRequest off to a se

Solution 1:

There are a number of possibilities. One likely culprit is the difference in the information you're sending to the server. Browsers typically send various headers, cookies, etc., that a web request does not have unless you explicitly add it.

For example, when I hit www.google.com with my web browser, I get stuff from Google Plus, and I get the fanciest experience possible because I'm on an evergreen browser. Roughly 139000 characters appear in my View Source page.

However, when I do a web request to the same URL (using the following code), I get only 45000 characters in the response stream:

asyncvoidMain()
{
    var result = await GetTextAsync("https://www.google.com");
    Console.Write(result.Length);
}

publicasync Task<string> GetTextAsync(string url){
    var result = await WebRequest.Create(url).GetResponseAsync();
    using (var stream = result.GetResponseStream())
    using (var reader = new StreamReader(stream))
    {
        returnawait reader.ReadToEndAsync();
    }
}

I suspect that if I were to set the WebRequest to send all the same cookies and headers that Chrome is sending to Google, my results would be much more similar.