If you would like to hire my services, you can now do so by visiting the following link:
Website Design Darlington
Article Statistics
Code Bank Statistics
Summary: An example of how we can make a request to a web page and retrieve the resulting HTML
There are a few situations where it would be useful to be able to retrieve the HTML from a web page via code. Fortunately, this is made relatively easy by the HttpWebRequest and httpWebResponse classes.Firstly, we need to decide which type of form the web page is using. The two different methods are GET and POST and for this article, I'm going to assume you have a basic understanding of these methods and I'll simply show you an example of how to implement either method rather than go into the differences between them. For both methods we'll need a simple page to actually display the results so let's start by making a page with a TextBox on it: <%@ Page Language="VB" AutoEventWireup="false" CodeFile="Default1.aspx.vb" Inherits="Default1" %> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" > <head runat="server"> <title>Retrieve data from a web page</title> </head> <body> <form id="form1" runat="server"> <div> <asp:TextBox ID="TextBox1" runat="server" Rows="40" Columns="100" TextMode="MultiLine"></asp:TextBox> </div> </form> </body> </html> Now, we need to write the code based on whichever method we have decided to use.Using the GET methodThis is the shorter of the two methods as since we don't have to post any data we can simply set the URL and add any querystring values to that URL. We can then simply make a request using this URL and read the response given back to us: Imports System.IO Imports System.Net Partial Class Default1 Inherits System.Web.UI.Page Protected Sub Page_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Load Dim strURL As String Dim strResult As String Dim wbrq As HttpWebRequest Dim wbrs As HttpWebResponse Dim sr As StreamReader ' Set the URL (and add any querystring values) strURL = "http://aspnetlibrary.com/articles.aspx?Page=1" ' Create the web request wbrq = WebRequest.Create(strURL) wbrq.Method = "GET" ' Read the returned data wbrs = wbrq.GetResponse sr = New StreamReader(wbrs.GetResponseStream) strResult = sr.ReadToEnd.Trim sr.Close() ' Write the returned data out to the page TextBox1.Text = strResult End Sub End Class Using the POST methodThis method is slightly longer than the GET method due to the fact that we actually have to specify the data that is going to be sent to the page. We also have to set the actual type and length of this content before posting it so the page in question knows what it will be receiving: Imports System.IO Imports System.Net Partial Class Default1 Inherits System.Web.UI.Page Protected Sub Page_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Load Dim strURL As String = "" Dim strPostData As String = "" Dim strResult As String = "" Dim wbrq As HttpWebRequest Dim wbrs As HttpWebResponse Dim sw As StreamWriter Dim sr As StreamReader ' Set the URL to post to strURL = "http://www.webcom.com/cgi-bin/form" ' Post some values to the page strPostData = String.Format("your_name={0}&userid={1}&form_name={2}", "Mark Smith", "webcom", "tutortest") ' Create the web request wbrq = WebRequest.Create(strURL) wbrq.Method = "POST" ' We don't always need to set the Referer but in this case ' the page we are posting to will only issue a response if we do wbrq.Referer = "http://www.webcom.com/cgi-bin/form" wbrq.ContentLength = strPostData.Length wbrq.ContentType = "application/x-www-form-urlencoded" ' Post the data sw = New StreamWriter(wbrq.GetRequestStream) sw.Write(strPostData) sw.Close() ' Read the returned data wbrs = wbrq.GetResponse sr = New StreamReader(wbrs.GetResponseStream) strResult = sr.ReadToEnd.Trim sr.Close() ' Write the returned data out to the page TextBox1.Text = strResult End Sub End Class Both of these simple examples demonstrate an easy method of reading the results of a request to a web page. You can obviously parse the results if necessary and only display certain information from the web page but this article will at least get you up to that point.
Posted on 03/04/2007 03:12:53
1. DUB 03/04/2007 12:42:22
This is very useful. I've learned a lot about c#.When adapting this code for my own needs strResult is equal to the "view page source" of the page im contacting.It dosnt seem as though its actually "posting".Any Ideas?
2. Mark Smith 04/04/2007 00:29:02
"When adapting this code for my own needs strResult is equal to the "view page source" of the page im contacting."That is what I would expect to happen and that's what does happen in the example above. However, you then go on to say that you don't think the form is posting. Why do you think that? Doesn't the page return the HTML you were expecting? If that is the case, I imagine you haven't set the correct form values that the page you are posting to expects.
3. TCDooM 14/04/2007 01:31:05
nice, simple and understandable.Thanks.
4. wooncherk 14/04/2007 03:06:05
i wonder how can we write code to retrieve the static link of a media... for example... KeepVid is able to retrieve the real download link of videos on YouTube... can we write something similar using WebRequest and WebResponse as well?Thanks.
5. Mohammad Azam 14/04/2007 09:18:23
Nice example! You can also use the WebClient method to download the html of the page.
6. atarikg 18/04/2007 13:54:43
Thanks this is very understandable example for dummies :)like me ? :)
7. kalai 14/08/2007 20:32:37
Thanks.. Very useful code..
8. Vinay 16/08/2007 08:31:25
Very useful code, infact i was looking for exactly the same thing!!
9. Craig 26/10/2007 16:20:15
Dear Sir,I have uploaded a text document (.txt) onto a server in it's wwwroot, and want to read from this using an ASP.NET program also residing on wwwroot of that server.Could you advise me as how best to read a textfile in this situation?
10. manish kapde 08/11/2007 09:28:21
It is nice but return the source-code not the actual output ?? why ??
11. Manish Garg 27/12/2007 01:50:56
HiI need the same code in C#
12. Kumar 10/01/2008 05:20:48
i am using this code in my projects. But i have encountered some problem when retrieving content from particular URL some content is missing. How to over come it.
13. Leonardo 13/03/2008 10:21:52
That was exactly what I was looking for!Thanks!
14. jay praaksh tiwari 15/03/2008 07:55:16
I have tried to data read from SQL sever data base, but i could not able to find out the c# language query set to find the data in our page retreived.ASP.net is used in my project. Take the example of retreival of all rows (no of rows) from the table xyz; i am waiting for yur responce
15. Susan Kefgen 08/05/2008 14:50:17
It won't work for http://www.spiritair.com/welcome.aspx?pg=salesinformation&number=570 It says something about active scripting. How can I get past this??Thanks! Susie2483 [at] hotmail.com
16. pinkpanter 11/06/2008 06:15:17
Thank you fot this code. It saved my day.
17. Ramanathan 06/01/2009 12:36:30
Good work it is useful for retrive image/data from webpage
Please keep your comments relevant to this page. Any inappropriate or purely promotional comments may be removed. Email addresses are never displayed but are required so you can confirm your comments.
