Between windsurf sessions and spikey haired adventures, Rick has been a software developer for over 25 years, developing business and Web applications since the very early days of the Web when you needed a hand crank or a pair of wire splicers to get online. NET stack on the back end. NET and the XML classes, there are still many needs to retrieve Web content directly and manipulate it as text or data downloaded into files. In this article, I will describe the functionality of the HttpWebRequest and HttpWebResponse classes and provide an easy to use wrapper class. The class simplifies the HTTP access and provides most of the common features in a single interface while still providing full access to the base functionality of the HttpWebRequest class. In the process, I will describe some sticky issues like string encoding and Cookie handling and some related topics like implementing events and running multiple threads to service Web requests.
|Published (Last):||4 October 2007|
|PDF File Size:||20.43 Mb|
|ePub File Size:||2.53 Mb|
|Price:||Free* [*Free Regsitration Required]|
Between windsurf sessions and spikey haired adventures, Rick has been a software developer for over 25 years, developing business and Web applications since the very early days of the Web when you needed a hand crank or a pair of wire splicers to get online. NET stack on the back end. NET and the XML classes, there are still many needs to retrieve Web content directly and manipulate it as text or data downloaded into files. In this article, I will describe the functionality of the HttpWebRequest and HttpWebResponse classes and provide an easy to use wrapper class.
The class simplifies the HTTP access and provides most of the common features in a single interface while still providing full access to the base functionality of the HttpWebRequest class. In the process, I will describe some sticky issues like string encoding and Cookie handling and some related topics like implementing events and running multiple threads to service Web requests. Last week I decided I needed a good, useful project to throw at.
NET to continue my learning curve while actually building something I can use. A few years back I wrote a Web Monitoring package that monitors a set of Web sites and sends out alerts when the sites are down and not responding.
I thought this would be a good 'training' application to re-build for. This application exercises HTTP functionality built into the. NET Framework, requires setting up and running multiple threads, hooking up events, managing a small set of data without a database backend, and finally provides a Windows Form user interface UI. In the future, converting this application to work as a Windows service would also be a nice feature.
This application lets explores a wide variety of features of a programming environment. In this month's article, I'll describe a few of the features I needed to build, specifically focusing on the HTTP retrieval mechanism. If you've ever worked in pre-.
These tools invariably all worked in some situations, but none of them really fit the bill for all instances. For example, WinInet can't scale on the server with no multi-threading support. WinHTTP, which is the latest Microsoft tool for COM, solves many of these problems, but doesn't work at all on Win9x, making it a bad choice for a client tool integrated into broadly distributed applications.
These classes handle most of the functionality provided through the HTTP protocol in a straightforward manner. The task of returning content from the Web requires very little code see Listing 1. Pretty simple, right? But beneath this simplicity lies a lot of power, too. Let's start by looking at how this works. Start by creating the HttpWebRequest object, which is the base object used to initiate a Web request. A call to the static WebRequest. The request portion controls how the outbound HTTP request is structured.
As such it handles configuration of the HTTP headers, the most common of which are expressed as properties of the HttpWebRequest object. A few examples are UserAgent, ContentType, Expires and even a Cookies collection that map directly to header values that are set when the response is sent. Headers can also be set explicitly using the Headers string collection, to which you can add either a whole header string or a key value pair.
Generally, the properties address all common headers, so you'll rarely need to resort to setting headers explicitly, most likely only to support special protocols for example, SoapAction for SOAP requests. The code Listing 1 , does nothing much with the request other than set a couple of the optional properties? If you need to POST data to the server, you'll need to do a little more work? I'll talk about this later. When the code performs the GetResponse call, only a small chunk of data is returned from the Web server.
The first chunk contains the HTTP header and the very first part of the data, which is simply buffered internally until read from the stream itself. The data from this initial request is used to set the properties of the HttpWebResponse object, so you can look at things like ContentType , ContentLength , StatusCode , Cookies and much more. Next, a stream is returned using the GetResponseStream method. Streams provide you with a lot of flexibility in handling how data is retrieved from the Web server.
As mentioned, the call to GetResponse returned only an initial internal buffer. To retrieve the actual data and read the rest of the result document from the Web server, you have to read the stream. The code in Listing 1 uses a StreamReader object to return a string from the data in a single operation.
Since a stream is returned, I could access the stream directly and read smaller chunks to provide other information, such as the progress status of the HTTP download. Also notice that when the StreamReader is created, I explicitly provide an encoding type? This is important because if the data is transferred as a byte stream and without the encoding, it will result in invalid character translations for any extended characters.
CodePage works fairly well for English or European language content, as well as binary content. Ideally, you will need to decide at runtime which encoding to use. For example, a binary file probably should write a stream out to a file or other location rather than converting to a string, while a page from Japan should use the appropriate Unicode encoding for that language.
The following code is useful for testing the CodePage and setting a default. For more information about encosing refer to the Encoding sidebar. This example uses the StreamReader object, which provides an easy mechanism to retrieve the contents of a stream into strings or arrays of characters. It also provides the handy ReadToEnd method, which retrieves the entire stream in a single batch.
The operation of reading the stream is what actually retrieves the data from the Web server except for the initial block that was read to retrieve the headers. In this case, a single read operation is called and retrieves all of the requested data. If you wanted to provide feedback, you can also read the data in chunks using the StreamReader's Read method, which lets you specify the size of the data to read.
You would run this in a loop and provide whatever status info you need on each read. With this mechanism, you can retrieve the data and provide progress information. POSTing data refers to the process of taking data and sending it to the Web server as part of the request payload. A POST operation both sends data to the server and retrieves a response from the server.
Posting uses a stream to send the data to the server, so the process of posting data is pretty much the reverse of retrieving data see listing 2 :. GetResponse call. All other manipulation of the Request object has no effect, as the headers get sent with the POST buffer. The rest of the code is identical to what was shown before? POST data needs to be properly encoded when sent to the server. You can utilize the static method System. UrlEncode to encode the data. In this case, make sure to include the System.
Web namespace in your project. Note this is necessary only if you're posting to a typical HTML page? This is all much easier to do using a custom class like the one included in this article. To send the actual data in the POST buffer, the data has to be converted to a byte buffer first. Again, we need to properly encode the string. Using the static Encoding. ASCII object with the GetBytes method, which returns a byte array, you should then set the ContentLength property so the server can handle the size of the data stream.
Simply write the entire byte array out to the stream in one Write method call with the appropriate size of the byte array. This writes the data and waits for completion. As with the retrieval operation, the stream operations actually cause data to be sent to the server, so if you want to provide progress information you can send smaller chunks and provide feedback to the user, if needed.
But if you build typical applications that use HTTP access quite a bit, you'll find that you have to set a number of additional properties, such as object properties. One nice feature of the. NET Framework is the consistency of common objects that are reused by many areas of the framework. If you use authentication in your ASP. NET applications, the objects used on the server have the same interface as on the client side. In this section, I'll address the topics of authentication, proxy configuration and using Cookies.
User authentication is a very common security measure for Web applications. Web authentication usually consists of either basic authentication which is application driven and typically prompts the user for an operating system account or NTLM integrated file security. If you're using basic authentication, only the username and password are meaningful, while with NTLM you can also pass a domain name.
If you're authenticating against an NTLM resource permissions are set on the server's file system from a Windows client application, you can also use the credentials of the user who is currently logged on like this:.
HttpWebRequest handles navigation of the Authentication HTTP protocol requests, so an authenticated request operates like any other if it validates. If the request fails due to authentication, an exception occurs.
Luckily, HttpWebRequest makes this fairly painless with a WebProxy class member that handles proxy information. To configure a proxy, you can use code like the following:. How much detail is provided to the Proxy object depends on the particular proxy server.
For example, a bypass list is not required and most proxies don't require a username and password, which means you don't need to provide the credentials. However, I would advise against this because your application will likely have to check which objects to set and pass based on the option the user requires. Cookies work by storing tokens on the client, making the client responsible for the management of any cookie created.
Normally, a browser manages all of this for you, but in this case, there's no browser for an application front end, so we're responsible for tracking this state. This means that when the server assigns a cookie for one request, the client must retain it and send it back to the server on the next request where it applies based on the Web site and virtual directory.
HttpWebRequest and HttpWebResponse provide the container to hold cookies, both for the sending and receiving ends, but they don't automatically persist them, so that becomes your responsibility.
Because the Cookie collections are nicely abstracted in these objects, they are fairly easy to save and restore.
Subscribe to RSS
Once you have retrieved data from a web service you will need to do something with it. NET applications you have an easy and powerful set of tools. NET 2. The simplest way to view the returned data is to get the response stream and put it into a string.
Retrieving HTTP content in .NET
How to: Request data by using the WebRequest class
The following procedure describes the steps to request a resource, such as a Web page or a file, from a server. The resource must be identified by a URI. Create a WebRequest instance by calling WebRequest. Create with the URI of a resource. For example:. If you need to set or read protocol-specific properties, you must cast your WebRequest or WebResponse object to a protocol-specific object type. For more information, see Programming pluggable protocols.