Making HTTP Requests in Node.js


In previous articles, I’ve focused on creating a Node.js server to handle HTTP requests. This article looks at the problem in reverse, by showing you how to make HTTP requests from your Node.js applications. You may be asking yourself why you would want to do that. Two applications come to mind right away –web scraping and proxying. Scrapers are pieces of software which download web pages and programatically extract information from them. Proxy servers act as intermediaries, forwarding client requests to other servers and returning the responses.

The Request Module

The simplest way to create HTTP requests in Node.js is by using the request module. Written by Mikeal Rogersrequest allows you to make all types of HTTP requests, including GETPOSTPUT, and DELETE. Its flexibility makes the request module ideal for interacting with RESTful APIs. You can install request using the following npm command.

npm install request

Once you have installed request, create a new Node.js file containing the code shown below. The first line of the example is used to import request into the program. Making an HTTP request is as simple as calling the request() function. In this example, we are making a request to download the JSPro homepage. The second argument to request() is, of course, a callback function.

1 var request = require("request");
2 request("http://www.jspro.com"function(error, response, body) {
3   console.log(body);
4 });

The request() callback function takes three arguments – errorresponse, and body. As the name implies, error contains information regarding any problems that occurred during the HTTP request. Theresponse argument is an object containing a variety of data returned from the call, including the status code, headers, and much more. The third callback argument, body, is the body of the response. In the previous example, body contained the source code for the JSPro homepage.

The request module can also be combined with the file system module to stream HTTP requests to and from files. The following example again fetches JSPro. However, instead of displaying the source code in the console, it is written to a file named “jspro.htm”.

1 var request = require("request");
2 var fs = require("fs");
3 request("http://www.jspro.com").pipe(fs.createWriteStream("jspro.htm"));

Parameterizing Requests

The previous examples have created simple GET requests for URLs. However, we can modify the HTTP request by passing a configuration object as the first argument to request(). The following example uses the more powerful syntax to again download JSPro. Notice the numerous configuration parameters. Theuri and method parameters tell request to send a GET request to JSPro.

1 var request = require("request");
2 request({
3   uri: "http://www.jspro.com",
4   method: "GET",
5   timeout: 10000,
6   followRedirect: true,
7   maxRedirects: 10
8 }, function(error, response, body) {
9   console.log(body);
10 });

The timeout parameter tells request to wait 10,000 milliseconds (ten seconds) for a response before giving up. The final two parameters deal with following redirects. Many pages perform redirects behind the scenes without the user’s knowledge. The followRedirect and maxRedirects parameters instructrequest to follow up to ten 3xx HTTP redirects. Also, note that request() supports many other parameters which are not shown here.

Submitting Form Data

As previously mentioned, request supports more than just GET requests. One very common operation is the submission of form data using a POST request. The following example uses the form parameter ofrequest() to send a user’s name as part of a POST. The PHP page then responds by greeting the user by name.

1 var request = require("request");
2 request({
4   method: "POST",
5   form: {
6     name: "Bob"
7   }
8 }, function(error, response, body) {
9   console.log(body);
10 });

Dealing with Cookies

Many sites track various user interactions using cookies. Cookies are small pieces of data that are passed back and forth between the client and server with every HTTP transaction. To facilitate the use of cookies, request provides the jar() and cookie() methods. Individual cookies are created using thecookie() method. On line 3 of the following example, a cookie is created which specifies the user’s name. The cookie is then added to the cookie jar created on line 2. When the HTTP request is made, the jarparameter is used to send the cookie jar to the server.

1 var request = require("request");
2 var jar = request.jar();
3 var cookie = request.cookie("name=John");
4 jar.add(cookie);
5 request({
7   method: "GET",
8   jar: jar
9 }, function(error, response, body) {
10   console.log(body);
11 });

When the response is received, the user will be greeted as John, the name stored in the cookie. The cookie jar is also updated with any changes made by the server. If you print out the cookie jar before and after the request, you’ll see that the server changed the value of the cookie from "John" to "Stranger". The same cookie jar can be passed to consecutive HTTP requests in order to mimic a real user’s interactions with a site.

Conclusion

This article has introduced the simple, yet powerful, request module. As this article is not comprehensive, I recommend at least browsing over the documentation to see what else request can do. Stay tuned for an upcoming article which will cover web scraping using request in more depth.

Copy from http://www.sitepoint.com/making-http-requests-in-node-js/

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s