Scott Smith

Secure Node Apps Against OWASP Top 10 - Cross Site Request Forgery

2015-06-29T10:09:00-07:00

Welcome to part 4 of the OWASP security series

Injection
Broken Authentication & Session Management
Cross Site Scripting (XSS)
Cross Site Request Forgery (CSRF)
Using Components with Known Vulnerabilities (Coming soon)

In this multipart series, we will explore some of the the OWASP top web application security flaws including how they work and best practices to protect your application from them. The focus will be on Express web applications in Node, but the principles shown can be applied to any framework or environment.

This part will cover cross site request forgery (CSRF).

Cross Site Request Forgery (CSRF)

So what exactly is a cross site request forgery vulnerability?

A CSRF attack occurs when an attacker is able to create forged HTTP requests and trick the victim into making those requests via image tags, XSS, and many other ways. When the user makes these malicious requests and is authenticated with the application, the attack can be even more devastating. The attacker is able to get the user to perform state changing operations that the user is authorized to do in the application such as updating account details, making purchases, transferring money, and even deleting the account. Essentially, the attacker takes advantage of the website’s trust in the user.

We will go over a few different scenarios and show how to protect against them.

Scenario 1: Changes allowed via GET requests

This first scenario occurs when a website allows changes to be done via GET requests.

Login

The first two steps show the victim logging into their bank. This is important because the user must be logged in for the CSRF attack to work.

GET /index.html

The attacker has a page on a site they control and shares a link on social media. Our user clicks the link to see what it is and their browser makes a request to /index.html on the attacker’s site.

HTTP/1.1 200 OK

Requests to /index.html returns the following HTML content. If you look at the image tag, you will see it will make a request directly to the bank’s transfer page along with some query string parameters specifying an account to transfer an amount of money to.

   src="https://bank.com/transfer?to=12345&dollars=1000000" width="0" height="0">

GET /transfer?to…

This is where things get bad. Because the bank supports changes via GET requests, the user browsing to the attackers site will automatically make a GET request to https://bank.com/transfer?to=12345&dollars=1000000. Because the user has already logged in, their session cookie is passed along in the request and the attack will succeed.

Solution

The solution here is very simple. Never allow changes to occur on GET requests. Only allow changes to be made when the HTTP method is POST, PUT, or DELETE.

Scenario 2: Posting exploited data to sites

We have now locked our applications down to no longer allow changes via GET requests. This next attack occurs by getting the user to automatically submit a malicious POST request to the web application.

Login

The first two steps show the victim logging into their bank.

GET /index.html

HTTP/1.1 200 OK

Requests to /index.html returns the following HTML content. What is going on here is the attacker is creating a form that when submitted will POST to our transfer endpoint on our web application. The content then adds a script that will automatically trigger that POST.

   name="bad" method="post" action="https://bank.com/transfer">
     type="hidden" name="to" value="12345">
     type="hidden" name="dollars" value="1000000">

POST /transfer?to…

This is where things get bad, again. Because the user is logged in and the attacker was able to get the user’s browser to automatically POST a malicious request to our web application, the user’s bank account will have been successfully hacked.

Solution

The solution here is to implement the synchronizer token pattern.

The following sample Express application shows how to implement this using the csurf npm package.

var express = require('express');
var csrf    = require('csurf');

var app = express();

app.use(csrf());

app.use(function(req, res, next) {
  res.locals._csrf = req.csrfToken();
  next();
});

//Add _csrf to rendered HTML forms as hidden field (see HTML below)

app.listen(80);

   method="post" action="transfer">
     type="hidden" name="_csrf" value="_csrf">
     type="text" name="to">
     type="test" name="dollars">
  

What this code does is makes it so that all POSTs made to our application MUST include a CSRF token or nonce. This token is set when the user makes a request to our page that contains the form and expects that same token when a POST is made. If the token is not there, the POST is not allowed. This will defeat the scenario we just went over because the attacker will not be able to generate a token that our application is aware of.

Scenario 3: Bypassing CSRF protections

Now that we do not make changes via GET requests as well as implementing CSRF protection via nonce tokens, we are still vulnerable to some attacks that can get around CSRF protection.

Login

The first two steps show the victim logging into their bank.

GET /index.html

HTTP/1.1 200 OK

Requests to /index.html returns the following HTML content. What is going on here is the attacker is creating an iframe that loads up our page along with our form that includes the fields and the CSRF nonce token.

   src="https://bank.com/transfer?to=12345&dollars=1000000">

For the sake of this scenario, the resulting form that would be loaded will look like the following.

 method="post">
  type="text" name="to" value="">
  type="text" name="dollars" value="">
  type="hidden" name="csrf" value="a0d73b12">

And finally, here is our code on our server handling POSTs to the transfer endpoint.

app.post('/transfer', function (req, res) {
  if (isValid(req.body.csrfToken)) {
    var to = req.params.to || req.body.to;
    var dollars = req.params.dollars || req.body.dollars;

    //Transfer money
  }
});

POST /transfer?to…

This is where things get bad, yet again. Like before, our user is logged in. The content from our page including the form is now loaded within an attackers site. Through clickjacking techniques the attack could potentially get the user to submit the form.

Solution 1

Before we discuss the solution, we need to understand what is wrong with our current setup. Even if the page loads up our form, the user would have to manually enter a bank account along with money for the form submission to work, right? Well, not exactly. Because we are not explicitly setting the form action, the default URL that will be POSTed to will be the Url the page was loaded from. In our case this will be the Url the attacker entered which contains query string parameters for the account to send money to and the amount.

If you look at how we coded our handling of the POST, we are using parameters from either the query string or the POSTed data. In this example and attack, the query string parameters would get used and the attack would succeed.

The first solution to this problem is to always set the form action. Instead of leaving it blank, you should always set it to the endpoint you want to submit to.

 method="post" action="https://bank.com/transfer">
  type="text" name="to" value="">
  type="text" name="dollars" value="">
  type="hidden" name="csrf" value="a0d73b12">

This will make it so the attacker cannot get the user to submit the form to an endpoint along with query string parameters.

Solution 2

The second solution goes along with the first. This one is to never use query string parameters for user input. This will force our code to only use parameters sent via the POSTed form data.

app.post('/transfer', function (req, res) {
  if (isValid(req.body.csrfToken)) {
    var to = req.body.to;
    var dollars = req.body.dollars;

    //Transfer money
  }
});

Solution 3

The final solution is one that is good to implement on all your applications in order to control how your application behaves and to help avoid this type of attack. The solution is to use the X-Frame-Options header.

By taking advantage of this header, you can tell web browsers (that support it) to never allow your application within frames. With this in place, the attacker will not be able to load your page within theirs.

Here is an example Express application using the helmet npm package to do this.

var express = require('express');
var helmet  = require('helmet');

var app = express();

app.use(helmet.xframe('deny'));       //never allow in frames

app.listen(80);

This code will send back a response header that will tell the browser to never allow it to be within a frame. Here is what the header looks like.

X-Frame-Options: DENY

Wrap up

I hope that by learning how CSRF attacks are performed you have a better understanding of how to protect your applications. I have shown a few ways in which you can protect yourself but it is important to learn more on this subject using the links I shared above. Also, these solutions are not exclusive. You should implement many layers of protection for your application.

I have a lot more tutorials coming so be sure to subscribe to my RSS feed or follow me on Twitter. Also, if there are certain topics you would like me to write on, feel free to leave comments and let me know.

Secure Node Apps Against OWASP Top 10 - Cross Site Scripting

2015-06-22T07:44:00-07:00

Welcome to part 3 of the OWASP security series

Injection
Broken Authentication & Session Management
Cross Site Scripting (XSS)
Cross Site Request Forgery (CSRF)
Using Components with Known Vulnerabilities (Coming soon)

This part will cover cross site scripting (XSS).

Cross Site Scripting (XSS)

So what exactly is a cross site scripting vulnerability?

XSS, the most prevalent web application security flaw, occurs when an application includes user supplied data in a page sent to the browser without properly validating, escaping, or sanitizing that content.

An XSS attack occurs when an attacker sends text-based scripts that exploit the interpreter in the browser. Essentially, the attacker takes advantage of the user’s trust in the website. By this, I mean the end user viewing your web application trusts the content you are sending back. Because of this implicit trust, an attacker can exploit it by having your web application return malicious content.

There are two main types of XSS attacks:

Non-persistent
Persistent

Non-persistent

Non-persistent, also referred to as reflected, are attacks where injected scripts are sent to a user’s browser via the user’s own request to the web application. These types of attacks are done by tricking the victim into initiating the attack via email, some other website, social media, etc. The result is the user clicks a malicious link, submits a form, or makes a request to a malicious site. In doing so, the user’s browser is used as the medium to send the injected code to the web application through the request which is then reflected back to user in the response. The browser then executes the injected code because it came from the web application which it trusts.

The following diagram shows how this type of attack is done. We will go over it step by step.

In this example, Site (our web application) implements a search function at the endpoint /search?q=searchTerm. An attacker has found that when they do a normal search on the site, the results page shows the search term along with the results. After some testing, the attacker finds that the site is not escaping or sanitizing the search term before showing it in the results page. The attacker has just found an XSS vulnerability.

Here is one way the attacker could exploit this vulnerability.

GET /index.html

HTTP/1.1 200 OK

Requests to /index.html returns the following HTML content. If you look at the anchor tag, you will see it links directly to the search page on our web application and supplies JavaScript as the search term.

   href="https://site.com/search?q=">
    Click Here
  

GET /search?q=…

If the user clicks the link, and many will, their browser will make a request to our search page along with the malicious JavaScript in the search term.

https://site.com/search?q=

HTTP/1.1 200 OK

Because our site is not properly escaping the search term being supplied by the user, we will return a response to the user that includes the injected JavaScript. In this example, it would just create an alert, but the attacker could do anything including injecting script tags that fetch other JavaScript.

Here is the response in a normal case

  
    You searched for: Puppies
  

Here is the response in our exploited case

  
    You searched for: 
  

Send valuable data

The end user’s browser will now execute the script that was injected into the page. The attacker could send valuable information embedded on the page, include other scripts to further the exploit, and more.

In this example, the attacker setup a page that the user had to click a link on. This attack could be done easier by simply returning a 302 response with the Location header set to the search page on our web application. This takes out the step of the user needing to click a link and makes the attack more probable. The user’s browser would get redirected to the exploited link and the attack would occur.

Persistent

Persistent, also referred to as stored, are attacks where the injected script is permanently stored on the target web application’s database. Victims are then tricked into clicking a link to the web application that returns back the injected script in the response after retrieving it from the database or persistent storage.

The following diagram shows how this type of attack is done. We will go over it step by step.

In this example, Site (our web application) allows users to POST comments at the endpoint /comment. An attacker has found that when they POST comments on the site, the posted data is not escaped or sanitized before storing it and/or showing it. The attacker has just found an XSS vulnerability.

Here is one way the attacker could exploit this vulnerability.

POST /comment

The attacker knows they can POST comments to the site that allow scripts to be injected. The attacker will POST an exploited comment to the site with the following content.

This comment is now persisted because the content is stored in the database. Anytime a user views the comment, the injected script will be executed.

GET /comment?id=1

A user is browsing the site and views the exploited comment. They could get to this comment by just being on the site or could be directly to it via email, social share, etc.

HTTP/1.1 200 OK

Because our site is not properly escaping or sanitizing input or output, the resulting response to the user will include the injected script.

Send valuable data

The injected script will be executed on the end user’s browser which allows the attacker to get a hold of valuable information embedded on the page, include other scripts to further the exploit, and more.

Solution 1

The first solution is to always validate, escape, or sanitize user input. As a rule, all user input should be treated as malicious. If you build your application in a way that distrusts user input, you will end up with a more secure system.

This can be done by either sanitizing all user input as it enters the system or as it leaves. The following code is one way to do this in an Express application. What we are doing here is setting up middleware that will process all posted data by sanitizing it. I like this pattern because it avoids the case where newly added forms could be missed if you are sanitizing per input.

var express    = require('express');
var bodyParser = require('body-parser');
var validator  = require('express-validator');

var app = express();

app.use(bodyParser.urlencoded());

app.use(validator());
app.use(function(req, res, next) {
  for (var item in req.body) {
    req.sanitize(item).escape();
  }
  next();
});

app.listen(80);

With this code in place, our previous exploit examples of injecting JavaScript would turn

into the following

>script<alert('hacked')>/script<

One issue with sanitizing user input as it enters the system is if someone is able to get data into your database or persistent storage through another means. If this happens, your application can still be vulnerable to XSS attacks. In this case, you will want to consider sanitizing data as it leaves your application.

Solution 2

The second solution is to tell the web browser to allow content only from trusted sources.

This can be done by taking advantage of a powerful response header called Content Security Policy or CSP. CSP is a response header that tells the browser the domains it should consider as valid sources of content.

This can be used in two ways for our applications. First, we can tell the browser which domains it is allowed to trust and run JavaScript from. Second, we can tell the browser to not allow any in-line script. These two methods can protect many of our users (the ones with browsers that support CSP) from XSS attacks. If the attacker cannot get the browser to load or execute the injected JavaScript, then the attack is stopped.

The following code shows an example Express application using the helmet package to implement CSP.

var express = require('express');
var helmet  = require('helmet');

var app = express();

app.use(helmet.csp({
  defaultSrc: ["'self'"],
  scriptSrc: ['*.google-analytics.com'],
  styleSrc: ["'unsafe-inline'"],
  imgSrc: ['*.google-analytics.com'],
  connectSrc: ["'none'"],
  fontSrc: [],
  objectSrc: [],
  mediaSrc: [],
  frameSrc: []
}));

app.listen(80);

In this example, we are telling the browser to only allow scripts from our own domain as well as *.google-analytics.com.

Here is what the response headers look like with newlines added for readability.

Content-Security-Policy: 
  default-src 'self';
  script-src *.google-analytics.com;
  object-src ;
  img-src *.google-analytics.com;
  media-src ;
  frame-src ;
  font-src ;
  connect-src 'none';
  style-src 'unsafe-inline'

Solution 3

The final solution is to do a deep analysis of your application and understand all the ways in which you could/should protect yourself. This cheat sheet_Prevention_Cheat_Sheet) provided by OWASP does a great job of discussing all the rules you should follow.

Wrap up

I hope that by learning how XSS attacks are performed you have a better understanding of how to protect your applications. I have shown a few ways in which you can protect yourself but it is important to learn more on this subject using the links I shared above. Also, these solutions are not exclusive. You should implement many layers of protection for your application.

Secure Node Apps Against OWASP Top 10 - Authentication & Sessions

2015-06-15T10:42:00-07:00

Welcome to part 2 of the OWASP security series

Injection
Broken Authentication & Session Management
Cross Site Scripting (XSS)
Cross Site Request Forgery (CSRF)
Using Components with Known Vulnerabilities (Coming soon)

This part will cover broken authentication and session management.

Broken Authentication and Session Management

So what exactly is a broken authentication or session management vulnerability? OWASP defines it as follows:

“Attacker uses leaks or flaws in the authentication or session management functions (e.g., exposed accounts, passwords, session IDs) to impersonate users.”

“An attacker can be an anonymous external attacker or a user with their own account who may attempt to steal accounts from others.”

“Such flaws may allow some or even all accounts to be attacked. Once successful, the attacker can do anything the victim could do. Privileged accounts are frequently targeted.”

The following scenarios will highlight some of the ways in which this type of vulnerability can be exploited and best practices to protect against them. Like all things security related, this list should not be considered the only ways in which you could be vulnerable.

Scenario 1: Plain text passwords

Surprisingly, there are still many sites that store user passwords in plain text. This becomes obvious when a site gets hacked and all their user’s information along with passwords are exposed.

This is bad for obvious reasons. Once an attacker knows a user’s password, they can now log in as that user and do whatever they want.

You should always build your applications assuming you will get hacked or data will get leaked. By taking this approach, you will be much more likely to take precautions to secure sensitive information. User passwords are one of the most important pieces of data to secure.

The best way to secure user passwords is with a strong hash. Encrypting a password is better than plain text but still not as secure as a hash. With encryption, all the attacker needs do is get a hold of the keys and they can determine the password.

One way to hash and verify passwords is with the following code.

var bcrypt = require('bcrypt-nodejs');

var hashPassword = function(password, callback) {
  bcrypt.genSalt(10, function(err, salt) {
    if (err) return callback(err);

    bcrypt.hash(password, salt, null, function(err, hash) {
      if (err) return callback(err);

      callback(null, hash);
    });
  });
};

var verifyPassword = function(password, hash, callback) {
  bcrypt.compare(password, hash, function(err, isMatch) {
    if (err) return callback(err);

    callback(null, isMatch);
  });
};

Just because your user’s passwords are hashed doesn’t mean you shouldn’t worry about locking down your application. Even with hashed passwords, an attacker can eventually determine the passwords of some of your user’s accounts with enough time.

Scenario 2: Session IDs in the URL

Session IDs are unique numbers web applications assign to a user for the duration of the user’s visit. It is used so the web server knows the identify of the user making the request without the user having to log in again.

Because the session ID can be used to identify a user as well as grant permissions, protecting it is very important. By passing the session ID via the URL, you are exposing your application and users to a hacker. Imagine the following request being made to your bank:

https://bank.com/account?sessionid=1234567

Passing session IDs like this can be bad for the following reasons:

Sharing of link grants others full access
Stored on cache servers
Stored in browser history
Leaked through the Referer header
Leaked through logs not properly protected
Much more visible and thus more dangerous

The solution to this problem is pretty simple. Pass session IDs via cookies and not the URL. The following code is an example Express application that uses cookies for the session ID.

var express = require('express');
var session = require('express-session');

var app = express();

app.use(session({
  secret: 'our super secret session secret',
  cookie: { maxAge: 3600000 } // 2 hours in milliseconds
}));

app.listen(80);

Scenario 3: Site accessed over HTTP

The next scenario where vulnerabilities can occur is if your site is accessed over HTTP.

This can be bad if you are not taking the necessary precautions to protect your users. Imagine if your user is on a public unencrypted wifi network. If your site is over HTTP, it is very easy for someone else on that wifi network to monitor and obtain your session ids. Even if the user is on a secure network, there is no guarantee that a third party somewhere along the route to the web application is not monitoring traffic.

Solution 1

The first and most obvious solution here is to run your site over HTTPS. If you have user accounts, users logging in, or session sate you should not run your site over HTTP. It is very affordable and even free solutions exist. By running over HTTPS, you will reduce a significant attack vector within your application.

Solution 2

The second solution is to not allow cookies to be sent over HTTP. Even if your site runs over HTTPS, you will very likely still be listening for HTTP requests in order to redirect them to the HTTPS endpoint. Because of this, unless you take measures, a user can still send their session id via the cookie over an unsecured channel.

In order to stop this, you need to use the secure flag for your cookies. This flag, when set in the response that sets the cookie, will tell the browser to only send this cookie over an HTTPS request.

The following Express application from previous examples is updated to show how to do this.

var express = require('express');
var session = require('express-session');

var app = express();

app.set('trust proxy', 1);

app.use(session({
  secret: 'our super secret session secret',
  cookie: {
    maxAge: 3600000,
    secure: true
  }
}));

app.listen(80);

Here is an example of what the header looks like with both the secure and httpOnly flags set.

Set-Cookie: connect.sid=fjdskfjdsksdfjksa; Domain=favatron.com; Path=/; Expires=Sun, 28 Jun 2015 02:12:41 GMT; Secure

Solution 3

The third solution is to not allow client side scripts access to your cookies. By default, client side JavaScript is able to read your cookies. This means that any third party JavaScript you are including has the potential to read your cookies and use the session id for nefarious purposes.

In order to stop this, you need to use the httpOnly flag for your cookies. This flag, when set in the response that sets the cookie, will tell the browser to not allow any client side JavaScript access to it. The cookie is only sent over requests.

The following Express application from previous examples is updated to show how to do this.

var express = require('express');
var session = require('express-session');

var app = express();

app.use(session({
  secret: 'our super secret session secret',
  cookie: {
    maxAge: 3600000,
    secure: true,
    httpOnly: true
  }
}));

app.listen(80);

Here is an example of what the header looks like with both the secure and httpOnly flags set.

Set-Cookie: connect.sid=fjdskfjdsksdfjksa; Domain=favatron.com; Path=/; Expires=Sun, 28 Jun 2015 02:12:41 GMT; HttpOnly; Secure

Solution 4

The fourth solution is to tell the browser to never make HTTP requests again. There is a great response header you can send back on all responses that tells the browser to only make requests over HTTPS for a certain amount of time. This response is called HTTP Strict Transport Security (HSTS).

This is a great feature, because it negates the problem where users may still make requests over HTTP to your site. Then at the most, users would make one request over HTTP. From that point forward, they would only make requests over HTTPS.

The following Express application from previous examples is updated to show how to do this using the helmet package.

var express = require('express');
var session = require('express-session');
var helmet  = require('helmet');

var app = express();

app.use(helmet.hsts({
  maxAge: 7776000000,
  includeSubdomains: true
}));

app.set('trust proxy', 1);

app.use(session({
  secret: 'our super secret session secret',
  cookie: {
    maxAge: 3600000,
    secure: true,
    httpOnly: true
  }
}));

app.listen(80);

Here is what the response header would look like.

Strict-Transport-Security: max-age=7776000; includeSubDomains

Along with using the response headers to control this, there is a registry where you can add your domain and have it automatically added to preload lists of Chrome, Firefox, Safari, and a future version of IE. This way, even that initial request over HTTP could be avoided. You can find the form here.

Wrap up

There are many more ways in which you can create broken authentication and session management. Hopefully this article has given you some insight into areas of your application to pay attention to.

Secure Node Apps Against OWASP Top 10 - Injection

2015-06-08T12:06:00-07:00

Welcome to part 1 of the OWASP security series

Injection
Broken Authentication & Session Management
Cross Site Scripting (XSS)
Cross Site Request Forgery (CSRF)
Using Components with Known Vulnerabilities (Coming soon)

This part of the series will cover Injection.

Injection

So what exactly is an Injection attack? An Injection attack occurs when an attacker sends text-based attacks that exploit the syntax of the targeted interpreter.

An attacker can be anyone capable of sending untrusted data to the system such as external users, internal users, administrators, etc.

Injection attacks can be very bad for an application. They can result in data loss, data corruption, data access, denial of access, and even complete takeover.

SQL Injection

When you hear about Injection attacks, the one you might think of first is SQL injection. SQL injection has been around for almost 20 years and is still a big issue for many web applications. A study done in 2012 by Imperva observed that average web applications get at least 4 SQL injection attacks per month.

Most commonly known as an attack vector for web applications, SQL injection can also be used to attack any application using SQL databases. Like Injection attacks in general, SQL injection is done by injecting code (via text) into data-driven applications. The goal of these attacks is to inject SQL statements and have them executed for the purpose of dumping the database contents, deleting data, and more.

Some of the systems that can be affected by this attack are SQL Server, PostgreSQL, MySQL, and any SQL based database.

Below you will see an example Express application using a MySQL database that is vulnerable to SQL injection.

var express    = require('express');
var bodyParser = require('body-parser');
var mysql      = require('mysql');
var connection = mysql.createConnection();

var app = express();

app.use(bodyParser.urlencoded());

app.post('/login', function (req, res) {
  var user = req.body.user;
  var pass = req.body.pass;
  var sql  = "SELECT * FROM users WHERE user = '" + user + "' AND pass = '" + pass + "'";

  connection.query(sql, function(err, results) {
    // ...
  });
});

app.listen(80);

The problem with this code is that we are building our SQL statement using user input. You can see this on lines 11 to 13. We are reading in user supplied input for the user and pass variables. These are then used via string concatentation to build our SQL statement.

So why is this bad? Let me show you an example of how bad this is. Imagine someone making the following HTTP request to the /login endpoint.

POST /login HTTP/1.1
Host: example.com
Content-Type: application/x-www-form-urlencoded

user=admin'--&pass=whatever

The resulting SQL statement would look like this.

SELECT * FROM users WHERE user = 'admin'--' AND pass = 'whatever'

Yeah, that is very bad. Assuming there is an account named admin, this attacker would now have access and permissions that the admin account has. But this is only a simple example. The attacker could do a multitude of things given this vulnerability.

Solution #1: Escape user input

In this example, we are using the mysql npm package. It offers functionality to properly escape user input which we can use to lock down our application. Many other libraries out there offer this type of functionality or you can find libraries solely for escaping user input. Here is our previous example now escaping user input.

var express    = require('express');
var bodyParser = require('body-parser');
var mysql      = require('mysql');
var connection = mysql.createConnection();

var app = express();

app.use(bodyParser.urlencoded());

app.post('/login', function (req, res) {
  var user = connection.escape(req.body.user);
  var pass = connection.escape(req.body.pass);
  var sql  = "SELECT * FROM users WHERE user = '" + user + "' AND pass = '" + pass + "'";

  connection.query(sql, function(err, results) {
    // ...
  });
});

app.listen(80);

Now, when someone sends an HTTP request like our previous example, the resulting SQL statement will look like this.

SELECT * FROM users WHERE user = 'admin''--' AND pass = 'whatever'

You can see that the resulting SQL statement no longer has the exploit in it.

While escaping user input is better than not, it is still not full proof. Escaping algorithms can have bugs, not cover all cases, or miss newly found exploits.

Solution #2: Parameterized SQL queries

Parameterize SQL queries is an even better way to secure your application. Instead of building a SQL statement using concatenation, we let a function replace the parameters within the statement and perform sanitation. Here is what our example would look like using this method. Please note this is a bit pseudo code but helps get the point across.

var express    = require('express');
var bodyParser = require('body-parser');
var db         = require('db');

var app = express();

app.use(bodyParser.urlencoded());

app.post('/login', function (req, res) {
  var user = req.body.user;
  var pass = req.body.pass;
  var sql  = "SELECT * FROM users WHERE user = $1 AND pass = $2";

  db.query(sql, [user, pass], function(err, results) {
    // ...
  });
});

app.listen(80);

What happens here is $1 and $2 get replaced with user and pass when we make the call to db.query. With our previous example, this is what the SQL statement would look like.

SELECT * FROM users WHERE user = 'admin''--' AND pass = 'whatever'

Node Injection

SQL is not the only way in which an injection attack can occur. Node can also be an attack vector. This can be done by getting it to execute JavaScript submitted as user input.

Attacks are done by taking advantage of applications that use eval() with user input. The following code shows an Express application vulnerable to this type of attack.

var express    = require('express');
var bodyParser = require('body-parser');

var app = express();

app.use(bodyParser.urlencoded());

app.post('/run', function (req, res) {
  eval(req.body.cmd);
});

app.listen(80);

If someone were to make an HTTP request like the following, things would be bad in the application. It would be stuck in an infinite loop outputting to the console.

POST /run HTTP/1.1
Host: example.com
Content-Type: application/x-www-form-urlencoded

cmd=while(1){console.log("HACKED")}

This is a more benign exploit example but if this hole existed in an application, an attacker could execute anything they want.

The solution here is pretty simple. Never use eval() with user input. If you have to, be aware of the risks and attempt to mitigate them as much as possible.

MongoDB Injection

Another type of injection attack you need to be aware of is when working with MongoDB. These attacks usually take advantage of query selectors or the fact that they do not get explicitly set. The following Express application currently has a vulnerability to this type of attack.

var express    = require('express');
var bodyParser = require('body-parser');

var app = express();

app.use(bodyParser.urlencoded());
app.use(bodyParser.json());

app.post('/login', function (req, res) {
  var user = req.body.user;
  var pass = req.body.pass;

  db.users.find({user: user, pass: pass});
});

app.listen(3000);

The exploit occurs because of the couple things. First, we are allowing JSON data to be posted to our application. Second, we are not specifying the query selectors to use (line 13). If an attacker were to make the following HTTP request to our application, a successful attack would occur.

POST /login HTTP/1.1
Host: example.com
Content-Type: application/json

{
    "user": {"$gt": ""},
    "pass": {"$gt": ""}
}

What happens here is the attacker is passing JSON objects for the user and pass parameters. When those are passed in to the db.user.find() call, the following occurs.

db.users.find({user: { '$gt': '' }, pass: { '$gt': '' }});

When this executes, we will find all users with user greater than ” and pass greater than ”. This will return all users within the user table. This happened because we are not explicitly setting the query selector so the attacker was able to specify one themself.

The solution to this exploit is simple. You need to explicity set the query selector. Here is our previous code updated to fix this issue. See line 13 for the changes.

var express    = require('express');
var bodyParser = require('body-parser');

var app = express();

app.use(bodyParser.urlencoded());
app.use(bodyParser.json());

app.post('/login', function (req, res) {
  var user = req.body.user;
  var pass = req.body.pass;

  db.users.find({user: { $in: [user] }, pass: { $in: [pass] }});
});

app.listen(3000);

Here is what the result is now if someone were to make the same exploit HTTP request.

db.users.find({user: { $in: [{ '$gt': '' }] }, pass: { $in: [{ '$gt': '' }] }});

Now, all users will not be returned for this attack and our code works as expected.

Wrap up

Injection attacks are one the most prevalent attacks out there, easiest to exploit, and can have a severe impact. If you can take one thing away from this article is to never trust user input. It is the trust in the input that allows these attacks to occur.

Twitatron: Building a production web app with Node - User Accounts

2015-05-26T07:55:00-07:00

Welcome to part 3 of the Twitaron series

In our previous article we leared how to add views, layouts, partials, controllers, and more.

In this installment of the Twitatron series, we will be diving into how to implement user accounts. By the end of this article you will have learned how to connect to MongoDB, used Mongoose for object modeling, implemented Passport for user authentication, allow users to login with their Twitter account, and have full support for user accounts.

Secrets

Before we go further into setting up support for logging in with Twitter, we need to add a way to easily develop locally and run in production. There are going to be settings that are different locally versus production and we don’t want these production values in our source code. There are many ways to handle this, but one way I like is to use a secrets module.

If you don’t already have a config directory in the root of your application, create one now. Inside this directory, create a new filed named secrets.js. Update this file to contain the following. We will be using many of these items in this and future tutorials.

module.exports = {
  db: process.env.MONGODB || 'mongodb://localhost:27017/twitatron',

  cryptos: {
    algorithm: 'aes256',
    key: process.env.CRYPTO_KEY || 'Your crypto key goes here'
  },

  sessionSecret: process.env.SESSION_SECRET || 'Your session secret goes here',

  twitter: {
    consumerKey: process.env.TWITTER_KEY || 'Your Twitter consumer key',
    consumerSecret: process.env.TWITTER_SECRET  || 'Your Twitter consumer secret',
    callbackURL: process.env.TWITTER_CALLBACK || 'http://localhost:3000/auth/twitter/callback',
    passReqToCallback: true
  }
};

When your application runs in production, you can setup all the necessary environment variables so they are used within your application. When you run locally, it will use the values specified within this module.

The last thing we need to do is use this module within our application. Update the code in server.js from our previous article to look like the following.

// Load required packages
var path = require('path');
var express = require('express');
var compression = require('compression');
var secrets = require('./config/secrets');

Connecting to MongoDB

If you don’t already have MondgoDB installed and running, you will want to go their official site and follow their installation instructions.

There are three things we need to do to connect to MongoDB.

Install the Mongoose package
Load the Mongoose package
Connect to it using our connection string

Install the package manually using the following command:

npm install mongoose --save

Update the code in server.js from our previous article to look like the following.

// Load required packages
var path = require('path');
var express = require('express');
var compression = require('compression');
var secrets = require('./config/secrets');
var mongoose = require('mongoose');

Connect to MongoDB

// Load required packages
var path = require('path');
var express = require('express');
var compression = require('compression');
var secrets = require('./config/secrets');
var mongoose = require('mongoose');

// Connect to the twitatron MongoDB
mongoose.connect(secrets.db);

If all goes well, your application should start up just fine. You will notice we are already using our secrets module for the MongoDB connection string.

User Model

We now need a model to store our user. Inside the models directory, create a file named user.js and add the following code to it. If you don’t have a models directory, go ahead and create one in the root of your application.

// Load required packages
var mongoose = require('mongoose');
var crypto = require('crypto');
var secrets = require('../config/secrets');

// Define our user schema
var UserSchema = new mongoose.Schema({
  twitterId: { type: String, unique: true, required: true },
  username: { type: String, unique: true, lowercase: true, required: true },
  email: { type: String, lowercase: true },
  name: { type: String, default: '' },
  created: { type: Date, default: new Date() },
  accessToken: { type: String, required: true },
  tokenSecret: { type: String, required: true }
});

UserSchema.methods.encrypt = function(text) {
  var algorithm = secrets.cryptos.algorithm;
  var key = secrets.cryptos.key;

  var cipher = crypto.createCipher(algorithm, key);
  return cipher.update(text, 'utf8', 'hex') + cipher.final('hex');
};

UserSchema.methods.decrypt = function(text) {
  var algorithm = secrets.cryptos.algorithm;
  var key = secrets.cryptos.key;

  var decipher = crypto.createDecipher(algorithm, key);
  return decipher.update(text, 'hex', 'utf8') + decipher.final('utf8');
};

// Export the Mongoose model
module.exports = mongoose.model('User', UserSchema);

So what is going on here?

We loaded the Mongoose package
Created a Mongoose schema which maps to a MongoDB collection and defines the shape of the documents within that collection.
We defined our schema to contain twitterId, username, email, name, created date, access token, and token secret.
We exported the Mongoose user model for use within our application.
We created two methods on our schema that we will use to encrypt and decrypt the access token and token secret.

Auth Controller

npm install passport --save
npm install passport-twitter --save

This will install the standard passport package along with passport-twitter. Passport-twitter will provide our application with Twitter authentication strategies. It will allow us to easily add Twitter login to our app.

In the controllers directory, add a file named auth.js with the following contents.

// Load required packages
var passport = require('passport');
var TwitterStrategy = require('passport-twitter').Strategy;
var User = require('../models/user');
var secrets = require('../config/secrets');

passport.serializeUser(function(user, done) {
  done(null, user);
});

passport.deserializeUser(function(user, done) {
  done(null, user);
});

passport.use(new TwitterStrategy(secrets.twitter, function(req, accessToken, tokenSecret, profile, done) {
  User.findOne({ twitterId: profile.id }, function(err, existingUser) {
    if (existingUser) return done(null, existingUser);

    var user = new User();

    user.twitterId = profile.id;
    user.username = profile.id;
    user.email = '';
    user.name = profile.displayName;
    user.created = new Date();
    user.accessToken = user.encrypt(accessToken);
    user.tokenSecret = user.encrypt(tokenSecret);

    user.save(function(err) {
      done(err, user);
    });
  });
}));

exports.twitter = passport.authenticate('twitter');
exports.twitterCallback = passport.authenticate('twitter', { failureRedirect: '/' });

What we are doing here is setting up passport to use the Twitter authentication strategy provided by the passport-twitter package. For our TwitterStrategy, we are defining a callback that will attempt to look up the user using the Twitter profile id and if found not found, create a new user. If all works well, it will return an existing user or create a new user.

The final piece of this is exporting the auth and authCallback functions which will be used within our application as route endpoints responsible for creating and logging users in via Twitter. Open up server.js and set it to the following code.

Also, because Passport Twitter strategy requires sessions, be sure to install the express-session package.

npm install express-session --save

// Load required packages
var path = require('path');
var express = require('express');
var compression = require('compression');
var secrets = require('./config/secrets');
var mongoose = require('mongoose');
var passport = require('passport');
var session = require('express-session');

// Connect to the twitatron MongoDB
mongoose.connect(secrets.db);

// Load controllers
var homeController = require('./controllers/home');
var authController = require('./controllers/auth');

// Create our Express application
var app = express();

// Tell Express to use sessions
app.use(session({
  secret: secrets.sessionSecret,
  resave: false,
  saveUninitialized: false,
}));

// Use the passport package in our application
app.use(passport.initialize());
app.use(passport.session());

// Add content compression middleware
app.use(compression());

// Add static middleware
var oneDay = 86400000;
app.use(express.static(path.join(__dirname, 'public'), { maxAge: oneDay }));

// Add jade view engine
app.set('views', path.join(__dirname, 'views'));
app.set('view engine', 'jade');

// Create our Express router
var router = express.Router();

// Landing page route
router.get('/', homeController.index);

// Auth routes
router.get('/auth/twitter', authController.twitter);
router.get('/auth/twitter/callback', authController.twitterCallback, function(req, res) {
  res.redirect(req.session.returnTo || '/');});

// Register all our routes
app.use(router);

// Start the server
app.listen(3000);

What we did here was to include the passport, express-session, and authController modules. After that we setup our Express application to use passport and passport session as middleware. Finally, we create two new endpoints responsible for logging users in via Twitter.

In order to test this, you will need to head to Twitter and register an application. You can do that here: https://apps.twitter.com/. Once you have an application, update the consumer key and secret inside secrets.js.

You can now test things out by making a request to http://localhost:3000/auth/twitter

Clean up our views

Before we update our views to support logging in and out, we need to clean up our views and some of the code behind it first.

Open up homeController.js and delete this line from the index action: res.locals.ip = req.ip;.

Open up home.jade and delete this line from the view: h2 You are visiting from #{ip}.

Allow users to login and logout

To know whether or not a user is currently logged in, we need to add a little code to our Express application. One of the nice things Passport provides is that it automatically adds a user object to the Express request object when someone is logged in. We can take advantage of this by passing it to our views. Open up server.js and update the code as follows right after we use passport.session.

// Setup objects needed by views
app.use(function(req, res, next) {
  res.locals.user = req.user;
  next();
});

What we are doing is adding the user object to the locals object in order to make it available in our views.

Next, we will want to update navigation.jade to show login or logout depending on the user’s state.

header
  div
    a(href='/') Twitatron
    if !user
      a(href="/auth/twitter") Login with Twitter
    else
      a(href="/auth/logout") Logout

The last thing we need to implement is a controller action for the route /auth/logout. Open up authController.js and add the following to the very end.

exports.logout = function(req, res) {
  req.logout();
  req.session.destroy();
  res.redirect('/');
};

Now, just define your route within server.js as follows.

// Auth routes
router.get('/auth/twitter', authController.twitter);
router.get('/auth/twitter/callback', authController.twitterCallback, function(req, res) {
  res.redirect(req.session.returnTo || '/');});
router.get('/auth/logout', authController.logout);

Go ahead and try things out. You should be able to click Login with Twitter, get redirected to Twitter, authorize access to your Twitter account, and have a User created or get logged in as an existing user.

Wrap up

We covered a lot of areas in this tutorial. First, we added a configuration module which allows easy configuration between development and production. Second, we learned about Mongoose and connected to MongoDB. Third, we created a Mongoose User model and created helper methods that allow us to encrypt and decrypt sensitive information such as access tokens and token secrets. Finally, we added the ability to log in with Twitter, have a user account created, and then log out.

If you found this article or others useful be sure to subscribe to my RSS feed or follow me on Twitter. Also, if there are certain topics you would like me to write on, feel free to leave comments and let me know.

Source code for this part can be found here on GitHub.

Algolia Real Time Search With ASP.NET MVC & Windows Phone 8.1

2015-04-04T20:11:00-07:00

In this article, we are going to explore how to easily add Algolia search to a Windows Phone 8.1 application with an ASP.NET MVC backend. To do this, we will take advantage of Algolia’s C# client to add indexing capabilities to the ASP.NET MVC application and search capabilities to the Windows Phone application.

The application we will be building is called PackageTrack. It is a simple web application where a user can create, read, update, and delete packages they like and use. The web application will be capable of managing the package information, indexing that data with Algolia, and providing a simple REST API for the data. We will also create a Windows Phone application with an auto suggest search box hooked into the Algolia search service to provide search results. Users will then be able to select one of the search results and have that information shown by requesting the data from the web application API.

Prerequisites

In order to follow along well, it will be helpful to share versions of tooling, operating system, and technology used for the tutorial. You will need to use (at the very least) Visual Studio 2013 with update 4. Also, in order to develop Windows Phone 8.1 applications, you will need to be developing on Windows 8 or greater. Finally, the application is being built using .NET framework 4.5.

Create a new ASP.NET Web Application

With Visual Studio running, create a new solution and choose an ASP.NET Web Application for your initial project. If you want to follow along exactly, name your solution PackageTrack and your project PackageTrack.Web.

To keep this tutorial simple, change the authentication to No Authentication.

You will also want to make sure MVC and Web API are selected.

Install necessary NuGet packages

This application requires two new packages in order to work.

Entity Framework

Install the Entity Framework package. The version used in this tutorial is 6.1.3.

PM> Install-Package EntityFramework

Algolia Search

Install the Algolia Search package. The version used in this tutorial is 3.0.5.

PM> Install-Package Algolia.Search

Finally, we need to update all installed packages to make sure we are using the latest libraries and scripts. You can do this by right clicking the web project in Solution Explore and choosing Manage NuGet Packages....

Cleanup

Now that we have our packages installed and updated, we need to perform a little cleanup in our project.

First, let’s delete the Home Controller, HomeController.cs, and the Home directory within Views. We are removing the controller and its views because we will be creating our own later.

Update our layout

Our layout needs a little updating to improve the look and feel a bit. Update \Vieews\Shared\_Layout.cshtml with the following.




     charset="utf-8" />
     name="viewport" content="width=device-width, initial-scale=1.0">
    </span>Package Track<span class="nt">
    @Styles.Render("~/Content/css")
    @Scripts.Render("~/bundles/modernizr")


     class="navbar navbar-inverse navbar-fixed-top">
         class="container">
             class="navbar-header">
                @Html.ActionLink("Package Track", "Index", "Home", new { area = "" }, new { @class = "navbar-brand" })
            
        
    
     class="container body-content">
        @RenderBody()
         />
        
            © @DateTime.Now.Year -  href="https://www.algolia.com/" target="_blank">Algolia
        
    

    @Scripts.Render("~/bundles/jquery")
    @Scripts.Render("~/bundles/bootstrap")
    @RenderSection("scripts", required: false)

Entity Framework and Database

We are now ready to create our model and DbContext in order to take advantage of Entity Framework. What we will do is a create a simple model to represent our package that we want to store in the database. After that, we will create our DbContext which will allow us to easily add, remove, update, and more on our packages stored in the database.

By default, the project is setup to use a local MDF database. You are welcome to change it but for simplicity this tutorial will not.

To create our package model, create a new file named Package.cs inside the Models directory. Update the created class to look like the following.

public class Package
{
    public int Id { get; set; }
    public string Name { get; set; }
    public string Link { get; set; }
    public int Count { get; set; }
}

Next, we need to create our DbContext. To do this, create a new directory called Data at the root of the project. Then create a new file named PackageTrackContext.cs inside this new directory. Update the code to the following.

using PackageTrack.Web.Models;
using System;
using System.Collections.Generic;
using System.Data.Entity;
using System.Linq;
using System.Web;

namespace PackageTrack.Web.Data
{
    public class PackageTrackDbContext : DbContext
    {
        public DbSet<Package> Packages { get; set; }
    }
}

Before moving on, be sure to build your solution. This will be necessary for the next steps where we use Visual Studio tooling to scaffold our CRUD controller and views for us.

Package CRUD

We now need to create the controller, actions, and views in order to create, read, update, and delete packages. To do this we will use Visual Studio tooling. This will make it super simple. The following 3 screenshots show what to do.

First, right click the Controller directory and choose Add followed by Controller.

Next, choose MVC 5 Controller with views, using Entity Framework.

Finally, be sure to choose the Package model we created for the Model class, the PackageTrackDbContext we created for the Data context class, check Use async controller actions, and set the Controller name to HomeController.

Visual Studio should create a new HomeController along with a set of views. You can now build and run your project. You should be able to create, read, update, and delete packages. Go head and do this now and create a few packages. We will need them later.

Build an Admin controller

We now need to create an Admin controller that will provide us the ability to issue index, re-index, and delete commands to our Algolia index.

Inside the Controllers directory, create a file named AdminController.cs. Update it to contain the following code.

using Algolia.Search;
using PackageTrack.Web.Data;
using PackageTrack.Web.Models;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using System.Web;
using System.Web.Mvc;

namespace PackageTrack.Web.Controllers
{
    public class AdminController : Controller
    {
        private PackageTrackDbContext db = new PackageTrackDbContext();

        public ActionResult Index()
        {
            return View();
        }

        [HttpPost]
        [ValidateAntiForgeryToken]
        public async Task<ActionResult> ReIndexData()
        {
            return View();
        }

        [HttpPost]
        [ValidateAntiForgeryToken]
        public async Task<ActionResult> IndexData()
        {
            return View();
        }

        [HttpPost]
        [ValidateAntiForgeryToken]
        public async Task<ActionResult> DeleteData()
        {
            return View();
        }

        protected override void Dispose(bool disposing)
        {
            if (disposing)
            {
                db.Dispose();
            }
            base.Dispose(disposing);
        }
    }
}

Inside the Views directory, create a new directory named Admin. Now that we have a new place for our admin views, create a new view there named Index.cshml. This is where we will setup our ability to issue index commands. Update the view with the following code.

Admin


     />

     class="form-actions no-color">
        @using (Html.BeginForm("reindexdata", "admin"))
        {
            @Html.AntiForgeryToken()
             type="submit" value="Re-Index Data" class="btn btn-default" />
        }
         />
        @using (Html.BeginForm("indexdata", "admin"))
        {
            @Html.AntiForgeryToken()
             type="submit" value="Index Data" class="btn btn-default" />
        }
         />
        @using (Html.BeginForm("deletedata", "admin"))
        {
            @Html.AntiForgeryToken()
             type="submit" value="Delete Data" class="btn btn-default" />
        }
    

Now we need to create 3 views that can be used to show when our index commands have completed. Create the following views with their markup inside the Views\Admin directory.

ReIndexData.cshtml

Data Re-Indexed

IndexData.cshtml

Data Indexed

DeleteData.cshtml

Data Deleted

Algolia time

Up to this point, we have been doing standard .NET development and nothing with Algolia (except adding the NuGet package). We are now ready to start.

Since our web application allows our users to create, update, and delete packages, we need to be able to do the same things to our Algolia index. On top of that, there are cases where we need to perform an index on all of our data, do a full re-index where data is removed and then added, and also remove all data from our index. We do all this from our web application rather than the Windows Phone application because it is much more secure to have our server talking to Algolia rather than each client. We don’t want our credentials that allow changing index data to fall into a user’s hand.

Create reusable Algolia client

The AlgoliaClient allows our application to easily interact with our Algolia index. In order to keep performance optimal, we don’t want to create an instance of this client each time we need it. To address this, we will create one AlgoliaClient and store it in an Application Variable.

Let’s go ahead and do this now.

Open up your Global.asax.cs file and update it as follows.

using Algolia.Search;
using PackageTrack.Web.Data;
using PackageTrack.Web.Models;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.Http;
using System.Web.Mvc;
using System.Web.Optimization;
using System.Web.Routing;

namespace PackageTrack.Web
{
    public class MvcApplication : System.Web.HttpApplication
    {
        protected void Application_Start()
        {
            AreaRegistration.RegisterAllAreas();
            GlobalConfiguration.Configure(WebApiConfig.Register);
            FilterConfig.RegisterGlobalFilters(GlobalFilters.Filters);
            RouteConfig.RegisterRoutes(RouteTable.Routes);
            BundleConfig.RegisterBundles(BundleTable.Bundles);

            // Create our Algolia client
            var algoliaClient = new AlgoliaClient("", "");

            // Create our index helper
            var indexHelper = new IndexHelper<Package>(algoliaClient, "packages", "Id");

            // Store our index helper in an application variable.
            // We don't want to create a new one each time
            // because it will impact performance.
            Application.Add("PackageIndexHelper", indexHelper);
        }
    }
}

What we are doing here is first creating an AlgoliaClient using our application id and key. You will need to update these two values using your own credentials. You can either use the Admin key already created for you or create your own key and choose the permissions allowed.

Next, we create an IndexHelper and inject our AlgoliaClient into it along with the name of the index and the value of our Package model identifier that should be mapped to the Algolia index object Id. The reason we want to specify the identifier in our model is because the IndexHelper will automatically build and map our model to the Algolia index object.

Finally, we add our IndexHlper to the application variable so we can use it elsewhere in our application.

One thing to note is the index we are using in Algolia is called packages. You can either create it ahead of time or it will be automatically created when you issue index commands.

Update our Admin controller to work with Algolia

Now that we have an Algolia IndexHelper to use, we need to update our Admin controller to talk to Algolia when someone issues an indexing command.

Update each of the 3 following actions inside AdminController.cs. Make sure and add a using statement for Algolia.Search.

[HttpPost]
[ValidateAntiForgeryToken]
public async Task<ActionResult> ReIndexData()
{
    // Get the package index helper from Application variable
    var packageIndexHelper = HttpContext.Application.Get("PackageIndexHelper") as IndexHelper<Package>;
    await packageIndexHelper.OverwriteIndexAsync(db.Packages, 1000);

    return View();
}

[HttpPost]
[ValidateAntiForgeryToken]
public async Task<ActionResult> IndexData()
{
    // Get the package index helper from Application variable
    var packageIndexHelper = HttpContext.Application.Get("PackageIndexHelper") as IndexHelper<Package>;
    await packageIndexHelper.SaveObjectsAsync(db.Packages, 1000);

    return View();
}

[HttpPost]
[ValidateAntiForgeryToken]
public async Task<ActionResult> DeleteData()
{
    // Get the package index helper from Application variable
    var packageIndexHelper = HttpContext.Application.Get("PackageIndexHelper") as IndexHelper<Package>;
    await packageIndexHelper.DeleteObjectsAsync(db.Packages, 1000);

    return View();
}

The IndexHelper is very nice. It will automatically handle the steps needed to perform indexing, re-indexing, and deleting of data. The way it works is it already knows the identifier to look for in your model so when you make an OverwriteIndex, SaveObjects, or DeleteObjects call it automatically converts your model to JSON, adds an objectId field, sets the objectId field to the identifier you told it to use, and makes the necessary calls to Algolia.

Now is a great time to test things out. You should already have packages in your database that are not in your Algolia index. Try out the different functions under the /admin path. You should see your index get populated, re-populated, and cleared out depending on which function you use.

Update our CRUD to work with Algolia

Having administrative ability to manage our index is great, but we really need to have our index kept up to date as users create, update, and delete packages from our database.

We will do this by updating the actions inside our Home controller we created earlier to support our CRUD operations on packages.

Update each of the 3 following actions inside HomeController.cs. Make sure and add a using statement for Algolia.Search.

[HttpPost]
[ValidateAntiForgeryToken]
public async Task<ActionResult> Create([Bind(Include = "Id,Name,Link,Count")] Package package)
{
    if (ModelState.IsValid)
    {
        db.Packages.Add(package);
        await db.SaveChangesAsync();

        // Get the package index helper from Application variable
        var packageIndexHelper = HttpContext.Application.Get("PackageIndexHelper") as IndexHelper<Package>;
        await packageIndexHelper.SaveObjectAsync(package);

        return RedirectToAction("Index");
    }

    return View(package);
}

[HttpPost]
[ValidateAntiForgeryToken]
public async Task<ActionResult> Edit([Bind(Include = "Id,Name,Link,Count")] Package package)
{
    if (ModelState.IsValid)
    {
        db.Entry(package).State = EntityState.Modified;
        await db.SaveChangesAsync();

        // Get the package index helper from Application variable
        var packageIndexHelper = HttpContext.Application.Get("PackageIndexHelper") as IndexHelper<Package>;
        await packageIndexHelper.SaveObjectAsync(package);

        return RedirectToAction("Index");
    }
    return View(package);
}

[HttpPost, ActionName("Delete")]
[ValidateAntiForgeryToken]
public async Task<ActionResult> DeleteConfirmed(int id)
{
    Package package = await db.Packages.FindAsync(id);
    db.Packages.Remove(package);
    await db.SaveChangesAsync();

    // Get the package index helper from Application variable
    var packageIndexHelper = HttpContext.Application.Get("PackageIndexHelper") as IndexHelper<Package>;
    await packageIndexHelper.DeleteObjectAsync(package);

    return RedirectToAction("Index");
}

What we did here was whenever a package is created, updated, or deleted, we use the IndexHelper to add, update, or remove the data within our index.

Go ahead and try it out.

Implement real time search

With all of our indexing in place, we are ready to add a real time search experience to our web application.

Download JavaScript libraries

We will need some JavaScript libraries in order to create the search feature. Download each of the following and save them in the Scripts directory with the names I indicate.

typeahead.jquery.js
https://cdn.jsdelivr.net/typeahead.js/0.10.5/typeahead.jquery.min.js

hogan.common.js
https://cdn.jsdelivr.net/hogan.js/3.0.2/hogan.common.js

algoliasearch.min.js
https://cdn.jsdelivr.net/algoliasearch/3/algoliasearch.min.js

Bundle new JavaScript

With the new scripts added to our project, we need to update our bundles to include them. Open up \App_Start\BundleConfig.cs and add the following new bundle.

bundles.Add(new ScriptBundle("~/bundles/main").Include(
          "~/Scripts/typeahead.jquery.min.js",
          "~/Scripts/hogan.common.js",
          "~/Scripts/algoliasearch.min.js",
          "~/Scripts/main.js"));

Update our layout

Now we need to update our layout to include a search box and the new script bundle we created.

Open up _Layout.cshtml and modify the code as follows.

....

 class="navbar navbar-inverse navbar-fixed-top">
     class="container">
         class="navbar-header">
            @Html.ActionLink("Package Track", "Index", "Home", new { area = "" }, new { @class = "navbar-brand" })
        
         class="typeahead-container">
             id="typeahead-algolia" class="typeahead" type="text" placeholder="Search...">
        
    


...

@Scripts.Render("~/bundles/jquery")
@Scripts.Render("~/bundles/bootstrap")
@Scripts.Render("~/bundles/main")
@RenderSection("scripts", required: false)

Create JavaScript to tie UI with Algolia

The following code needs to be added to a new file named main.js within the Scripts directory. This sets us up to perform searches with our packages index on Algolia. Be sure to update the application id and key using your search only key.

$(document).ready(function () {
    var client = algoliasearch('', '');
    var index = client.initIndex('');

    var template = Hogan.compile(
        '' +
            '' +
                '' +
                    '} ' +
                '
' +
                '' +
                ': }
' +
                '' +
            '
' +
        '');

    $('#typeahead-algolia').typeahead({
        highlight: false,
        hint: true,
        minLength: 1
    },
    {
        source: index.ttAdapter({ "hitsPerPage": 10 }),
        displayKey: 'Name',
        templates: {
            suggestion: function (hit) {
                // select matching attributes only
                hit.attributes = [];
                for (var attribute in hit._highlightResult) {
                    if (attribute === 'Name') {
                        // already handled by the template
                        continue;
                    }
                    // all others attributes that are matching should be added in the attributes array
                    // so we can display them in the dropdown menu. Non-matching attributes are skipped.
                    if (hit._highlightResult[attribute].matchLevel !== 'none') {
                        hit.attributes.push({ attribute: attribute, value: hit._highlightResult[attribute].value });
                    }
                }

                // render the hit using Hogan.js
                return template.render(hit);
            }
        }
    });
});

Style the search

Finally, we want to style our search. Add the following CSS to \Content\Site.css.

.typeahead-container {
    float: right;
}

.typeahead {
    padding: 0 5px;
    margin-top: 10px;
    width: 300px;
    height: 30px;
    font-size: 14px;
    line-height: 30px;
    max-width: none;
}

.tt-input,
.tt-hint {
    width: 396px;
    height: 30px;
    padding: 8px 12px;
    font-size: 14px;
    line-height: 30px;
    border: 2px solid #ccc;
    border-radius: 8px;
    outline: none;
}

.tt-input {
    box-shadow: inset 0 1px 1px rgba(0, 0, 0, 0.075);
}

.tt-hint {
    color: #999;
}

.tt-dropdown-menu {
    width: 422px;
    margin-top: 12px;
    padding: 8px 0;
    background-color: #fff;
    border: 1px solid #ccc;
    border: 1px solid rgba(0, 0, 0, 0.2);
    border-radius: 8px;
    box-shadow: 0 5px 10px rgba(0,0,0,.2);
}

.tt-suggestion {
    padding: 3px 20px;
    font-size: 14px;
    line-height: 24px;
}

    .tt-suggestion.tt-cursor {
        color: #fff;
        background-color: #0097cf;
    }

    .tt-suggestion p {
        margin: 0;
    }

    .tt-suggestion em {
        font-weight: bold;
        font-style: normal;
    }

.name {
    font-weight: bold;
}

.attribute {
    margin-left: 10px;
}

If you would like to get a more in depth tutorial on the real time search piece we just added, you can read other articles I wrote here.

http://scottksmith.com/blog/2014/10/21/getting-started-with-algolia-real-time-search/

http://scottksmith.com/blog/2014/10/29/algolia-real-time-search-with-twitters-typeaheadjs/

Create REST API for packages

The last thing we need to do in our web application is to build a REST based API for our package information. This will be used by our Windows Phone application.

First, create a new directory within the Controllers directory and name it Api.

Right click on the Api folder and choose Add followed by Controller. Follow the next few screenshots to see what settings to use and values to set.

Because we want to only return JSON even when the request header is not application/json we need to update our WebApiConfig.cs file in the App_Start directory.

using System.Net.Http.Headers;

  ...

  config.Formatters.JsonFormatter.SupportedMediaTypes.Add(new MediaTypeHeaderValue("text/html"));

We are now ready to create our Windows Phone application that will perform searches using Algolia and show data from our web application.

Create a new Windows Phone Application

In our current solution, create a new project and choose Blank App (Windows Phone). You can name it whatever you like, but I went with PackageTrack.Phone.

Install necessary NuGet package

This application requires one new package in order to work.

Algolia Search

Install the Algolia Search package. The version used in this tutorial is 3.0.5.

PM> Install-Package Algolia.Search

Configure startup projects

In order for both applications to work, we need to configure our solution to start the web and phone application when we build and run. The following screenshots show how to do this.

Debug or run your applications now to make sure both projects start.

Build our application UI

Now that our project and solution are ready to go, we can add some components to our application to build out our UI.

If MainPage.xaml is not open, you will want to open it now. This will load the designer view. The next two steps and screenshots show the two components we want to add to our view. Just drag each of the components from the Toolbox within Visual Studio into your UI as shown below.

Add AutoSuggestBox

Add WebView

Configure WebView

We need to give our WebView a name so we can reference it within our code. You will need to name it SearchWebView. The following code shows the updated XAML for this change.

 Name="SearchWebView" HorizontalAlignment="Left" Height="576" Margin="10,54,0,0" VerticalAlignment="Top" Width="380"/>

Configure AutoSuggestBox events

Our AutoSuggestBox is where the user will start typing a search query and will be responsible for searching our index on Algolia. We need to setup two events to handle when the text changes and when a suggestion is selected.

Update the MainPage XAML to the following for the AutoSuggestBox.

 HorizontalAlignment="Left" Margin="10,10,0,0" VerticalAlignment="Top" Width="380" TextChanged="AutoSuggestBox_TextChanged" SuggestionChosen="AutoSuggestBox_SuggestionChosen"/>

When adding new event handlers in this fashion, the code behind is automatically created in MainPage.xaml.cs. If they are not there, be sure to add them as follows.

private void AutoSuggestBox_TextChanged(AutoSuggestBox sender, AutoSuggestBoxTextChangedEventArgs args)
{

}

private void AutoSuggestBox_SuggestionChosen(AutoSuggestBox sender, AutoSuggestBoxSuggestionChosenEventArgs args)
{

}

Add Algolia to our application

If you don’t have MainPage.xaml.cs open, open it now.

We are going to add the necessary code to add Algolia search capabilities to our application.

The first thing we need is to add two private members to our MainPage class. These will be an AlgoliaClient and Index.

private AlgoliaClient algoliaClient = null;
private Index algoliaIndex = null;

Next, we need to instantiate these objects within the constructor. Be sure to change the application id and search only key to the ones you used in the web project.

public MainPage()
{
    this.InitializeComponent();

    this.NavigationCacheMode = NavigationCacheMode.Required;

    algoliaClient = new AlgoliaClient("", "");
    algoliaIndex = algoliaClient.InitIndex("packages");
}

Finally, be sure to add the necessary using statement.

using Algolia.Search;

Tie it all together

We now need to create a class that can be used to deserialize the Algolia search responses. Inside MainPage.xaml.cs create the following class. You will notice it defines Name, Link, Count, and objectID. The first 3 are fields we defined in our web application. The last one is the ID that Algolia uses in its indexed data. This is the field we mapped our models ID field to.

Also, you will see we are overriding ToString(). This is because we will be adding each Hit object to our AutoSuggestBox and want to control how the data is shown.

private class PackagesResult
{
    public class Hit
    {
        public string Name { get; set; }
        public string Link { get; set; }
        public string Count { get; set; }
        public string objectID { get; set; }

        public override string ToString()
        {
            return Name + Environment.NewLine +
                "  Link:\t" + Link + Environment.NewLine +
                "  Count:\t" + Count;
        }
    }

    public List<Hit> hits { get; set; }
}

Next we need to implement our search code each time a user enters a character in our AutoSuggestBox. Update the AutoSuggestBox event handler for TextChanged as follows.

private void AutoSuggestBox_TextChanged(AutoSuggestBox sender, AutoSuggestBoxTextChangedEventArgs args)
{
    if (args.Reason == AutoSuggestionBoxTextChangeReason.UserInput)
    {
        var searchTerm = sender.Text;
        var query = new Query(searchTerm);
        var result = algoliaIndex.SearchAsync(query).Result;
        var packagesResult = result.ToObject<PackagesResult>();

        sender.ItemsSource = packagesResult.hits;
    }
}

What we are doing here is creating a new Algolia Query using the entered text. We then issue a search, get back the results as JSON, and convert it to a PackagesResult object. The last thing we do is add the list of Hits as the ItemsSource of our AutoSuggestBox. This will automatically show the results to the user.

Finally, we need to update the SuggestionChosen event for our AutoSuggestBox when a user selects one of the results we showed them. Update it as follows.

private void AutoSuggestBox_SuggestionChosen(AutoSuggestBox sender, AutoSuggestBoxSuggestionChosenEventArgs args)
{
    var hit = (PackagesResult.Hit)args.SelectedItem;
    SearchWebView.Navigate(new Uri("http://localhost:8671/api/packages/" + hit.objectID));
}

All we are doing here is taking the selected item, reference that items object ID, and then navigating our WebView to that page. The page we are showing is the API endpoint we created in our web application. This is just a simple example of how to tie the two applications together. A more robust approach would be to process the data from the API and build a UI capable of viewing, editing, and deleting the data.

Wrap up

That wraps up this tutorial on adding Algolia to an ASP.NET MVC application along with a Windows Phone 8.1 application. There was a lot going on here, but if you take out the standard stuff for our web and phone application, the Algolia part is very simple and straightforward.

Source code for tutorial can be found here on GitHub.

Twitatron: Building a production web app with Node - Views & Controllers

2015-03-23T14:59:00-07:00

Welcome to part 2 of the Twitaron series

In our previous article we started with the basics and built a web application capable of serving static content, compressing that content, and implementing cache headers.

In this installment of the Twitatron series, we will be diving into Views and Controllers.

Getting setup

Our view engine of choice will be Jade. Jade is a terse language for writing HTML templates. It is capable of producing HTML, supports dynamic code, and supports reusability. You can find a tutorial here to learn more about Jade.

First thing we need to do is create a directory to store our views. If you don’t already have a directory named views you will need to create one now.

We will now need to tell Express about the view engine we wish to use.

Open up server.js and update it with the following code after the static middleware code.

...

// Add static middleware
var oneDay = 86400000;
app.use(express.static(__dirname + '/public', { maxAge: oneDay }));

// Add jade view engine
app.set('views', __dirname + '/views');
app.set('view engine', 'jade');

// Create our Express router
var router = express.Router();

...

Next, we need to install the Jade npm package.

npm install jade --save

What we did was add two pieces of middleware to our Express application. First, we told it to set the directory for our views to /views. Second, we told it to use Jade as our view engine.

We are now ready to create our first view.

Our first view

In the views directory create a new file named home.jade and add the following code to it.

doctype html
html
  head
    title Twitatron
  body
    h1 A Twitatron view has been born!
    img(src="/img/birdatron-small.jpg")

Next, we need to create a route at the root of our application to render this view. Open up server.js and update it with the following code. Be sure to remove the “dummy” route we made in the first tutorial that just returned the text “Twitatron”.

...

// Create our Express router
var router = express.Router();

// Landing page route
router.get('/', function(req, res) {
  res.render('home');
});

// Register all our routes
app.use(router);

...

Finally, because we now have a view, we do not need the index.html file inside our public directory. Go ahead and delete it.

What we have done is setup a new route to handle GET requests to /. When a browser requests http://localhost:3000/ it will execute the anonymous function we defined to render the view named home. Because we already defined the directory for our views to be views, Express will look for a view named home.jade, render it into html, and return it back to the requesting client.

Go ahead and test out your code to make sure everything is working. You should get back a response with the text “A Twitatron view has been born!” along with the Twitatron bird image.

Layout and partials

The next thing to update in our application is to implement a layout view and some partial views. This will help us reduce a significant amount of view code through reuse.

The first thing we will create is our layout view. This will define the general layout of our application and will be used in most of our other views. Inside the views directory, create a filed named layout.jade. Update it with the following code.

doctype html
html
  head
    title Twitatron
    include partials/head
  body
    include partials/navigation
    block content
    include partials/footer

You will notice that our layout contains references to 3 partial views. Let’s go ahead and create these now. First, create a sub-directory inside the views directory named partials. Next, create three new views named: head.jade, navigation.jade, and footer.jade. Update each one as follows.

Head

meta(charset='utf-8')
meta(http-equiv='X-UA-Compatible', content='IE=edge')
meta(name='viewport', content='width=device-width, initial-scale=1.0')
meta(name='description', content='Automatically monitor your Twitter account for mentions.')
meta(name='keywords' content='twitter, mentions, api, rss, email, storage, bookmark')
meta(name='csrf-token', content=_csrf)
link(rel='apple-touch-icon', sizes='57x57', href='/apple-touch-icon-57x57.png')
link(rel='apple-touch-icon', sizes='114x114', href='/apple-touch-icon-114x114.png')
link(rel='apple-touch-icon', sizes='72x72', href='/apple-touch-icon-72x72.png')
link(rel='apple-touch-icon', sizes='144x144', href='/apple-touch-icon-144x144.png')
link(rel='apple-touch-icon', sizes='60x60', href='/apple-touch-icon-60x60.png')
link(rel='apple-touch-icon', sizes='120x120', href='/apple-touch-icon-120x120.png')
link(rel='apple-touch-icon', sizes='76x76', href='/apple-touch-icon-76x76.png')
link(rel='apple-touch-icon', sizes='152x152', href='/apple-touch-icon-152x152.png')
link(rel='icon' type='image/png', href='/favicon-196x196.png', sizes='196x196')
link(rel='icon' type='image/png', href='/favicon-160x160.png', sizes='160x160')
link(rel='icon' type='image/png', href='/favicon-96x96.png', sizes='96x96')
link(rel='icon' type='image/png', href='/favicon-16x16.png', sizes='16x16')
link(rel='icon' type='image/png', href='/favicon-32x32.png', sizes='32x32')
meta(name='msapplication-TileColor' content='#9f00a7')
meta(name='msapplication-TileImage' content='/mstile-144x144.png')

This view contains all our information for the head section of our HTML. Right now there are references to many icons for our application that do not yet exist. Don’t worry about that for now as we will be making these in future tutorials.

Navigation

header
  div
    a(href='/') Twitatron

Pretty simple navigation piece for now. This will become more robust as we progress through the series.

Footer

footer
  div
    small © Twitatron 2015
    small Created by
      a(href="http://scottksmith.com", target="_blank") Scott Smith

The last thing we need to do is update our current view for the homepage to use the new layout view. Update home.jade with the following.

extends layout

block content
  h1 A Twitatron view has been born!
  img(src="/img/birdatron-small.jpg")

Our view now says to extend our layout view and use the content defined within the content block within the block content section in the layout.

Now, all future views we create can extend the layout view and thus provide a head section, navigation, and footer. Later we will update our layout to add things like CSS, JavaScript, and more.

Adding some dynamic code

By themselves, views are helpful but they are much more powerful when you support dynamic code. This can be done easily taking advantage of Express. Let’s go ahead and add a new element to our view that shows the IP address of the person making the request to our page.

Open up server.js and update our route handler as follows.

...

// Landing page route
router.get('/', function(req, res) {
  res.locals.ip = req.ip;
  res.render('home');
});

...

What this does is add the IP address of the client making the request to the res.locals.ip object which makes it available to our views. The res.locals object is where you can add anything you want available in views.

Next, update home.jade as follows with a new element.

extends layout

block content
  h1 A Twitatron view has been born!
  h2 You are visiting from #{ip}
  img(src="/img/birdatron-small.jpg")

This example is more to illustrate how to add dynamic code to your view from within your application. In future articles, we will dive more deeply into how to take advantage of this.

Controllers

The next thing we need to help make our application easier to understand and maintain are controllers. As our application grows, our route handlers will increase in quantity and complexity. We will do this by pulling out most functionality and placing them within separate modules.

Let’s start by updating the current route handler for we have the the root of our application.

If you don’t already have a directory named controllers you will need to create one now. Inside this folder, create a new file named home.js and update it with the following code.

exports.index = function(req, res) {
  res.locals.ip = req.ip;
  res.render('home');
};

What we have done is put our route handler code into a separate home controller module. Now we just need to update server.js to import the module and use it. Update server.jswith the following code.

// Load required packages
var path = require('path');
var express = require('express');
var compression = require('compression');

// Load controllers
var homeController = require('./controllers/home');

...

// Landing page route
router.get('/', homeController.index);

...

It may not seem like much now, but this type of pattern will help our application as we add a lot more functionality.

Using path to normalize paths

In two places, we have referenced the local file system using the __dirname global and a directory. To make our code more robust, we will take advantage of the core Node module path. The path module provides the function join that will join all the arguments into a normalized path.

For example, the following code would result in the path /Users/scott/Projects/twitatron/public

path.join(__dirname, 'public')

Update the two places within server.js where we are using the dirname global to use the path module.

//Old
app.use(express.static(__dirname + '/public', { maxAge: oneDay }));

//New
app.use(express.static(path.join(__dirname, 'public'), { maxAge: oneDay }));

//Old
app.set('views', __dirname + '/views');

//New
app.set('views', path.join(__dirname, 'views'));

Wrap up

While our application may not yet be pretty, we have a solid base to build and grow from. We have views, layouts, partials, controllers, and more. Stay tuned for more articles on this tutorial series on building production ready Node web applications.

Source code for this part can be found here on GitHub.

5 Fantastic & Free Screencast Sites For Node Developers

2015-02-26T06:47:00-08:00

While learning and working in Node, screencasts have been very beneficial. Over time I have come across many sites offering videos. Here is a list of my current top 5 every Node developer should know about.

KnowThen

James Moore, a personal friend of mine, is the newest kid on the block with KnowThen. He created the site to challenge himself to become a better instructor while at the same time creating quality videos for the community. What I really like about KnowThen is how it brings a fresh new look at Node.js by focusing on the bleeding edge. You will find videos on topics such as io.js, JSON Web Tokens, Koajs, Marko, and Generators to name a few.

http://knowthen.com/

Node Tuts

After a bit of a hiatus, Node Tuts by YLD is back and planning to start adding more videos to their site. Node Tuts was born back in the early days of Node (version 0.1) because there weren’t many resources around to learn from. There are currently four series to learn from: asynchronous, TCP servers, HTTP API servers, and tools, tips, & tricks.

http://nodetuts.com/

Tagtree

Tagtree, created by Hendrik Swanepoel, is focused on growing your full stack JavaScript skills. While not solely focused on Node.js, Tagtree brings fantastic screencasts along with some courses on topics such as Node streams, Express, promises, REST APIs, ES6, and lots more.

http://tagtree.tv/

Learn All The Nodes

An oldie but goodie, Learn All The Nodes by Big-Oh Studios has been around since 2013. Their series are about sharing the things they learn and love. Their videos focus more on short demonstrations of useful tips, tricks, and packages.

http://www.learnallthenodes.com/

StrongLoop

StrongLoop, whose focus is to deliver expertise and software solutions to help developers be successful with their Node projects, has a great set of videos available to learn from. They currently have over 30 videos focusing on topics such as clustering, performance, debugging, web frameworks, and more.

http://strongloop.com/node-js/videos/

Have Favorites of your Own?

If you have any favorites of you own, please share them in the comments. I am always looking to find more resources for myself and to share with others.

Algolia Real Time Search With Firebase

2014-12-09T16:17:00-08:00

One key feature of Firebase is building realtime applications by using their backend to store and sync data. Algolia further enhances that by providing realtime search capabilities. In a few simple steps, this tutorial will teach you how to import your existing data, index new data as it is added to Firebase, and remove indexed data when it is removed from Firebase.

Algolia’s Node.js client simplifies the integration of your Firebase applications with Algolia’s real time search service. The module makes it easy for you to use Algolia’s search capabilities in a manner that will be familiar to those already developing Firebase and Node.js applications.

Prerequisites

Familiar with Firebase

This tutorial assumes you are familiar with Firebase, how it works, and how to build Firebase applications. If you would like to learn more before continuing with this tutorial, I suggest reading the following documentation and tutorials:

Create a Node.js Application

In order to index your Firebase data and continually add/update index information, you will need to create a Node.js application. This application will be responsible for getting data out of Firebase and indexing it with Algolia. It can then be run anywhere like Heroku, Nodejitsu, AWS, Azure, etc.

In this tutorial, we will be indexing contact information in a Firebase application. Be sure to change ‘YourApplicationID’ and ‘YourAPIKey’ to your account values here. Because we are making calls that require more than read access, you will need to create a new key or use an existing one that can Add Records, Delete Records, and Delete Index (for reindexing example). You will also need to set your Firebase ‘INSTANCE’ to the one your application uses.

Here is the intial portion of the Node.js application.

var Firebase = require('firebase');
var algoliasearch = require('algoliasearch');
var client = algoliasearch('YourApplicationID', 'YourAPIKey');
var index = client.initIndex('contacts');

// Connect to our Firebase contacts data
var fb = new Firebase('.firebaseio.com/contacts');

Be sure to install the necessary packages so your application will run.

npm install firebase --save
npm install algoliasearch --save

Import Existing Data

In many cases, you may already have data within your Firebase application. In order to integrate with Algolia, you will want to index that data. We will use contact information being stored within Firebaseio as our example. Add the following code to your Node.js application.

// Get all data from Firebase
fb.on('value', initIndex);

function initIndex(dataSnapshot) {
  // Array of data to index
  var objectsToIndex = [];

  // Get all objects
  var values = dataSnapshot.val();

  // Process each Firebase ojbect
  for (var key in values) {
    if (values.hasOwnProperty(key)) {
      // Get current Firebase object
      var firebaseObject = values[key];

      // Specify Algolia's objectID using the Firebase object key
      firebaseObject.objectID = key;

      // Add object for indexing
      objectsToIndex.push(firebaseObject);
    }
  }

  // Add or update new objects
  index.saveObjects(objectsToIndex, function(err, content) {
    if (err) {
      throw err;
    }

    console.log('Firebase<>Algolia import done');
  });
}

To ensure the indexing performs well it is suggested you limit the number of items indexed per call between 1,000 and 10,000 depending on the object size.

Once you run this code, you will have all of your existing Firebase data indexed with Algolia. You will want to remove this code once is is done because the event will continue to fire each time data is added.

Reindex Data

Sometimes, you may have the need to completely reindex your data. This means removing data from the index that may not longer exist, adding new data, and updating existing data. The following code can be added to the Node.js application to perform a reindexing. You will want to remove or comment out the initial index code if currently present.

// Get all data from Firebase
fb.on('value', reindexIndex);

function reindexIndex(dataSnapshot) {
  // Array of objects to index
  var objectsToIndex = [];

  // Create a temp index
  var tempIndexName = 'contacts_temp';
  var tempIndex = client.initIndex(tempIndexName);

  // Get all objects
  var values = dataSnapshot.val();

  // Process each Firebase object
  for (var key in values) {
    if (values.hasOwnProperty(key)) {
      // Get current Firebase object
      var firebaseObject = values[key];

      // Specify Algolia's objectID using the Firebase object key
      firebaseObject.objectID = key;

      // Add object for indexing
      objectsToIndex.push(firebaseObject);
    }
  }

  // Add or update new objects
  index.saveObjects(objectsToIndex, function(err, content) {
    if (err) {
      throw err;
    }

    // Overwrite main index with temp index
    client.moveIndex(tempIndexName, 'contacts', function(err, content) {
      if (err) {
        throw err;
      }

      console.log('Firebase<>Algolia reimport done');
    });
  });
}

To ensure the reindexing performs well it is suggested you limit the number of items indexed per call between 1,000 and 10,000 depending on the object size.

Once you run this code, you will have all of your existing Firebase data reindexed with Algolia. You will want to remove this code once is is done because the event will continue to fire each time data is added.

Add or Update Data

Now, we need to handle the case where data is being added or updated. We can easily setup our code to automatically add or update data to our search index by attaching to the ‘child_added’ and ‘child_changed’ events. This will allow us to define code that will be called after data is stored in Firebase. Add the following code to your Node.js application.

// Listen for changes to Firebase data
fb.on('child_added', addOrUpdateObject);
fb.on('child_changed', addOrUpdateObject);

function addOrUpdateObject(dataSnapshot) {
  // Get Firebase object
  var firebaseObject = dataSnapshot.val();

  // Specify Algolia's objectID using the Firebase object key
  firebaseObject.objectID = dataSnapshot.key();

  // Add or update object
  index.saveObject(firebaseObject, function(err, content) {
    if (err) {
      throw err;
    }

    console.log('Firebase<>Algolia object saved');
  });
}

Now, whenever contact data is saved in Firebase, it will automatically be indexed with Algolia.

Delete Data

Next, we need to handle the case where data is deleted from your Firebase application. In order to do this, we can attach to the ‘child_removed’ event. This will allow us to define code that will be called after data is removed from Firebase. Add the following code to your Node.js application.

// Listen for changes to Firebase data
fb.on('child_removed', removeIndex);

function removeIndex(dataSnapshot) {
  // Get Algolia's objectID from the Firebase object key
  var objectID = dataSnapshot.key();

  // Remove the object from Algolia
  index.deleteObject(objectID, function(err, content) {
    if (err) {
      throw err;
    }

    console.log('Firebase<>Algolia object deleted');
  });
}

Now, whenever contact data is removed from Firebase, it will automatically get removed from Algolia.

Next Steps

Algolia Real Time Search With Parse

2014-12-08T08:42:00-08:00

One key feature of Parse is for applications to use Parse Core as their data store. In a few simple steps, this tutorial will teach you how to import your existing data, index new data as it is added to Parse, and remove indexed data when it is removed from Parse.

The Algolia Parse Module simplifies the integration of your Parse based applications with Algolia’s real time search service. The module makes it easy for you to use Algolia’s search capabilities in a manner that will be familiar to those already using the Algolia Node.js client APIs.

Prerequisites

Familiar with Parse

This tutorial assumes you are familiar with Parse, how it works, and how to build Cloud Code applications. If you would like to learn more before continuing with this tutorial, I suggest reading the following documentation and tutorials:

Add Agolia Real Time Search to the Project

In order to integrate Aloglia within your Parse application, you will need to add the Algolia Node.js client. Copy algoliasearch.parse.js to cloud/algoliasearch.parse.js within your Parse Cloud Code directory.

Import Existing Data

In many cases, you may already have data within your Parse application. In order to integrate with Algolia, you will want to index that data. We will use contact information being stored within Parse as our example.

var algoliasearch = require('cloud/algoliasearch.parse.js');
var client = algoliasearch('YourApplicationID', 'YourAPIKey');
var index = client.initIndex('contacts');

function indexData() {
  var objectsToIndex = [];

  //Create a new query for Contacts
  var query = new Parse.Query('Contact');

  // Find all items
  query.find({
    success: function(contacts) {
      // prepare objects to index from contacts
      objectsToIndex = contacts.map(function(contact) {
        // convert to regular key/value JavaScript object
        contact = contact.toJSON();

        // Specify Algolia's objectID with the Parse.Object unique ID
        contact.objectID = contact.objectId;

        return contact;
      });

      // Add or update new objects
      index.saveObjects(objectsToIndex, function(err, content) {
        if (err) {
          throw err;
        }

        console.log('Parse<>Algolia import done');
      });
    },
    error: function(err) {
      throw err;
    }
  });
}

You can now use this function within your own Parse Cloud Code functions in order to index your existing data.

Be sure to change ‘YourApplicationID’ and ‘YourAPIKey’ to your account values here. Because we are making calls that require more than read access, you will need to create a new key or use an existing one that can Add Records, Delete Records, and Delete Index (for reindexing example). If you create a new key, you will need to make sure it can Add Records and Delete Records.

Reindex Data

Sometimes, you may have the need to completely reindex your data. This means removing data from the index that may not longer exist, adding new data, and updating existing data. The following code can be used within your own Parse Cloud Code functions to perform a reindexing.

var tempIndexName = 'contacts_temp';
var mainIndexName = 'contacts';
var algoliasearch = require('cloud/algoliasearch.parse.js');
var client = algoliasearch('YourApplicationID', 'YourAPIKey');
var tempIndex = client.initIndex(tempIndexName);

var reindexData = function() {
  var objectsToIndex = [];

  // Create a temp index
  var tempIndex = client.initIndex(tempIndexName);

  // Create a new query for Contacts
  var query = new Parse.Query('Contact');

  // Find all items
  query.find({
    success: function(contacts) {
      // prepare objects to index from contacts
      objectsToIndex = contacts.map(function(contact) {
        // convert to regular key/value JavaScript object
        contact = contact.toJSON();

        // Specify Algolia's objectID with the Parse.Object unique ID
        contact.objectID = contact.objectId;

        return contact;
      });

      // Add new objects to temp index
      tempIndex.saveObjects(objectsToIndex, function(err, content) {
        if (err) {
          throw err;
        }

        // Overwrite main index with temp index
        client.moveIndex(tempIndexName, mainIndexName, function(err, content) {
          if (err) {
            throw err;
          }

          console.log('Parse<>Algolia reimport done');
        });
      });
    },
    error: function(err) {
      throw err;
    }
  });
};

Add or Update Data

Now, we need to handle the case where data is being added or updated. We can easily setup our code to automatically add or update data to our search index by using the afterSave Parse function. This will allow us to define code that will be called after data is stored in Parse.

var algoliasearch = require('cloud/algoliasearch.parse.js');
var client = algoliasearch('YourApplicationID', 'YourAPIKey');
var index = client.initIndex('contacts');

Parse.Cloud.afterSave('Contact', function(request) {
  // Convert Parse.Object to JSON
  var objectToSave = request.object.toJSON();

  // Specify Algolia's objectID with the Parse.Object unique ID
  objectToSave.objectID = objectToSave.objectId;

  // Add or update object
  index.saveObject(objectToSave, function(err, content) {
    if (err) {
      throw err;
    }

    console.log('Parse<>Algolia object saved');
  });
});

Now, whenever contact data is saved in Parse, it will automatically be indexed with Algolia.

Delete Data

Next, we need to handle the case where data is deleted from your Parse application. In order to do this, we can use the afterDelete Parse function. This will allow us to define code that will be called after data is removed from Parse.

var algoliasearch = require('cloud/algoliasearch.parse.js');
var client = algoliasearch('YourApplicationID', 'YourAPIKey');
var index = client.initIndex('contacts');

Parse.Cloud.afterDelete('Contact', function(request) {
  // Get Algolia objectID
  var objectID = request.object.id;

  // Remove the object from Algolia
  index.deleteObject(objectID, function(err, content) {
    if (err) {
      throw err;
    }

    console.log('Parse<>Algolia object deleted');
  });
});

Now, whenever contact data is removed from Parse, it will automatically get removed from Algolia.

Next Steps

Algolia Real Time Search with Twitter typeahead.js

2014-10-29T11:35:00-07:00

In our previous article we learned how to easily add Algolia’s real time search to our web application. We kept it simple and showed the results in a functional but not very pretty result list.

In this article, we will see how to quickly and easily add some awesomesauce to our application by integrating our search results with Twitter’s typeahead.js. If you haven’t read the previous article, you will want to so you can create your Algolia account and initial data index which will be used in this article.

Initial code

To help start this tutorial, we will start with a basic application structure. Create the following directory structure and files.

user-search-typeahead/
  css/
    styles.css
  js/
    main.js
  index.html

Add the following code to index.html, styles.css, and main.js.

 lang="en">

  </span>Algolia | User Search with typeahead.js<span class="nt">
   rel="stylesheet" type="text/css" href="/css/styles.css">

   class="search">
    Search users in real time with typeahead.js
     id="typeahead-algolia" class="typeahead" type="text">

.typeahead {
  width: 300px;
  height: 30px;
  font-size: 24px;
  line-height: 30px;
}

$(document).ready(function() {
  var client = algoliasearch('', '');
  var index = client.initIndex('');
});

You will need to update the JavaScript file and set your own application id, search-only api key, and index name. These will be the same ones you used for the previous tutorial we did.

Algolia’s support for Twitter typeahead.js

One of the many nice things about Algolia’s APIs is the built in adapter for Twitter typeahead.js.

/*
 * Get a Typeahead.js adapter
 * @param searchParams contains an object with query parameters (see search for details)
 */
ttAdapter: function(params) {
    var self = this;
    return function(query, cb) {
        self.search(query, function(success, content) {
            if (success) {
                cb(content.hits);
            }
        }, params);
    };
}

This adapter makes it very simple to hook into typeahead.js functionality for building beautiful auto complete UI components. In essence it is building the necessary hooks to automatically initiate a search and return the results as a user types within the typeahead.js component.

Add typeahead.js to our application

In order to add typeahead.js to our application, we need to include the script. In the index.html file, add the following typeahead.js script tag just before the Algolia script tag.

Now that we have the necessary scripts, we can tie the typeahead.js functionality with our Algolia search code. Update main.js to look like the following.

$(document).ready(function() {
  var client = algoliasearch('', '');
  var index = client.initIndex('');

  $('#typeahead-algolia').typeahead(null, {
    source: index.ttAdapter({ "hitsPerPage": 10 }),
    displayKey: 'name'
  });
});

So that is all we need in order to have nice type ahead/auto complete functionality tied to our Algolia real time search. What is happening here is as we type in the input field, the typeahead component is firing off searches using the ttAdapter we hooked in as the source. As results come back, the typeahead component pulls out the name field from each hit and shows that in the dropdown.

Make it a little prettier

Some final touches will be to make the results look a little nicer with some CSS. Open up styles.css and update it to the following.

.typeahead {
  width: 300px;
  height: 30px;
  font-size: 24px;
  line-height: 30px;
}

.tt-input,
.tt-hint {
  width: 396px;
  height: 30px;
  padding: 8px 12px;
  font-size: 24px;
  line-height: 30px;
  border: 2px solid #ccc;
  border-radius: 8px;
  outline: none;
}

.tt-input {
  box-shadow: inset 0 1px 1px rgba(0, 0, 0, 0.075);
}

.tt-hint {
  color: #999
}

.tt-dropdown-menu {
  width: 422px;
  margin-top: 12px;
  padding: 8px 0;
  background-color: #fff;
  border: 1px solid #ccc;
  border: 1px solid rgba(0, 0, 0, 0.2);
  border-radius: 8px;
  box-shadow: 0 5px 10px rgba(0,0,0,.2);
}

.tt-suggestion {
  padding: 3px 20px;
  font-size: 18px;
  line-height: 24px;
}

.tt-suggestion.tt-cursor {
  color: #fff;
  background-color: #0097cf;
}

.tt-suggestion p {
  margin: 0;
}

.tt-suggestion em {
  font-weight: bold;
  font-style: normal;
}

.name {
  font-weight: bold;
}

.attribute {
  margin-left: 10px;
}

Your results should look like this now if everything was done correctly.

Customize our typeahead.js options

There are three options we can control when setting up typeahead.js.

highlight - If true, when suggestions are rendered, pattern matches for the current query in text nodes will be wrapped in a strong element with tt-highlight class. Defaults to false.
hint - If false, the typeahead will not show a hint. Defaults to true.
minLength - The minimum character length needed before suggestions start getting rendered. Defaults to 1.

Right now we are not supplying options when creating our typeahead object. Let’s change that by updating our code in main.js to look like this.

$(document).ready(function() {
  var client = algoliasearch('', '');
  var index = client.initIndex('');

  $('#typeahead-algolia').typeahead({
    highlight: false,
    hint: true,
    minLength: 1
  },
  {
    source: index.ttAdapter({ "hitsPerPage": 10 }),
    displayKey: 'name'
  });
});

What we have done is passed in an object with the options explicitly set to the defaults. This will give you and idea of what you can tweak and play with. As the code stands now, the application will behave the same. Feel free to try different settings out to see how they behave.

Enhancing our dataset

The second parameter to typeahead.js is an object that is a dataset. Datasets in typeahead.js have 4 fields that can be set.

source – The backing data source for suggestions. Expected to be a function with the signature (query, cb). It is expected that the function will compute the suggestion set (i.e. an array of JavaScript objects) for query and then invoke cb with said set. cb can be invoked synchronously or asynchronously. A Bloodhound suggestion engine can be used here, to learn how, see Bloodhound Integration. Required.
name – The name of the dataset. This will be appended to tt-dataset- to form the class name of the containing DOM element. Must only consist of underscores, dashes, letters (a-z), and numbers. Defaults to a random number.
displayKey – For a given suggestion object, determines the string representation of it. This will be used when setting the value of the input control after a suggestion is selected. Can be either a key string or a function that transforms a suggestion object into a string. Defaults to value.
templates – A hash of templates to be used when rendering the dataset. Note a precompiled template is a function that takes a JavaScript object as its first argument and returns a HTML string.

We have already used source and displayKey. We will now take advantage of templates to further enhance our application.

Typeahead.js templates

For our template, we will use Hogan.js. What we essentially need is a function that can take a JavaScript object and return an HTML string. Hogan will give us that functionality.

First thing we need to do is include Hogan.js in our application. Open up index.html and add the following script tag before the Algolia script tag.

Next we will need to create our template for use by typeahead.js. Open up main.js and add the following code before the typeahead.js code.

var template = Hogan.compile('' +
  '' +
    '} ' +
  '
' +
  '' +
    ': }
' +
  '' +
  '
');

This template will serve to show our results by displaying the name along with a sub list of attributes that match out search criteria.

The final piece we need is to tell typeahead.js to use our template. Update the typeahead.js code like the following.

$('#typeahead-algolia').typeahead({
  highlight: false,
  hint: true,
  minLength: 1
},
{
  source: index.ttAdapter({ "hitsPerPage": 10 }),
  displayKey: 'name',
  templates: {
    suggestion: function(hit) {
      hit.attributes = [];

      // check each hit for highlighted results
      for (var attribute in hit._highlightResult) {
        if (attribute === 'name') {
          // already shown, no need to add a second time
          continue;
        }

        // all highlighted results with match level not equal to non should
        // be added in order to show up in the results drop down
        if (hit._highlightResult[attribute].matchLevel !== 'none') {
          hit.attributes.push({
            attribute: attribute,
            value: hit._highlightResult[attribute].value
          });
        }
      }

      // render a template for each hit
      return template.render(hit);
    }
  }
});

What we have done is defined what we want to show for the suggestion in the drop down by building our template. The really cool thing going on here is that we are taking advantage of the _highlightResult object for each of the hits provided by Algolia. For each of the highlighted results that have a match level not equal to none, we add it to our attributes object which will end up being shown as part of our template. Here is an example of a hit with the highlighted results. You will notice that email, web, and name have highlighted results when searching for the letter ‘k’.

{
  "company": "Frances Meyer Inc",
  "address": "2505 Congress St",
  "email": "kelli@varrato.com",
  "web": "http://www.kellivarrato.com",
  "name": "Kelli Varrato",
  "objectID": "3769410",
  "_highlightResult": {
    "company": {
      "value": "Frances Meyer Inc",
      "matchLevel": "none",
      "matchedWords": []
    },
    "address": {
      "value": "2505 Congress St",
      "matchLevel": "none",
      "matchedWords": []
    },
    "email": {
      "value": "kelli@varrato.com",
      "matchLevel": "full",
      "matchedWords": ["k"]
    },
    "web": {
      "value": "http://www.kellivarrato.com",
      "matchLevel": "full",
      "matchedWords": ["k"]
    },
    "name": {
      "value": "Kelli Varrato",
      "matchLevel": "full",
      "matchedWords": ["k"]
    }
  }
}

Final result

If everything is working properly, this is what your results should look like.

Wrap up

I absolutely love what Algolia is making for real time search. The ease of development is amazing and the speed of the search is unparalleled. Look forward to more articles as I dig deeper into the guts, options, and more on this fantastic developer service.

Getting Started With Algolia Real Time Search

2014-10-21T16:09:00-07:00

Ever heard of Stripe? Stripe makes it dead simple for developers to add payment processing and management to their applications through a set of tools and APIs. Now, imagine the speed and simplicity of Stripe’s payment integration for real time search. That is where Algolia comes in. Algolia does for search what Stripe does for payments.

Algolia is a service that provides APIs and tools for developers to facilitate and simplify the process of adding real time search to mobile and web applications.

A few of the features provided by Algolia are:

Real time database search
Get up and running in minutes
Client side integrations
Server side integrations
Expert help to build the perfect configuration

Enough spouting about how great it is, let’s start using it!

Creating an account

The first thing we need to do is create an account. You will be able to sign up and try out their service using the free Hacker Plan.

After you create your account, you will be asked to choose the location of your data center. This is an important step as it will determine performance and latency. As of writing this article it is not possible to change the location of your data center so you will want to choose wisely. The team has informed me this will be possible in the future along with multiple locations. The best advice now is to pick a data center closest to your servers.

After you pick the best data center, in my case North America West, you will be presented with a page to create your first data index.

Creating your first index

Indexes are entities within Algolia that allow you to define the attributes you want be be searched, the rank/order of those attributes, how to display that data, and much more.

You have three options for creating your first index.

Quick start with sample data
Upload your own data via the web application
Use of the API clients to get started

For this tutorial we will use the sample data option. Go ahead and pick the middle option: User Profile. You will see a loading screen as your data is being imported. Once it is done, you should see something similar to this. You are now ready to make your first search.

Algolia provides a very nice interface here that allows you to explore and search your newly created index. In the section titled “Explore your index”, you can enter search terms and see your results instantly below. Here is what it looks like when searching for the name “Kevin”.

You also have the option to view the data as raw JSON along with specifying numerous query parameters to control the search.

Adding real time to search to a web app

Let’s explore now how to integrate Algolia into a web application in order to provide real time search.

We will first create our HTML file for the web application. In order to keep this article simple to follow, I will not be placing CSS or JavaScript in separate files. Go ahead and create an index.html file with the following content.

 lang='en'>

  </span>Algolia | User Search<span class="nt">
  
   class="search">
    Search users in real time
     type="text">
  
   class="results">
     id="users">

What we have here is a very simple page with an input box that will allow us to search our data indexed with Algolia. We are including jQuery to simplify the code and the latest Algolia client side script to facilitate our integration.

The next thing we will do is add code to make queries to Algolia as a user enters search terms. Update your index.html file with the following code within the

There are 3 items in the code we just added that need to be configured for your account. You can find your APPLICATION ID and SEARCH-ONLY API KEY on the Credentials page of your account. Once you find them, update and with the values from your account. The final value needing to be updated is INDEX NAME. You can find the name of your index on the Indices page. Mine is called test_drive_contacts. Once you find yours, update with it.

There are a few things going on here in the code we just added. First, we are setting up a function to handle our callback from performing an Agolia search. We will return to this function in a bit and add functionality to it. For now, we will just log the JSON response to the console. Next, once the document is ready, we create an Algolia Search object with our API credentials. We then initialize our index with our Algolia Search object using the index name from our account.

Finally, we hook into the keyup event on our input field in order to fire off a search on our Algolia Index object. When this call completes, it will call the searchCallback function we defined.

With this bit of code we have successfully hooked into Algolia’s real time search. If you test this out in your web browser and open up the developer tools, you will see response data coming back in your console as you type characters in the search box.

Show our search results

Now that we have the core code in place to talk to and query our data from Agolia, we need to show that data to the user. To do this we will update the searchCallback function to show the resulting names. Update the searchCallback function to the following.

function searchCallback(err, content) {
  if (err) {
    console.error(err);
    return;
  }

  var $users = $('#users');
  $users.empty();

  for (var i = 0; i < content.hits.length; i++) {
    $users.append('' + content.hits[i].name + '
');
  }
};

This is simple web code not specific to Algolia. We are using jQuery to add the names returned in the search to an unordered list. First we clear out the list and then add the name of the user from the result. Once you have this done, try it out. You should see very fast results being returned and shown.

Where to go from here

We have only scratched the surface of what is available from Algolia’s real time search. Some of the many things we have not touched on are Sorting, Filtering, Faceting, and Geo-Search. To learn more, check out these tutorials and explore their documentation.

Also, in case you run into any issue with this tutorial, here is the final code.

 lang="en">

  </span>Algolia | User Search<span class="nt">
  
   class="search">
    Search users in real time
     type="text">
  
   class="results">
     id="users">

Twitatron: Building a production web app with Node

2014-10-05T09:30:00-07:00

Welcome to part 1 of the Twitaron series

While writing the Beer Locker tutorials many readers commented on how helpful it was to have a full walk through on creating RESTful APIs. I am going to continue this trend and start a multi part tutorial series on how to create a production ready Node web application.

Some of you may know about my latest project Favatron. It is a Node web application that integrates with Twitter to provide an automated bookmark and read later service for Twitter favorites. Using Favatron as a guide, I will take you through the entire process of creating a fully functional production ready Node web application. Some of the many parts we will touch on include authorization, user accounts, data storage, background workers, security, email, APIs, RSS, third party service integrations, and much more.

I am proud to introduce Twitatron, the web application we will be making. Twitatron will automatically monitor your Twitter account for mentions. When it finds those mentions, it will process them, store them, show them in the UI, expose them via an API endpoint and RSS feed, send an email digest, and share weekly stats with integrated social networks.

Enough talking, let’s get started!

Prerequisites

Before starting the tutorial, you will need to make sure and have Node.js and MongoDB installed.

Install the latest version of Node. This will also automatically install npm which will be used to manage packages.
Install and run MongoDB. If you would prefer to not install MongoDB locally, you can always sign up for a free MongoLab account and use it instead.

Initialize our Node application

First thing we need to do is create a directory for our application. I went ahead and named mine twitatron. Feel free to name yours whatever you want.

In your console, navigate into your newly create folder and run the following command:

npm init

You will be asked a few questions in order to have npm automatically create a package.json file for you. The only one that matters at this time is the entry point. You will want to specify server.js. For reference, here is what mine looked like.

npm init
This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sane defaults.

See `npm help json` for definitive documentation on these fields
and exactly what they do.

Use `npm install  --save` afterwards to install a package and
save it as a dependency in the package.json file.

Press ^C at any time to quit.
name: (twitatron)
version: (1.0.0)
description: Twitter mention tracker
entry point: (index.js) server.js
test command:
git repository:
keywords:
author: Scott Smith
license: (ISC)
About to write to /Users/scott/Projects/twitatron/package.json:

{
  "name": "twitatron",
  "version": "1.0.0",
  "description": "Twitter mention tracker",
  "main": "server.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "Scott Smith",
  "license": "ISC"
}


Is this ok? (yes)

Directory structure

As we progress through this tutorial series, we will be adding many directories to our application. Here is what the final structure will be.

twitatron/
  config/         // hold our configuration values
  controllers/    // hold our controllers
  models/         // holds our models
  node_modules/   // npm packages (auto created by npm)
  public/         // hold our static html, images, css, javascript, etc
  views/          // hold our jade views
  package.json    // defines our node app and dependencies
  server.js       // main application logic

Don’t worry too much now about creating all those directories. We will be creating them as we progress through the tutorial.

Simple HTTP server

Now it is time to start building our application. Within the application directory, create a file named server.js. This is the main entry point for our application. It is what will be run by Node. Let’s start our application as a super simple HTTP listener that just returns back ‘Twitatron’ for all HTTP requests.

var http = require('http');

http.createServer(function (req, res) {
  res.end('Twitatron!');
}).listen(3000);

What we are doing here is using the built in Node http module to start an HTTP server on port 3000. We then respond back to all requests with with the string ‘Twitatron!’. Start your application with the command node server.js and browse to it at http://localhost:3000.

Replace HTTP with Express

Our basic application is cool, but using the http module will get very difficult as the complexity increases. That is where Express comes in! Express is a minimal and flexible Node.js web application framework that provides a robust set of features for web and mobile applications.

We will use npm to install the Express module. Execute the following command and make sure to include the --save flag. This flag will add Express to the list of Node dependencies for your application. This is useful because in most cases you won’t add the node_modules directory to source control which means for other people that use your application, they will need to know which packages to install. This is done by executing the npm install command which will download and install any packages not in the local node_modules directory.

npm install express --save

You should now have Express added to the node_modules directory and the package.json file should have it added to the dependencies object as well. Here is what my package.json file now has.

{
  "name": "twitatron",
  "version": "1.0.0",
  "description": "Twitter mention tracker",
  "main": "server.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "Scott Smith",
  "license": "ISC",
  "dependencies": {
    "express": "^4.9.5"
  }
}

Now that we have Express installed, we can update our server code to use it. Replace your server.js file with the following code.

// Load required packages
var express = require('express');

// Create our Express application
var app = express();

// Create our Express router
var router = express.Router();

// Initial dummy route for testing
router.get('/', function(req, res) {
  res.end('Twitatron');
});

// Register all our routes
app.use(router);

// Start the server
app.listen(3000);

Let’s explore what we added to our application in detail.

Require

// Get the packages we need
var express = require('express');

This block of code will load the Express package and allow us to use it within our application. We will be requiring more packages here as we add more complexity to our application.

Express Application

// Create our Express application
var app = express();

The Express Application is the main component for your web application. Among other things, it is used to define routes, start listening for http connections, and perform routing for requests.

Express Router

// Create our Express router
var router = express.Router();

A router is an isolated instance of middleware and routes. Routers can be thought of as “mini” applications only capable of performing middleware and routing. Every express application has a builtin app router. Routers behave like middleware themselves and can be “.use()’d” by the app or in other routers.

Create Route

// Initial dummy route for testing
router.get('/', function(req, res) {
  res.end('Twitatron');
});

Here we are creating a route for ‘/’ to return ‘Twitatron’. All HTTP requests made to ‘/’ will execute this portion of code and return back a string.

Register Routes and Start Server

// Register all our routes
app.use(router);

// Start the server
app.listen(3000);

Finally we register our previously defined routes with the application.

The last step is to call listen on port 3000 to start listening for incoming connections and requests.

Test it out

If you did everything correctly, you should be able to start your server and make your first request. In the root folder of your application run:

node server.js

Open your browser and browse to http://localhost:3000/. You should get back a response with the word ‘Twitatron’.

Nodemon

Now is a good time to segue into some tooling that will make things much easier when developing Node applications. Nodemon is a utility that will monitor for any changes in your source and automatically restart your server.

You can install it using npm using the following command. I like to install it globally so I can use it for all projects, but you can remove the -g to install it locally instead.

npm install -g nodemon

Now instead of using node server.js to run your application, you can use nodemon server.js. It will watch for any changes in your application and automatically restart your server for you.

Node Inspector

Node Inspector is a debugger interface for Node.js applications that uses the Blink Developer Tools. The really cool thing is that it works almost exactly as the Chrome Developer Tools.

Some of the things you can do with Node Inspector are:

Navigate in your source files
Set breakpoints (and specify trigger conditions)
Step over, step in, step out, resume (continue)
Inspect scopes, variables, object properties
Hover your mouse over an expression in your source to display its value in a tooltip
Edit variables and object properties
Continue to location
Break on exceptions
Disable/enable all breakpoints

Just like Nodemon, you can install it locally or globally. Use the following command to install it globablly:

npm install -g node-inspector

Once it is installed, you can run it using the following command. This will start the debugger and open your browser.

node-debug server.js

Serving static content

The final piece of this part will be adding the ability to serve static content such as HTML, images, CSS, JavaScript, etc. To do this we will need to use the Express static middleware.

Update server.js with the following code.

...

// Create our Express application
var app = express();

// Add static middleware
app.use(express.static(__dirname + '/public'));

// Create our Express router
var router = express.Router();

...

What we have done here is register the static middleware with our Express application and told it our static content should be served from the public directory. If you didn’t already create the public directory, please do so now. You will also need to create js, img, and css as subdirectories.

Save this image (right click and save as) to your newly created public/img directory and name it birdatron-small.jpg. Image is Creative Commons licensed by Keoni Cabral and can be found here.

In the public directory, create a file named index.html and add the following markup to it.

  </span>Twitatron<span class="nt">


  Twitatron has been born!
   src="/img/birdatron-small.jpg">

Open up your web browser and browse to http://localhost:3000/. If everything was done correctly, you should see the following. You will notice now that our route that returned back ‘Twitatron’ is no longer getting executed. This is because the static middleware is automatically looking for index.html in the root of the public directory and returning it. If that were not there or named anything but index.html, the original route would execute.

Compress our static content

If you were to open up your DevTools and request http://localhost:3000/ you would notice the response is not being compressed with gzip. This is very inefficient yet surprisingly easy to fix. First thing we need to do is install the compression npm package.

npm install compression --save

Once installed, we can update our server.js to use this package.

// Load required packages
...
var compression = require('compression');

// Create our Express application
var app = express();

// Add content compression middleware
app.use(compression());

...

If you were to make a request again, you wouldn’t see the response headers indicating the content has been compressed with gzip. The reason for this is that the compression package defaults the bytes threshold to 1024. Our index.html file is below that threshold so it isn’t compressed. If you want to force compression on all responses regardless of size, you can update the code like this.

app.use(compression({
  threshold: false
}));

You can also control it by the number of bytes for the threshold like this.

app.use(compression({
  threshold: 256
}));

I would suggest leaving the defaults as is because there tends to be no gain when making the threshold too small. There are cases where your response could actually be bigger after compression.

Cache our static content

The last part of this tutorial will be caching our static content client side. This helps reduce unnecessary requests to your server and makes page load times much faster for client browsers. To do this, we will use the Express cache middleware.

If you were to examine the response headers now, you would see the following.

Cache-Control: public, max-age=0

This is telling the client browser to not cache the content. In order to change this, we need to update our Express static middleware in server.js.

...

// Add static middleware
var oneDay = 86400000;
app.use(express.static(__dirname + '/public', { maxAge: oneDay }));

...

What we have done here is tell our Express application to set the Cache-Control header to one day using milliseconds. Here is what the response headers look like after this change.

Cache-Control: public, max-age=86400

Wrap up

We now have the tools to create a simple, yet fully functional web application capable of serving static content that can be compressed and cached client side. This is just the beginning of a thorough and detailed tutorial series on building production ready Node web applications.

Source code for this part can be found here on GitHub.

3 Essential Sublime Text Plugins for Node & JavaScript developers

2014-09-29T14:20:00-07:00

Check out these 3 great and essential Sublime Text plugins every JavaScript and Node developer should know about and use.

JsFormat

https://github.com/jdc0589/JsFormat

JsFormat is a JavaScript formatting plugin. Behind the scenes, it uses the command line formatter from jsbeautifier.org to format full or portions of JavaScript and JSON files.

Features

JavaScript formatting
JSON formatting
Full file formatting
Selected text formatting
Customizable settings for formatting options
Customize per project with .jsbeautifyrc settings file

Usage

Either cmd+alt+f on OS X or ctrl+alt+f on Linux/Windows

JSHint

https://github.com/uipoet/sublime-jshint

“JSHint is a community-driven tool to detect errors and potential problems in JavaScript code and to enforce your team’s coding conventions. It is very flexible so you can easily adjust it to your particular coding guidelines and the environment you expect your code to execute in. JSHint is open source and will always stay this way.” - JSHint

Usage

ctrl+j on OS X or alt+j on Linux/Windows

If you would like to have JSHint run anytime you save a JavaScript file (highly suggest this), you will need to install the SublimeOnSaveBuild package.

JavaScriptNext

https://github.com/Benvie/JavaScriptNext.tmLanguage

This plugin is a better syntax highlighter for JavaScript. Not only does it improve syntax highlighting for current ES5, it also adds syntax highlighting for new ES6 syntax such as modules, succinct methods, arrow functions, classes, and generators.

Here is the original JavaScript syntax highlighter:

Here is the new JavaScript syntax highlighter:

Usage

You can either set individual JavaScript files to use this syntax highlighter by changing it in the “View -> Syntax” menu or you can change it for all JavaScript files in the “View -> Syntax -> Open all with current extension as”.

Wrap up

These 3 plugins have been very beneficial to me as a JavaScript and Node developer. If you know of other useful plugins, feel free to share them in the comments.

Protect Your Node App's Noggin With Helmet

2014-09-21T20:48:00-07:00

There are many ways in which attackers will attempt to exploit your applications and users. Some of the more famous are cross-site scripting (XSS), script injection, clickjacking, insecure requests, and identifying web application frameworks to name a few.

In my previous article, 4 Simple Steps to Secure Your Express Node Application, we learned how to use HTTPS to avoid man in the middle attacks, secure our cookies from being read by client side JavaScript, prevent our cookies from being sent on an HTTP request, and how to prevent cross-site request forgery attacks.

This article will focus more on middleware we can add to Express to further lock down our application.

Helmet

We will be using Helmet, a set of Express middleware, to help lock down and secure our web applications.

To install Helmet, simply run the following command in your application.

npm install helmet --save

Here is a simple sample Express application to get us started using Helmet. You will notice the main addition is requiring the Helmet module.

// Load required modules
var express = require('express');
var helmet = require('helmet');

// Create our Express application
var app = express();

// Simple endpoint
app.get('/', function(req, res) {
  res.send('Time to secure your application...');
});

// Start the server
app.listen(3000);

The following sections will dive into each of the security features provided by Helmet.

Content Security Policy (CSP)

Content Security Policy (CSP) is a promising defense against the risk and impact of XSS attacks. Its main goal is to prevent anything unintended being injected into our page. This can include frames, images, tracking scripts, and opening the door to XSS vulnerabilities.

CSP works by setting whitelists in the Content-Security-Policy response header to define sources of trusted content. The browser is then only allowed to execute or render resources from those trusted sources. This means that if someone were to successfully inject their script into your page, the browser would not execute it as the source would not be in the whitelist.

The main types of resources that can be controlled with CSP are:

JavaScript
Stylesheets
Images
AJAX, WebSockets, etc.
Fonts
Plugins
HTML5 Media Elements
Frames

So enough about what CSP is and what it provides. You are here to learn how to protect your web application. If you want to learn more, I would suggest a great article on HTML5 Rocks. Let’s dive into how we can implement CSP with Helmet.

// Load required modules
var express = require('express');
var helmet = require('helmet');

// Create our Express application
var app = express();

// Implement CSP with Helmet
app.use(helmet.csp({
  defaultSrc: ["'self'"],
  scriptSrc: ['*.google-analytics.com'],
  styleSrc: ["'unsafe-inline'"],
  imgSrc: ['*.google-analytics.com'],
  connectSrc: ["'none'"],
  fontSrc: [],
  objectSrc: [],
  mediaSrc: [],
  frameSrc: []
}));

// Simple endpoint
app.get('/', function(req, res) {
  res.send('Time to secure your application...');
});

// Start the server
app.listen(3000);

So what I have done here is told our Express application to use helmet.csp() and passed in some optional parameters. Helmet does have a way to use it with defaults, but I would suggest being explicit so you know exactly what it is being done. If you do want to use the defaults, you can do it with app.use(helmet());.

defaultSrc

This is the default policy for loading all content. Whatever is defined here applies to all the other type unless you set them to 'none'. In our case we are saying that any content coming from the same origin is allowed.

scriptSrc

This is the policy for controlling the valid sources of JavaScript. In our case we are setting *.google-analytics.com as an allowed source for scripts. This is in addition to the same origin rule we defined in defaultSrc.

styleSrc

This is the policy for controlling the valid sources of stylesheets. What we are doing here is using the 'unsafe-inline' keyword to allow inline stylesheets. If this was not set, any inline stylesheets would not work.

imgSrc

This is the policy for controlling the valid sources of images. Just like scriptSrc, we are setting the allowed domain for images.

connectSrc

This is the policy for controlling the valid sources of AJAX, WebSockets, or EventSource. In this case, we are using the 'none' keyword to state that no content of this type should be allowed. This overrides the defaultSrc directive.

fontSrc

This is the policy for controlling the valid sources of fonts. It is blank to state the only whitelist to use is the one defined in defaultSrc.

objectSrc

This is the policy for controlling the valid sources of plugins like , , or . It is blank to state the only whitelist to use is the one defined in defaultSrc. mediaSrc This is the policy for controlling the valid sources of HTML5 media types like or . It is blank to state the only whitelist to use is the one defined in defaultSrc. frameSrc This is the policy for controlling the valid sources of frames. It is blank to state the only whitelist to use is the one defined in defaultSrc. There are four special keywords you can use when defining whitelists: 'none' will match nothing 'self' will match the current origin but not subdomains 'unsafe-inline' allows inline JavaScript and CSS 'unsafe-eval' allows things like eval() to work There are also 5 other directives you can use. If interested, you can read more about them here. sandbox eportUri reportOnly setAllHeaders safari5 And finally here is what the response header looks like for our example (newlines added for better reading). 1 2 3 4 5 6 7 8 9 10 Content-Security-Policy: default-src 'self'; script-src *.google-analytics.com; object-src ; img-src *.google-analytics.com; media-src ; frame-src ; font-src ; connect-src 'none'; style-src 'unsafe-inline' XSS Filter The XSS Filter was created to help protect against XSS attack. Ironically, it actually helps attackers perform XSS attacks against older versions of Internet Explorer! You can read more about it here. What we need to do is have the X-XSS-Protection header set to 0 in order to disable it for older versions of IE. Helmet provides a piece of middleware to do just that. It will detect the web browser and either set it to 1; mode=block or 0. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 // Load required modules var express = require('express'); var helmet = require('helmet'); // Create our Express application var app = express(); // Implement X-XSS-Protection app.use(helmet.xssFilter()); // Simple endpoint app.get('/', function(req, res) { res.send('Time to secure your application...'); }); // Start the server app.listen(3000); Here is what the response header looks like when using a modern web browser without the exploit. 1 X-XSS-Protection: 1; mode=block And here is what it would look like on an older version of Internet Explorer with the exploit. 1 X-XSS-Protection: 0 Frame Options To help mitigate the risk of clickjacking attacks, Helmet offers a piece of middleware to help control the X-Frame response header. What this does is allow you to control if and where your page can be put into a or . This is also nice even from a non security point of view. You may not want your site to be loaded within anyone else’s frames from a purely aesthetic point of view. There are three options with the X-Frame response header: Deny - Does not allow your page to be served inside any frames. SameOrigin - Allows your page to be served inside a frame with the same origin. Allow-From - Allows your page to be served inside a frame from a specific URL. Deny 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 // Load required modules var express = require('express'); var helmet = require('helmet'); // Create our Express application var app = express(); // Implement X-Frame: Deny app.use(helmet.xframe()); //app.use(helmet.xframe('deny')); //Same thing // Simple endpoint app.get('/', function(req, res) { res.send('Time to secure your application...'); }); // Start the server app.listen(3000); Here are the response headers. 1 X-Frame-Options: DENY SameOrigin 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 // Load required modules var express = require('express'); var helmet = require('helmet'); // Create our Express application var app = express(); // Implement X-Frame: SameOrigin app.use(helmet.xframe('sameorigin')); // Simple endpoint app.get('/', function(req, res) { res.send('Time to secure your application...'); }); // Start the server app.listen(3000); Here are the response headers. 1 X-Frame-Options: SAMEORIGIN Allow-From 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 // Load required modules var express = require('express'); var helmet = require('helmet'); // Create our Express application var app = express(); // Implement X-Frame: Allow-From app.use(helmet.xframe('allow-from', 'http://example.com')); // Simple endpoint app.get('/', function(req, res) { res.send('Time to secure your application...'); }); // Start the server app.listen(3000); Here are the response headers. 1 X-Frame-Options: ALLOW-FROM http://example.com HTTP Strict Transport Security (HSTS) HTTP Strict Transport Security (HSTS) allows a web server to tell user agents to interact with it in the future only over HTTPS. It controls it by defining a period of time with the Strict-Transport-Security response header. This is great if you are running an HTTPS site but still have an HTTP endpoint in order to provide redirection to the HTTPS endpoint. This can help reduce the chance of Man-In-The-Middle (MITM) attacks by reducing the frequency of requests being made over insecure channels. Helmet provides middleware to help control setting and configuring the HSTS headers. You can specify how long the user agent should make requests over HTTPS and whether or not to include subdomains. Here it is in action. We are telling it to set the maximum age to 90 days (value is in milliseconds). While the code uses milliseconds, the response header will be in seconds. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 // Load required modules var express = require('express'); var helmet = require('helmet'); // Create our Express application var app = express(); // Implement Strict-Transport-Security app.use(helmet.hsts({ maxAge: 7776000000, includeSubdomains: true })); // Simple endpoint app.get('/', function(req, res) { res.send('Time to secure your application...'); }); // Start the server app.listen(3000); Here are the response headers. 1 Strict-Transport-Security: max-age=7776000; includeSubDomains You should be aware that this only works if your site is running over HTTPS. The middleware will check to see if req.secure is set to true which is automatically populate by Express. Hide X-Powered-By By default, Express will add the X-Powered-By header to each response. The actual response header will look like this: 1 X-Powered-By: Express While by itself it doesn’t cause any security holes, it does give potential attacker useful information. With this information they can focus their attack to exploit known vulnerabilities in Express and Node. Helmet provides middleware to either strip out the X-Powered-By header or set it to anything else you want. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 // Load required modules var express = require('express'); var helmet = require('helmet'); // Create our Express application var app = express(); // Hide X-Powered-By app.use(helmet.hidePoweredBy()); // Simple endpoint app.get('/', function(req, res) { res.send('Time to secure your application...'); }); // Start the server app.listen(3000); If you don’t want to remove the header, you can set it to anything else using. 1 app.use(helmet.hidePoweredBy({ setTo: 'all your base are belong to us' })); Which results in the following response header. 1 X-Powered-By: all your base are belong to us Other There are a few more things Helmet provides. The documentation on the GitHub repository does a good job of explaining them. If you want to learn more, I would suggest checking each of them out. IE, Restrict Untrusted HTML Don’t Infer The MIME Type Turn Off Caching (I would highly advise against this. Caching is very good for web performance.) Restrictive Crossdomain.xml Wrap Up Helmet is an amazing piece of Express middleware that can help you easily lock down your web application. I would suggest you take the time to better understand the types of attacks and fixes talked about so you can better secure your applications. I have a lot more tutorials coming so be sure to subscribe to my RSS feed or follow me on Twitter. Also, if there are certain topics you would like me to write on, feel free to leave comments and let me know. Beer Locker: Building a RESTful API with Node - Username & Password 2014-09-18T20:31:00-07:00 Welcome to part 6 of the Beer Locker series Getting started CRUD Passport OAuth2 Server Digest Username & Password In our previous article we explored implementing Digest authentication in place of Basic. This article will now delve into implementing username and password authentication using the passport-local module. There is a new set of code this article will start from rather than where we left off in the previous article. The previous article required us to remove our password hashing which we don’t have to do for username/password. You can find the starting point of code here on GitHub. The only difference here is that we added back password hashing and verification along with using Basic instead of Digest for our authentication. The Digest strategy implementation is still there but isn’t being used. Username and Password One of the most widely used authentication mechanism on the web is via a username and password submitted in a POST request or in the query string. This authentication mechansim can be used for APIs but it requires submitting the credentials on each call. You could implement session state for your API but it technically goes against REST being stateless. By session state, I mean making a direct login type call with the username and password and getting back some form of session id. This session id can then be transmitted on each API call to allow the server to verify the authenticated user. Update our Auth Controller To get started with the passport-local module (Local) we need to install it using npm. 1 npm install passport-local --save The next thing we need to do is update our Auth Controller. Open controllers/auth.js and implement the Local strategy as follows. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 // Load required packages var LocalStrategy = require('passport-local').Strategy; ... passport.use(new LocalStrategy( function(username, password, callback) { User.findOne({ username: username }, function (err, user) { if (err) { return callback(err); } // No user found with that username if (!user) { return callback(null, false); } // Make sure the password is correct user.verifyPassword(password, function(err, isMatch) { if (err) { return callback(err); } // Password did not match if (!isMatch) { return callback(null, false); } // Success return callback(null, user); }); }); } )); ... exports.isAuthenticated = passport.authenticate(['local', 'bearer'], { session : false }); We have done 3 main things here. First, we required the LocalStrategy provided by the passport-local module. Second, we told Passport to use LocalStrategy. If you are paying close attention, you will notice that the strategy implementation for Local is the same as Basic. This is because we are essentially working with usernames and passwords for both strategies. The main difference here is that the Basic strategy is pulling the username and password out of the Authorization header. Local allows us to submit the username and password via POST data or within the query string. Third, we updated isAuthenticated to use local instead of basic. Test it using query strings Yep, you heard me right. We are ready to test out this implementation. The first way we can test out our new authentication is through the use of query strings. You are welecome to test this using any of the endpoints in our API. I am going to use http://localhost:3000/api/beers/. Because we are using query strings, it is very simple to test this using a web browser. Here is what I am using to test: 1 http://localhost:3000/api/beers/?username=denver&password=rocks If you implemented everything correctly, you should get back a list of beer for the authentication user. Test it using POST data The next test we can do is using POST data rather than query strings. This is where things break down a little bit when you don’t have session state. In calls where we are POSTing, a username and password could be added. But we would be adding them along with other data we are submitting to in the POST. For exampe, when we are adding new beer to our locker, we would be posting beer name, type, and quantity. But we would also be posting a username and password. Also to note that this method only works for POST and PUT. We cannot submit data with GET or DELETE requests. Here is an example where I am adding a new beer to my locker using Postman. Customize our implementation As noted before, the Local Strategy is looking to find the credentials in the body or query string parameters named ‘username’ and ‘password’. But what if you want to name them something else? You are in luck because you can configure it to specify the name to use for both username and password. This is done by passing in an options object when defining the LocalStrategy. Open up controllers/auth.js and update it to look like this: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 ... passport.use(new LocalStrategy({ usernameField: 'email', passwordField: 'pass' }, function(username, password, callback) { User.findOne({ username: username }, function (err, user) { if (err) { return callback(err); } // No user found with that username if (!user) { return callback(null, false); } // Make sure the password is correct user.verifyPassword(password, function(err, isMatch) { if (err) { return callback(err); } // Password did not match if (!isMatch) { return callback(null, false); } // Success return callback(null, user); }); }); } )); ... What this does is tell the LocalStrategy to look for the named parameter ‘email’ for the username and ‘pass’ for the password. Wrap up You now have the tools needed to implement username and password authentication. As stated before, this type of authentication may not make the most sense for a RESTful API. Of the two, query string makes more sense but it is still risky because you are sending the username and password within the URL. Even over an HTTPS connection, it makes it potentially more visible to someone ‘watching over your shoulder’, browser history, etc. Hopefully this shows you how to use this strategy in other situations such as a web UI with a login form. I have a lot more tutorials coming so be sure to subscribe to my RSS feed or follow me on Twitter. Also, if there are certain topics you would like me to write on, feel free to leave comments and let me know. Source code for this part can be found here on GitHub. Beer Locker: Building a RESTful API with Node - Digest 2014-09-14T13:28:00-07:00 Welcome to part 5 of the Beer Locker series Getting started CRUD Passport OAuth2 Server Digest Username & Password In our previous article we ended wtih a functional API capable of creating user accounts, locking down API endpoints, only allowing access to a user’s own beer locker, and an OAuth2 server. Many readers have asked questions about how to use different authentication strategies so I am going to continue this series and delve into many of those strategies. This article will explore the use of Digest authentication instead of Basic. Digest Like the Basic scheme, Digest uses a username and password to authenticate a user. The benefit it provides over Basic is that it uses a challenge-response paradigm to avoid sending the password in the clear. Here is how it works: Client sends an unauthenticated request to a server. Server responds with a 401 “Unauthorized” reponse code along with a special code (called a nonce) and another string representing the authentication realm. Client responds with the nonce and an encrypted version of the username, password and realm. Server responds with a 200 OK and the response data if the authentication passes. Update our Routes There is something odd going on with the way in which we defined our routes in the original tutorials. In most cases it isn’t an issue but it is for Digest auth. When we tell Express to use our router, we were prefixing it with /api. The problem is that Digest auth uses the URI as part of the scheme and the request object within Express is seeing it without the /api portion of the URI. This will cause authentication to fail. You will need to update server.js routes like this: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 ... // Create endpoint handlers for /beers router.route('/api/beers') .post(authController.isAuthenticated, beerController.postBeers) .get(authController.isAuthenticated, beerController.getBeers); // Create endpoint handlers for /beers/:beer_id router.route('/api/beers/:beer_id') .get(authController.isAuthenticated, beerController.getBeer) .put(authController.isAuthenticated, beerController.putBeer) .delete(authController.isAuthenticated, beerController.deleteBeer); // Create endpoint handlers for /users router.route('/api/users') .post(userController.postUsers) .get(authController.isAuthenticated, userController.getUsers); // Create endpoint handlers for /clients router.route('/api/clients') .post(authController.isAuthenticated, clientController.postClients) .get(authController.isAuthenticated, clientController.getClients); // Create endpoint handlers for oauth2 authorize router.route('/api/oauth2/authorize') .get(authController.isAuthenticated, oauth2Controller.authorization) .post(authController.isAuthenticated, oauth2Controller.decision); // Create endpoint handlers for oauth2 token router.route('/api/oauth2/token') .post(authController.isClientAuthenticated, oauth2Controller.token); // Register all our routes app.use(router); ... Update our Auth Controller The first thing we need to do is update our Auth Controller. Open controllers/auth.js and implement the Digest strategy as follows. You can leave the Basic auth strategy implementation if you want. We will be updating the isAuthenticated to use Digest instead of Basic. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 // Load required packages var DigestStrategy = require('passport-http').DigestStrategy; ... passport.use(new DigestStrategy( { qop: 'auth' }, function(username, callback) { User.findOne({ username: username }, function (err, user) { if (err) { return callback(err); } // No user found with that username if (!user) { return callback(null, false); } // Success return callback(null, user, user.password); }); }, function(params, callback) { // validate nonces as necessary callback(null, true); } )); ... exports.isAuthenticated = passport.authenticate(['digest', 'bearer'], { session : false }); We have done 3 main things here. First, we required the DigestStrategy provided by the passport-http module. Second, we told Passport to use DigestStrategy. Inside this we told it to use qop: 'auth' which is quality of protection along with two anonymous functions. The first function does our verification of the user and if success we call the callback and supply the user along with their password. The second anonymous function can be used to protect against replay attacks. Nonces should be validated to make sure they are not used again. You can do this by storing issued nonces and removing them as they are used. This will increase the security of your authentication. Third, we updated isAuthenticated to use digest instead of basic. Remove hashing of passwords :( This is the part of using Digest and more specifically the implementation provided by the passport-http module that I like the least. You cannot store the password as a hash. You need to get to the original password in order for the authentication to work. At the very least you will want to encrypt the password. I highly advise you analyze this type of approach for your application. If you must use Digest, you may want to consider implementing your own Digest strategy or updating the existing one where you don’t pass in the password in the success callback. You could instead pass in the MD5 hash of the username:realm:password which you have pre calculated and stored. So on to the very unsafe update to our User model. Again, for the tutorial we are going to store the password in plain text. You NEVER want to do this in a real situation. You will want to look into cryptography modules such as crypto to do this. Open up models/user.js and update it to look like this: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 // Load required packages var mongoose = require('mongoose'); // Define our user schema var UserSchema = new mongoose.Schema({ username: { type: String, unique: true, required: true }, password: { type: String, required: true } }); // Export the Mongoose model module.exports = mongoose.model('User', UserSchema); Testing it out Fire up your trusted tool such as Postman and create a new user as the old ones will have hashed passwords which will not work. I found using Postman to test Digest a bit unfriendly so I opted instead for curl. Here is a command you can use to test your API using Digest: 1 curl -v --user smith:smith --digest http://127.0.0.1:3000/api/users Just change the username:password to whatever you used. This command is great in curl because it will issue the intial unauthenticated request, process the reponse along with the nonce, qop, realm, etc, and finally issue another authenticated request. Here is what my test looked like: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 curl -v --user smith:smith --digest http://127.0.0.1:3000/api/users * About to connect() to 127.0.0.1 port 3000 (#0) * Trying 127.0.0.1... * Adding handle: conn: 0x7ffb71804000 * Adding handle: send: 0 * Adding handle: recv: 0 * Curl_addHandleToPipeline: length: 1 * - Conn 0 (0x7ffb71804000) send_pipe: 1, recv_pipe: 0 * Connected to 127.0.0.1 (127.0.0.1) port 3000 (#0) * Server auth using Digest with user 'smith' > GET /api/users HTTP/1.1 > User-Agent: curl/7.30.0 > Host: 127.0.0.1:3000 > Accept: */* > < HTTP/1.1 401 Unauthorized < X-Powered-By: Express < WWW-Authenticate: Digest realm="Users", nonce="x9hR8HiV9szqnvzUU7DsS5ekfq4xNM2p", algorithm=MD5, qop="auth" < WWW-Authenticate: Bearer realm="Users" < Date: Sun, 14 Sep 2014 23:05:02 GMT < Connection: keep-alive < Transfer-Encoding: chunked < * Ignoring the response-body * Connection #0 to host 127.0.0.1 left intact * Issue another request to this URL: 'http://127.0.0.1:3000/api/users' * Found bundle for host 127.0.0.1: 0x7ffb71415150 * Re-using existing connection! (#0) with host 127.0.0.1 * Connected to 127.0.0.1 (127.0.0.1) port 3000 (#0) * Adding handle: conn: 0x7ffb71804000 * Adding handle: send: 0 * Adding handle: recv: 0 * Curl_addHandleToPipeline: length: 1 * - Conn 0 (0x7ffb71804000) send_pipe: 1, recv_pipe: 0 * Server auth using Digest with user 'smith' > GET /api/users HTTP/1.1 > Authorization: Digest username="smith", realm="Users", nonce="x9hR8HiV9szqnvzUU7DsS5ekfq4xNM2p", uri="/api/users", cnonce="ICAgICAgICAgICAgICAgICAgICAgIDE0MTEzMjk4NDQ=", nc=00000001, qop=auth, response="a9b8ab69c3a44c253a7834bdfe45b26d", algorithm="MD5" > User-Agent: curl/7.30.0 > Host: 127.0.0.1:3000 > Accept: */* > < HTTP/1.1 200 OK < X-Powered-By: Express < Content-Type: application/json < Content-Length: 354 < ETag: "-1301455500" < Date: Sun, 14 Sep 2014 23:05:02 GMT < Connection: keep-alive < * Connection #0 to host 127.0.0.1 left intact [...] Wrap up You now have the tools needed to implement Digest authentication. I still strongly caution against the use as it currently stands. I highly dislike and advise against storing passwords encrypted. It is only slightly better than plain text since it is encrypted, but it is only one step away from becoming plain text. The best is to implement your own Digest strategy so you can store your passwords in a way that they can be hashed. If you have thoughts on this, please share them in the comments section. I have a lot more tutorials coming so be sure to subscribe to my RSS feed or follow me on Twitter. Also, if there are certain topics you would like me to write on, feel free to leave comments and let me know. Source code for this part can be found here on GitHub. 4 Simple Steps To Secure Your Express Node Application 2014-09-04T21:35:00-07:00 While not an exhaustive article on securing web applications, the four steps shown here will get you pretty far along in building a secure web application. HTTPS One of the first steps you should take to secure your web application is to use HTTPS. For those of you that think it is too hard, too expensive, or too compute intensive, hopefully I can convince you otherwise. So why would we want to use HTTPS? The number one reason is to keep people and devices from viewing or modifying content being sent and received. There are so many hacks and exploits that can be done when not using a secure connection. It is foolhardy to not use HTTPS. So how would you get up and running with HTTPS without shelling out a lot of money? Let me introduce you to StartSSL. StartSSL offers free Class 1 certificates you can use to implement HTTPS on your site. While their UI is a bit clunky and cumbersome to work with, the fact that you can get a valid certificate is well worth it. Cookies Now that you have HTTPS setup and communication to your server is secure, we need to look at securing your cookies. Assuming your web application has some form of authentication, it is likely you are using cookies to maintain session state. HttpOnly HttpOnly is a flag that can be included in a Set-Cookie response header. The presence of this flag will tell browsers to not allow client side script access to the cookie (if the browser supports it). This is important because it helps protect your cookie data from malicious scripts and helps mitigate the most common XSS attacks. Here is how you can tell Express to set your cookie using the HttpOnly flag: 1 res.cookie('sessionid', '1', { httpOnly: true }); Secure Equally important as the HttpOnly flag is the Secure flag. This too is included in a Set-Cookie response header. The presence of the secure flag tells web browsers to only send this cookie in requests going to HTTPS endpoints. This is very important, as the cookie information will not be sent on an unencrypted channel. This helps mitigate some exploits where your browser is redirected to the HTTP endpoint for a site rather than the HTTPS endpoint and thus potentially exposing your cookies to someone in the middle of the traffic. Here is how you can tell Express to set your cookie using the Secure flag: 1 res.cookie('sessionid', '1', { secure: true }); Cross-site Request Forgery “Cross-Site Request Forgery (CSRF) is an attack that tricks the victim into loading a page that contains a malicious request. It is malicious in the sense that it inherits the identity and privileges of the victim to perform an undesired function on the victim’s behalf, like change the victim’s e-mail address, home address, or password, or purchase something. CSRF attacks generally target functions that cause a state change on the server but can also be used to access sensitive data.” - Open Web Application Security Project) To mitigate these types of attacks, you can use a secret, unique and random token that is embedded by the web application in all HTML forms and then verified on the server side when submitted. Here is how you can implement this protection using Express and Jade: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 var express = require('express'); var session = require('express-session'); var csrf = require('csurf'); var app = express(); app.use(session({ secret: 'My super session secret', cookie: { httpOnly: true, secure: true } })); app.use(csrf()); app.use(function(req, res, next) { res.locals._csrf = req.csrfToken(); next(); }); 1 2 3 4 5 6 7 form(action='/account/profile', method='POST') input(type='hidden', name='_csrf', value=_csrf) div label(for='username') Username input(type='text', name='username', id='username') div input(type='submit', value='Update Profile') This example is a little more detailed because we need to see how it all works with Express, Sessions, and our View. The csurf module requires either session middleware or cookie-parser to be initialized first. Learn more about the csurf module here and the express-session module here. Also worth mentioning (thanks to Evan Johnson for pointing this out) is that XHR requests need to include the CSRF token. Here is an example of how you could do that. 1 2 3 4 5 6 7 var csrf_token = '<%= token_value %>'; $("body").bind("ajaxSend", function(elm, xhr, s){ if (s.type == "POST") { xhr.setRequestHeader('X-CSRF-Token', csrf_token); } }); Wrap up By taking these first 4 steps, you can help secure your web application. Just using HTTPS alone is a great first step by itself. There are many more things you can and must do in order to fully secure your web applications. Think of this article as the first steps in your long journey to securing your web apps. Using secure cookies in Node on Azure 2014-08-22T12:36:00-07:00 In most cases, using secure cookies works just fine. As long as you are using a secure endpoint over HTTPS, the server application will send back cookies to the client with the secure flag set (assuming you have it configured to do so). When using systems like Heroku and Azure, the secure endpoint is terminated before your application and proxied to you. This means that your application is actually not running on a secure port or protocol. The way your application knows the connection could be considered secure is from the addition of the X-Protocol-Proto header from the proxy. In most cases this is just fine because web application frameworks like Express will watch for this header to know the application is running behind a reverse proxy. If you have configured Express to trust this proxy it will consider the request secure. Once it considers it secure, it will allow the setting of cookies that are marked to be secure. There is a problem with this on Azure that isn’t obvious at first. The Azure proxy does not add the X-Protocol-Proto header. Instead it adds x-arr-ssl. This means that even though the request could be considered secure, Express does not know that so it will not set any cookies in the response that are marked as secure. In order to get secure cookies to work in Azure you will need to trick Express. What you need to do is add the following anonymous function as middleware before you attempt to set any secure cookies. 1 2 3 4 5 6 7 8 9 10 11 12 //Tell Express that we're running behind a //reverse proxy that supplies https for you app.set('trust proxy', 1); //Add middleware that will trick Express //into thinking the request is secure app.use(function(req, res, next) { if(req.headers['x-arr-ssl'] && !req.headers['x-forwarded-proto']) { req.headers['x-forwarded-proto'] = 'https'; } return next(); }); What we have done is setup code to watch for the x-arr-ssl header and set the x-forwarded-proto header. Express will now see this header set as https and secure cookies will be sent to the client. Edit August 23rd, 2014: As mentioned in the comments by Ranjith, there is a setting called enableXFF in the web.config/iisde.yml that can be set to true to enable the addition of the X-Forwarded-Proto header. Beer Locker: Building a RESTful API with Node - OAuth2 Server 2014-07-02T08:23:00-07:00 Welcome to part 4 of the Beer Locker series Getting started CRUD Passport OAuth2 Server Digest Username & Password In our previous article we ended wtih a functional API capable of creating user accounts, locking down API endpoints, and only allowing access to a user’s own beer locker. In this part we will dive into creating an OAuth2 server and allowing access to API endpoints for the authorized user or authorized applications. We will do this by integrating OAuth2orize into our application. Security I realized I wasn’t explicitly clear about what steps ones should take in regards to security. This article was meant more on how to get an OAuth2 server up and running. When implementing an OAuth2 server you MUST make sure to secure your application. This means running all OAuth2 endpoints over HTTPS and hashing the client secret, authorization code, and access token. All three of those values should be treated the same way you would a password for a user account. If you are unsure about how best to secure your applications, you should seek out the assistance of someone who does. Application Client The first thing we need to do is add a new model, controller, and endpoints to allow us to create new application clients. An application client is what would request access to a user account. Perhaps something like a service that wants to help manage your beer collection to notify you when you are running low. Create a new file called client.js in the models directory and add the following code to it. 1 2 3 4 5 6 7 8 9 10 11 12 13 // Load required packages var mongoose = require('mongoose'); // Define our client schema var ClientSchema = new mongoose.Schema({ name: { type: String, unique: true, required: true }, id: { type: String, required: true }, secret: { type: String, required: true }, userId: { type: String, required: true } }); // Export the Mongoose model module.exports = mongoose.model('Client', ClientSchema); There isn’t too much going on here that differs from what we already did in previous articles. We have a name to help identify the application client. The id and secret are used as part of the OAuth2 flow and should always be kept secret. In this post we aren’t adding any encryption, but it would be a good practice to hash the secret at the very least. Finally we have a userId field to identify which user owns this application client. You could also consider auto generating the client id and secret in order to enforce uniqueness, randomness, and strength. The next thing we will add is the controller to facilitate adding and viewing application clients. Create a new file called client.js in the controllers directory and add the following code to it. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 // Load required packages var Client = require('../models/client'); // Create endpoint /api/client for POST exports.postClients = function(req, res) { // Create a new instance of the Client model var client = new Client(); // Set the client properties that came from the POST data client.name = req.body.name; client.id = req.body.id; client.secret = req.body.secret; client.userId = req.user._id; // Save the client and check for errors client.save(function(err) { if (err) res.send(err); res.json({ message: 'Client added to the locker!', data: client }); }); }; // Create endpoint /api/clients for GET exports.getClients = function(req, res) { // Use the Client model to find all clients Client.find({ userId: req.user._id }, function(err, clients) { if (err) res.send(err); res.json(clients); }); }; These two methods will allow us to create new application clients and get all existing ones for the authenticated user. Finally, in the server.js file we need to require the new controller and add some new routes for the two endpoints. The new route can be added just after the /users route. 1 2 3 4 5 6 7 8 var clientController = require('./controllers/client'); ... // Create endpoint handlers for /clients router.route('/clients') .post(authController.isAuthenticated, clientController.postClients) .get(authController.isAuthenticated, clientController.getClients); Using Postman, let’s go ahead and create a new application client. If for some reason you forgot your password for your user, you should make a new one by posting to the /users endpoint with username and password. Authenticate our application client We already created the ability to authenticate a user in our previous article using the BasicStrategy. We need to do the same here so we can lock down our token exchange endpoint which we will implement later. Update the controllers/auth.js file to require the Client model, add a new BasicStrategy to passport, and setup an export that can be used to verify the client is authenticated. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 var Client = require('../models/client'); ... passport.use('client-basic', new BasicStrategy( function(username, password, callback) { Client.findOne({ id: username }, function (err, client) { if (err) { return callback(err); } // No client found with that id or bad password if (!client || client.secret !== password) { return callback(null, false); } // Success return callback(null, client); }); } )); ... exports.isClientAuthenticated = passport.authenticate('client-basic', { session : false }); The one thing to note here is that when we call passport.use() we are not just supplying a BasicStrategy object. Instead we are also giving it the name client-basic. Without this, we would not be able to have two BasicStragies running at the same time. The actual implementation for our new BasicStrategy is to lookup a client using the supplied client id and verify the password is correct. Authorization Codes We need to create another model that will store our authorization codes. These are the codes generated in the first part of the OAuth2 flow. These codes are then used in later steps by getting exchanged for access tokens. Create a new file called code.js in the models directory and add the following code to it. 1 2 3 4 5 6 7 8 9 10 11 12 13 // Load required packages var mongoose = require('mongoose'); // Define our token schema var CodeSchema = new mongoose.Schema({ value: { type: String, required: true }, redirectUri: { type: String, required: true }, userId: { type: String, required: true }, clientId: { type: String, required: true } }); // Export the Mongoose model module.exports = mongoose.model('Code', CodeSchema); It is a pretty simple model with the value field used to store our authorization code. redirectUri is there to store the redirect uri supplied in the initial authorization process so we can add a bit more security later on to make sure the token exchange is legitimate. The userId and clientId fields are used to know what user and application client own this code. It is also worth noting, that to be extra secure, you should consider hashing the authorization code. Access Tokens Now we need to create the model that will store our access tokens. Access tokens are the final step in the OAuth2 process. With an access token, an application client is able to make a request on behalf of the user. We will implement the code a little later that creates and validates them. Create a new file called token.js in the models directory and add the following code to it. 1 2 3 4 5 6 7 8 9 10 11 12 // Load required packages var mongoose = require('mongoose'); // Define our token schema var TokenSchema = new mongoose.Schema({ value: { type: String, required: true }, userId: { type: String, required: true }, clientId: { type: String, required: true } }); // Export the Mongoose model module.exports = mongoose.model('Token', TokenSchema); The value field will be of the most interest here. It is the actual token value used when accessing the API on behalf of the user. The userId and clientId fields are used to know what user and application client own this token. Just like we did for user passwords, you should implement a strong hashing scheme for the access token. Never store them as plain text as we are in this example. Authentication using access tokens Earlier, we added a second BasicStrategy so we can authenticate requests from clients. Now we need to setup a BearerStategy which will allow us to authenticate requests made on behalf of users via an OAuth token. This is done via the Authorization: Bearer header. First we need to install another npm package that will provide us with the BearerStrategy for Passport. 1 npm install passport-http-bearer --save Update the controllers/auth.js file to require the passport-http-bearer package and Token model, add a new BearerStrategy to passport, and setup an export that can be used to verify the application client request is authenticated. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 var BearerStrategy = require('passport-http-bearer').Strategy var Token = require('../models/token'); ... passport.use(new BearerStrategy( function(accessToken, callback) { Token.findOne({value: accessToken }, function (err, token) { if (err) { return callback(err); } // No token found if (!token) { return callback(null, false); } User.findOne({ _id: token.userId }, function (err, user) { if (err) { return callback(err); } // No user found if (!user) { return callback(null, false); } // Simple example with no scope callback(null, user, { scope: '*' }); }); }); } )); ... exports.isBearerAuthenticated = passport.authenticate('bearer', { session: false }); This new strategy will allow us to accept requests from application clients using OAuth tokens and for us to validate those requests. Simple UI for granting application client access Up to this point in our series, we have not added any UI. We need to add a simple page with a form that will allow a user to grant or deny access to their account for any application client requesting access. There are a lot of template engines to pick from like jade, handlebars, ejs, and more.For this series, I went with ejs. First, we need to install the ejs npm package. 1 npm install ejs --save Next, we need to update our express application to tell it to use ejs as its view engine. Add the following require and app.set statements in server.js. 1 2 3 4 5 6 7 8 9 var ejs = require('ejs'); ... // Create our Express application var app = express(); // Set view engine to ejs app.set('view engine', 'ejs'); Finally, we need to create our view that will let the user grant or deny the application client access to their account. Create a new directory called views and add a file named dialog.ejs. Add the following code to the dialog.ejs file. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 </span>Beer Locker<span class="nt"> Hi <%= user.username %>! <%= client.name %> is requesting full access to your account. Do you approve? action="/api/oauth2/authorize" method="post"> name="transaction_id" type="hidden" value="<%= transactionID %>"> type="submit" value="Allow" id="allow"> type="submit" value="Deny" name="cancel" id="deny"> We will come back to this page later as we do a full walkthrough of how everything works. For now, we have this in place and can move on to the next piece. Enable sessions for our express application OAuth2orize requires session state for the express application in order to properly complete the authorization transaction. In order to do this, we need to install the express-session package. 1 npm install express-session --save Next we need to require the package and use it in our express application. Update server.js with the following code. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 var session = require('express-session'); ... // Set view engine to ejs app.set('view engine', 'ejs'); // Use the body-parser package in our application app.use(bodyParser.urlencoded({ extended: true })); // Use express session support since OAuth2orize requires it app.use(session({ secret: 'Super Secret Session Key', saveUninitialized: true, resave: true })); Create our OAuth2 controller We are finally ready to create our OAuth2 controller that will facilitate the OAuth2 flow. First, install the oauth2orize package. 1 npm install oauth2orize --save Next, create a new file called oauth2.js in the controllers directory. We will add the code to this file in steps. Load required packages 1 2 3 4 5 6 // Load required packages var oauth2orize = require('oauth2orize') var User = require('../models/user'); var Client = require('../models/client'); var Token = require('../models/token'); var Code = require('../models/code'); Create our OAuth2 server 1 2 // Create OAuth 2.0 server var server = oauth2orize.createServer(); Register serialization and deserialization functions 1 2 3 4 5 6 7 8 9 10 11 12 // Register serialialization function server.serializeClient(function(client, callback) { return callback(null, client._id); }); // Register deserialization function server.deserializeClient(function(id, callback) { Client.findOne({ _id: id }, function (err, client) { if (err) { return callback(err); } return callback(null, client); }); }); When a client redirects a user to user authorization endpoint, an authorization transaction is initiated. To complete the transaction, the user must authenticate and approve the authorization request. Because this may involve multiple HTTP request/response exchanges, the transaction is stored in the session. Register authorization code grant type 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 // Register authorization code grant type server.grant(oauth2orize.grant.code(function(client, redirectUri, user, ares, callback) { // Create a new authorization code var code = new Code({ value: uid(16), clientId: client._id, redirectUri: redirectUri, userId: user._id }); // Save the auth code and check for errors code.save(function(err) { if (err) { return callback(err); } callback(null, code.value); }); })); OAuth 2.0 specifies a framework that allows users to grant client applications limited access to their protected resources. It does this through a process of the user granting access, and the client exchanging the grant for an access token. We are registering here for an authorization code grant type. We create a new authorization code model for the user and application client. It is then stored in MongoDB so we can access it later when exchanging for an access token. Exchange authorization codes for access tokens 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 // Exchange authorization codes for access tokens server.exchange(oauth2orize.exchange.code(function(client, code, redirectUri, callback) { Code.findOne({ value: code }, function (err, authCode) { if (err) { return callback(err); } if (authCode === undefined) { return callback(null, false); } if (client._id.toString() !== authCode.clientId) { return callback(null, false); } if (redirectUri !== authCode.redirectUri) { return callback(null, false); } // Delete auth code now that it has been used authCode.remove(function (err) { if(err) { return callback(err); } // Create a new access token var token = new Token({ value: uid(256), clientId: authCode.clientId, userId: authCode.userId }); // Save the access token and check for errors token.save(function (err) { if (err) { return callback(err); } callback(null, token); }); }); }); })); What we are doing here is registering for the exchange of authorization codes for access tokens. We first look up to see if we have an authorization code for the one supplied. If we do we perform validation to make sure everything is as it should be. If all is well, we remove the existing authorization code so it cannot be used again and create a new access token. This token is tied to the application client and user. It is finally saved to MongoDB. User authorization endpoint 1 2 3 4 5 6 7 8 9 10 11 12 13 14 // User authorization endpoint exports.authorization = [ server.authorization(function(clientId, redirectUri, callback) { Client.findOne({ id: clientId }, function (err, client) { if (err) { return callback(err); } return callback(null, client, redirectUri); }); }), function(req, res){ res.render('dialog', { transactionID: req.oauth2.transactionID, user: req.user, client: req.oauth2.client }); } ] This endpoint, initializes a new authorization transaction. It finds the client requesting access to the user’s account and then renders the dialog ejs view we created eariler. User decision endpoint 1 2 3 4 // User decision endpoint exports.decision = [ server.decision() ] This endpoint is setup to handle when the user either grants or denies access to their account to the requesting application client. The server.decision() function handles the data submitted by the post and will call the server.grant() function we created earlier if the user granted access. Application client token exchange endpoint 1 2 3 4 5 // Application client token exchange endpoint exports.token = [ server.token(), server.errorHandler() ] This endpoint is setup to handle the request made by the application client after they have been granted an authorization code by the user. The server.token() function will initiate a call to the server.exchange() function we created earlier. Utility functions to generate unique identifiers 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 function uid (len) { var buf = [] , chars = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789' , charlen = chars.length; for (var i = 0; i < len; ++i) { buf.push(chars[getRandomInt(0, charlen - 1)]); } return buf.join(''); }; function getRandomInt(min, max) { return Math.floor(Math.random() * (max - min + 1)) + min; } Add routes to OAuth2 endpoints Now that we have the controller made for our OAuth2 endpoints, we need to update our express application to add the necessary routes to those endpoints. In server.js require the new oauth2 controller and add a few new routes. 1 2 3 4 5 6 7 8 9 10 11 12 var oauth2Controller = require('./controllers/oauth2'); ... // Create endpoint handlers for oauth2 authorize router.route('/oauth2/authorize') .get(authController.isAuthenticated, oauth2Controller.authorization) .post(authController.isAuthenticated, oauth2Controller.decision); // Create endpoint handlers for oauth2 token router.route('/oauth2/token') .post(authController.isClientAuthenticated, oauth2Controller.token); Access token authorization on API endpoints At this point we have everything in place for a fully functioal OAuth2 server. The last piece we need is to update our endpoints that require authorization. Currently, we are authorizing with the BasicStrategy which uses username/password. We need to update that to also allow it to use the BearerStrategy which will allow the use of the access token. Change the exports.isAuthenticated call in controllers/auth.js to use either basic or bearer strategies. 1 exports.isAuthenticated = passport.authenticate(['basic', 'bearer'], { session : false }); We are already using the isAuthenticated function on our endpoints so this change will allow authorization with usernamne/password and access tokens. Let’s use our OAuth2 server! That was a lot of code! Still far less than it would have been had we not used OAuth2orize. Now it is time to actually try it out. Open up your favorite web browser and browse to: http://localhost:3000/api/oauth2/authorize?client_id=this_is_my_id&response_type=code&redirect_uri=http://localhost:3000. If you used a different client id, then change it in the query string. Also, if you are running on a different port, be sure to change that in both places. When prompted, enter your username and password. You can test it out by clicking Deny if you want and should see it not continue the OAuth2 flow. Go ahead and click Allow to continue to the next step. So why did we get a 404? This is part of the tutorial where we are hacking things together a bit. Normally with OAuth2 you would have an endpoint in the application requesting access to a user’s account. That is the query string redirect_uri that we supplied. So when a user grants access, that URI is requested and passed the authorization code. This then allows the requesting application to exchange that code for an access token. To continue this tutorial, we will fake an application server using Postman. Go ahead and copy the authorization code from the query string code. Mine in this example would be S7VlbvRQW1aIC5X5. In Postman, we will want to POST to http://localhost:3000/api/oauth2/token, set the Basic Auth username and password to the client id and client secret for your application client, add set post data values code, grant_type, and redirect_uri. Code needs to be set the code you copied from the browser request. Grant_type needs to be set to authorization_code because that is the type we are using. Redirect_uri needs to be set to the same redirect_uri you used in the authorization code request. See that value field in the response access_token object? That is our access token which we can now use to make API requests on behalf of the user! Let’s test our access token by making a request to our API endpoints. All you have to do is make GET, POST, PUT, or DELETE requests to the API endpoints we made in earlier tutorials. The only difference is you don’t have to supply a username or password. Instead, you will add an Authorization header with the value set to Bearer Add beer to the user’s locker Get beer from the user’s locker Feel free to play around a bit. You should be able to alter the access token and find you are unauthorized. Switch back to username and password to verify the user still has access. Wrap up You now have a fully functional OAuth2 server done with just a little bit of work. OAuth2orize is an amazing library that makes building our server very straightfoward. I have a lot more tutorials coming so be sure to subscribe to my RSS feed or follow me on Twitter. Also, if there are certain topics you would like me to write on, feel free to leave comments and let me know. Source code for this part can be found here on GitHub.