JSON Web API Consumption

The web is packed full of web APIs just waiting to be consumed. Sites such as Meetup, GitHub, Twitter, LinkedIn, and Facebook to name a few all have developer APIs that can be used programmatically to consume the data contained within their site. By implementing their APIs in similar fashions using REST, JSON, and OAuth, the ease at which these APIs can be consumed is amazing.

For this post, I am going to discuss how to consume GitHub’s RESTful web API using web resources, NuGet packages, the .NET framework, and a few lines of C# code.

The first thing that should always be done is to seek out documentation for the web API. The current version for GitHub’s is v3 and can be found here. By looking at the documentation, you will notice that the first page discusses topics such as Schema, Client Errors, HTTP Verbs, Authentication, Pagination, Rate Limiting, and a few others. While all are equally important, we will be focusing on Schema and later on Authentication.

Schema

All API access is over HTTPS, and accessed from the api.github.com domain. All data is sent and received as JSON.

$ curl -i https://api.github.com

HTTP/1.1 302 Found
Server: nginx/1.0.12
Date: Mon, 20 Feb 2012 11:15:49 GMT
Content-Type: text/html;charset=utf-8
Connection: keep-alive
Status: 302 Found
X-RateLimit-Limit: 5000
ETag: "d41d8cd98f00b204e9800998ecf8427e"
Location: http://developer.github.com
X-RateLimit-Remaining: 4999
Content-Length: 0

Blank fields are included as null instead of being omitted.

All timestamps are returned in ISO 8601 format:

YYYY-MM-DDTHH:MM:SSZ

So what exactly does all this mean? The schema provides the information needed to know where and how to get the data along with how to process the data once you have it. In our case the schema defines that all API access is over HTTPS and can be accessed from the domain api.github.com. The data sent to and received from this API is JSON (JavaScript Object Notation). JSON is a very versatile data format that can be consumed easily both server and client side.

The rest of the information shown has an example request to the API using curl and discusses two important points on the data. The first states that blank fields are included as null. This is very important because it makes it even easier for discovering the data returned by the API without having to refer to the documentation. Any call to the API will return all possible fields even if they are null. The second item describes the format for timestamps so that they can be processed properly.

Now that we know where to access the API and the type of data to expect, we can make our first call. Fire up your preferred web browser and navigate to the following location https://api.github.com/users/scottksmith95/repos. If you are wondering where the path /users/scottksmith95/repos came from, it came from the GitHub API documentation for Repos found here. Here is a trimmed down version of what this call returns:

[
  {
    "description": "GitHub API access and OAuth connect support for the github.com API",
    "has_wiki": true,
    "svn_url": "https://github.com/scottksmith95/CSharp.GitHub",
    "open_issues": 0,
    "language": "C#",
    "watchers": 1,
    "fork": false,
    "homepage": "",
    "git_url": "git://github.com/scottksmith95/CSharp.GitHub.git",
    "clone_url": "https://github.com/scottksmith95/CSharp.GitHub.git",
    "pushed_at": "2012-04-27T20:54:27Z",
    "size": 3112,
    "private": false,
    "created_at": "2012-04-17T01:54:00Z",
    "html_url": "https://github.com/scottksmith95/CSharp.GitHub",
    "owner": {
      "avatar_url": "https://secure.gravatar.com/avatar/4400061d4ff1f220a9b5f060368cd869",
      "login": "scottksmith95",
      "url": "https://api.github.com/users/scottksmith95",
      "gravatar_id": "4400061d4ff1f220a9b5f060368cd869",
      "id": 1346512
    },
    "name": "CSharp.GitHub",
    "url": "https://api.github.com/repos/scottksmith95/CSharp.GitHub",
    "mirror_url": null,
    "has_downloads": true,
    "has_issues": true,
    "ssh_url": "git@github.com:scottksmith95/CSharp.GitHub.git",
    "updated_at": "2012-04-27T20:54:27Z",
    "id": 4047943,
    "forks": 1
  }
]

In the case of GitHub, the JSON returned is formatted very nicely. In many cases JSON will be returned without indentation or line breaks. If you run into this, there are many options available for formatting the JSON into a readable form. One such way is using JSON Formatter and Validator. This tool will not only pretty up the JSON, it will also validate it. A great tool to have in your toolbox.

It is all well and good to pull down JSON from an API using a web browser, but we are here to consume the data programmatically.

First we will create a class that will contain a static method for retrieving the JSON from the API as a string.

using System;
using System.Net;

public class ApiRequest
{
  readonly static WebClient WebClient = new WebClient();

  public static string GetJson(Uri uri)
  {
    return WebClient.DownloadString(uri);
  }
}

using System;

class Program
{
  const string GitHubPath = "https://api.github.com/users/scottksmith95/repos";

  static void Main(string[] args)
  {
    var gitHubUri = new Uri(GitHubPath);

    var json = ApiRequest.GetJson(gitHubUri);
  }
}

Now that we have the ability to make requests for JSON from web APIs, we need to do something useful with it.

The .NET framework has the System.Web.Script.Serialization namespace. This namespace contains the JavaScriptSerializer class that can be used to serialize and deserialize JSON. Make sure you add a reference to the assembly System.Web.Extensions. Here it is in action in the updated Program class.

using System;
using System.Web.Script.Serialization;

class Program
{
  const string GitHubPath = "https://api.github.com/users/scottksmith95/repos";

  static readonly JavaScriptSerializer Serializer = new JavaScriptSerializer();

  static void Main(string[] args)
  {
    var gitHubUri = new Uri(GitHubPath);

    var json = ApiRequest.GetJson(gitHubUri);

    var jsonObject = Serializer.Deserialize&lt;object&gt;(json);
  }
}

If you were to step through the execution of this code with a debugger, you would notice that jsonObject is an object array containing arrays of key value pairs that correspond to the JSON. This is definitely better than handling the JSON as a string, but we can do much better.

First off, we are going to switch to a different deserializer. The JavaScriptSerializer we were using will do the job, but there are much more robust and higher performance deserializers available. The one we are going to use is Json.Net. Json.Net is available as a Nuget package so we will add it to our project via the NuGet package manager console in Visual Studio 2010 using the command

PM> Install-Package Newtonsoft.Json

With the Json.Net package installed, we will alter our existing Program class to use it.

sing System;
using Newtonsoft.Json;

class Program
{
  const string GitHubPath = "https://api.github.com/users/scottksmith95/repos";

  static void Main(string[] args)
  {
    var gitHubUri = new Uri(GitHubPath);

    var json = ApiRequest.GetJson(gitHubUri);

    var jsonObject = JsonConvert.DeserializeObject&lt;object&gt;(json);
  }
}

So the code is now using Json.Net but it isn’t too different than before. We still have an object that has been created by deserializing the JSON. What we really need is a strongly typed object to store the data. For this we will create classes that will be used to store the JSON data. We could take the time to look at the JSON and manually create the classes. Instead, we are going to use a tool on the web that will do it for us. The tool is called json2csharp. We simply paste in the JSON we want classes for and it will generate them for us. Here are the classes it creates for us.

public class Owner
{
  public string avatar_url { get; set; }
  public string login { get; set; }
  public string url { get; set; }
  public string gravatar_id { get; set; }
  public int id { get; set; }
}

public class RootObject
{
  public string description { get; set; }
  public bool has_wiki { get; set; }
  public string svn_url { get; set; }
  public int open_issues { get; set; }
  public string language { get; set; }
  public int watchers { get; set; }
  public bool fork { get; set; }
  public string homepage { get; set; }
  public string git_url { get; set; }
  public string clone_url { get; set; }
  public string pushed_at { get; set; }
  public int size { get; set; }
  public bool @private { get; set; }
  public string created_at { get; set; }
  public string html_url { get; set; }
  public Owner owner { get; set; }
  public string name { get; set; }
  public string url { get; set; }
  public object mirror_url { get; set; }
  public bool has_downloads { get; set; }
  public bool has_issues { get; set; }
  public string ssh_url { get; set; }
  public string updated_at { get; set; }
  public int id { get; set; }
  public int forks { get; set; }
}

If you examine the generated classes you may notice that the mirror_url member is not strongly typed. It is of type object because the JSON we submitted to the class generator contained null for the data. Because of this, the generated class used object instead of string (which is the actual type). When you run into cases like this, you will want to make sure and change them to the appropriate type. For our example it is not an issue so we will leave it alone.

The second item of note is the name of the root class. The tool named it RootObject because it is the root level object. We know what this is so we will rename it to GitHubRepo.

Finally, now that we have classes to represent the JSON data, we will update the Program class to deserialize the JSON into objects of these types.

using System;
using System.Collections.Generic;
using Newtonsoft.Json;

class Program
{
  const string GitHubPath = "https://api.github.com/users/scottksmith95/repos";

  static void Main(string[] args)
  {
    var gitHubUri = new Uri(GitHubPath);

    var json = ApiRequest.GetJson(gitHubUri);

    var jsonObject = JsonConvert.DeserializeObject&lt;List&lt;GitHubRepo&gt;&gt;(json);
  }
}

public class Owner
{
  public string avatar_url { get; set; }
  public string login { get; set; }
  public string url { get; set; }
  public string gravatar_id { get; set; }
  public int id { get; set; }
}

public class GitHubRepo
{
  public string description { get; set; }
  public bool has_wiki { get; set; }
  public string svn_url { get; set; }
  public int open_issues { get; set; }
  public string language { get; set; }
  public int watchers { get; set; }
  public bool fork { get; set; }
  public string homepage { get; set; }
  public string git_url { get; set; }
  public string clone_url { get; set; }
  public string pushed_at { get; set; }
  public int size { get; set; }
  public bool @private { get; set; }
  public string created_at { get; set; }
  public string html_url { get; set; }
  public Owner owner { get; set; }
  public string name { get; set; }
  public string url { get; set; }
  public object mirror_url { get; set; }
  public bool has_downloads { get; set; }
  public bool has_issues { get; set; }
  public string ssh_url { get; set; }
  public string updated_at { get; set; }
  public int id { get; set; }
  public int forks { get; set; }
}

Now we are getting somewhere. We now have a List of GitHubRepo objects that we can use programmatically. There is still one things bothering me. The member names of the JSON objects are not the greatest and don’t conform to most naming guidelines. What I want is to have properly named members yet still have the deserializer code able to deserialize each member. Json.Net has many different attributes that can be applied, but the one we will use is JsonProperty. This property allows us to specify the name of the field to deserialize and to change the member name to anything we want. Here is an example.

public string avatar_url{ get; set; }

becomes

[JsonProperty("avatar_url")]
public string AvatarUrl { get; set; }

That about wraps up this walk through on how to consume a JSON web API programmatically.

The final source code can be found here on GitHub.

Scott Smith

Blog Tutorials Projects Speaking RSS

JSON Web API Consumption

Schema