WhitherApps

There’s been a hiatus in our efforts to rewrite the BBC News app in HTML5 – no thanks to the odd tropical storm, a consuming day-job project, and kids who’ve decided that boomerang toucans take priority over their father’s Javascript hacking. So let’s get back to it.

We left things at the point where we had a static HTML skeleton that sort of looked like it could be the BBC News app if you squinted at it. Now it’s time to make it work.

To start with, we need to get our HTML file to be able to make calls to the BBC servers and retrieve JSON- and XML-based payloads. Immediately we run into a problem here, because browsers won’t make requests to different domains (or will, but probably won’t receive a successful response). I can’t upload my humble little HTML file onto the BBC servers, and of course I can’t get them to respond with JSONP or play the cross-origin preflight request game.

So I need to do a little server-side proxying of my own. It seems that Safari refuses to set the user-agent header in AJAX calls anyway – and if you recall from our earlier discovery, that’s important for this app. The proxy will be able to do this user-agent rewriting for us too.

Since I’m running whitherapps.com on an nginx server, running such a proxy is incredibly easy. I’ve simply added this to the server’s config file:

location /apps/bbc-news/proxy {
  resolver 8.8.8.8;
  proxy_pass $http_x_bbc_url;
  proxy_set_header User-Agent $http_x_bbc_ua;
}

…which basically means that any request to http://whitherapps.com/apps/bbc-news/proxy will be proxied on to the URL present in the X-BBC-URL header (as resolved by Google’s DNS server), and the user-agent will be rewritten to whatever is in the X-BBC-UA header. (Not using nginx for your own web server? Why not?)

In the Javascript for the app, I’d better provide a small AJAX handler that sets these headers appropriately and calls back to a provided success function (with the response text) or a failure function (with the XHR instance) as required. I also append a timestamp to the end of the URL, since I want to be sure that the browser doesn’t cache it, confused by my burying of the changing URL inside the headers.

var BBC_PROXY = "/apps/bbc-news/proxy";
var BBC_USER_AGENT = "BBC News 1.2.1 (iPad; iPhone OS 3.2; en_US)";
function bbcXhr(url, success, failure) {
  var xhr = new XMLHttpRequest();
  xhr.open("GET", BBC_PROXY + '?' + new Date().getTime(), true);
  xhr.onreadystatechange = function() {
    if (xhr.readyState == 4) {
      if (xhr.status == 200) {
        success(xhr.responseText);
      } else {
        failure(xhr);
      }
    }
  }
  xhr.setRequestHeader("X-BBC-URL", url);
  xhr.setRequestHeader("X-BBC-UA", BBC_USER_AGENT);
  xhr.send("");
}

(If I was the BBC and wanted to do this properly, I would simply need to alter this function and/or accept requests from other user-agents, and/or respond to cross-origin requests on these URLs… confusingly the app needs to access a range of different domains.)

So, with the app-to-server communication more or less in place, let’s start using it.

If you recall from our original investigation into the app traffic, the first request is to a bootstrap URL that returns some helpful pointers in a JSON file. We need to call that when the app is refreshed (or indeed when it first loads up):

var BBC_BOOTSTRAP = "http://www.live.bbc.co.uk/moira/feeds/ipad/news/en/v1";
function bbcRefresh() {
  bbcXhr(BBC_BOOTSTRAP, function(responseText) {
    var bootstrap = JSON.parse(responseText);
  }, function(xhr) {});
}

addEventListener("load", function() {
  bbcRefresh();
}, false);

While we are at it, let’s create a crude refresh button in the control bar at the top of the app. That should help with testing things:

[<a href='#' id='bbc_refresh'>Refresh</a>]

And in the page load event, add a handler for it:

document.getElementById('bbc_refresh').onclick = function() {
  bbcRefresh();
  return false;
};

Put those in our app file, and… yes, it works: I get the slab of JSON back, and it’s parsed into the bootstrap object by the JSON parser available in Safari – which, although I suppose I do trust the BBC, feels far safer than eval().

Now, there’s a bunch of interesting things in that structure, such as links to privacy URLs and the like, but for now, we’ll just iterate through the list of feeds. These will populate the sections of the UI.

I’ve noticed that the real BBC News app loads the news from the top six or so categories preemptively, but when you click on the other, slightly more obscure categories (like the one for that little-known place, “Europe”), it goes and fetches them on-demand. I presume this is the default=true property set on the major feeds and that the other feeds just don’t work if you’ve never opened them and then go offline.

Rather than taking the feed information and creating the UI elements directly, I’ve decided I’ll do it in two logical steps. First get the feeds and stash the data about them in a local data store, and then secondly construct the UI from the contents of the data store. This is primarily to cater for users who go offline during the life of the app, or who start it up in offline mode and want to see what was there last time.

In fact, now I think about it, the easiest way to do this might be to wrap the local storage functionality in and around the AJAX function. We’ll key data in the browser’s local datastore off the URL. If the device is online, the browser gets fresh data, otherwise it gets it from the local storage.

So here’s a slightly beefier AJAX function that makes this happen behind the scenes:

function bbcCachedXhr(url, success, failure) {
  var xhr = new XMLHttpRequest();
  xhr.open("GET", BBC_PROXY + '?' + new Date().getTime(), true);
  xhr.onreadystatechange = function() {
    if (xhr.readyState == 4) {
      if (xhr.status == 200) {
        localStorage.removeItem(url);
        localStorage.setItem(url, xhr.responseText)
        success(xhr.responseText);
      } else {
        responseText = localStorage.getItem(url);
        if (responseText != null) {
          success(responseText);
        } else {
          failure(xhr);
        }
      }
    }
  }
  xhr.setRequestHeader("X-BBC-URL", url);
  xhr.setRequestHeader("X-BBC-UA", BBC_USER_AGENT);
  xhr.send("");
}

It still calls either a success or failure function, but the former will be called even if the AJAX request is unsuccessful (for whatever reason) but the data was still in the local storage.

This approach works for the BBC app because there are a relatively small number of URLs in play: all the articles for a given section are in one single structure at a stable URL. If there had been a URL for each article, I might have been worried about the cache filling up after a few weeks… but this seems OK for now. This week’s 10 articles will simply overwrite all of last week’s, since they are keyed off the same URL.

I have an idea that this function will be very useful both here and in future WhitherApps project.

Right then. Let’s take the feeds (either just fetched, or rescued from local storage), fetch the articles and create the UI. In part 2, we set up some dummy category panels. Let’s remove them from the static HTML and simple leave their containers:

<nav id='bbc_category_selector'></nav>
<div id='bbc_article_selectors'></div>

So we need a function that creates all the pieces needed for a given feed category selector. There are two titles (one which appears in the portrait mode, and one in landscape), and the selector panel itself. This function creates all three – if they are not present – and returns the main selector panel.

function bbcEnsureSelector(title) {
  var id = title.replace(/[^a-zA-Z0-9_]/g, '_')
  if (document.getElementById(id+'-t1') == null) {
    var selectorT1 = bbcCreateNode('a', id+'-t1', title);
    document.getElementById('bbc_category_selector').appendChild(selectorT1);
  }
  if (document.getElementById(id+'-t2') == null) {
    var selectorT2 = bbcCreateNode('h2', id+'-t2', title);
    document.getElementById('bbc_article_selectors').appendChild(selectorT2);
  }
  var selector = document.getElementById(id)
  if (selector == null) {
    selector = bbcCreateNode('nav', id);
    document.getElementById('bbc_article_selectors').appendChild(selector);
  }
  return selector;
}

function bbcCreateNode(tag, id, innerText) {
  var node = document.createElement(tag);
  if (id!=null) {
    node.id = id;
  }
  if (innerText!=null) {
    node.appendChild(document.createTextNode(innerText));
  }
  return node;
}

This may look a little clumsy, but it does include making a safe unique ID (from the title) and using DOM methods to ensure the resulting markup structure is well formed. (I’m thinking that a few of these functions might get moved into a separate library at some point.)

Then, as just after we parse the bootstrap in the refresh function, let’s call this for each feed:

var feeds = bootstrap.feeds[0].feeds;
for (var f in feeds) {
  var feed = feeds[f];
  var selector = bbcEnsureSelector(feed.title);
}

Seems to work (“At last! A screenshot!”):

Now, each of these category selector panels need to have the article thumbnails placed in them. This should happen when the bootstrap is loaded (for those that have default=true in the bootstrap, it seems), and also when a selector is shown that hasn’t previously been populated. So we’ll do it in a separate function that can be called in either case.

function bbcPopulateSelector(selector, url) {
  bbcCachedXhr(url, function(responseText){
    var feed = new DOMParser().parseFromString(responseText, 'text/xml').documentElement;
    if (feed.nodeName=='feed') {
      var feedId = feed.getElementsByTagName('id')[0].textContent;
      var entries = feed.getElementsByTagName('entry');
      for (e=0; e < entries.length; e++) {
        entry = entries[e];
        var entryId = entry.getElementsByTagName('id')[0].textContent;

        var id = feedId + '-' + entryId;
        var thumbnail = bbcCreateNode('a', id+'-thumbnail');

        var thumbnailImage = bbcCreateNode('img', '');
        thumbnailImage.src = bbcRewriteMediaUrl(
          entry.getElementsByTagNameNS('*', 'thumbnail')[0].getAttribute('url'),
          entryId
        );
        thumbnail.appendChild(thumbnailImage);

        var thumbnailTitle = bbcCreateNode('h3', null,
          entry.getElementsByTagName('title')[0].textContent
        );
        thumbnail.appendChild(thumbnailTitle);

        var content = entry.getElementsByTagName('content')[0];
        var contentNodes = content.childNodes;
        var article = bbcCreateNode('div', id+'-article');
        for (var i=0; i < contentNodes.length; i++) {
          article.appendChild(document.importNode(contentNodes[i], true));
        }

        thumbnail.article = article;

        selector.appendChild(thumbnail);
      }
    }
  }, function(xhr){});
}

function bbcRewriteMediaUrl(url, base) {
  if (url.substr(0, 11)!='bbcimage://') {
    return url;
  }
  url = url.substr(11);
  url = decodeURIComponent(url);
  if (url.substr(0, base.length)!=base) {
    return url;
  }
  url = url.substr(base.length);
  url = url.replace('{device}', 'ipad');
  return 'http://' + url;
}

So what’s going on here? Firstly we call the feed URL (again, cached via localStorage if offline), and then parse the results. For each article entry in the feed, we get its ID, and create a thumbnail link that will sit in the article-selecting category tray. We use the thumbnail image to actually get a picture for it (using the bbcRewriteMediaUrl function to transform its rather strange syntax, as observed in part I), and add the title of the article beneath it. We have to use getElementsByTagNameNS to get the thumbnail image URL, due to the media RSS namespace on that element.

Then, embedded in the Atom is a slab of HTML under the <content> element for the article itself. We get its children out and lash them into a container held under the thumbnail’s DOM using the ‘article’ attribute. I’m not entirely sure if this is the most kosher way to do things, but it will do for now. Basically we want it in a place where we can switch it into the main reading panel whenever someone clicks on – or rather, touches – the thumbnail.

After the selector panel is known to be in place in the refresh sequence, let’s call this with URL of every default feed. We have to use ‘default’ as an array index when examining the feed object, rather than an object property, because it’s a reserved word in Javascript and Safari complains otherwise:

if (feed['default']) {
  bbcPopulateSelector(selector, feed.feed_url);
}

OK, so let’s run that up and see what happens. Hopefully, the bootstrap and then the default feeds will be fetched. We should then see a panel appear for each of the categories, and the images and titles of the articles within them. And…

Well, OK. It did what I claimed, more or less, but we obviously have a few cosmetic problems here. Our category trays are of a constrained height, and the thumbnails are not at all styled to fit them. But it shouldn’t take a lot of CSS to sort this out. Firstly, let’s sort out the tray overflow, and then the display layout of the thumbnails themselves. (We’ll deal with making scrolling look nice in due course – the iPad doesn’t show scroll bars for overflow:scroll, although you can use two fingers to move the trays left and right).

So let’s make the following small additions to the CSS:

#bbc_article_selectors nav {
  ...
   overflow-x:scroll; overflow-y:hidden;
   white-space:nowrap;
}
#bbc_article_selectors nav a {
  display:inline-block;
  width:115px; height:137px;
  margin:5px;
  white-space:normal;
  overflow:hidden;
  background:#000000;
  border:1px solid #ffffff;
}

What a difference that makes. Arnie looks a bit odd (which is surely setting somebody up for a punchline) for an as-yet-unknown reason… but most of the thumbnails now look respectable. We’ll sort out the typography soon enough, but even with the default <h3> styling we can still read most of the headlines.

Of course we should make these <a> thumbnails actually do something. After all, we have the text of the article hanging off an expando DOM property, so let’s put it to good use. We can bind an article display function onto its click event at the end of the bbcPopulateSelector function. It’s tempting to use an anonymous function here, but I think I’ll need this in other places eventually (since in the app you can swipe articles to change them too).

...
  thumbnail.onclick = bbcDisplayArticle;
...

function bbcDisplayArticle(event, thumbnail) {
  if (thumbnail==null) {
    thumbnail = this;
  }
  var articlePanel = document.getElementById('bbc_article');
  articlePanel.innerHTML = '';
  articlePanel.appendChild(thumbnail.article);
  return false;
}

Crude for now, but this latter function will be a good place to put in some swanky page sliding animation at some point.

So at least we can now pull up articles:

Lovely. Seems to be one final thing to do though: rewrite the images that appear in articles. It makes sense to do this once – probably at the time we are stealing the article out of the Atom DOM and putting it into the HTML one. Let’s just do a quick scan for <img> tags and rewrite their src attributes, just before the importNode function puts them into the DOM (this means that Safari won’t try to go and get them before we can get them into the right syntax).

var contentImages = content.getElementsByTagName('img');
for (var i=0; i < contentImages.length; i++) {
  contentImages[i].src = bbcRewriteMediaUrl(contentImages[i].src, entryId);
}

It turns out that some of the images are placeholders for BBC video files too: these are <a> tags with bbcvideo:// schemes and bandwidth metadata. Let’s look for those too, and enhance our media URL rewriter:

var contentVideos = content.getElementsByTagName('a');
for (var i=0; i < contentVideos.length; i++) {
  contentVideos[i].href = bbcRewriteMediaUrl(contentVideos[i].href, entryId);
}

if (url.substr(0, 11)!='bbcimage://' &&
    url.substr(0, 11)!='bbcvideo://') {
...
   url = url.replace('{bandwidth}', 'wifi');
...

There’s more to the video thing than meets the eye. If you try the app you’ll be confronted with a cryptic MP4 URL in a manifest of some sort, and so I’ll need to do a bit more hacking to get the videos showing. Something to do with British TV license payers or some such nonsense. But at least we now have images within the articles:

And, if you compare it with a screen shot of the real thing, you’ll see we’re actually not doing too badly:

That’s it for this time. But we have an app you can just about use. Please check it out on your iPads here (but don’t all go too crazy at the same time, since it’s using my server as the proxy!) and see what you think. Yes, I know the portrait version doesn’t work yet, and the scrolling is pretty poor… but hey, come on.

In the next episode, after all this heavy lifting, I think I deserve to play around with some CSS. Lots of lovely -webkit-gradient coming up (I think that’s what you were all waiting for, anyway).

Back soon.

• • Comments [23]
  • Categories [BBC News]