How to specify the firefox binary for Selenium Webdriver under node.js

My test suite was failing because selenium-webdriver 2.41.0 doesn’t support sendKeys for <input type='number'> with Firefox 29.  But downgrading my system Firefox to an older version just to run a test suite seems crazy.  Why not just download an older version of Firefox, and set the firefox binary to use, as you would with selenium webdriver in python?

Alas, the webdriver-js docs give no hint as to how this might be done.  After digging into the source code for selenium-webdriver remote and the documentation for the Firefox remote webdriver, here’s how it’s done:

var SeleniumServer = require("selenium-webdriver/remote"),
webdriver = require("selenium-webdriver");

var seleniumPath = "/path/to/selenium-server-standalone.jar";
var firefoxPath = "/path/to/firefox-28/firefox";
var server = new SeleniumServer(seleniumPath, {
port: 4444,
jvmArgs: ["-Dwebdriver.firefox.bin=" + firefoxPath]
});
server.start().then(function() {
var browser = new webdriver.Builder().usingServer(
server.address()
).withCapabilities(
webdriver.Capabilities.firefox()
).build();
// browser is now tied to firefox-28!
});

As a bonus, while testing this, I also discovered that you can enable verbose logging for the standalone webdriver by adding the option stdio: "inherit" to the constructor for SeleniumServer, e.g.:

var server = new SeleniumServer(seleniumPath, {
port: 4444,
stdio: "inherit",
...
});

This ought to be helpful in debugging issues around getting the standalone server going.

Posted in Uncategorized | Leave a comment

How to get an email every time your node.js app logs an error

This was surprisingly hard to Google for, and docs left a bit to be desired.  So here’s how I got my production Node.js app to send me an email whenever it logs an error.

1. Use log4js. I picked this logging library because it’s got a vote of confidence from etherpad, is fairly light weight (few dependencies, small code base), and very flexible.

 npm install log4js

2. Install nodemailer. It’s not listed as a dependency, so it doesn’t automatically install with log4js, but it’s necessary to use the SMTP appender with log4js, which is how to send emails.

npm install nodemailer

3. Configure log4js. While it’s endlessly flexible with different ways to configure it, the method I prefer is to just have a logging.json file that I load in. log4js works with “appenders” — each appender is a destination for log messages to go, which could include the console, a file, email, or whatever else.  I set mine up to log to a console (which supervisord, under which I run node, appends to a file), and to send errors by email.  Here’s a logging config with comments — remove these comments to be valid JSON.

{
  "replaceConsole": true,     // Optional: if true, console.log 
                              // will be replaced by log4js
  "appenders": [{
    "type": "console"         // This appender just sends
                              // everything to the console.
  }, {
    "type": "logLevelFilter", // This is a recursive appender,
                              // filters log messages and
                              // sends them to its own appender.

    "level": "ERROR",         // Include only error logs.

    "appender": {             // the filter's appender, smtp
      "type": "smtp",
      "recipients": "you@example.com",
      "sender": "sender@serverhost.com",
      "sendInterval": 60,     // Batch log messages, and send via 
                              // this interval in seconds (0 sends 
                              // immediately, unbatched).
      "transport": "SMTP",
      "SMTP": {
        "host": "localhost",  // Other SMTP options here.
        "port": 25
      }
    }
  }]
}

Finally, you’re ready to go.  In your app, log errors like so:

log4js = require("log4js");
log4js.configure(__dirname + "/config/logging.json") // Path to the 
                                                     // config above
logger = log4js.getLogger();

logger.error("This will send you an email!");
logger.info("But this won't.");
Posted in Uncategorized | Tagged , , | 1 Comment

SSL, Apache, VirtualHosts, Django, and SuspiciousOperation’s

I recently upgraded to Django 1.4.5, which fixes security issues relating to malicious HTTP “Host” headers. Since my Django site does use the host header occasionally, I took the recommended step of adding an ALLOWED_HOSTS setting which whitelists the hosts that are allowed to access the site.

My server logs then started filling up with SuspiciousOperations, being triggered by none other than GoogleBot — hundreds of hits a day. I checked with Google’s webmaster tools, but there were no listed crawl errors for my domain, despite my hundreds of 500’s in the logs.

The first breakthrough came when I added an additional field to the Apache combo log directive, to see the host header:

Original:

LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined

Add host header:

LogFormat "%h %l %u %t %{Host}i \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined

Once I did this, I started seeing that the host names for the SuspiciousOperation’s were all valid names for other virtual hosts on this server.

When using VirtualHost’s, apache searches for a virtualhost with a ServerName that matches the request’s Host header. If it doesn’t find such a VirtualHost, it just uses the first defined VirtualHost. This can lead to weird results if your first-listed VirtualHost is a production site that usually goes by a different name. It seems the “best practice” is to define a catch-all 404 virtual host directive before all the others, so that it collects the bad host names. For example, this one returns a 404 for any URL:

    # /etc/apache2/sites-enabled/000-default
    <VirtualHost *:80>
        ServerName bogus # it doesn't matter what this is
        RewriteEngine on
        RewriteRule ^/.*$ [R=404]
    </VirtualHost>

However, this didn’t fix my problem. I manually checked all of these virtual hosts, and they all were resolving correctly and as expected — yet the SuspiciousOperations kept pouring in, with Host headers matching various of my virtualhost’s. Somehow Django was receiving requests from other virtualhosts.

Finally, I found the answer: SSL. The Django site I have is configured to use both HTTP and HTTPS, and these are separate worlds for apache. My catch-all virtualhost, and the definitions for all the other virtualhosts, only matched on port 80; but SSL requests come in at port 443. So GoogleBot was requesting SSL variants of the virtualhost names, and Apache was shunting these to the Django app, which was the only virtualhost configured for port 443. It’s interesting, but perhaps unsurprising, that GoogleBot ignores the certificate errors it would have gotten if it tried to validate the cert for those host names.

The trouble with VirtualHosts and apache and SSL, is that you can’t define two different VirtualHost sections for a single port, in the way that you can with non-SSL, due to how Apache parses SSL requests. So you can’t just define a default 404 VirtualHost like you could on port 80. We need some way to limit which hosts are sent to Django to the ones that it should actually get, within a single VirtualHost.

The solution is to use mod_rewrite to check the HTTP_HOST header, and to explicitly send 404’s, redirects, or whatever else if an unconfigured virtualhost is used. Here’s what I ended up doing:

    # /etc/apache2/sites-enabled/my-ssl-django-site
    <VirtualHost *:443>
        RewriteEngine On

        # Send a 404 for anything that isn't www.example.com or example.com.
        RewriteCond %{HTTP_HOST} !^(www\.)?example\.com$
        RewriteRule /.* [R=404]

        # Optional -- canonicalize the URL by redirecting "www" variants to
        # non-www. You could also do the reverse. If you skip the 404 RewriteRule
        # above, this would also redirect other virtualhost's to your functioning
        # SSL virtualhost if you wanted.
        RewriteCond %{HTTP_HOST} !^example\.com$
        RewriteRule /.* https://example.com/ [R]

        SSLEngine on
        ...
    </VirtualHost>

This worked great. The SuspiciousOperation’s went away, and if I boldly visit the non-SSL virtualhost’s with https, ignoring the certificate errors, I get a reasonable error response.

Posted in apache, django | Leave a comment

Mocking Persona (browserid) in node using zombie.js

I wanted to be able to run headless integration tests in a nodejs app that require users to log in.  I’m using Mozilla Persona for authentication, which means that a full login and authorization workflow is a little complex:

flow

Given the heavy reliance on client-side javascript to request the assertion, request verification, and then respond to the application server’s authorization, it’s hard to just declare that “Jane is logged in” at the start of a test.  There’s a lot of state in there that needs to be set up.  We could write tests to use live persona servers (either using Mozilla’s, or downloading and running your own for offline testing), but this is heavy-weight and slow.  It can take several seconds for the whole chain to complete, and for headless integration tests that we want to incorporate into our development toolchains, it’s a bit much.

A better approach for my use case is to mock the persona requests and responses, and to stub them out of the flow.  The client side code still makes the same calls – firing navigator.id.request and naviator.id.watch – but we fake the requests and responses. I did this using ZombieJS, a headless browser emulator for Node.

1. Mock include.js

The first step is to mock the Mozilla Persona shim, include.js, which sets up navigator.id. We can replace it with an extremely simple shim that cuts out all the actual network traffic:

// include.js replacement:
var handlers = {};
navigator.id = {
    watch: function(obj) { handlers = obj; },
    request: function() { handlers.onlogin("faux-assertion"); },
    logout: function() { handlers.onlogout(); }
};

To get this stub to replace Mozilla’s include.js, we use the undocumented but very useful browser.resources.mock function in ZombieJS:

// Set up Zombie browser for tests
var Browser = require("zombie");
var browser = new Browser();
browser.resources.mock("https://login.persona.org/include.js", {
    statusCode: 200,
    headers: { "Content-Type": "text/javascript" },
    body: "var handlers = {};" +
      "navigator.id = { " +
      "  request: function() { handlers.onlogin("faux-assertion"); }," +
      "  watch: function(obj) { handlers = obj; }," +
      "  logout: function() { handlers.onlogout(); }" +
      "};"
});

This will short-circuit any attempt for a page to load https://login.persona.org/include.js, and will deliver the response we’ve provided instead.

2. Mock the server verification

I use my own node-browserid-consumer for my verification function, but the same principle should apply to whatever you’re using. The goal: have the test runner swap out the verification logic with something that just returns the address you expect.

var browserid = require("browserid-consumer");
browserid.verify = function(assertion, callback) {
    callback(null, {
        status: "okay",
        email: "test@mock",
        audience: "http://localhost:9000",
        expires: new Date().getTime() + 60*60*1000,
        issuer: "mock-stub"
    });
}

Replace test@mock with whatever email address you want to authenticate. And with that, we should be ready to go.

All together now

Now we’re ready to sign someone in rapidly in a test:

var Browser = require("zombie");
// Set up Zombie browser for tests
var browser = new Browser();
browser.resources.mock("https://login.persona.org/include.js", {
    statusCode: 200,
    headers: { "Content-Type": "text/javascript" },
    body: "var handlers = {};" +
      "navigator.id = { " +
      "  request: function() { handlers.onlogin("faux-assertion"); }," +
      "  watch: function(obj) { handlers = obj; }," +
      "  logout: function() { handlers.onlogout(); }" +
      "};"
});
var browserid = require("browserid-consumer");
browserid.verify = function(assertion, callback) {
    callback(null, {
        status: "okay",
        email: "test@mockmyid.com",
        audience: "http://localhost:9000",
        expires: new Date().getTime() + 60*60*1000,
        issuer: "mock-stub"
    });
}

// Now log the user in.
browser.visit("http://localhost:9000/");
browser.evaluate("$('signin').click();");

// .. a few milliseconds later.. you're authenticated!

To wait for auth to complete, I use a simple “await” function that just polls the browser to see if the user it’s done:

function awaitLogin(browser, callback) {
    // Replace this conditional with whatever you need to do to
    // see that a user is logged in.
    if (browser.evaluate("window.user")) {
       callback();
       return;
    }
    setTimeout(function() { awaitLogin(browser, callback) }, 100);
}

// Wait for login, then continue testing:
awaitLogin(browser, function() {
    // more tests here, with browser now logged in as 
    // test@mockmyid.com
});
Posted in Uncategorized | Leave a comment

Making Django ORM more DRY with prefixes and Q’s

This post builds on Jamie Mathews’ excellent Building a higher-level query API: the right way to use Django’s ORM, which makes the solid argument that the “Manager API is a Lie”.  If you haven’t read that post, head over that way to hear why it’s a fantastic idea to build high level interfaces to your models, and because of limitations in Django’s manager API, to do so in QuerySet’s rather than Managers.

This post tackles the problem: how do we get a high level interface across relationships?

Suppose we have the following models:

from django.db import models
class Membership(models.Model):
    user = models.ForeignKey(User)
    organization = models.ForeignKey(Organization)
    start = models.DateTimeField()
    end = models.DateTimeField(null=True)

class Organization(models.Model):
    name = models.CharField(max_length=25)

The high-level notion we’d like to get at with these models is whether a membership is “current”.  Suppose the definition of a “current membership” is:

A membership is current if “now” is after the start date, and either the end date is null (a perpetual membership), or “now” is before the end date (not yet expired).

This is Django ORM 101 here: in order to get that “or” logic, we either need to combine two querysets with `|`, or to do the same with “Q” objects.  For reasons that will become obvious below, I’ll go the Q route.  As in Jamie Mathews’ post, we’ll use the PassThroughManager from django-model-utils to get our logic into both the QuerySet’s and the Manager.

from django.db import models
from django.db.models import Q
from datetime import datetime
from model_utils.managers import PassThroughManager

class MembershipQuerySet(models.query.QuerySet):
    def current(self):
        now = datetime.now()
        return self.filter(start__lte=now, 
            Q(end__isnull=True) | Q(end__gte=now))

class Membership(models.Model):
    user = models.ForeignKey(User)
    group = models.ForeignKey(Organization)
    start = models.DateTimeField()
    end = models.DateTimeField(null=True)

    objects = PassThroughManager.for_queryset_class(MembershipQuerySet)()

class Organization(models.Model):
    name = models.CharField(max_length=25)

This works well – we can now get current memberships with the high-level, conceptually friendly:

 >>> Membership.objects.current()

But suppose we want to retrieve all the Organizations which have current members?  Whether using a Manager class or QuerySet class to define our filtering logic, we’re stuck: the notion of “current” is baked into the QuerySet (or manager) of the original class.  If we come from a related class, we have to repeat the logic, prefixing all of the keys:

>>> Organization.objects.filter(membership__start__lte=now,
...     Q(membership__end__isnull=True) | Q(membership__end__gte=now))

This breaks DRY – if we ever need to change the logic for “current” (say, to add `dues_payed=True`), we have to find all the instances and fix it.  Bug magnet!

Prefixed Q Objects

Here’s one possible solution to this problem.  The idea is to build the logic for a query using a custom “Q” class, which dynamically prefixes its arguments:

from django.db import models
from django.db.models import Q

class PrefixedQ(Q):
    accessor = ""
    def __init__(self, **kwargs):
        # Prefix all the dictionary keys
        super(PrefixedQ, self).__init__(**dict(
            (self.prefix(k), v) for k,v in kwargs.items()
        ))

    def prefix(self, *args):
        return "__".join(a for a in (self.accessor,) + args if a)

class MembershipQuerySet(models.query.QuerySet):
    class MQ(PrefixedQ): # "membership" Q
        accessor = ""    # Use an empty accessor -- no prefix.

    def membership_current_q(self):
        now = datetime.now()
        return self.MQ(start__lte=now) & (
            self.MQ(end__isnull=True) | self.MQ(end__gte=now)
        )

    def current(self):
        return self.filter(self.membership_current_q())

Now that we’ve abstracted the definition of “current” into the prefixed-Q class, we can subclass this QuerySet, and override the prefix in our related class:

class OrganizationQuerySet(MembershipQuerySet):
    # override the superclass's MQ definition, to add our prefix:
    class MQ(PrefixedQ):
        accessor = "membership"

    def with_current_members(self):
        return self.filter(self.membership_current_q())

>>> Organization.objects.with_current_members()
# should do the right thing!

This trick will work with multiple different relations — you’ll just need to use a different “Q” subclass name for each relation, so they don’t conflict, and mix in super classes:

class OrganizationQuerySet(MembershipQuerySet, ExecutiveQuerySet):
    class MQ(PrefixedQ):
        accessor = "membership"
    class EQ(PrefixedQ):
        accessor = "executives"

While this definitely has the feel of slightly noodley trickery to it, I think it starts to get at the core of what we might want in a more robust future ORM for Django: the ability to define high-level logic to assign meaning to particular collections of fields, and to preserve that logic across relations and chains.

Does this meet a use case you have?

Posted in django | Leave a comment

Getting access to a phone’s camera from a web page

For a web application I’m developing, I wanted to be able to allow mobile users to seamlessly push a button on the web page, take a photo, and post that photo to the page. Since the W3C draft specification for media capture isn’t yet implemented by any mobile browser except perhaps some Opera versions, it’s necessary to use a native application to get access to the camera. But I really wanted the lightest weight possible application, as I’m targeting tablets, phones, and computers for this site. I didn’t want to end up having to develop a different version for each device.

Enter phone gap! Phone gap is a wrapper application framework that lets you code mobile applications in HTML and javascript. It exposes javascript APIs to hardware features like the camrea. But I didn’t want to have to recompile every time I made a change to the code — really, I just want a thin wrapper around the page to add the one missing feature (camera access). Could my phonegap app be as simple as this?

<!-- phonegap html file for app -->
<html>
  <head>
    <!-- phonegap shim -->
    <script type="text/javascript" charset="utf-8" src="cordova-1.6.0.js"></script>
  </head>
  <body>
    &lt;iframe src="http://myapp.com"></iframe>
  </body>
</html>

(the &lt; is a result of wordpress.com’s apparent inability to render a literal iframe tag, even when properly escaped.)

The problem is that the iframe and the phonegap app’s page run on different domains, and thus they can’t see each other. The inner iframe can’t trigger a camera event on the outer frame directly.

Stack Overflow commenters alluded vaguely that it might be possible to do this with cross-domain messaging. Several hours later, here’s how, in detail!

Cross-document messaging

The main difficulty with accessing the camera from within an iframe in a PhoneGap application is that the document inside the iframe (which contains your remote webpage) has a different origin from the local web page (which has the phone gap shim). Consequently, the remote page can’t access navigator.camera. Cross-document messaging makes it possible for them to communicate even so. Here’s a decent writeup on the topic.

Basically, the parent document can send messages to the iframe (if it’s listening) like this:

iframe.contentWindow.postMessage({data: "stuff"}, "http://iframepage.com")

Replace "iframepage.com" with the URL for the page the iframe is accessing. The iframe can talk to the parent document (the phonegap window which has access to the camera) like this:

iframe.parent.postMessage({stuff: "rad"}, "file://")

Yes, that’s right — the PhoneGap’s page identifies as "file://", with no other path.

Listening for messages is fairly straight-forward. In both the phonegap file and the remote webpage, listen for cross-document messages by attaching an event listener to window:

window.addEventListener("message", function(event) {
  // Check that the message is coming from an expected sender --
  // "file://" if you're in the iframe, or your remote URL for
  // the phonegap file.
  if (event.origin == url) {
    // do something with the message
  }
});

Putting it all together

So the plan is to have the phonegap page do nothing other than announce to the iframe that a camera is available, and respond if the camera is found. The only additional wrinkle is just delaying your events until both the phonegap page and the iframe have loaded.

Putting it all together, here is the entire phonegap html page:

<!DOCTYPE HTML>
<html>
<head>
<title>My app</title>
<script charset="utf-8" type="text/javascript" src="cordova-1.6.0.js"></script><script type="text/javascript">// <![CDATA[
  document.addEventListener("deviceready",function() {
    var iframe = document.getElementById('iframe');
    var url = "http://example.com";
    // Announce that we have a camera.
    iframe.addEventListener("load", function(event) {
      iframe.contentWindow.postMessage({
        cameraEnabled: navigator.camera != null && navigator.camera.getPicture != null
      }, url);
    }, false);
    // Listen for requests to use it.
    window.addEventListener("message", function(event) {
      if (event.origin == url) {
        if (event.data == "camera") {
          navigator.camera.getPicture(function(imageData) {
            iframe.contentWindow.postMessage({
              image: imageData
            }, url);
          }, function(message) {
            iframe.contentWindow.postMessage({
              error: message
            }, url);
          }, {
            quality: 50,
            destinationType: Camera.DestinationType.DATA_URL,
            targetWidth: 640,
            targetHeight: 640
          });
        }
      }
    }, false);
    iframe.src= url;
  }, false);
// ]]></script>
<style type='text/css'>
  body,html,iframe {
    margin: 0; padding: 0; border: none; width: 100%; height: 100%;
  }
</style>
</head>
<body>
  &lt;iframe src='' id='iframe'></iframe>
</body>
</html>

In the remote webpage, you can seamlessly handle devices that have cameras or that don’t:

var cameraAvailable = false;

window.addEventListener('message', function (event) {
  if (event.origin == "file://") {
    if (event.data.cameraEnabled) {
      cameraAvailable = true;
    } else if (event.data.image) {
      image = $("<img/>");
      $("#app").prepend(image)
      image.attr("src", "data:image/jpg;base64," + event.data.image);
    } else if (event.error) {
      alert("Error! " + event.error);
    }
  }
}, false);
// When you want to get a picture:
if (cameraAvailable) {
    window.parent.postMessage("camera", "file://");
}

And that’s it! Not too bad. I don’t know yet whether the resulting app is something the iPhone app store would tolerate, but it flies for Android.

(now if only it were as easy to post html code snippets into wordpress without their getting munged!)

Posted in Uncategorized | 14 Comments

The real danger of data consolidation

To all the apathetic defeatists who won’t delete their search data: your threat model is wrong.

Yes, it sucks that your personal details are already owned and exploited by major corporations. And I agree, aggressively protecting yourself against this may not have the best value trade-off.

But there’s another player who probably doesn’t (yet) own all of your data, but could in a moment: the government.  Unlike corporations whose only interest is to sell you products, governments are interested in prosecuting you based on any evidence they can find that you have done anything wrong.  The more your data is centralized in one location, the easier it is to sobpoena.  If google has your search history, your photos, your email, your calendar, your instant messenger conversations, your documents, and your social network, it’s an easy one-stop-shop for collecting all of your data.

If you do any activist work – whether fighting injustice, exposing wrongdoing, or supporting freedom of speech – you can bet that at some point in your career, you will have a file, and possibly some digital surveillance to go along with it. This isn’t tin-foil hattery; governments regularly investigate anything they view as a threat, and people trying to change the unjust status quo are threats.  The more fractured, fragmented, incomplete, and scattered your personal data is, the harder it will be for you to be persecuted for one of your three daily felonies.

Deleting your search history is a small start, it’s easy, and doesn’t hurt.   You might also consider getting your email out of google’s hands too – if it’s older than 6 months, it is considered “abandoned”, and can be read without a warrant.  You might also consider setting up multiple google accounts and using them for different purposes (docs, mailing lists, etc).

TL;DR: Don’t delete your data for the corporations: delete it for the government.

Posted in Uncategorized | Leave a comment