Making Django ORM more DRY with prefixes and Q’s

This post builds on Jamie Mathews’ excellent Building a higher-level query API: the right way to use Django’s ORM, which makes the solid argument that the “Manager API is a Lie”.  If you haven’t read that post, head over that way to hear why it’s a fantastic idea to build high level interfaces to your models, and because of limitations in Django’s manager API, to do so in QuerySet’s rather than Managers.

This post tackles the problem: how do we get a high level interface across relationships?

Suppose we have the following models:

from django.db import models
class Membership(models.Model):
    user = models.ForeignKey(User)
    organization = models.ForeignKey(Organization)
    start = models.DateTimeField()
    end = models.DateTimeField(null=True)

class Organization(models.Model):
    name = models.CharField(max_length=25)

The high-level notion we’d like to get at with these models is whether a membership is “current”.  Suppose the definition of a “current membership” is:

A membership is current if “now” is after the start date, and either the end date is null (a perpetual membership), or “now” is before the end date (not yet expired).

This is Django ORM 101 here: in order to get that “or” logic, we either need to combine two querysets with `|`, or to do the same with “Q” objects.  For reasons that will become obvious below, I’ll go the Q route.  As in Jamie Mathews’ post, we’ll use the PassThroughManager from django-model-utils to get our logic into both the QuerySet’s and the Manager.

from django.db import models
from django.db.models import Q
from datetime import datetime
from model_utils.managers import PassThroughManager

class MembershipQuerySet(models.query.QuerySet):
    def current(self):
        now = datetime.now()
        return self.filter(start__lte=now, 
            Q(end__isnull=True) | Q(end__gte=now))

class Membership(models.Model):
    user = models.ForeignKey(User)
    group = models.ForeignKey(Organization)
    start = models.DateTimeField()
    end = models.DateTimeField(null=True)

    objects = PassThroughManager.for_queryset_class(MembershipQuerySet)()

class Organization(models.Model):
    name = models.CharField(max_length=25)

This works well – we can now get current memberships with the high-level, conceptually friendly:

 >>> Membership.objects.current()

But suppose we want to retrieve all the Organizations which have current members?  Whether using a Manager class or QuerySet class to define our filtering logic, we’re stuck: the notion of “current” is baked into the QuerySet (or manager) of the original class.  If we come from a related class, we have to repeat the logic, prefixing all of the keys:

>>> Organization.objects.filter(membership__start__lte=now,
...     Q(membership__end__isnull=True) | Q(membership__end__gte=now))

This breaks DRY – if we ever need to change the logic for “current” (say, to add `dues_payed=True`), we have to find all the instances and fix it.  Bug magnet!

Prefixed Q Objects

Here’s one possible solution to this problem.  The idea is to build the logic for a query using a custom “Q” class, which dynamically prefixes its arguments:

from django.db import models
from django.db.models import Q

class PrefixedQ(Q):
    accessor = ""
    def __init__(self, **kwargs):
        # Prefix all the dictionary keys
        super(PrefixedQ, self).__init__(**dict(
            (self.prefix(k), v) for k,v in kwargs.items()
        ))

    def prefix(self, *args):
        return "__".join(a for a in (self.accessor,) + args if a)

class MembershipQuerySet(models.query.QuerySet):
    class MQ(PrefixedQ): # "membership" Q
        accessor = ""    # Use an empty accessor -- no prefix.

    def membership_current_q(self):
        now = datetime.now()
        return self.MQ(start__lte=now) & (
            self.MQ(end__isnull=True) | self.MQ(end__gte=now)
        )

    def current(self):
        return self.filter(self.membership_current_q())

Now that we’ve abstracted the definition of “current” into the prefixed-Q class, we can subclass this QuerySet, and override the prefix in our related class:

class OrganizationQuerySet(MembershipQuerySet):
    # override the superclass's MQ definition, to add our prefix:
    class MQ(PrefixedQ):
        accessor = "membership"

    def with_current_members(self):
        return self.filter(self.membership_current_q())

>>> Organization.objects.with_current_members()
# should do the right thing!

This trick will work with multiple different relations — you’ll just need to use a different “Q” subclass name for each relation, so they don’t conflict, and mix in super classes:

class OrganizationQuerySet(MembershipQuerySet, ExecutiveQuerySet):
    class MQ(PrefixedQ):
        accessor = "membership"
    class EQ(PrefixedQ):
        accessor = "executives"

While this definitely has the feel of slightly noodley trickery to it, I think it starts to get at the core of what we might want in a more robust future ORM for Django: the ability to define high-level logic to assign meaning to particular collections of fields, and to preserve that logic across relations and chains.

Does this meet a use case you have?

This entry was posted in django. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s