April 2009

On Design Minimalism

    ... or, How to implement "includes" on top of JSON Template

    ... or, Avoiding the Dinosaur Snaggletooth

After introducing JSON Template, the most common feature request was for some kind of include or macro system. The language isn't frozen yet by any means, but it's meant to have a minimal design, while still retaining power.

This article includes two detailed examples that show how JSON Template solves the problem of reuse, without any new features. I'll then use this as a general illustration of the principle of design minimalism.

Contents

Two Template Reuse Patterns

Notice that an "include" or "macro" system is actually a solution. I want know what problem the solution addresses. And then I can figure out how to solve these problems, without blindly copying what other template systems do.

The problem is reuse, or avoiding duplication between template files. This can be broken down into two more specific use cases:

  1. Reusing a template "inside" another template. For example, you might want to define an HTML "panel" for a user profile, and include that panel across multiple HTML pages.

    Nearly all template systems have an include feature that implements this. For example, in Django you use the {% include %} statement.

  2. Reusing the "outside" of a template, specifying this "shell" in another template. For example, most sites have a common page header, footer, or sidebar. You don't want to repeat this across pages.

    In Django, template inheritance or the {% block %} and {% extends %} tags are what you use to implement this. [1]

I should note that I'm not only avoiding new features -- I see problems with these "standard" solutions. First, include statements break encapsulation. You've broken up your templates into modules (files), but what about the data dictionary? A lot of web apps seem to have a lot of what are essentially global template variables stuffed into a "servlet base class".

Also, if inclusion is implemented by naive textual substitution, then we haven't moved beyond the C preprocessor and its well known problems.

I have less experience with the "inheritance" model. But I have dealt with inheritance in specialized languages (as well as in all the popular general purpose languages), and it inevitably suffers from the fragile base class problem. When you can override anything, there's no possibility of hiding data. A change to your "base class" requires you (in theory) to test every page on your site.

Anyway, for each of these two problems, I've implemented a small dynamic page that shows how to express the reuse pattern in JSON Template.

Sample program for reuse on the "outside"

The second case is easier, so we'll start with that: reusing_the_outside.py.

The idea is simple:

  1. Create the <body></body> section of the page, using a template expansion.
  2. Put that HTML in a JSON dictionary.
  3. Use another template expansion to generate the complete page (making sure to use the raw formatter to avoid double escaping).

This involves very little code: A template expansion involes just constructing a dictionary and then a single line of code for the API call.

In fact, this article itself is generated using this method. The "shell" has substitutions for the <title> and <body>, as in the example. It also has repeated sections for external JavaScript and CSS (e.g. for google-code-prettify), and a section for Google Analytics.

Sample program for reuse on the "inside"

The first case is shown in reusing_the_inside.py.

The idea here is a bit more subtle. We already know how to substitute variables with the {variable-name|formatter} syntax (if you don't, go back to Introducing JSON Template).

JSON Template's execution model involves a cursor, which is moved by the names and types of the sections in the template. {section foo} pushes the cursor into the JSON node called foo. {repeated section bar} runs the cursor over each element in the JSON node bar.

Usually you expand simple variables under the cursor, but there's no reason you can't expand entire JSON (sub) dictionaries. Just like a variable, the subdictionary is passed to the formatters you specify in the template.

We can define our own application-specific formatters. So in this way, we can substitute an entire user profile data dictionary for user profile HTML -- moreover, we can do it multiple times in the same page, with different profiles.

Read the example for details.

It may seem long, but keep in mind the core of what you have to do to implement this reuse pattern: 

def MoreFormatters(formatter_name):
  if formatter_name == 'user-profile':
    return USER_PROFILE_TEMPLATE.expand
  else:
    return None  # consult default formatters

def TemplateThatCanRenderProfiles(template_str):
  return jsontemplate.Template(template_str,
                               more_formatters=MoreFormatters,
                               default_formatter='html')

This is such a small amount of code that "folding it into the language" would make your system bigger rather than smaller (accounting for the size of JSON template itself).

If you don't like this extra level of indirection, you can also implement a straightforward include mechanism. Let's follow the convention that formatters starting with % load template files. 

def MoreFormatters(formatter_name):
  if formatter_name.startswith('%'):
    filename = formatter_name[1:]
    # Put your application-specific template-finding logic here
    # In a production app, you will want to cache the file
    # open/read and template compilation
    return jsontemplate.FromFile(open(filename)).expand
  else:
    return None  # consult default formatters

Now you can write panels like this: 

{owner|%user-profile.jsont}
{member|%user-profile.jsont}

... where user-profile.jsont is a template filename. Beautifully simple, right? JSON Template is meant to be a library and not a framework. It's small and extensible, in order to allow each person and each application to use it in the most appropriate way.

To get familiar with the more_formatters argument, it's a good exercise to modify the MoreFormatters definition above to cache the included template compilations.

Functional Programming

Notice that formatters are quite flexible. HTML escaping (which takes "<" to &gt;, etc.) is a formatter, and constructing an entire user profile is also a formatter. From the template system's point of view, there's no difference.

This works because formatters are simply pure functions which map JSON nodes to strings -- and TEMPLATES THEMSELVES are also pure functions which map JSON nodes to strings. This is a key point.

In other words, defining a Template is defining a function. In mathematical notation, you can write:

    'Hello {name}' ( {'name': 'World'} ) = 'Hello World'

Here we are applying the argument {'name': 'World'} to the function 'Hello {name}' (a template), and getting the string 'Hello World' as a result.

JSON Template is more or less a functional programming language (but don't tell anyone). I will probably expound more on this in a future article.

Now onto Design Minimalism

 Design philosophies can be characterized along this axis:
  1. At one end is what I'll call the dinosaur snaggletooth method of design.

This is named after a conversation I had with a former coworker about the good old days of Legos. You had a bunch of blocks, and you combined them in novel ways to build new things. Today you have these prepackaged sets instead, and you can make the cool Jurassic Forest. But to make it, you need the dinosaur snaggletooth Lego piece, and then the Mongol loincloth Lego piece, the Mandolin coke spoon, etc.

Using this method means coming up with a specific solution for each specific problem you encounter. Seeing the Perl 6 periodic table of operators gives me this sinking feeling. [2]

This method ends up being used a lot because it's easy to copy an existing design, and then add the new features you need for new problems. Copying certainly lets you get things done much faster. But the issue is that you haven't internalized why the copied design chose the solutions it did. You end up with a much larger solution than you would than when starting from first principles.

  1. At the other end is what I'll call problem-oriented rather than solution-oriented design. You figure out all the problems that need to be solved with the system, and try to solve them with a few orthogonal constructs which can be combined in novel ways.

I'm going to be immodest here and say that I'm surprised that the second ("inside") reuse pattern fell out of JSON Template. I certainly didn't design it with that intent -- only after pondering the include feature pretty hard did this occur to me. I had envisioned formatters as ways to escape variables, just as in Django (with filters) and google-ctemplate. But having a well-defined data model (of JSON), and a well-defined execution model (the cursor traversing a node tree, using a stack), seems to have paid off. Orthogonal designs are the ones that seem to admit unexpective and creative uses.

Another example of multiple problems addressed by a single solution

Here is the justification behind what might be seen as another unusual design choice.

Problems:

  • It should be able to generate any type of content -- and the template must be readable in all cases.
  • It should be easy to write ad hoc tools to process templates.
  • The implementation should be small, especially since it's meant to be implemented in multiple programming languages.

Solution that addresses all three problems:

    The grammar of JSON Template is context free, but it has customizable metacharacters ({}). [3]

This means that a template can be be tokenized by a regular expression. A single re.split() call splits the strings into alternating literal strings and template directives.

A consequence is that it's easy to write tools like (correct) search/replace tools, checks for correctness or style, the syntax highlighter, maybe template security tools (for escaping), etc.

The {.meta-left} and {.meta-right} constructs are admittedly awkard, but customizable metacharacters let you avoid them easily. Moreover, any characters will be awkward for generating some kind of content (e.g. <> for HTML, {} and [] for C, etc.). In particular, generating templates from other templates is not uncommon, so there you'll have ugliness no matter what fixed characters you choose (how does it look generating a Django template from another Django template?). So we sacrifice a little consistency and allow this small degree of customizability.

If I had used a parser library/framework, I could have used a more complicated grammar and kept implementation the small. But since JSON Template is meant to be implemented in many languages, and every language has a different native parser technology, I avoided this.

Conclusion

Don't create a dinosaur snaggletooth just to solve the problem you have now. Step back a bit, and see if you address multiple problems with a single solution. Copy solutions where appropriate, but also judiciously avoid bloat.

Feedback

Please send mail to the mailing list or comment on reddit.

[1] Both of these are described in the Django Book, under "The Include Template Tag" and "Template Inheritance".

[2] Language War Disclaimer: Perl had a lot of great and influential ideas. (OK, but really, does anyone want to learn, much less remember, all that?)

[3] google-ctemplate also has customizeable metacharacters, but the grammar isn't context-free because the metacharacters can be changed at any time in the template. Note: I'm not claiming this is a fantastic or original idea -- it's just a design choice, that's all.