Python templates and custom mapping types

24 Aug 2017

In the Python world, the Right Way to render a template is to pass it a mapping.

Python has exactly one built-in mapping type: good old dict.

This means there’s no consensus on the Right Way for templates to handle custom mapping types. Some downcast them to dict, some take them as they are. This post looks at which template libraries do which.

Why?

In the Python world, the Right Way to render a template is to pass it a mapping — or some keyword arguments that it can use to build a mapping.

Python has exactly one built-in mapping type: good old dict.

But what about mapping types other than dict? Duck typing says templates should accept any type of object, no matter how odd, so long as it supports __getitem__ and maybe a few other basic methods. But this isn’t what all of them do.

This becomes especially clear when you pass in a mapping object that can’t be converted to a dict — say, because the mapping it implements has a large or even infinite domain. In the examples below, infiniteMap maps strings containing any number of as to strings with the same number of bs.

              >>> import jinja2 
>>> from fsmcontainers import FsmMap
>>> infiniteMap = FsmMap({"a": "b"}).closure() 
>>> infiniteMap["a"] 

'b'

>>> infiniteMap["aaa"] 

'bbb'

>>> infiniteMap["a"*1000]

'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb'

            

Getting values from the mapping works fine. But passing it as an argument to Jinja’s rendering function doesn’t work.

              >>> jinja2.Template("").render(infiniteMap) 

Traceback (most recent call last): 
... pywrapfst.FstArgError: FST is not acyclic

The first thing jinja2.Template.render does with its arguments is call dict on them. If it’s passed an actual dict, or something that the dict constructor can be called on, then we have no problem. But if we pass it a mapping with an infinite number of keys — one that is meant to be evaluated lazily on one finitely-long key at a time — the eager dict constructor tries to iterate over every key and value in it, and that doesn’t end well. If FsmMap hadn’t thrown an exception, the result would have been an infinite loop.

Python 3’s built-in string.format_map, on the other hand, handles infinite mappings with ease:

              >>> "{a}".format_map(infiniteMap) 

'b'

>>> "{a} {aaaa} {aaaaaaa}".format_map(infiniteMap) 

'b bbbb bbbbbbb'

>>> ("{" + ("a" * 1000) + "}").format_map(infiniteMap)

'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb'

            

Why would you do that?

Let’s be clear. Doing this with an infinite mapping is a terrible idea.

Templates aren’t supposed to do much logic (or indeed, if you’re a purist, any logic at all). They’re really not supposed to handle weird Lovecraftian macro-language-in-disguise “mapping” objects that can expand unlimitedly large keys in arbitrary ways, even if those objects perversely insist on offering a dict-like interface. They’re just meant to accept a small — and definitely finite — collection of key–value pairs, and a normal finite dict is a convenient way to pass one of those around.

But there are other, less-Lovecraftian cases where using a template might make sense, but where downcasting its context to dict is still undesirable.

In my last post I fantasized about a Fancy Linguistic Resource that would let you write something like this :

              jinja2.Template(
  "The word  is pronounced ."
).render(resource)

            

and get back a string like this:

"The word <i>llegues</i> is pronounced /'je.ges/."

In this application, we don’t need resource to be an infinite mapping. But it’s likely to be a pretty big one, stored as a trie or a finite state transducer or some other time- and space-efficient structure. Eagerly iterating over the whole thing to convert it into a dict, while possible, is still not a great use of resources.

A purist might say we should separate out logic and presentation and do something like this:

              word = str(resource['llegues'])
pronunciation = str(resource['llegues'].pron)
jinja2.Template(
  "The word <i></i> is pronounced "
).render(word=word, pronunciation=pronunciation)

            

But if all the words I want to look up can be found in a single resource — and if that resource is already efficiently stored, and we’re not working in a web app environment where we need everything to be massively parallelizable — then this separation of concerns is overkill.

A purist might also say that what I’m doing here really doesn’t call for a templating system at all — that really this is more like text macro expansion or string interpolation, and I should just use a system for those. But templating systems have a better ecosystem: lots of available filters for useful tasks, good integration with Python text-processing tools like Sphinx, and so on. It sure would be nice if I didn’t have to reinvent those wheels.

I can imagine other situations where we might not want our template engine to convert a custom mapping object to a dict:

The mapping’s __getitem__ method has side effects. Maybe it logs the fact that a lookup happened, for instance.
The mapping has a special behavior for missing keys, and that behavior will be lost if we turn it into a vanilla dict.

What works

In these examples, finiteMap is an ordinary dictionary mapping a to b, and infiniteMap is a custom mapping object mapping any number of as to any number of bs. The expected result is for templates containing the placeholder a to render it to 'b' whether they are given finiteMap or infiniteMap. These functions do the right thing:

              from fsmcontainers import FsmMap 
finiteMap = {"a": "b"} 
infiniteMap = FsmMap({"a": "b"}).closure()

            

string.format_map:

              >>> '{a}'.format_map(finiteMap) 

'b'

>>> '{a}'.format_map(infiniteMap) 

'b'

string.Template.substitute:

              >>> import string 
>>> string.Template("$a").substitute(finiteMap) 

'b'

>>> string.Template("$a").substitute(infiniteMap) 

'b'

pymustache.render:

              >>> import pymustache 
>>> pymustache.render("", finiteMap) 

'b'

>>> pymustache.render("", infiniteMap) 

'b'

airspeed.Template.merge:

              >>> import airspeed 
>>> airspeed.Template("$a").merge(finiteMap) 

'b'

>>> airspeed.Template("$a").merge(infiniteMap) 

'b'

Cheetah.Template

              >>> from Cheetah.Template import Template 
>>> str(Template("$a", finiteMap)) 

'b'

>>> str(Template("$a", infiniteMap)) 

'b'

What doesn’t

Internal iteration over mapping argument

jinja2.Template.render:

              >>> import jinja2 
>>> jinja2.Template("").render(finiteMap) 

'b'

>>> jinja2.Template("").render(infiniteMap) 

Traceback (most recent call last): 
... pywrapfst.FstArgError: FST is not acyclic

Silent failure with empty result

ashes.AshesEnv.render:

              >>> from ashes import AshesEnv 
>>> env = AshesEnv()
>>> env.register_source("test", "{a}") 

<Template name='test'> 

>>> env.render("test", finiteMap) 

'b'

>>> env.render("test", infiniteMap) 

''

pystache.render:

              >>> import pystache 
>>> pystache.render("", finiteMap) 

'b'

>>> pystache.render("", infiniteMap) 

''

Forced unpacking

string.format:

              >>> '{a}'.format(**finiteMap) 

'b'

>>> '{a}'.format(**infiniteMap) 

Traceback (most recent call last): 
... pywrapfst.FstArgError: FST is not acyclic

>>> '{a}'.format(finiteMap) 

Traceback (most recent call last):
... KeyError: 'a'

>>> '{a}'.format(infiniteMap) 

Traceback (most recent call last): 
... KeyError: 'a'

moody.render:

              >>> import moody 
>>> moody.render("", **finiteMap) 

'b'

>>> moody.render("", **infiniteMap) 

Traceback (most recent call last): 
... pywrapfst.FstArgError: FST is not acyclic

>>> moody.render("", finiteMap) 

Traceback (most recent call last): 
... TypeError: render() takes 1 positional argument 
but 2 were given

>>> moody.render("", infiniteMap) 

Traceback (most recent call last): 
... TypeError: render() takes 1 positional argument 
but 2 were given

            

mako.template.Template.render

              >>> from mako.template import Template 
>>> Template("${a}").render(**finiteMap) 

'b'

>>> Template("${a}").render(**infiniteMap) 
Traceback (most recent call last): 
... pywrapfst.FstArgError: FST is not acyclic

>>> Template("${a}").render(finiteMap) 

Traceback (most recent call last): 
... TypeError: render_body() takes 1 positional 
argument but 2 were given

>>> Template("${a}").render(infiniteMap) 

Traceback (most recent call last): 
... TypeError: render_body() takes 1 positional 
argument but 2 were given

            

mako.template.Template.render_context:

              >>> import sys 
>>> from mako.runtime import Context
>>> Template("${a}").render_context(
        Context(sys.stdout, **finiteMap)) 

b

>>> import sys 
>>> from mako.runtime import Context
>>> Template("${a}").render_context(
        Context(sys.stdout, **infiniteMap)) 

Traceback (most recent call last): ...
pywrapfst.FstArgError: FST is not acyclic

>>> import sys 
>>> from mako.runtime import Context
>>> Template("${a}").render_context(
        Context(sys.stdout, finiteMap)) 

Traceback (most recent call last): ... 
TypeError: __init__() takes 2 positional 
arguments but 3 were given

>>> import sys 
>>> from mako.runtime import Context
>>> Template("${a}").render_context(
        Context(sys.stdout, infiniteMap)) 

Traceback (most recent call last): ... 
TypeError: __init__() takes 2 positional 
arguments but 3 were given

            

« Can linguists and doc writers share tools?Finite state transducers and The Two Kinds of Suck »