Line # Revision Author
1 158 abw Introduction
2 ------------
3
4 This is a port to Python of the Perl Template Toolkit (TT): a fast, flexible
5 and extensible template processing system. The Python port is by Sean McAfee
6 and is based on the original Perl version by Andy Wardley. For more
7 information about the Template Toolkit, please see:
8
9 http://template-toolkit.org/
10
11 159 abw For up-to-date information relating to the Python port, please see:
12
13 http://template-toolkit.org/python/
14
15 157 sean This file documents many of the design differences between the Perl
16 and Python implementations of the Template Toolkit.
17
18 In no particular order:
19
20 Documentation
21 -------------
22
23 All source-code-level documentation from the Perl Template Toolkit has
24 been copied to the Python version, with all Perl-specific terminology
25 translated to its Python equivalent. Documentation appears in Python
26 docstrings, as appropriate. The main body of POD documentation for
27 each Perl module appears as the corresponding Python module's
28 docstring. For example:
29
30 $ python
31 >>> import template.stash
32 >>> help(template.stash)
33
34 This prints the main documentation for the Python version of the
35 Template::Stash module.
36
37
38 Grammar
39 -------
40
41 A casual search revealed no well-established Python parser module to
42 compare with Parse::Yapp, so I simply translated the generated grammar
43 module Template::Grammar by hand.
44
45 Late experimentation suggests that it may be possible to coerce a
46 Parse::Yapp source file to emit Python code, or at least code in some
47 format that is easily transformable to Python. This avenue should be
48 explored.
49
50
51 Template Input and Output
52 -------------------------
53
54 One can specify an input template to Perl's Template::process() either
55 as a string (to indicate a source file or block name) or a reference
56 to a string (which contains the input text). Perl's reference
57 semantics are awkward to emulate in Python, so an alternate scheme is
58 used. Rather than specifying template text via reference, the text is
59 wrapped in an instance of the template.util.Literal class:
60
61 from template import Template
62 from template.util import Literal
63
64 Template().process(Literal("[% x %]"), { "x": 42 })
65
66 Alternatively, one may call the processString method instead, which
67 simply wraps its first argument in a Literal and calls process().
68
69 Template().processString("[% x %]", { "x": 42 })
70
71 Unlike the Perl Toolkit, the Python version of process() does not
72 accept a third argument indicating a destination for the processed
73 template text; instead, the text is returned. The caller must take
74 responsibility for directing the text to the desired destination.
75 While the Perl version of process() returns false to indicate an error
76 condition, which must be fetched with the error() method, the Python
77 version simply raises an exception.
78
79 Perl:
80
81 use Template;
82
83 my $template = Template->new;
84 $template->process("someblock") or die $template->error;
85
86 Python:
87
88 from template import Template
89
90 template = Template()
91 print template.process("someblock")
92
93 Or just:
94
95 print Template().process("someblock")
96
97 You can, of course, catch a raised exception to take special action.
98
99 from template import Template, TemplateException
100
101 try:
102 print Template().process("someblock")
103 except TemplateException, e:
104 print >>sys.stderr, "Got exception: ", e
105
106 The OUTPUT and OUTPUT_PATH options are honored, but do not otherwise
107 change the behavior of process().
108
109
110
111 Error Reporting
112 ---------------
113
114 As it makes for more idiomatic Python, errors are reported by raising
115 an exception, rather than returning a status code and/or setting an
116 instance or module variable. Since the primary purpose of
117 Template::Base is to provide such an error-reporting structure, it has
118 been omitted from the Python implementation.
119
120 The (result, error-code) return-value scheme is a public interface of
121 the Template::Iterator module, so it has been retained by
122 template.iterator.
123
124 Provider-type interfaces (template.filters, template.plugins, etc.)
125 return None to indicate that the requested resource has been declined,
126 rather than returning STATUS_DECLINED.
127
128
129 Subroutines
130 -----------
131
132 In Python, it isn't so straightforward to put subroutines in a data
133 structure, as, for example, with $Template::Stash::SCALAR_OPS:
134
135 our $SCALAR_OPS = {
136 ...
137 'match' => sub { ... },
138 'search' => sub { ... },
139 ...
140 };
141
142 Python function definitions are classed as "statements," and may not
143 appear in an expression. (If the function body consists of only a
144 single expression, a lambda expression may be used, but this does not
145 suffice in many cases.) My original approach was simply to split the
146 definition and registration into two steps:
147
148 def scalar_match():
149 ...
150
151 def scalar_search():
152 ...
153
154 SCALAR_OPS = {
155 'match': scalar_match,
156 'search': scalar_search,
157 ...
158 }
159
160 This scheme suffers from unnecessarily distributed information; it
161 would be easy to add a new function but forget to update SCALAR_OPS.
162 I eventually hit upon the idea of using function decorators to perform
163 registration:
164
165 @scalar_op("match")
166 def scalar_match(...):
167 ...
168
169 @list_op("push")
170 def list_push(...):
171 ...
172
173 @hash_op("keys")
174 def hash_keys(...):
175 ...
176
177 This has the added advantage that the decorators may be employed by
178 users to add custom vmethods. For example:
179
180 from template.stash import scalar_op
181
182 @scalar_op("double")
183 def my_double(x):
184 return x * 2
185
186 As opposed to the slightly more obscure:
187
188 165 sean $Template::Stash::SCALAR_OP->{double} = sub { shift * 2 };
189 157 sean
190
191
192 Filters
193 -------
194
195 Dynamic filter factories in the Perl TT are indicated with a
196 two-element array reference, where the second element is true, eg:
197
198 sub password_filter_factory {
199 my $char = shift;
200 return sub {
201 160 sean return $char x length $_[0];
202 157 sean }
203 }
204
205 my $filters = Template::Filters->new({
206 FILTERS => {
207 password => [ \&password_filter_factory, 1 ]
208 }
209 });
210
211 In Python this is accomplished more simply by setting an attribute on
212 the function object:
213
214 def password_filter_factory(char):
215 def password_filter(str):
216 return char * len(str)
217 return password_filter
218
219 password_filter_factory.dynamic_filter = True
220
221 filters = template.filters.Filters({
222 'FILTERS': {
223 'password': password_filter_factory
224 }
225 })
226
227 Even easier, a function decorator is provided to set the attribute:
228
229 from template.filters import dynamic_filter
230
231 @dynamic_filter
232 def password_filter_factory(char):
233 def password_filter(str):
234 return char * len(str)
235 return password_filter
236
237
238 Scalar semantics
239 ----------------
240
241 Python variables have very different semantics from Perl scalars. The
242 most important differences from the perspective of generated code are
243 these:
244
245 * Python variables do not automatically convert to strings or numbers
246 as appropriate as Perl scalars do.
247
248 * Notions of booleanness differ. Empty Python lists or dictionaries
249 are considered false, while in Perl all references are true, even
250 references to empty arrays or hashes. The string "0" is true in
251 Python, but false in Perl.
252
253 Perl's scalar semantics are expressed in the Python class PerlScalar,
254 found in the template.util module, and available to generated code
255 under the name "scalar". A PerlScalar wraps a Python value and
256 provides Perl-like semantics via special methods like __add__,
257 __nonzero__, etc. PerlScalars are employed ubiquitously in generated
258 code. Constants are wrapped explicitly in a scalar; for example, this
259 template:
260
261 [% IF "0"; "yes"; ELSE; "no"; END %]
262
263 ...would be translated into this code:
264
265 if scalar("0"):
266 output.write("yes")
267 else:
268 output.write("no")
269
270 Note that without the scalar wrapper, a bare Python
271
272 if "0":
273
274 ...would take the opposite branch of the if statement.
275
276 Operations involving two PerlScalars result in another PerlScalar.
277 Arithmetic operations result in Perl-style conversion from string to
278 number, if necessary.
279
280 Python values are wrapped in a PerlScalar on being retrieved from a
281 Stash object, and are unwrapped on being stored in one.
282
283 See the PerlScalar documentation in template/util.py for more
284 exhaustive information and examples.
285
286
287 Classes
288 -------
289
290 Perl classes typically occur in one-to-one correspondence with module
291 source files, and so they can be uniquely identified by a package
292 identifier like "Template::Plugin::File". In Python the situation is
293 a little more complicated; there is no conventional relationship
294 between the path to a module and the primary class of interest it
295 exposes. Therefore, in situations where the user may identify a class
296 (template.config, template.plugins), the class may be identified
297 unambiguously using a two-element tuple: the module name, followed by
298 the class name within that module. For example, from
299 template.plugins:
300
301 STD_PLUGINS = {
302 "datafile": ("template.plugin.datafile", "Datafile"),
303 "date": ("template.plugin.date", "Date"),
304 ...
305 }
306
307 A user-supplied class should be identified by using this scheme, or
308 one of two alternates. First, the class object may be given directly:
309
310 class MyFilters:
311 ...
312
313 template.config.Config.FILTERS = MyFilters
314
315 class MyCustomPlugin:
316 ...
317
318 tt = Template({ "PLUGINS": { "custom": MyCustomPlugin } })
319
320 This of course requires that the class be loaded prior to processing
321 the template. The second alternate is to supply a module name as a
322 plain string. A simple heuristic is applied to guess a class name:
323 the last component of the module name is capitalized.
324
325 tt = Template({ "PLUGINS": { "custom": "my.org.plugins.custom" } })
326
327 Here, the name of the plugin class is assumed to be "Custom".
328
329 This last way of identifying classes should typically be avoided, as
330 less precise than the other options.
331
332
333 Arrays and Hashes
334 -----------------
335
336 Perl's arrays and hashes are fundamental data types, and the type that
337 a reference points to can be inferred at compile time from the way
338 it's used (eg. $ref->[$index] or $ref->{$key}). In Python the
339 situation is more nebulous. Any class can expose a list-like or
340 dict-like interface, and functions that expect a list or dict will
341 usually happily accept an object of such a class. Such effects can be
342 achieved in Perl via "use overload '@{}'" et al, but seemingly are
343 comparatively rarer than in Python, and the Perl Template Toolkit
344 would reject such hijinks. It would not be valid to apply the
345 [% FOREACH %] construct to a stash variable that is not an
346 honest-to-goodness array reference, but an object that overloaded
347 '@{}'.
348
349 I have tried to be as permissive as possible in not requiring objects
350 to be instances of list, tuple, or dict, but to pass along any object
351 that can raise an exception if used in an inappropriate manner. For
352 example, the function is_seq() in template.util guesses that an object
353 is a sequence type if it supports iteration but is not a string. This
354 isn't a perfect solution (I just recently discovered that Python's
355 base Exception class is iterable--what the heck?), and cannot be
356 universally applied (such as in deciding whether a stash object can
357 have scalar, array, or hash vmethods called on it), but it seems to
358 work well enough for the time being. In more recent vintages of
359 Python it's possible to derive classes directly from the built-in
360 container types; one more permanent solution might involve requiring
361 classes that expect to be transparently treated as containers by the
362 Template Toolkit to be so derived.
363
364
365 Hash Keys
366 ---------
367
368 Perl hash keys are always converted to strings; Python dictionary keys
369 aren't. The following dict contains two distinct items:
370
371 mydict = { 1: "number", "1": "string" }
372
373 Since the Template Toolkit assumes Perlish hash semantics, there is
374 some unavoidable ambiguity when it comes to retrieving items from a
375 dict in the stash:
376
377 [% x = 1; mydict.x # is this "number" or "string"? %]
378
379 I have tried to follow a principle of least surprise here. Dict
380 lookup is attempted up to three times, first using the given key, then
381 using the stringified key if possible, then using the key converted to
382 an integer if possible. Therefore the previous template snippet would
383 print "number", and the other cases are exhibited thusly:
384
385 mydict = { 1: "one", "2": "two" }
386
387 160 sean [% x = "1"; y = 2 %]
388 157 sean [% mydict.x # "one" %]
389 [% mydict.y # "two" %]
390
391 In the more complicated case of an object that supported conversion to
392 both string and integer, the string version would win out.
393
394 class Dubious:
395 def __str__(self): return "1"
396 def __int__(self): return 1
397
398 dubious = Dubious()
399
400 mydict = { 1: "number", "1": "string" }
401
402 [% mydict.dubious # prints "string" %]
403
404
405 Unit Tests
406 ----------
407
408 All unit tests have been translated using the standard Python unittest
409 module. This is the framework with which I am most familiar, but it
410 transpires that the Perl unit tests don't lend themselves particularly
411 well to this approach. A typical unittest test should examine each
412 aspect of the object or class under test, one per method. For
413 example:
414
415 class MyTest(unittest.TestCase):
416 def testMethodOne(self):
417 ...
418 def testMethodTwo(self):
419 ...
420
421 No order of test evaluation is defined, which encourages tests to be
422 written independently of each other. In contrast, the Perl unit tests
423 typically include a long sequence of template/output pairs that must
424 be processed in sequence, since later templates depend on state
425 established by earlier templates. The Python tests therefore don't
426 exploit the full power of the unittest approach, and have the
427 appearance of mere boilerplate. Many test programs share a very
428 similar structure:
429
430 class FooTest(TestCase):
431 def testFoo(self):
432 # some setup
433 self.Expect(DATA, templates, variables)
434
435 # No other test methods!
436
437 DATA = r"""
438 -- test --
439 ...
440 -- expect --
441 ...
442 """
443
444 main()
445
446 In my opinion, the existing tests provide a good, but far from
447 exhaustive, level of test coverage. For example, on a few occasions
448 I've noticed Perl functions that I neglected to translate into Python,
449 and the omission went undetected by the relevant unit test. Many more
450 tests could and should be written in the mold of the unittest module,
451 and backported into the Perl version of the Toolkit.
452
453
454 PERL/RAWPERL/evalperl
455 ---------------------
456
457 For obvious reasons, the Python version of the Template Toolkit does
458 not support the PERL or RAWPERL template directives. It does,
459 however, have the parallel directives PYTHON and RAWPYTHON, and the
460 parallel configuration option EVAL_PYTHON. The variables "context"
461 and "stash" are available, mirroring the $context and $stash Perl
462 variables--but note that stash.get() returns a PerlScalar wrapping
463 object, which can be upwrapped by calling its value() method.
464
465 Perl version:
466
467 [% PERL %]
468 print $context->include('myfile');
469 $stash->set(foo => 'bar');
470 print 'foo value: ', $stash->get('foo');
471 [% END %]
472
473 Python version:
474
475 [% PYTHON %]
476 print context.include('myfile'),
477 stash.set('foo', 'bar')
478 print 'foo value:', stash.get('foo'),
479 [% END %]
480
481 Note that Python's print statement has semantics that may be
482 considered nontrivial; according to the online documentation, "a space
483 is written before each object is (converted [to a string] and)
484 written, unless the output system believes it is positioned at the
485 beginning of a line." Limited experimentation suggests that the
486 "output system" may become confused in the context of template
487 evaluation. To eliminate any ambiguity, one may alternately produce
488 output by calling the "write" method of the variable "stdout".
489
490 Alternate Python version:
491
492 [% PYTHON %]
493 stdout.write(context.include('myfile'))
494 stash.set('foo', 'bar')
495 stdout.write('foo value: ')
496 stdout.write(stash.get('foo'))
497 [% END %]
498
499 The print statement and the stdout variable both send output to
500 sys.stdout, which is temporarily set to a StringIO object during the
501 evaluation of the block. Note that the write() method of StringIO,
502 like that of ordinary Python file objects, accepts only a single
503 argument. For convenience, output may alternately be sent to the
504 write() method of the variable "output", which accepts any number of
505 arguments.
506
507 Another alternate Python version:
508
509 [% PYTHON %]
510 output.write(context.include('myfile'))
511 stash.set('foo', 'bar')
512 output.write('foo value: ', stash.get('foo'))
513 [% END %]
514
515 In RAWPYTHON blocks, only the "output" variable is available, taking
516 the place of the RAWPERL $output variable.
517
518 [% RAWPERL %]
519 $output .= foo() . bar();
520 [% END %]
521
522 Vs:
523
524 [% RAWPYTHON %]
525 output.write(foo(), bar())
526 [% END %]
527
528 Astute readers may be wondering how Python's (in)famous
529 indentation-as-block-delimiter feature figures into the evaluation of
530 PYTHON/RAWPYTHON blocks. The answer is that prior to passing the
531 contents of such blocks to the Python interpreter, each line of code
532 in such blocks has a number of leading whitespace characters stripped
533 which is equal to the smallest number of leading whitespace characters
534 found on any line in the block, not including empty lines and lines
535 consisting solely of whitespace. This should produce unsurprising
536 results, as long as all lines in the block are indented by a
537 consistent amount. Inconsistent indentation will likely produce a
538 syntax error, and should be avoided.
539
540 An example of a block that will cause a syntax error:
541
542 [% PYTHON %]
543 print "line 1"
544 print "line 2"
545 [% END %]
546
547 Another block that will cause a syntax error:
548
549 [% PYTHON %]
550 print "line 1"
551 print "line 2"
552 [% END %]
553
554 Note that leading whitespace is stripped indiscriminately; tab and
555 space characters are not distinguished at all. One shouldn't mix
556 tabs and spaces in leading whitespace, or problems are very likely to
557 occur.
558
559 For convenience in PYTHON blocks, a standard filter called "repr" is
560 provided which passes the stringified version of its argument to the
561 built-in Python function repr(). Example:
562
563 [% x = 1; y = 2 %]
564 [% PYTHON %]
565 print "x =", [% x | repr %], "; y =", [% y | repr %]
566 # Or:
567 print "x = %s; y = %s" % ([% x | repr %], [% y | repr %])
568 [% END %]
569
570 Finally, there is a standard filter "python", but it is more limited
571 than its "evalperl" counterpart. The latter filter produces its final
572 expression as output; for example, this prints "hello world":
573
574 [% FILTER perl %]
575 x = f()
576 y = g()
577 "hello world"
578 [% END %]
579
580 The same effect cannot be achieved in Python readily, if at all;
581 Python's dynamic code evaluation feature does not return the final
582 expression as Perl's does. One must produce output explicitly by
583 printing it, just as if the filtered text appeared in a PYTHON block,
584 as described above.
585
586 [% FILTER python %]
587 x = f()
588 y = g()
589 print "hello world"
590 [% END %]
591
592
593 Precompiled Templates
594 ---------------------
595
596 The Python Template Toolkit supports the COMPILE_DIR and COMPILE_EXT
597 options, just as the Perl version does. Another level of optimization
598 that is possible, but not yet exploited, is to byte-compile the
599 precompiled template source code. A future version Python Toolkit
600 should take advantage of this capability.
601
602