Summary

A powerful and robust templating system for Python.

Overview

EmPy is a system for embedding Python expressions and statements in template text; it takes an EmPy source file, processes it, and produces output. This is accomplished via expansions, which are special signals to the EmPy system and are set off by a special prefix (by default the at sign, @). EmPy can expand arbitrary Python expressions and statements in this way, as well as a variety of special forms. Textual data not explicitly delimited in this way is sent unaffected to the output, allowing Python to be used in effect as a markup language. Also supported are callbacks via hooks, recording and playback via diversions, and dynamic, chainable filters. The system is highly configurable via command line options and embedded commands.

Expressions are embedded in text with the @(...) notation; variations include conditional expressions with @(...?...!...) and the ability to handle thrown exceptions with @(...$...). As a shortcut, simple variables and expressions can be abbreviated as @variable, @object.attribute, @function(arguments), @sequence [index], and combinations. Full-fledged statements are embedded with @{...}. Control flow in terms of conditional or repeated expansion is available with @[...]. A @ followed by a whitespace character (including a newline) expands to nothing, allowing string concatenations and line continuations. Comments are indicated with @# and consume the rest of the line, up to and including the trailing newline. @% indicate "significators," which are special forms of variable assignment intended to specify per-file identification information in a format which is easy to parse externally. Context name and line number changes can be done with @? and @! respectively. Escape sequences analogous to those in C can be specified with @\..., and finally a @@ sequence expands to a single literal at sign.

Getting the software

The current version of empy is 3.2.

The latest version of the software is available in a tarball here: http://www.alcyone.com/software/empy/empy-latest.tar.gz.

The official URL for this Web site is http://www.alcyone.com/software/empy/.

Requirements

EmPy should work with any version of Python from 1.5.2 onward. It has been tested with all major versions of CPython from 1.5 up, and Jython from 2.0 up (using Java runtimes 1.3 and 1.4). The included test script is intended to run on Unix-like systems with a Bourne shell.

License

This code is released under the LGPL.

Mailing lists

There are two EmPy related mailing lists available. The first is a receive-only, very low volume list for important announcements (including releases). To subscribe, send an email to empy-announce-list-subscribe@alcyone.com.

The second is a general discussion list for topics related to EmPy, and is open for everyone to contribute; announcements related to EmPy will also be made on this list. The author of EmPy (and any future developers) will also be on the list, so it can be used not only to discuss EmPy features with other users, but also to ask questions of the author(s). To subscribe, send an email to empy-list-subscribe@alcyone.com.

Basics

EmPy is intended for embedding Python code in otherwise unprocessed text. Source files are processed, and the results are written to an output file. Normal text is sent to the output unchanged, but markups are processed, expanded to their results, and then written to the output file as strings (that is, with the str function, not repr). The act of processing EmPy source and handling markups is called "expansion."

Code that is processed is executed exactly as if it were entered into the Python interpreter; that is, it is executed with the equivalent of eval (for expressions) and exec (for statements). EmPy is intended to be a very thin (though powerful) layer on top of a running Python system; Python and EmPy files can be mixed together (via command line options) without complications.

By default the embedding prefix is the at sign (@), which appears neither in valid Python code nor commonly in arbitrary texts; it can be overridden with the -p option (or with the empy.setPrefix function). The prefix indicates to the EmPy interpreter that a special sequence follows and should be processed rather than sent to the output untouched (to indicate a literal at sign, it can be doubled as in @@).

When the interpreter starts processing its target file, no modules are imported by default, save the empy pseudomodule (see below), which is placed in the globals; the empy pseudomodule is associated with a particular interpreter; it is important that it not be removed from that interpreter's globals, nor that it be shared with other interpreters running concurrently (a name other than empy can be specified with the -m option). The globals are not cleared or reset in any way. It is perfectly legal to set variables or explicitly import modules and then use them in later markups, e.g., @{import time} ... @time.time(). Scoping rules are as in normal Python, although all defined variables and objects are taken to be in the global namespace.

Indentation is significant in Python, and therefore is also significant in EmPy. EmPy statement markups (@{...}), when spanning multiple lines, must be flush with the left margin. This is because (multiline) statement markups are not treated specially in EmPy and are simply passed to the Python interpreter, where indentation is significant.

Activities you would like to be done before any processing of the main EmPy file can be specified with the -I, -D, -E, -F, and -P options. -I imports modules, -D executes a Python variable assignment, -E executes an arbitrary Python (not EmPy) statement, -F executes a Python (not EmPy) file, and -P processes an EmPy (not Python) file. These operations are done in the order they appear on the command line; any number of each (including, of course, zero) can be used.

Expansions

The following markups are supported. For concreteness below, @ is taken for the sake of argument to be the prefix character, although this can be changed.

@# COMMENT NEWLINE

A comment. Comments, including the trailing newline, are stripped out completely. Comments should only be present outside of expansions. The comment itself is not processed in any way: It is completely discarded. This allows @# comments to be used to disable markups. Note: As special support for "bangpaths" in Unix-like operating systems, if the first line of a file (or indeed any context) begins with #!, and the interpreter has a processBangpaths option set to true (default), it is treated as a @# comment. A #! sequence appearing anywhere else will be handled literally and unaltered in the expansion. Example:

          @# This line is a comment.
          @# This will NOT be expanded: @x.

@? NAME NEWLINE

Set the name of the current context to be the given string. Variables are not allowed here; the name is treated as a literal. (If you wish to use arbitrary expressions, use the empy.setContextName function instead.) Example:

          @?NewName
          The context name is now @empy.identify()[0] (NewName).

@! INTEGER NEWLINE

Set the line number of the current context to be the given integer value; this is similar to the #line C preprocessor directive. This is done in such a way that the next line will have the specified numeric value, not the current one. Expressions are not allowed here; the number must be a literal integer. (If you wish to use arbitrary expressions, use the empy.setContextLine function instead.) Example:

          @!100
          The context line is now @empy.identify()[1] (100).

@ WHITESPACE

A @ followed by one whitespace character (a space, horizontal tab, vertical tab, carriage return, or newline) is expanded to nothing; it serves as a way to explicitly separate two elements which might otherwise be interpreted as being the same symbol (such as @name@ s to mean @(name)s -- see below). Also, since a newline qualifies as whitespace here, the lone @ at the end of a line represents a line continuation, similar to the backslash in other languages. Coupled with statement expansion below, spurious newlines can be eliminated in statement expansions by use of the @{...}@ construct. Example:

          This will appear as one word: salt@ water.
          This is a line continuation; @
          this text will appear on the same line.

@\ ESCAPE_CODE

An escape code. Escape codes in EmPy are similar to C-style escape codes, although they all begin with the prefix character. Valid escape codes include:

@\0

NUL, null

@\a

BEL, bell

@\b

BS, backspace

@\d

three-digital decimal code DDD

@\e

ESC, escape

@\f

FF, form feed

@\h

DEL, delete

@\n

LF, linefeed character, newline

@\oOOO

three-digit octal code OOO

@\qQQQQ

four-digit quaternary code QQQQ

@\r

CR, carriage return

@\s

SP, space

@\t

HT, horizontal tab

@\v

VT, vertical tab

@\xHH

two-digit hexadecimal code HH

@\z

EOT, end of transmission

@^X

the control character ^X

Unlike in C-style escape codes, escape codes taking some number of digits afterward always take the same number to prevent ambiguities. Furthermore, unknown escape codes are treated as parse errors to discourage potential subtle mistakes. Note that, while @\0 represents the NUL character, to represent an octal code, one must use @\o..., in contrast to C. Example:

          This embeds a newline.@\nThis is on the following line.
          This beeps!@\a
          There is a tab here:@\tSee?
          This is the character with octal code 141: @\o141.

@@

A literal at sign (@). To embed two adjacent at signs, use @@@@, and so on. Any literal at sign that you wish to appear in your text must be written this way, so that it will not be processed by the system. Note: If a prefix other than @ has been chosen via the command line option, one expresses that literal prefix by doubling it, not by appending a @. Example:

          The prefix character is @@.
          To get the expansion of x you would write @@x.

@), @], @}

These expand to literal close parentheses, close brackets, and close braces, respectively; these are included for completeness and explicitness only. Example:

          This is a close parenthesis: @).

@"...", @"""...""", etc.

These string literals expand to the literals themselves, so @"test" expands to test. Since they are inherently no-operations, the only reason for their use is to override their behavior with hooks.

@( EXPRESSION )

Evaluate an expression, and expand with the string (via a call to str) representation evaluation of that expression. Whitespace immediately inside the parentheses is ignored; @( expression ) is equivalent to @(expression). If the expression evaluates to None, nothing is expanded in its place; this allows function calls that depend on side effects (such as printing) to be called as expressions. (If you really do want a None to appear in the output, then use the Python string "None".) Note: If an expression prints something to sys.stdout as a side effect, then that printing will be spooled to the output before the expression's return value is. Example:

          2 + 2 is @(2 + 2).
          4 squared is @(4**2).
          The value of the variable x is @(x).
          This will be blank: @(None).

@( TEST ? THEN (! ELSE)_opt ($ EXCEPT)_opt )

A special form of expression evaluation representing conditional and protected evaluation. Evaluate the "test" expression; if it evaluates to true (in the Pythonic sense), then evaluate the "then" section as an expression and expand with the str of that result. If false, then the "else" section is evaluated and similarly expanded. The "else" section is optional and, if omitted, is equivalent to None (that is, no expansion will take place). Note: For backward compatibility, the "else" section delimiter, !, may be expressed as a :. This behavior is supported but deprecated.

If the "except" section is present, then if any of the prior expressions raises an exception when evaluated, the expansion will be replaced with the evaluation of the except expression. (If the "except" expression itself raises, then that exception will be propagated normally.) The except section is optional and, if omitted, is equivalent to None (that is, no expansion will take place). An exception (cough) to this is if one of these first expressions raises a SyntaxError; in that case the protected evaluation lets the error through without evaluating the "except" expression. The intent of this construct is to except runtime errors, and if there is actually a syntax error in the "try" code, that is a problem that should probably be diagnosed rather than hidden. Example:

          What is x? x is @(x ? "true" ! "false").
          Pluralization: How many words? @x word@(x != 1 ? 's').
          The value of foo is @(foo $ "undefined").
          Division by zero is @(x/0 $ "illegal").

@ SIMPLE_EXPRESSION

As a shortcut for the @(...) notation, the parentheses can be omitted if it is followed by a "simple expression." A simple expression consists of a name followed by a series of function applications, array subscriptions, or attribute resolutions, with no intervening whitespace. For example:

• a name, possibly with qualifying attributes (e.g., @value, @os.environ).

• a straightforward function call (e.g., @min(2, 3), @time.ctime()), with no space between the function name and the open parenthesis.

• an array subscription (e.g., '@array[index]', '@os.environ[name]', with no space between the name and the open bracket.

• any combination of the above (e.g., '@function(args).attr[sub].other[i](foo)').

In essence, simple expressions are expressions that can be written ambiguously from text, without intervening space. Note that trailing dots are not considered part of the expansion (e.g., @x. is equivalent to @(x)., not @(x.), which would be illegal anyway). Also, whitespace is allowed within parentheses or brackets since it is unambiguous, but not between identifiers and parentheses, brackets, or dots. Explicit @(...) notation can be used instead of the abbreviation when concatenation is what one really wants (e.g., @(word)s for simple pluralization of the contents of the variable word). As above, if the expression evaluates to the None object, nothing is expanded. Note that since a curly appearing where EmPy would expect an open parenthesis or bracket in is meaningless in Python, it is treated as a parse error (e.g., @x{1, 2} results in an error). Example:

          The value of x is @x.
          The ith value of a is @a[i].
          The result of calling f with q is @f(q).
          The attribute a of x is @x.a.
          The current time is @time.ctime(time.time()).
          The current year is @time.localtime(time.time())[0].
          These are the same: @min(2,3) and @min(2, 3).
          But these are not the same: @min(2, 3) vs. @min (2, 3).
          The plural of @name is @(name)s, or @name@ s.

@` EXPRESSION `

Evaluate a expression, and expand with the repr (instead of the str which is the default) of the evaluation of that expression. This expansion is primarily intended for debugging and is unlikely to be useful in actual practice. That is, a @`...` is identical to @(repr(...)). Example:

          The repr of the value of x is @`x`.
          This print the Python repr of a module: @`time`.
          This actually does print None: @`None`.

@: EXPRESSION : DUMMY :

Evaluate an expression and then expand to a @:, the original expression, a :, the evaluation of the expression, and then a :. The current contents of the dummy area are ignored in the new expansion. In this sense it is self-evaluating; the syntax is available for use in situations where the same text will be sent through the EmPy processor multiple times. Example:

          This construct allows self-evaluation:
          @:2 + 2:this will get replaced with 4:

@{ STATEMENTS }

Execute a (potentially compound) statement; statements have no return value, so the expansion is not replaced with anything. Multiple statements can either be separated on different lines, or with semicolons; indentation is significant, just as in normal Python code. Statements, however, can have side effects, including printing; output to sys.stdout (explicitly or via a print statement) is collected by the interpreter and sent to the output (unless this behavior is suppressed with the -n option). The usual Python indentation rules must be followed, although if the statement consists of only one statement, leading and trailing whitespace is ignored (e.g., @{ print time.time() } is equivalent to @{print time.time()}). Example:

          @{x = 123}
          @{a = 1; b = 2}
          @{print time.time()}
          @# Note that extra newlines will appear above because of the
          @# newlines trailing the close braces.  To suppress them
          @# use a @ before the newline:
          @{
          for i in range(10):
              print "i is %d" % i
          }@
          @{print "Welcome to EmPy."}@

@% KEY (WHITESPACE VALUE)_opt NEWLINE

Declare a significator. Significators consume the whole line (including the trailing newline), and consist of a key string containing no whitespace, and than optional value prefixed by whitespace. The key may not start with or contain internal whitespace, but the value may; preceding or following whitespace in the value is stripped. Significators are totally optional, and are intended to be used for easy external (that is, outside of EmPy) identification when used in large scale environments with many EmPy files to be processed. The purpose of significators is to provide identification information about each file in a special, easy-to-parse form so that external programs can process the significators and build databases, independently of EmPy. Inside of EmPy, when a significator is encountered, its key, value pair is translated into a simple assignment of the form __KEY__ = VALUE , where "__KEY__" is the key string with two underscores on either side and "VALUE" is a Python expression. Example:

          @%title     "Gravitation"
          @%author    "Misner", "Thorne", "Wheeler"
          @%publisher "W.H. Freeman and Company"
          @%pages     1279
          @%keywords  'physics', 'gravity', 'Einstein', 'relativity'
          @%copyright 1970, 1971

Control

EmPy version 3 and above includes the ability to direct conditional and repeated expansion of blocks of EmPy code with control markups (the obsolescent "substitution" markups are unavailable as of version 3.0). Control markups have analogs to control flow structures in Python such as if/elif/else, for, and while. Control markups are set off with the @[...] notation.

Control markups are designed to be used in precisely the same way that their internal Python analogues are used, except that the control markups are intended to be used where there is much more markup than control structure.

Some control markups are considered "primary," (e.g., if, for, while) as they begin a control markup. Others are considered "secondary," since they can only appear inside control flow markups delineated by primary markups (e.g., elif, else, continue, break).

Since EmPy, unlike Python, cannot use indentation to determine where control structures begin and end, all primary control markups must be followed by a corresponding terminating control markup:

        @[PRIMARY ...]...@[end PRIMARY]

(where PRIMARY represents one of the primary keywords). The end markup is mandatory, as is the space between the end and the starting keyword. For instance:

        @# If `person' is alive, show their age.
        @person.name is @
        @[if person.isAlive]@person.age@[else]dead@[end if].

All primary markups must be terminated in this way, and the keyword appearing in the appropriate end markup must match the primary markup it corresponds to; if either of these conditions are not satisfied, the result is a parse error. Everything between the starting control flow marker (@[PRIMARY ...]) and the ending marker (@[end PRIMARY]) -- including other markups, even control markups -- is considered part of the markup. Control markups can be nested:

        @# Print all non-false elements on separate lines.
        @[for elem in elements]@[if elem]@elem@\n@[end if]@[end for]

Three major types of primary control markups are available: conditional (e.g., if, try), looping (e.g., for, while), and definitional (e.g., def, discussed below). Conditional control markups conditionally expand their contents, whereas looping control markups repeatedly expand their contents. The third type, definitional markups, will define new objects in the globals relating to their contents. Conditional and looping markups also differ in one substantial respect: Looping constructs support '@[continue]' and '@[break]' markups which, like their Python equivalents, continue with the next iteration or break out of the innermost looping construct, respectively ('@[continue]' and '@[break]' markups have no meaning inside conditional markups and are an error). Also like their Python equivalents, '@[continue]' and '@[break]' may appear inside nested markups, so long as they ultimately are contained by at least one looping control markup:

        @# Walk a long a linked list, printing each element.
        @[while 1]@
        @node
        @{node = node.next}@
        @[if not node]@[break]@[end if]@
        @[end while]

The provided markups are designed to mimic the internal Python control structures as closely as possible. The supported control markups are (the phrases in all uppercase are intended to signify user-selectable patterns):

        @[if CONDITION1]...@[elif CONDITION2]...@[else]...@[end if]
        @[try]...@[except ...]...@[except ...]...@[end try]
        @[try]...@[finally]...@[end try]
        @[for VARIABLE in SEQUENCE]...@[else]...@[end for]
        @[while CONDITION]...@[else]...@[end while]
        @[def SIGNATURE]...@[end def]

All recognizable forms behave like their Python equivalents; if can contain multiple elif secondary markups within it; the else markups are optional (but must appear at the end), the try form with the except clause can contain multiple ones which are handled in sequence, the try form can either contain one or more except clauses or one finally clause (but not both), and the for and while structures can contain continue or break clauses internally (even if contained within other markups).

The third type of primary control markup is "definitional," in that they create objects in the globals for later use (e.g., def). This allows the definition of a callable object which, when called, will expand the contained markup (which can in turn, of course, contain further markups). The argument to the markup can be any legal Python function signature:

        @[def f(x, y, z=2, *args, **keywords)]...@[end def]

would define a function in the globals named f that takes the given arguments. A macro markup of the form @[def SIGNATURE]CODE@[end def] is equivalent to the Python code:

        def SIGNATURE:
            r"""CODE""" # so it is a doc string
            empy.expand(r"""CODE""", locals())

That is, it creates a Python function with the same name and function arguments, whose docstring is the contents of the EmPy markup that will be expanded when called. And, when called, it will expand those contents, with the locals passed in.

Unicode support

EmPy version 3.1 and above includes intrinsic Unicode support. EmPy's Unicode support defers to Python's internal Unicode support, available in Python 2.0 and up, in order to allow seamless and transparent translation of different encodings to the native Python Unicode format.

Knowledge of Python's Unicode support is expected, although not completely required, to gain full benefit of EmPy's Unicode features. To enable Unicode support, start EmPy with the -u/--unicode option. EmPy will then transparently encode from the input stream, process markups internally with native Unicode, and then decode transparently to the output stream.

By default, Python sets sys.stdin and sys.stdout with a default encoding which is accessible via 'sys.getdefaultencoding()'; encodings are represented by string names. These streams have encodings set by the system and cannot be changed.

However, encodings for newly created files (files to be read when specified on the command line, and/or files to be written when used with the -o and -a arguments) can be specified for EmPy via command line options. The --unicode-encoding option simultaneously indicates the encoding to be used for both input and output, whereas the --unicode-input-encoding and --unicode-output-encoding options can each be used to specify different encodings for both input and output. (If an encoding is not explicitly indicated, it resorts to the system default in sys.getdefaultencoding(), which is locale dependent.)

Python's Unicode implementation has the concept of error handlers, registered with the codecs module, which can be specified to determine what action should take place if input cannot be decoded into Unicode, or Unicode cannot be encoded into output. EmPy uses these same "errors," as they are called, and can be specified via command line options. The three most common error handlers are: ignore, where invalid sequences are simply ignored; replace, where invalid sequences are replaced with an encoding-specific indicator, usually a question mark; and strict, where invalid sequences raise an error. The --unicode-errors command line option specifies the same error handler to be used for both input and output, and the --unicode-input-errors and --unicode-output-errors options can specify different error handlers for input and output. If an error handler is not explicitly specified, the strict handler (which will raise errors) is used.

Remember, to specify input encodings or errors that will take effect, one cannot take input from sys.stdin and must explicitly specify an EmPy file to process on the command line. Similarly, for output encodings or errors, sys.stdout cannot be used and an explicit output file must be specified with the -o or -a options. It is perfectly valid to enable the Unicode subsystem (-u option) while using sys.stdin and sys.stdout, but the encodings and errors of these preexisting streams cannot be changed.

Combined with the --no-prefix option, which disables all markup processing, EmPy can act merely as an encoding translator, relying on Python's Unicode facilities:

        em.py --no-prefix \
            --unicode-input-encoding=utf-8 \
            --unicode-output-encoding=latin-1 \
            -o filename.Latin-1 filename.UTF-8

Significators

Significators, introduced in EmPy version 1.2, are intended to represent special assignment in a form that is easy to externally parse. For instance, if one has a system that contains many EmPy files, each of which has its own title, one could use a title significator in each file and use a simple regular expression to find this significator in each file and organize a database of the EmPy files to be built. This is an easier proposition than, for instance, attempting to grep for a normal Python assignment (inside a @{...} expansion) of the desired variable.

Significators look like the following:

        @%KEY VALUE

including the trailing newline, where "key" is a name and "value" is a Python expression, and are separated by any whitespace. This is equivalent to the following Python code:

        __KEY__ = VALUE

That is to say, a significator key translates to a Python variable consisting of that key surrounded by double underscores on either side. The value may contain spaces, but the key may not. So:

        @%title "All Roads Lead to Rome"

translates to the Python code:

        __title__ = "All Roads Lead to Rome"

but obviously in a way that easier to detect externally than if this Python code were to appear somewhere in an expansion. Since significator keys are surrounded by double underscores, significator keys can be any sequence of alphanumeric and underscore characters; choosing 123 is perfectly valid for a significator (although straight), since it maps to the name __123__ which is a legal Python identifier.

Note the value can be any Python expression. The value can be omitted; if missing, it is treated as None.

Significators are completely optional; it is completely legal for a EmPy file or files to be processed without containing any significators. Significators can appear anywhere within a file outside of other markups, but typically they are placed near the top of the file to make them easy to spot and edit by humans.

A regular expression string designed to match significators (with the default prefix) is available as empy.SIGNIFICATOR_RE_STRING, and also is a toplevel definition in the em module itself.

Diversions

EmPy supports an extended form of diversions, which are a mechanism for deferring and recalling output on demand, similar to the functionality included in m4. Multiple "streams" of output can be diverted (deferred) and undiverted (recalled) in this manner. A diversion is identified with a name, which is any immutable object such an integer or string. When recalled, diverted code is not resent through the EmPy interpreter (although a filter could be set up to do this).

By default, no diversions take place. When no diversion is in effect, processing output goes directly to the specified output file. This state can be explicitly requested at any time by calling the empy.stopDiverting function. It is always legal to call this function.

When diverted, however, output goes to a deferred location which can then be recalled later. Output is diverted with the empy.startDiversion function, which takes an argument that is the name of the diversion. If there is no diversion by that name, a new diversion is created and output will be sent to that diversion; if the diversion already exists, output will be appended to that preexisting diversion.

Output send to diversions can be recalled in two ways. The first is through the empy.playDiversion function, which takes the name of the diversion as an argument. This recalls the named diversion, sends it to the output, and then erases that diversion. A variant of this behavior is the empy.replayDiversion, which recalls the named diversion but does not eliminate it afterwards; empy.replayDiversion can be repeatedly called with the same diversion name, and will replay that diversion repeatedly. empy.createDiversion create a diversion without actually diverting to it, for cases where you want to make sure a diversion exists but do not yet want to send anything to it.

The diversion object itself can be retrieved with empy.retrieveDiversion. Diversions act as writable file-objects, supporting the usual write, writelines, flush, and close methods. The data that has been diverted to them can be retrieved in one of two ways; either through the asString method, which returns the entire contents of the diversion as a single strong, or through the asFile method, which returns the contents of the diversion as a readable (not writable) file-like object.

Diversions can also be explicitly deleted without recalling them with the empy.purgeDiversion function, which takes the desired diversion name as an argument.

Additionally there are three functions which will apply the above operations to all existing diversions: empy.playAllDiversions, empy.replayAllDiversions, and empy.purgeAllDiversions. All three will do the equivalent of a empy.stopDiverting call before they do their thing.

The name of the current diversion can be requested with the empy.getCurrentDiversion function; also, the names of all existing diversions (in sorted order) can be retrieved with empy.getAllDiversions.

When all processing is finished, the equivalent of a call to empy.playAllDiversions is done.

Filters

EmPy also supports dynamic filters, introduced in version 1.3. Filters are put in place right "before" the final output file, and so are only invoked after all other processing has taken place (including interpreting and diverting). Filters take input, remap it, and then send it to the output.

The current filter can be retrieved with the empy.getFilter function. The filter can be cleared (reset to no filter) with empy.resetFilter and a special "null filter" which does not send any output at all can be installed with empy.nullFilter. A custom filter can be set with the empy.setFilter function; for convenience, specialized shortcuts for filters preexist and can be used in lieu of actual empy.Filter instances for the empy.setFilter or empy.attachFilter argument:

• None is a special filter meaning "no filter"; when installed, no filtering whatsoever will take place. empy.setFilter(None) is equivalent to empy.resetFilter().

• 0 (or any other numeric constant equal to zero) is another special filter that represents the null filter; when installed, no output will ever be sent to the filter's sink.

• A filter specified as a function (or lambda) is expected to take one string argument and return one string argument; this filter will execute the function on any input and use the return value as output.

• A filter that is a string is a 256-character table is substituted with the result of a call to string.translate using that table.

• A filter can be an instance of a subclass of empy.Filter. This is the most general form of filter. (In actuality, it can be any object that exhibits a Filter interface, which would include the normal file-like write, flush, and close methods, as well as next, attach, and detach methods for filter-specific behavior.)

• Finally, the argument to empy.setFilter can be a Python list consisting of one or more of the above objects. In that case, those filters are chained together in the order they appear in the list. An empty list is the equivalent of 'None'; all filters will be uninstalled.

Filters are, at their core, simply file-like objects (minimally supporting write, flush, and close methods that behave in the usual way) which, after performing whatever processing they need to do, send their work to the next file-like object or filter in line, called that filter's "sink." That is to say, filters can be "chained" together; the action of each filter takes place in sequence, with the output of one filter being the input of the next. Additionally, filters support a _flush method (note the leading underscore) which will always flush the filter's underlying sink; this method should be not overridden.

Filters also support three additional methods, not part of the traditional file interface: attach, which takes as an argument a file-like object (perhaps another filter) and sets that as the filter's "sink" -- that is, the next filter/file-like object in line. detach (which takes no arguments) is another method which flushes the filter and removes its sink, leaving it isolated. Finally, next is an accessor method which returns the filter's sink -- or None, if the filter does not yet have a sink attached.

To create your own filter, you can create an object which supports the above described interface, or simply derive from the empy.Filter class and override its write and possibly flush methods. You can chain filters together by passing them as elements in a list to the empy.setFilter function, or you can chain them together manually with the attach method:

        firstFilter.attach(secondFilter)
        empy.setFilter(firstFilter)

or just let EmPy do the chaining for you:

        empy.setFilter([firstFilter, secondFilter])

In either case, EmPy will walk the filter chain and find the end and then hook that into the appropriate interpreter stream; you need not do this manually. The function empy.attachFilter can be used to attach a single filter (or shortcut, as above) to the end of a currently existing chain. Note that unlike its cousin empy.setFilter, one cannot pass a sequence of filters (or filter shortcuts) to empy.attachFilter. (If there is no existing filter chain installed, empy.attachFilter will behave the same as empy.setFilter.)

Subclasses of empy.Filter are already provided with the above null, function, and string functionality described above; they are NullFilter, FunctionFilter, and StringFilter, respectively. In addition, a filter which supports buffering, BufferedFilter, is provided. Several variants are included: SizeBufferedFilter, a filter which buffers into fixed-sized chunks, LineBufferedFilter, a filter which buffers by lines, and MaximallyBufferedFilter, a filter which completely buffers its input.

Hooks

The EmPy system allows for the registry of hooks with a running EmPy interpreter. Originally introduced in version 2.0 and much improved in 3.2, hooks are objects, registered with an interpreter, whose methods represent specific callbacks. Any number of hook objects can be registered with an interpreter, and when a callback is invoked, the associated method on each one of those hook objects will be called by the interpreter in sequence.

Hooks are simply instances, nominally derived from the empy.Hook class. The empy.Hook class itself defines a series of methods, with the expected arguments, which would be called by a running EmPy interpreter. This scenario, much improved from the prior implementation in 2.0, allows hooks to keep state and have more direct access to the interpreter they are running in (the empy.Hook instance contains an interpreter attribute).

To use a hook, derive a class from empy.Hook and override the desired methods (with the same signatures as they appear in the base class). Create an instance of that subclass, and then register it with a running interpreter with the empy.addHook function. (This same hook instance can be removed with the empy.removeHook function.)

More than one hook instance can be registered with an interpreter; in such a case, the appropriate methods are invoked on each instance in the order in which they were registered. To adjust this behavior, an optional prepend argument to the empy.addHook function can be used dictate that the new hook should placed at the beginning of the sequence of hooks, rather than at the end (which is the default).

All hooks can be enabled and disabled entirely for a given interpreter; this is done with the empy.enableHooks and empy.disableHooks functions. By default hooks are enabled, but obviously if no hooks have been registered no hook callbacks will be made. Whether hooks are enabled or disabled can be determined by calling empy.areHooksEnabled. To get a (copy of) the list of registered hooks, call empy.getHooks. Finally, to invoke a hook manually, use empy.invokeHook.

For a list of supported hook callbacks, see the empy.Hook class definition.

As a practical example, this sample Python code would print a pound sign followed by the name of every file that is included with 'empy.include':

        class IncludeHook(empy.Hook):
            def beforeInclude(self, name, file, locals):
                print "# %s" % name

        empy.addHook(IncludeHook())

Data flow

input -> interpreter -> diversions -> filters -> output

Here, in summary, is how data flows through a working EmPy system:

1. Input comes from a source, such an .em file on the command line, or via an empy.include statement.

2. The interpreter processes this material as it comes in, expanding EmPy expansions as it goes.

3. After interpretation, data is then sent through the diversion layer, which may allow it directly through (if no diversion is in progress) or defer it temporarily. Diversions that are recalled initiate from this point.

4. Any filters in place are then used to filter the data and produce filtered data as output.

5. Finally, any material surviving this far is sent to the output stream. That stream is stdout by default, but can be changed with the -o or -a options, or may be fully buffered with the -b option (that is, the output file would not even be opened until the entire system is finished).

Pseudomodule

The empy pseudomodule is available only in an operating EmPy system. (The name of the module, by default empy, can be changed with the -m option or the EMPY_PSEUDO environment variable). It is called a pseudomodule because it is not actually a module, but rather exports a module-like interface.

The pseudomodule contains the following functions and objects (and their signatures, with a suffixed opt indicating an optional argument):

First, basic identification:

VERSION

A constant variable which contains a string representation of the EmPy version.

SIGNIFICATOR_RE_STRING

A constant variable representing a regular expression string (using the default prefix) that can be used to find significators in EmPy code.

SIGNIFICATOR_RE_SUFFIX

The portion of the significator regular expression string excluding the prefix, so that those using non-standard prefix can build their own custom regular expression string with myPrefix + empy.SIGNIFICATOR_RE_SUFFIX.

interpreter

The instance of the interpreter that is currently being used to perform execution.

argv

A list consisting of the name of the primary EmPy script and its command line arguments, in analogue to the sys.argv list.

args

A list of the command line arguments following the primary EmPy script; this is equivalent to empy.argv[1:].

identify() -> string, integer

Retrieve identification information about the current parsing context. Returns a 2-tuple consisting of a filename and a line number; if the file is something other than from a physical file (e.g., an explicit expansion with empy.expand, a file-like object within Python, or via the -E or -F command line options), a string representation is presented surrounded by angle brackets. Note that the context only applies to the EmPy context, not the Python context.

atExit(callable)

Register a callable object (such as a function) taking no arguments which will be called at the end of a normal shutdown. Callable objects registered in this way are called in the reverse order in which they are added, so the first callable registered with empy.atExit is the last one to be called. Note that although the functionality is related to hooks, empy.atExit does no work via the hook mechanism, and you are guaranteed that the interpreter and stdout will be in a consistent state when the callable is invoked.

Context manipulation:

pushContext(name_opt, line_opt)

Create a new context with the given name and line and push it on the stack.

popContext()

Pop the top context and dispose of it.

setContextName(name)

Manually set the name of the current context.

setContextLine(line)

Manually set the line number of the current context; line must be a numeric value. Note that afterward the line number will increment by one for each newline that is encountered, as before.

Globals manipulation:

getGlobals()

Retrieve the globals dictionary for this interpreter. Unlike when calling globals() in Python, this dictionary can be manipulated and you can expect changes you make to it to be reflected in the interpreter that holds it.

setGlobals(globals)

Reseat the globals dictionary associated with this interpreter to the provided mapping type.

updateGlobals(globals)

Merge the given dictionary into this interpreter's globals.

clearGlobals(globals_opt)

Clear out the globals (restoring, of course, the empy pseudomodule). Optionally, instead of starting with a refresh dictionary, use the dictionary provided.

saveGlobals(deep=True)

Save a copy of the globals onto an internal history stack from which it can be restored later. The optional deep argument indicates whether or not the copying should be a deep copy (default) or a shallow one. Copying is done with copy.deepcopy or copy.copy, respectively.

restoreGlobals(destructive=True)

Restore the most recently saved globals from the history stack to as the current globals for this instance. The optional destructive argument indicates whether or not the restore should remove the restored globals from the history stack (default), or whether it should be left there for subsequent restores.

Types:

Interpreter

The actual interpreter class.

Hook

The base hook class that can be used to create user-defined hooks, which are then registered with an interpreter to receive event-based callbacks.

Filter

The base Filter class which can be derived from to make custom filters.

NullFilter

A null filter; all data sent to the filter is discarded.

FunctionFilter

A filter which uses a function taking a string and returning another to perform the filtering.

StringFilter

A filter which uses a 256-character string table to map any incoming character. (The table must be a string and cannot be Unicode, for reasons which should be obvious.)

BufferedFilter

A filter which does not modify its input, but instead holds it until it is told to flush (via the filter's flush method). This also serves as the base class for the other buffered filters below.

SizeBufferedFilter

A filter which buffers into fixed-size chunks, with the possible exception of the last chunk. The buffer size is indicated as the sole argument to the constructor.

LineBufferedFilter

A filter which buffers into lines, with the possible exception of the last line (which may not end with a newline).

MaximallyBufferedFilter

A filter which does not flush any of its contents until it is closed. Note that since this filter ignores calls to its flush method, this means that installing this filter and then replacing it with another can result in loss of data.

The following functions allow direct execution; optional locals arguments, if specified, are treated as the locals dictionary in evaluation and execution:

defined(name, locals_opt)

Return true if the given name is defined either in the (optional) locals or the interpreter globals; return false otherwise.

evaluate(expression, locals_opt)

Evaluate the given expression.

serialize(expression, locals_opt)

Serialize the expression, just as the interpreter would: If it is not None, convert it to a string with the str builtin function, and then write out the result. If it evaluates to None, do nothing.

execute(statements, locals_opt)

Execute the given statement(s).

single(source, locals_opt)

Interpret the "single" source code, just as the Python interactive interpreter would.

import_(name, locals_opt)

Import a module.

atomic(name, value, locals_opt)

Perform a single, atomic assignment. In this case name is the string denoating the name of the (single) variable to be assigned to, and value is a Python object which the name is to be bound to.

assign(name, value, locals_opt)

Perform general assignment. This decays to atomic assignment (above) in the normal case, but supports "tuple unpacking" in the sense that if name string contains commas, it is treated as a sequence of names and memberwise assignment with each member of the value (still a Python object, but which must be a sequence). This function will raise a TypeError or ValueError just like Python would if tuple unpacking is not possible (that is, if the value is not a sequence or is of an incompatible length, respectively). This only supports the assignment of Python identifiers, not arbitrary Python lvalues.

significate(key, value_opt, locals_opt)

Do a manual signification. If value is not specified, it is treated as None.

The following functions relate to source manipulation:

include(file_or_filename, locals_opt)

Include another EmPy file, by processing it in place. The argument can either be a filename (which is then opened with open in text mode) or a file object, which is used as is. Once the included file is processed, processing of the current file continues. Includes can be nested. The call also takes an optional locals dictionary which will be passed into the evaluation function.

expand(string, locals_opt) -> string

Explicitly invoke the EmPy parsing system to process the given string and return its expansion. This allows multiple levels of expansion, e.g., @(empy.expand("@(2 + 2)")). The call also takes an optional locals dictionary which will be passed into the evaluation function. This is necessary when text is being expanded inside a function definition and it is desired that the function arguments (or just plain local variables) are available to be referenced within the expansion.

quote(string) -> string

The inverse process of empy.expand, this will take a string and return a new string that, when expanded, would expand to the original string. In practice, this means that appearances of the prefix character are doubled, except when they appear inside a string literal.

escape(string, more_opt) -> string

Given a string, quote the nonprintable characters contained within it with EmPy escapes. The optional more argument specifies additional characters that should be escaped.

flush()

Do an explicit flush on the underlying stream.

string(string, name_opt, locals_opt)

Explicitly process a string-like object. This differs from empy.expand in that the string is directly processed into the EmPy system, rather than being evaluated in an isolated context and then returned as a string.

Changing the behavior of the pseudomodule itself:

flatten(keys_opt)

Perform the equivalent of from empy import ... in code (which is not directly possible because empy is a pseudomodule). If keys is omitted, it is taken as being everything in the empy pseudomodule. Each of the elements of this pseudomodule is flattened into the globals namespace; after a call to empy.flatten, they can be referred to simple as globals, e.g., @divert(3) instead of @empy.divert(3). If any preexisting variables are bound to these names, they are silently overridden. Doing this is tantamount to declaring an from ... import ... which is often considered bad form in Python.

Prefix-related functions:

getPrefix() -> char

Return the current prefix.

setPrefix(char)

Set a new prefix. Immediately after this call finishes, the prefix will be changed. Changing the prefix affects only the current interpreter; any other created interpreters are unaffected. Setting the prefix to None or the null string means that no further markups will be processed, equivalent to specifying the --no-prefix command line argument.

Diversions:

stopDiverting()

Any diversions that are currently taking place are stopped; thereafter, output will go directly to the output file as normal. It is never illegal to call this function.

createDiversion(name)

Create a diversion, but do not begin diverting to it. This is the equivalent of starting a diversion and then immediately stopping diversion; it is used in cases where you want to make sure that a diversion will exist for future replaying but may be empty.

startDiversion(name)

Start diverting to the specified diversion name. If such a diversion does not already exist, it is created; if it does, then additional material will be appended to the preexisting diversions.

playDiversion(name)

Recall the specified diversion and then purge it. The provided diversion name must exist.

replayDiversion(name)

Recall the specified diversion without purging it. The provided diversion name must exist.

purgeDiversion(name)

Purge the specified diversion without recalling it. The provided diversion name must exist.

playAllDiversions()

Play (and purge) all existing diversions in the sorted order of their names. This call does an implicit empy.stopDiverting before executing.

replayAllDiversions()

Replay (without purging) all existing diversions in the sorted order of their names. This call does an implicit empy.stopDiverting before executing.

purgeAllDiversions()

Purge all existing diversions without recalling them. This call does an implicit empy.stopDiverting before executing.

getCurrentDiversion() -> diversion

Return the name of the current diversion.

getAllDiversions() -> sequence

Return a sorted list of all existing diversions.

Filters:

getFilter() -> filter

Retrieve the current filter. None indicates no filter is installed.

resetFilter()

Reset the filter so that no filtering is done.

nullFilter()

Install a special null filter, one which consumes all text and never sends any text to the output.

setFilter(shortcut)

Install a new filter. A filter is None or an empty sequence representing no filter, or 0 for a null filter, a function for a function filter, a string for a string filter, or an instance of empy.Filter (or a workalike object). If filter is a list of the above things, they will be chained together manually; if it is only one, it will be presumed to be solitary or to have already been manually chained together. See the "Filters" section for more information.

attachFilter(shortcut)

Attach a single filter (sequences are not allowed here) to the end of a currently existing filter chain, or if there is no current chain, install it as empy.setFilter would. As with empy.setFilter, the shortcut versions of filters are also allowed here.

Hooks:

areHooksEnabled()

Return whether or not hooks are presently enabled.

enableHooks()

Enable invocation of hooks. By default hooks are enabled.

disableHooks()

Disable invocation of hooks. Hooks can still be added, removed, and queried, but invocation of hooks will not occur (even explicit invocation with empy.invokeHook).

getHooks()

Get a (copy of the) list of the hooks currently registered.

clearHooks()

Clear all the hooks registered with this interpreter.

addHook(hook, prepend_opt)

Add this hook to the hooks associated with this interpreter. By default, the hook is appended to the end of the existing hooks, if any; if the optional insert argument is present and true, it will be prepended to the list instead.

removeHook(hook)

Remove this hook from the hooks associated with this interpreter.

invokeHook(_name, ...)

Manually invoke a hook method. The remaining arguments are treated as keyword arguments and the resulting dictionary is passed in as the second argument to the hooks.

Invocation

Basic invocation involves running the interpreter on an EmPy file and some optional arguments. If no file are specified, or the file is named -, EmPy takes its input from stdin. One can suppress option evaluation (to, say, specify a file that begins with a dash) by using the canonical -- option.

-h/--help

Print usage and exit.

-H/--extended-help

Print extended usage and exit. Extended usage includes a rundown of all the legal expansions, escape sequences, pseudomodule contents, used hooks, and supported environment variables.

-v/--verbose

The EmPy system will print all manner of details about what it is doing and what it is processing to stderr.

-V/--version

Print version and exit.

-a/--append (filename)

Open the specified file for append instead of using stdout.

-b/--buffered-output

Fully buffer processing output, including the file open itself. This is helpful when, should an error occur, you wish that no output file be generated at all (for instance, when using EmPy in conjunction with make). When specified, either the -o or -a options must be specified, and the -b option must precede them. This can also be specified through the existence of the EMPY_BUFFERED_OUTPUT environment variable.

-f/--flatten

Before processing, move the contents of the empy pseudomodule into the globals, just as if empy.flatten() were executed immediately after starting the interpreter. That is, e.g., empy.include can be referred to simply as include when this flag is specified on the command line. This can also be specified through the existence of the EMPY_FLATTEN environment variable.

-i/--interactive

After the main EmPy file has been processed, the state of the interpreter is left intact and further processing is done from stdin. This is analogous to the Python interpreter's -i option, which allows interactive inspection of the state of the system after a main module is executed. This behaves as expected when the main file is stdin itself. This can also be specified through the existence of the EMPY_INTERACTIVE environment variable.

-k/--suppress-errors

Normally when an error is encountered, information about its location is printed and the EmPy interpreter exits. With this option, when an error is encountered (except for keyboard interrupts), processing stops and the interpreter enters interactive mode, so the state of affairs can be assessed. This is also helpful, for instance, when experimenting with EmPy in an interactive manner. -k implies -i.

-n/--no-override-stdout

Do not override sys.stdout with a proxy object which the EmPy system interacts with. If suppressed, this means that side effect printing will not be captured and routed through the EmPy system. However, if this option is specified, EmPy can support multithreading.

-o/--output (filename)

Open the specified file for output instead of using stdout. If a file with that name already exists it is overwritten.

-p/--prefix (prefix)

Change the prefix used to detect expansions. The argument is the one-character string that will be used as the prefix. Note that whatever it is changed to, the way to represent the prefix literally is to double it, so if $ is the prefix, a literal dollar sign is represented with $$. Note that if the prefix is changed to one of the secondary characters (those that immediately follow the prefix to indicate the type of action EmPy should take), it will not be possible to represent literal prefix characters by doubling them (e.g., if the prefix were unadvisedly changed to # then ## would already have to represent a comment, so ## could not represent a literal #). This can also be specified through the EMPY_PREFIX environment variable.

-r/--raw-errors

Normally, EmPy catches Python exceptions and prints them alongside an error notation indicating the EmPy context in which it occurred. This option causes EmPy to display the full Python traceback; this is sometimes helpful for debugging. This can also be specified through the existence of the EMPY_RAW_ERRORS environment variable.

-u/--unicode

Enable the Unicode subsystem. This option only need be present if you wish to enable the Unicode subsystem with the defaults; any other Unicode-related option (starting with --unicode...) will also enable the Unicode subsystem.

-D/--define (assignment)

Execute a Python assignment of the form variable = expression. If only a variable name is provided (i.e., the statement does not contain an = sign), then it is taken as being assigned to None. The -D option is simply a specialized -E option that special cases the lack of an assignment operator. Multiple -D options can be specified.

-E/--execute (statement)

Execute the Python (not EmPy) statement before processing any files. Multiple -E options can be specified.

-F/--execute-file (filename)

Execute the Python (not EmPy) file before processing any files. This is equivalent to -E execfile("filename") but provides a more readable context. Multiple -F options can be specified.

-I/--import (module)

Imports the specified module name before processing any files. Multiple modules can be specified by separating them by commas, or by specifying multiple -I options.

-P/--preprocess (filename)

Process the EmPy file before processing the primary EmPy file on the command line.

--binary

Treat the file as a binary file, and read in chunks rather than line by line. In this mode, the "line" indicator represents the number of bytes read, not the number of lines processed.

--no-prefix

Disable the prefixing system entirely; when specified, EmPy will not expand any markups. This allows EmPy to merely act as a Unicode encoding translator..

--pause-at-end

If present, then raw_input will be called at the end of processing. Useful in systems where the output window would otherwise be closed by the operating system/window manager immediately after EmPy exited.

--relative-path

When present, the path the EmPy script being invoked is contained in will be prepended to sys.path. This is analogous to Python's internal handling of sys.path and scripts. If input is from stdin (- for a filename or no filename is specified), then nothing is added to the path.

--chunk-size (chunk)

Use the specific binary chunk size rather than the default; implies --binary.

--unicode-encoding (encoding)

Specify the Unicode encoding to be used for both input and output.

--unicode-input-encoding (encoding)

Specify the Unicode encoding to be used for input.

--unicode-output-encoding (encoding)

Specify the Unicode encoding to be used for output.

'--unicode-input-errors (errors)

Specify the Unicode error handling to be used for input.

'--unicode-errors (errors)

Specify the Unicode error handling to be used for both input and output.

'--unicode-output-errors (errors)

Specify the Unicode error handling to be used for output.

Environment variables

EmPy also supports a few environment variables to predefine certain behaviors. The settings chosen by environment variables can be overridden via command line arguments. The following environment variables have meaning to EmPy:

EMPY_OPTIONS

If present, the contents of this environment variable will be treated as options, just as if they were entered on the command line, before the actual command line arguments are processed. Note that these arguments are not processed by the shell, so quoting, filename globbing, and the like, will not work.

EMPY_PREFIX

If present, the value of this environment variable represents the prefix that will be used; this is equivalent to the -p command line option.

EMPY_PSEUDO

If present, the value of this environment variable represents the name of the pseudomodule that will be incorporated into every running EmPy system; this is equivalent to the -m command line option.

EMPY_FLATTEN

If defined, this is equivalent to including -f on the command line.

EMPY_RAW_ERRORS

If defined, this is equivalent to including -r on the command line.

EMPY_INTERACTIVE

If defined, this is equivalent to including -i on the command line.

EMPY_BUFFERED_OUTPUT

If defined, this is equivalent to including -b on the command line.

EMPY_UNICODE

If defined, this is equivalent to including -u on the command line.

EMPY_UNICODE_INPUT_ENCODING

If present, the value of this environment variable indicates the name of the Unicode input encoding to be used. This is equivalent to the --unicode-input-encoding command line option.

EMPY_UNICODE_OUTPUT_ENCODING

If present, the value of this environment variable indicates the name of the Unicode output encoding to be used. This is equivalent to the --unicode-output-encoding command line option.

EMPY_UNICODE_INPUT_ERRORS

If present, the value of this environment variable indicates the name of the error handler to be used for input. This is equivalent to the --unicode-input-errors command line option.

EMPY_UNICODE_OUTPUT_ERRORS

If present, the value of this environment variable indicates the name of the error handler to be used for output. This is equivalent to the --unicode-output-errors command line option.

Examples and testing EmPy

See the sample EmPy file sample.em which is included with the distribution. Run EmPy on it by typing something like:

         ./em.py sample.em

and compare the results and the sample source file side by side. The sample content is intended to be self-documenting, and even an introduction to the basic features of EmPy while simultaneously exercising them.

The file sample.bench is the benchmark output of the sample. Running the EmPy interpreter on the provided sample.em file should produce precisely the same results. You can run the provided test script to see if your EmPy environment is behaving as expected (presuming a Unix-like operating system):

        ./test.sh

By default this will test with the first Python interpreter available in the path; if you want to test with another interpreter, you can provide it as the first argument on the command line, e.g.:

        ./test.sh python2.1
        ./test.sh /usr/bin/python1.5
        ./test.sh jython

A more comprehensive test suite and set of real-world examples is planned for a future version.

Embedding EmPy

For atomic applications, the expand function is provided (the extra keyword arguments passed in are treated as locals):

        import em
        print em.expand("@x + @y is @(x + y).", x=2, y=3)

One can specify a globals dictionary and all the other interpreter options (below) as well. One can specify a globals dictionary that will be used if one wants persistence:

        import em
        g = {}
        em.expand("@{x = 10}", g)
        print em.expand("x is @x.", g)

The standalone expand function, however, creates and destroys an Interpreter instance each time it is called. For repeated expansions, this can be expensive. Instead, you will probably want to use the full-fledged features of embedding. An EmPy interpreter can be created with as code as simple as:

        import em
        interpreter = em.Interpreter()
        # The following prints the results to stdout:
        interpreter.string("@{x = 123}@x\n")
        # This expands to the same thing, but puts the results as a
        # string in the variable result:
        result = interpreter.expand("@{x = 123}@x\n")
        # This just prints the value of x directly:
        print interpreter.globals['x']
        # Process an actual file (and output to stdout):
        interpreter.file(open('/path/to/some/file'))
        interpreter.shutdown() # this is important; see below

One can capture the output of a run in something other than stdout by specifying the output parameter:

        import em, StringIO
        output = StringIO.StringIO()
        interpreter = em.Interpreter(output=output)
        # Do something.
        interpreter.file(open('/path/to/some/file'))
        interpreter.shutdown() # again, this is important; see below
        print output.getvalue() # this is the result from the session

When you are finished with your interpreter, it is important to call its shutdown method:

        interpreter.shutdown()

This will ensure that the interpreter cleans up all its overhead, entries in the sys.stdout proxy, and so forth. It is usually advisable that this be used in a try...finally clause:

        interpreter = em.Interpreter(...)
        try:
            ...
        finally:
            interpreter.shutdown()

The em.Interpreter constructor takes the following arguments; all are optional. Since options may be added in the future, it's recommended that you invoke them via keyword arguments, rather than their explicit order. They are:

output

The output file which the interpreter will be sending all its processed data to. This need only be a file-like object; it need not be an actual file. If omitted, sys.__stdout__ is used.

argv

An argument list analogous to sys.argv, consisting of the script name and zero or more arguments. These are available to executing interpreters via empy.argv and empy.args. If omitted, a non-descript script name is used with no arguments.

prefix

The prefix (a single-character string). Defaults to @. It is an error for this to be anything other than one character.

pseudo

The name (string) of the pseudmodule. Defaults to empy.

options

A dictionary of options that can override the default behavior of the interpreter. The names of the options are constant names ending in _OPT and their defaults are given in Interpreter.DEFAULT_OPTIONS.

globals

By default, interpreters begin with a pristine dictionary of globals (except, of course, for the empy pseudomodule). Specifying this argument will allow the globals to start with more.

hooks

A sequence of hooks (or None for none) to register with the interpreter at startup. Hooks can, of course, be added after the fact, but this allows the hooks to intercept the atStartup event.

Many things can be done with EmPy interpreters; for the full developer documentation, see the generated documentation for the em module.

Interpreter options

The following options (passed in as part of the options dictionary to the Interpreter constructor) have the following meanings. The defaults are shown below and are also indicated in an Interpreter.DEFAULT_OPTIONS dictionary.

BANGPATH_OPT

Should a bangpath (#!) as the first line of an EmPy file be treated as if it were an EmPy comment? Note that #! sequences starting lines or appearing anywhere else in the file are untouched regardless of the value of this option. Default: true.

BUFFERED_OPT

Should an abort method be called upon failure? This relates to the fully-buffered option, where all output can be buffered including the file open; this option only relates to the interpreter's behavior after that proxy file object has been created. Default: false.

RAW_OPT

Should errors be displayed as raw Python errors (that is, the exception is allowed to propagate through to the toplevel so that the user gets a standard Python traceback)? Default: false.

EXIT_OPT

Upon an error, should execution continue (although the interpreter stacks will be purged)? Note that even in the event this is set, the interpreter will halt upon receiving a KeyboardInterrupt. Default: true.

FLATTEN_OPT

Upon initial startup, should the empy pseudomodule namespace be flattened, i.e., should empy.flatten be called? Note this option only has an effect when the interpreter is first created; thereafter it is ignored. Default: false.

OVERRIDE_OPT

Should the sys.stdout object be overridden with a proxy object? If not, side effect output cannot be captured by the EmPy system, but EmPy will support multithreading. Default: true.

Author's notes

I originally conceived EmPy as a replacement for my Web templating system which uses m4 (a general macroprocessing system for Unix).

Most of my Web sites include a variety of m4 files, some of which are dynamically generated from databases, which are then scanned by a cataloging tool to organize them hierarchically (so that, say, a particular m4 file can understand where it is in the hierarchy, or what the titles of files related to it are without duplicating information); the results of the catalog are then written in database form as an m4 file (which every other m4 file implicitly includes), and then GNU make converts each m4 to an HTML file by processing it.

As the Web sites got more complicated, the use of m4 (which I had originally enjoyed for the challenge and abstractness) really started to become an impediment to serious work; while I am very knowledgeable about m4 -- having used it for for so many years -- getting even simple things done with it is awkward and difficult. Worse yet, as I started to use Python more and more over the years, the cataloging programs which scanned the m4 and built m4 databases were migrated to Python and made almost trivial, but writing out huge awkward tables of m4 definitions simply to make them accessible in other m4 scripts started to become almost farcical -- especially when coupled with the difficulty in getting simple things done in m4.

It occurred to me what I really wanted was an all-Python solution. But replacing what used to be the m4 files with standalone Python programs would result in somewhat awkward programs normally consisting mostly of unprocessed text punctuated by small portions where variables and small amounts of code need to be substituted. Thus the idea was a sort of inverse of a Python interpreter: a program that normally would just pass text through unmolested, but when it found a special signifier would execute Python code in a normal environment. I looked at existing Python templating systems, and didn't find anything that appealed to me -- I wanted something where the desired markups were simple and unobtrusive. After considering between choices of signifiers, I settled on @ and EmPy was born.

As I developed the tool, I realized it could have general appeal, even to those with widely varying problems to solve, provided the core tool they needed was an interpreter that could embed Python code inside templated text. As I continue to use the tool, I have been adding features as unintrusively as possible as I see areas that can be improved.

A design goal of EmPy is that its feature set should work on several levels; at each level, if the user does not wish or need to use features from another level, they are under no obligation to do so. If you have no need of diversions, for instance, you are under no obligation to use them. If significators will not help you organize a set of EmPy scripts globally, then you need not use them. New features that are being added are whenever possible transparently backward compatible; if you do not need them, their introduction should not affect you in any way. The use of unknown prefix sequences results in errors, guaranteeing that they are reserved for future use.

Glossary

control

A control markup, used to direct high-level control flow within an EmPy session. Control markups are expressed with the @[...] notation.

diversion

A process by which output is deferred, and can be recalled later on demand, multiple times if necessary.

document

The abstraction of an EmPy document as used by a processor.

escape

A markup designed to expand to a single (usually non-printable) character, similar to escape sequences in C or other languages.

expansion

The process of processing EmPy markups and producing output.

expression

An expression markup represents a Python expression to be evaluated, and replaced with the str of its value. Expression markups are expressed with the @(...) notation.

filter

A file-like object which can be chained to other objects (primarily the final stream) and can buffer, alter, or manipulate in any way the data sent. Filters can also be chained together in arbitrary order.

globals

The dictionary (or dictionary-like object) which resides inside the interpreter and holds the currently-defined variables.

hook

A callable object that can be registered in a dictionary, and which will be invoked before, during, or after certain internal operations, identified by name with a string.

interpreter

The application (or class instance) which processes EmPy markup.

markup

EmPy substitutions set off with a prefix and appropriate delimeters.

output

The final destination of the result of processing an EmPy file.

prefix

The ASCII character used to set off an expansions. By default, @.

processor

An extensible system which processes a group of EmPy files, usually arranged in a filesystem, and scans them for significators.

pseudomodule

The module-like object named empy which is exposed internally inside every EmPy system.

shortcut

A special object which takes the place of an instance of the Filter class, to represent a special form of filter. These include 0 for a null filter, a callable (function or lambda) to represent a callable filter, or a 256-character string which represents a translation filter.

significator

A special form of an assignment markup in EmPy which can be easily parsed externally, primarily designed for representing uniform assignment across a collection of files. Significators are indicated with the @% markup.

statement

A line of code that needs to be executed; statements do not have return values. In EmPy, statements are set off with @{...}.

Acknowledgements

Questions, suggestions, bug reports, evangelism, and even complaints from many people have helped make EmPy what it is today. Some, but by no means all, of these people are (in alphabetical order by surname):

• Biswapesh Chattopadhyay

• Beni Cherniavsky

• Dr. S. Candelaria de Ram

• Eric Eide

• Dinu Gherman

• Grzegorz Adam Hankiewicz

• Robert Kroeger

• Kouichi Takahashi

• Ville Vainio

Known issues and caveats

• EmPy was primarily intended for static processing of documents, rather than dynamic use, and hence speed of processing was not the primary consideration in its design.

• EmPy is not threadsafe by default. This is because of the need for EmPy to override the sys.stdout file with a proxy object which can capture effects of print and other spooling to stdout. This proxy can be suppressed with the -n option, which will result in EmPy being unable to do anything meaningful with this output, but will allow EmPy to be threadsafe.

• To function properly, EmPy must override sys.stdout with a proxy file object, so that it can capture output of side effects and support diversions for each interpreter instance. It is important that code executed in an environment not rebind sys.stdout, although it is perfectly legal to invoke it explicitly (e.g., @sys.stdout.write("Hello world\n")). If one really needs to access the "true" stdout, then use sys.__stdout__ instead (which should also not be rebound). EmPy uses the standard Python error handlers when exceptions are raised in EmPy code, which print to sys.stderr. (This behavior can be overridden with the -n option.)

• Due to Python's curious handling of the print statement -- particularly the form with a trailing comma to suppress the final newline -- mixing statement expansions using prints inline with unexpanded text will often result in surprising behavior, such as extraneous (sometimes even deferred!) spaces. This is a Python "feature," and occurs in non-EmPy applications as well; for finer control over output formatting, use sys.stdout.write or empy.interpreter.write directly.

• The empy "module" exposed through the EmPy interface (e.g., @empy) is an artificial module. It cannot be imported with the import statement (and shouldn't -- it is an artifact of the EmPy processing system and does not correspond to any accessible .py file).

• For an EmPy statement expansion all alone on a line, e.g., @{a = 1}, note that this will expand to a blank line due to the newline following the closing curly brace. To suppress this blank line, use the symmetric convention @{a = 1}@.

• When using EmPy with make, note that partial output may be created before an error occurs; this is a standard caveat when using make. To avoid this, write to a temporary file and move when complete, delete the file in case of an error, use the -b option to fully buffer output (including the open), or (with GNU make) define a .DELETE_ON_ERROR target.

• empy.identify tracks the context of executed EmPy code, not Python code. This means that blocks of code delimited with @{ and } will identify themselves as appearing on the line at which the } appears, and that pure Python code executed via the -D, -E and -F command line arguments will show up as all taking place on line 1. If you're tracking errors and want more information about the location of the errors from the Python code, use the -r command line option, which will provide you with the full Python traceback.

• The conditional form of expression expansion @(...?...!...) allows the use of a colon instead of an exclamation point, e.g., @(...?...:...). This behavior is supported for backward compatibility, but is deprecated. Due to an oversight, the colon was a poor choice since colons can appear legally in expressions (e.g., dictionary literals or lambda expressions).

• The '@[try]' construct only works with Python exceptions derived from Exception. It is not able to catch string exceptions.

• The '@[for]' variable specification supports tuples for tuple unpacking, even recursive tuples. However, it is limited in that the names included may only be valid Python identifiers, not arbitrary Python lvalues. Since the internal Python mechanism is very rarely used for this purpose (e.g., 'for (x, l[0], q.a) in sequence'), it is not thought to be a significant limitation.

Wish list

Here are some random ideas for future revisions of EmPy. If any of these are of particular interest to you, your input would be appreciated.

• Some real-world examples should really be included for demonstrating the power and expressiveness of EmPy first-hand.

• More extensive help (rather than a ridiculously long README), probably inherently using the EmPy system itself for building to HTML and other formats, thereby acting as a help facility and a demonstration of the working system.

• A "trivial" mode, where all the EmPy system does is scan for simple symbols to replace them with evaluations/executions, rather than having to do the contextual scanning it does now. This has the down side of being much less configurable and powerful but the upside of being extremely efficient. Perhaps this need not be a separate mode, but an additional prefix something of the form '@<(...)>', @<[...]>, and @<{...}>? Setting the trivial mode might simply disallow other expansions. Also, perhaps there might be a special mode where EmPy might convert normal expansions to their '@<...>' form in an automated fashion.

• A "debug" mode, where EmPy prints the contents of everything it's about to evaluate (probably to stderr) before it does?

• The ability to funnel all code through a configurable RExec for user-controlled security control. This would probably involve abstracting the execution functionality outside of the interpreter. [This suggestion is on hold until the rexec/Bastion exploits are worked out.]

• Optimized handling of processing would be nice for the possibility of an Apache module devoted to EmPy processing.

• An EmPy emacs mode.

• An optimization of offloading diversions to files when they become truly huge. (This is made possible by the abstraction of the Diversion class.)

• Support for mapping filters (specified by dictionaries).

• Support for some sort of batch processing, where several EmPy files can be listed at once and all of them evaluated with the same initial (presumably expensive) environment. empy.saveGlobals and empy.restoreGlobals have been introduced as a partial solution, but they need to be made more robust.

• A more elaborate interactive mode, perhaps with a prompt and readline support.

• A StructuredText and/or reStructuredText filter would be quite useful, as would SGML/HTML/XML/XHTML, s-expression, Python, etc. auto-indenter filters.

• An indexing filter, which can process text and pick out predefined keywords and thereby setup links to them.

• The ability to rerun diverted material back through the interpreter. (This can be done, awkwardly, by manually creating a filter which itself contains an interpreter, but it might be helpful if this was an all-in-one operation.)

• A caching system that stores off the compilations of repeated evaluations and executions so that in a persistent environment the same code does not have to be repeatedly evaluated/executed. This would probably be a necessity in an Apache module-based solution. Perhaps caching even to the point of generating pure PyWM bytecode?

• An option to change the format of the standard EmPy errors in a traceback.

• Support for some manner of implicitly processed /etc/empyrc and/or ~/.empyrc file, and of course an option to inhibit its processing. This can already be accomplished (and with greater control) via use of EMPY_OPTIONS, though.

• More uniform handling of the preprocessing directives (-I, -D, -E, -F, and -P), probably mapping directly to methods in the Interpreter class.

• Support for integration with mod_python.

• In simple expressions, a {...} suffix has no meaning in Python (e.g., in Python, @x(...) is a call, @x[...] is subscription, but @x{...} is illegal). This could be exploited by having a {...} suffix in a simple expression representing an encapsulation of an expanded string; e.g., @bullet{There are @count people here} would be equivalent to @bullet(empy.expand("There are @count people here", locals()))}.

• A tool to collect significator information from a hierarchy of .em files and put them in a database form available for individual scripts would be extremely useful -- this tool should be extensible so that users can use it to, say, build ordered hierarchies of their EmPy files by detecting contextual information like application-specific links to other EmPy documents.

• Extensions of the basic EmPy concepts to projects for other interpreted languages, such as Java, Lua, Ruby, and/or Perl.

• Ignore SystemExit when doing error handling, letting the exception progagate up? So far no one seems to worry about this; deliberately exiting early in a template seems to be an unlikely occurrence. (Furthermore, there are the os.abort and os._exit facilities for terminating without exception propagation.)

• An entirely unique and dedicated expansion for user-definable behavior (which would raise an exception if used without that behavior being defined).

• A new markup which is the equivalent of $...:...$ in source control systems, where the left-hand portion represents a keyword and the right-hand portion represents its value which is substituted in by the EmPy system.

• Would it make sense to eliminate the distinction between the pseudmodule and the interpreter itself -- that is, simply have the interpreter itself act as its own pseudomodule? This obviously would involve a circular reference, but circular references are already around and that's what the .shutdown interpreter method is for.

Release history

• 3.2; 2003 Oct 7. Reengineer hooks support to use hook instances; add -v/--verbose option; add --relative-path option; reverse PEP 317 style; modify Unicode support to give less confusing errors in the case of unknown encodings and error handlers; relicensed under LGPL.

• 3.1.1; 2003 Sep 20. Added literal @"..." markup; added --pause-at-end command line option; fixed improper globals collision error via the sys.stdout proxy.

• 3.1; 2003 Aug 8. Unicode support (Python 2.0 and above); added Document and Processor helper classes for processing significators; added --no-prefix option for suppressing all markups.

• 3.0.4; 2003 Aug 7. Somewhat more robust lvalue parsing for '@[for]' construct (thanks to Beni Cherniavsky for inspiration).

• 3.0.3; 2003 Jul 9. Fix bug regarding recursive tuple unpacking using '@[for]'; add empy.saveGlobals, empy.restoreGlobals, and empy.defined functions.

• 3.0.2; 2003 Jun 19. @? and @! markups for changing the current context name and line, respectively; add update method to interpreter; new and renamed context operations, empy.setContextName, empy.setContextLine, empy.pushContext, empy.popContext.

• 3.0.1; 2003 Jun 9. Fix simple bug preventing command line preprocessing directives (-I, -D, -E, -F, -P) from executing properly; defensive PEP 317 compliance [defunct].

• 3.0; 2003 Jun 1. Control markups with '@[...]'; remove substitutions (use control markups instead); support @(...?...!...) for conditional expressions in addition to the now-deprecated @(...?...:...) variety; add acknowledgements and glossary sections to documentation; rename buffering option back to -b; add -m option and EMPY_PSEUDO environment variable for changing the pseudomodule name; add -n option and EMPY_NO_OVERRIDE environment variable for suppressing sys.stdout proxy; rename main error class to 'Error'; add standalone expand function; add --binary and --chunk-size options; reengineer parsing system to use Tokens for easy extensibility; safeguard curly braces in simple expressions (meaningless in Python and thus likely a typographical error) by making them a parse error; fix bug involving custom Interpreter instances ignoring globals argument; distutils support.

• 2.3; 2003 Feb 20. Proper and full support for concurrent and recursive interpreters; protection from closing the true stdout file object; detect edge cases of interpreter globals or sys.stdout proxy collisions; add globals manipulation functions empy.getGlobals, empy.setGlobals, and empy.updateGlobals which properly preserve the empy pseudomodule; separate usage info out into easily accessible lists for easier presentation; have -h option show simple usage and -H show extened usage; add NullFile utility class.

• 2.2.6; 2003 Jan 30. Fix a bug in the Filter.detach method (which would not normally be called anyway).

• 2.2.5; 2003 Jan 9. Strip carriage returns out of executed code blocks for DOS/Windows compatibility.

• 2.2.4; 2002 Dec 23. Abstract Filter interface to use methods only; add @[noop: ...] substitution for completeness and block commenting [defunct].

• 2.2.3; 2002 Dec 16. Support compatibility with Jython by working around a minor difference between CPython and Jython in string splitting.

• 2.2.2; 2002 Dec 14. Include better docstrings for pseudomodule functions; segue to a dictionary-based options system for interpreters; add empy.clearAllHooks and 'empy.clearGlobals'; include a short documentation section on embedding interpreters; fix a bug in significator regular expression.

• 2.2.1; 2002 Nov 30. Tweak test script to avoid writing unnecessary temporary file; add Interpreter.single method; expose evaluate, execute, substitute [defunct], and single methods to the pseudomodule; add (rather obvious) EMPY_OPTIONS environment variable support; add empy.enableHooks and 'empy.disableHooks'; include optimization to transparently disable hooks until they are actually used.

• 2.2; 2002 Nov 21. Switched to -V option for version information; empy.createDiversion for creating initially empty diversion; direct access to diversion objects with 'empy.retrieveDiversion'; environment variable support; removed --raw long argument (use --raw-errors instead); added quaternary escape code (well, why not).

• 2.1; 2002 Oct 18. empy.atExit registry separate from hooks to allow for normal interpreter support; include a benchmark sample and test.sh verification script; expose empy.string directly; -D option for explicit defines on command line; remove ill-conceived support for @else: separator in @[if ...] substitution [defunct] ; handle nested substitutions properly [defunct] ; @[macro ...] substitution for creating recallable expansions [defunct].

• 2.0.1; 2002 Oct 8. Fix missing usage information; fix after_evaluate hook not getting called; add empy.atExit call to register values.

• 2.0; 2002 Sep 30. Parsing system completely revamped and simplified, eliminating a whole class of context-related bugs; builtin support for buffered filters; support for registering hooks; support for command line arguments; interactive mode with -i; significator value extended to be any valid Python expression.

• 1.5.1; 2002 Sep 24. Allow @] to represent unbalanced close brackets in @[...] markups [defunct].

• 1.5; 2002 Sep 18. Escape codes (@\...); conditional and repeated expansion substitutions [defunct] ; replaced with control markups]; fix a few bugs involving files which do not end in newlines.

• 1.4; 2002 Sep 7. Fix bug with triple quotes; collapse conditional and protected expression syntaxes into the single generalized @(...) notation; empy.setName and empy.setLine functions [deprecated] ; true support for multiple concurrent interpreters with improved sys.stdout proxy; proper support for empy.expand to return a string evaluated in a subinterpreter as intended; merged Context and Parser classes together, and separated out Scanner functionality.

• 1.3; 2002 Aug 24. Pseudomodule as true instance; move toward more verbose (and clear) pseudomodule functions; fleshed out diversion model; filters; conditional expressions; protected expressions; preprocessing with -P (in preparation for possible support for command line arguments).

• 1.2; 2002 Aug 16. Treat bangpaths as comments; empy.quote for the opposite process of 'empy.expand'; significators (@%... sequences); -I option; -f option; much improved documentation.

• 1.1.5; 2002 Aug 15. Add a separate invoke function that can be called multiple times with arguments to simulate multiple runs.

• 1.1.4; 2002 Aug 12. Handle strings thrown as exceptions properly; use getopt to process command line arguments; cleanup file buffering with AbstractFile; very slight documentation and code cleanup.

• 1.1.3; 2002 Aug 9. Support for changing the prefix from within the empy pseudomodule.

• 1.1.2; 2002 Aug 5. Renamed buffering option [defunct], added -F option for interpreting Python files from the command line, fixed improper handling of exceptions from command line options (-E, -F).

• 1.1.1; 2002 Aug 4. Typo bugfixes; documentation clarification.

• 1.1; 2002 Aug 4. Added option for fully buffering output (including file opens), executing commands through the command line; some documentation errors fixed.

• 1.0; 2002 Jul 23. Renamed project to EmPy. Documentation and sample tweaks; added empy.flatten. Added -a option.

• 0.3; 2002 Apr 14. Extended "simple expression" syntax, interpreter abstraction, proper context handling, better error handling, explicit file inclusion, extended samples.

• 0.2; 2002 Apr 13. Bugfixes, support non-expansion of Nones, allow choice of alternate prefix.

• 0.1.1; 2002 Apr 12. Bugfixes, support for Python 1.5.x, add -r option.

• 0.1; 2002 Apr 12. Initial early access release.

Author

This module was written by Erik Max Francis. If you use this software, have suggestions for future releases, or bug reports, I'd love to hear about it.

Even if you try out EmPy for a project and find it unsuitable, I'd like to know what stumbling blocks you ran into so they can potentially be addressed in a future version.

Version

Version 3.2 $Date: 2003/10/07 $ $Author: max $