Twisted is a big system. People are often daunted when they approach it. It's hard to know where to start looking.
This guide builds a full-fledged Twisted application from the ground up, using most of the important bits of the framework. There is a lot of code, but don't be afraid.
The application we are looking at is a "finger" service, along the lines of the familiar service traditionally provided by UNIX™ servers. We will extend this service slightly beyond the standard, in order to demonstrate some of Twisted's higher-level features.
This example only runs the reactor. Nothing at all will happen until we interrupt the program. It will not consume (almost) no CPU resources. Not very useful, perhaps -- but this is the skeleton inside which the Twisted program will grow.
You don't call Twisted, Twisted calls you. The reactor is Twisted's main event loop. There is exactly one reactor in any running Twisted application. Once started it loops over and over again, responding to network events, and making scheduled calls to code.
Here, we start listening on port 1079. The 1079 is a reminder that eventually, we want to run on port 79, the standard port for finger servers. We define a protocol which does not respond to any events. Thus, connections to 1079 will be accepted, but the input ignored.
Here we add to the protocol the ability to respond to the event of beginning a connection -- by terminating it. Perhaps not an interesting behavior, but it is already close to behaving according to the letter of the protocol. After all, there is no requirement to send any data to the remote connection in the standard. The only problem, as far as the standard is concerned, is that we terminate the connection too soon. A client which is slow enough will see his send() of the username result in an error.
Here we make FingerProtocol
inherit from LineReceiver
, so that we get data-based
events on a line-by-line basis. We respond to the event of receiving the line
with shutting down the connection.
Congratulations, this is the first standard-compliant version of the code. However, usually people actually expect some data about users to be transmitted.
Finally, a useful version. Granted, the usefulness is somewhat limited by the fact that this version only prints out a "No such user" message. It could be used for devastating effect in honey-pots, of course.
The same behavior, but finally we see what usefulness the factory has: as something that does not get constructed for every connection, it can be in charge of the user database. In particular, we won't have to change the protocol if the user database back-end changes.
Finally, a really useful finger database. While it does not supply information about logged in users, it could be used to distribute things like office locations and internal office numbers. As hinted above, the factory is in charge of keeping the user database: note that the protocol instance has not changed. This is starting to look good: we really won't have to keep tweaking our protocol.
But, here we tweak it just for the hell of it. Yes, while the previous version worked, it did assume the result of getUser is always immediately available. But what if instead of an in memory database, we would have to fetch result from a remote Oracle? Or from the web? Or, or...
...from running a local command? Yes, this version (safely!) runs finger locally with whatever arguments it is given, and returns the standard output. This will do exactly what the standard version of the finger server does -- without the need for any remote buffer overflows, as the networking is done safely.
The web. That invention which has infiltrated homes around the world finally gets through to our invention. Here we use the built-in Twisted web client, which also returns a deferred. Finally, we manage to have examples of three different database back-ends, which do not change the protocol class. In fact, we will not have to change the protocol again until the end of this talk: we have achieved, here, one truly usable class.
Up until now, we faked. We kept using port 1079, because really, who wants to run a finger server with root privileges? Well, the common solution is "privilege shedding": after binding to the network, become a different, less privileged user. We could have done it ourselves, but Twisted has a built-in way to do it. Create a snippet as above, defining an application object. That object will have uid and gid attributes. When running it (later we will see how) it will bind to ports, shed privileges and then run.
root% twistd -ny finger.tpy # just like before root% twistd -y finger.tpy # daemonize, keep pid in twistd.pid root% twistd -y finger.tpy --pidfile=finger.pid root% twistd -y finger.tpy --rundir=/ root% twistd -y finger.tpy --chroot=/var root% twistd -y finger.tpy -l /var/log/finger.log root% twistd -y finger.tpy --syslog # just log to syslog root% twistd -y finger.tpy --syslog --prefix=twistedfinger # use given prefix
This is how to run "Twisted Applications" -- files which define an 'application'. twistd (TWISTed Daemonizer) does everything a daemon can be expected to -- shuts down stdin/stdout/stderr, disconnects from the terminal and can even change runtime directory, or even the root filesystems. In short, it does everything so the Twisted application developer can concentrate on writing his networking code.
Now that port 1079 is free, maybe we can run on it a different server, one which will let people set their messages. It does no access control, so anyone who can login to the machine can set any message. We assume this is the desired behavior in our case. Testing it can be done by simply:
% nc localhost 1079 moshez Giving a talk now, sorry! ^D
The previous version had the setter poke at the innards of the finger factory. It's usually not a good idea: this version makes both factories symmetric by making them both look at a single object. Services are useful for when an object is needed which is not related to a specific network server. Here, we moved all responsibility for manufacturing factories into the service. Note that we stopped subclassing: the service simply puts useful methods and attributes inside the factories. We are getting better at protocol design: none of our protocol classes had to be changed, and neither will have to change until the end of the talk.
This version shows how, instead of just letting users set their messages, we can read those from a centrally managed file. We cache results, and every 30 seconds we refresh it. Services are useful for such scheduled tasks.
The same kind of service can also produce things useful for other protocols. For example, in twisted.web, the factory itself (the site) is almost never subclassed -- instead, it is given a resource, which represents the tree of resources available via URLs. That hierarchy is navigated by site, and overriding it dynamically is possible with getChild.
This is the first time there is client code. IRC clients often act a lot like servers: responding to events from the network. The reconnecting client factory will make sure that severed links will get re-established, with intelligent tweaked exponential back-off algorithms. The IRC client itself is simple: the only real hack is getting the nickname from the factory in connectionMade.
In Twisted, XML-RPC support is handled just as though it was another resource. That resource will still support GET calls normally through render(), but that is usually left unimplemented. Note that it is possible to return deferreds from XML-RPC methods. The client, of course, will not get the answer until the deferred is triggered.
The last version of the application had a lot of hacks. We avoided sub-classing, did not support things like user listings in the web support, and removed all blank lines -- all in the interest of code which is shorter. Here we take a step back, subclass what is more naturally a subclass, make things which should take multiple lines take them, etc. This shows a much better style of developing Twisted applications, though the hacks in the previous stages are sometimes used in throw-away prototypes.
In the last version, the service class was three times longer than any other class, and was hard to understand. This was because it turned out to have multiple responsibilities. It had to know how to access user information, by scheduling a reread of the file ever half minute, but also how to display itself in a myriad of protocols. Here, we used the component-based architecture that Twisted provides to achieve a separation of concerns. All the service is responsible for, now, is supporting getUser/getUsers. It declares its support via the __implements__ keyword. Then, adapters are used to make this service look like an appropriate class for various things: for supplying a finger factory to listenTCP, for supplying a resource to site's constructor, and to provide an IRC client factory for connectTCP. All the adapters use are the methods in FingerService they are declared to use: getUser/getUsers. We could, of course, skipped the interfaces and let the configuration code use things like FingerFactoryFromService(f) directly. However, using interfaces provides the same flexibility inheritance gives: future subclasses can override the adapters.
class MemoryFingerService(app.ApplicationService): __implements__ = IFingerService, IFingerSetterService def __init__(self, *args, **kwargs): app.ApplicationService.__init__(self, *args) self.users = kwargs def getUser(self, user): return defer.succeed(self.users.get(u, "No such user")) def getUsers(self): return defer.succeed(self.users.keys()) def setUser(self, user, status): self.users[user] = status application = app.Application('finger', uid=1, gid=1) # New constructor call f = MemoryFingerService(application, 'finger', moshez='Happy and well') application.listenTCP(79, IFingerFactory(f)) application.listenTCP(80, server.Site(resource.IResource(f))) i = IIRCClientFactory(f) i.nickname = 'fingerbot' application.connectTCP('irc.freenode.org', 6667, i) # New: run setter too application.listenTCP(1079, IFingerSetterFactory(f), interface='127.0.0.1')
Here we show just how convenient it is to implement new back-ends when we move to a component based architecture. Note that here we also use an interface we previously wrote, FingerSetterFactory, by supporting one single method. We manage to preserve the service's ignorance of the network.
class LocalFingerService(app.ApplicationService): __implements__ = IFingerService def getUser(self, user): return utils.getProcessOutput("finger", [user]) def getUsers(self): return defer.succeed([]) application = app.Application('finger', uid=1, gid=1) f = LocalFingerService(application, 'finger') application.listenTCP(79, IFingerFactory(f)) application.listenTCP(80, server.Site(resource.IResource(f))) i = IIRCClientFactory(f) i.nickname = 'fingerbot' application.connectTCP('irc.freenode.org', 6667, i)
We have already wrote this, but now we get more for less work: the network code is completely separate from the back-end.
import pwd class LocalFingerService(app.ApplicationService): __implements__ = IFingerService def getUser(self, user): try: entry = pwd.getpwnam(user) except KeyError: return "No such user" try: f=file(os.path.join(entry[5],'.plan')) except (IOError, OSError): return "No such user" data = f.read() f.close() return data def getUsers(self): return defer.succeed([]) application = app.Application('finger', uid=1, gid=1) f = LocalFingerService(application, 'finger') application.listenTCP(79, IFingerFactory(f)) application.listenTCP(80, server.Site(resource.IResource(f))) i = IIRCClientFactory(f) i.nickname = 'fingerbot' application.connectTCP('irc.freenode.org', 6667, i)
Not much to say about that, except to indicate that by now we can be churning out back-ends like crazy. Feel like doing a back-end for Advogato, for example? Dig out the XML-RPC client support Twisted has, and get to work!
At last, an example of aspect-oriented programming that isn't about logging or timing. This code is actually useful! Watch how aspect-oriented programming helps you write less code and have fewer dependencies!
Here we convert to using Woven, instead of manually constructing HTML snippets. Woven is a sophisticated web templating system. Its main features are to disallow any code inside the HTML, and transparent integration with deferred results.
We add support for perspective broker, Twisted's native remote object protocol. Now, Twisted clients will not have to go through XML-RPCish contortions to get information about users.
All we need to do to code an HTTPS site is just write a context factory (in this case, which loads the certificate from a certain file) and then use the listenSSL method. Note that one factory (in this case, a site) can listen on multiple ports with multiple protocols.
class FingerClient(protocol.Protocol): def connectionMade(self): self.transport.write(self.factory.user+"\r\n") self.buf = [] def dataReceived(self, data): self.buf.append(data) def connectionLost(self): self.factory.gotData(''.join(self.buf)) class FingerClientFactory(protocol.ClientFactory): protocol = FingerClient def __init__(self, user): self.user = user self.d = defer.Deferred() def clientConnectionFailed(self, _, reason): self.d.errback(reason) def gotData(self, data): self.d.callback(data) def finger(user, host, port=79): f = FingerClientFactory(user) reactor.connectTCP(host, port, f) return f.d class ProxyFingerService(app.ApplicationService): __implements__ = IFingerService def getUser(self, user): user, host = user.split('@', 1) ret = finger(user, host) ret.addErrback(lambda _: "Could not connect to remote host") return ret def getUsers(self): return defer.succeed([]) application = app.Application('finger', uid=1, gid=1) f = ProxyFingerService(application, 'finger') application.listenTCP(79, IFingerFactory(f))
Writing new clients with Twisted is much like writing new servers. We implement the protocol, which just gathers up all the data, and give it to the factory. The factory keeps a deferred which is triggered if the connection either fails or succeeds. When we use the client, we first make sure the deferred will never fail, by producing a message in that case. Implementing a wrapper around client which just returns the deferred is a common pattern. While being less flexible than using the factory directly, it is also more convenient.
Now this code, while being quite modularized and well-designed, isn't
properly organized. Everything above the application=
belongs in a
module, and the HTML templates all belong in separate files.
We can use the templateFile and templateDirectory attributes to indicate what HTML template file to use for each Page, and where to look for it.
from twisted.internet import app from finger import FingerService, IIRCclient, ServerContextFactory, \ IFingerFactory, IPerspectiveFinger from twisted.web import resource, server from twisted.spread import pb application = app.Application('finger', uid=1, gid=1) f = FingerService('/etc/users', application, 'finger') application.listenTCP(79, IFingerFactory(f)) r = resource.IResource(f) r.templateDirectory = '/usr/share/finger/templates/' site = server.Site(r) application.listenTCP(80, site) application.listenSSL(443, site, ServerContextFactory()) i = IIRCClientFactory(f) i.nickname = 'fingerbot' application.connectTCP('irc.freenode.org', 6667, i) application.listenTCP(8889, pb.BrokerFactory(IPerspectiveFinger(f))
Note that our program is now quite separated. We have:
We can also supply easy configuration for common cases
# in finger.py module def updateApplication(app, **kwargs): f = FingerService(kwargs['users'], application, 'finger') application.listenTCP(79, IFingerFactory(f)) r = resource.IResource(f) r.templateDirectory = kwargs['templates'] site = server.Site(r) app.listenTCP(80, site) if kwargs.get('ssl'): app.listenSSL(443, site, ServerContextFactory()) if kwargs.has_key('ircnick'): i = IIRCClientFactory(f) i.nickname = kwargs['ircnick'] ircServer = kwargs['ircserver'] application.connectTCP(ircserver, 6667, i) if kwargs.has_key('pbport'): application.listenTCP(int(kwargs['pbport']), pb.BrokerFactory(IPerspectiveFinger(f))
And we can write simpler files now:
# simple-finger.tpy from twisted.internet import app import finger application = app.Application('finger', uid=1, gid=1) finger.updateApplication(application, users='/etc/users', templatesDirectory='/usr/share/finger/templates', ssl=1, ircnick='fingerbot', ircserver='irc.freenode.net', pbport=8889 )
Note: the finger user still has ultimate power: he can use updateApplication, or he can use the lower-level interface if he has specific needs (maybe an IRC server on some other port? maybe we want the non-SSL webserver to listen only locally? etc. etc.) This is an important design principle: never force a layer of abstraction: allow usage of layers of abstractions.
The pasta theory of design:
So far, the user had to be somewhat of a programmer to be able to configure stuff. Maybe we can eliminate even that? Move old code to "finger/service.py", put in an empty "__init__.py" and...
# finger/tap.py from twisted.python import usage from finger import service class Options(usage.Options): optParams = [ ['users', 'u', '/etc/users'], ['templatesDirectory', 't', '/usr/share/finger/templates'], ['ircnick', 'n', 'fingerbot'], ['ircserver', None, 'irc.freenode.net'], ['pbport', 'p', 8889], ] optFlags = [['ssl', 's']] def updateApplication(app, config): service.updateApplication(app, **config)
And register it all:
#finger/plugins.tml register('Finger', 'finger.tap', type='tap', tapname='finger')
And now, the following works
% mktap finger --users=/usr/local/etc/users --ircnick=moshez-finger % sudo twistd -f finger.tap
If we already have the "finger" package installed, we can achieve easy integration:
% tap2deb --unsigned -m "Foo <foo@example.com>" --type=python finger.tpy % sudo dpkg -i .build/*.deb
% tap2rpm --type=python finger.tpy #[maybe other options needed] % sudo rpm -i .build/*.rpm
Will properly register configuration files, init.d sripts, etc. etc.
If it doesn't work on your favorite OS: patches accepted!
This document originally authored by Moshe Zadka.
Edited by Jonathan Lange