Thiago Bellini's homepage Free software development, Open source software, Linux – by Thiago Bellini

Writing asynchronous Python code with Twisted using inlineCallbacks

A few weeks ago, I started using Twisted for developing a plugin for synchronization between Stoq and Magento.

Twisted is really a great tool, and it makes it very easy to write asynchronous code. The only problem is that your code has a high probability of becoming Spaghetti code . And anyone who knows me knows that I'm crazy when it comes to code organization.

Well, those days, specially after a hint by a work mate, Johan Dahlin, I started to take a look on inlineCallbacks decorator. It's just beautiful and solve all my problems .

Since I found the documentation a lot hard to understand, and little examples on the web, I decided to try to make one of my own. Hope you will enjoy!

Now, consider the following piece of code (utilizing the classic Twisted way):

from twisted.internet import reactor
 
def on_failure(err):
    print "Error:", err
    reactor.stop()
 
def on_success(*args):
    print "Success. Shutting down"
    reactor.stop()
 
def print_file(file_):
    d = async_print_file(file_) # This will return a Deferred
 
    d.addCallback(on_success)
    d.addErrback(on_failure)
 
def get_file():
    d = async_get_file() # This will return a Deferred
 
    d.addCallback(print_file)
    d.addErrback(on_failure)
 
if __name__ == '__main__':
    get_file()
    reactor.run()

On this example, we want to get a file, print it, and then shutdown the application. Yeah, it's ugly, a little spaghetti (could be a lot more if the code wasn't a simple example)...but it works.

Just for a fast explanation, the async_*() functions are fictitious functions that will return a Deferred. When it's fired, it'll call the function added by addCallback, or, in case of failure, the one added by addErrback. If more than one callback (or errback) is added, when the first one returns, that return value will be passed to the second function, and so on, as a chain of callbacks.

Now, take a look at the following piece of code (utilizing the inlineCallbacks way):

from twisted.internet import defer, reactor
 
@defer.inlineCallbacks
def print_file():
    try:
        file_ = yield async_get_file() # This will return a Deferred
        yield async_print_file(file_)
        print "Success."
    except Exception as err:
        print "Error", err
    finally:
        print "Shutting down"
        reactor.stop()
 
if __name__ == '__main__':
    print_file()
    reactor.run()

Beautiful, isn't it? What does all the magic is the yield statement (without using it, file_ would still be a Deferred)

When the code inside a function decorated by the inlineCallbacks decorator yields a Deferred (in that case, a function that returns a Deferred), the code goes on and the reactor will come back after the Deferred fires. It's return value will be returned on the yield statement, and, if any errors occurred, the exception will be raised (that's why I yielded inside a try/except clause).

Note that, because yield is captured by inlineCallbacks, there's no way to use that function as an iterator generator.

And if we need to call another function decorated by inlineCallbacks? How to get it's return value, as the return statement won't work? Well, that's why there is a function called returnValue. Take a look at the following piece of code:

from twisted.internet import defer, reactor
 
@defer.inlineCallbacks
def get_arg():
    retval = yield another_sync_func()
    defer.returnValue(retval)
 
@defer.inlineCallbacks
def print_file():
    try:
        arg = yield get_arg()
    except Exception as err:
        arg = None
 
    try:
        file_ = yield async_get_file(arg)
        yield async_print_file(file_)
        print "Success."
    except Exception as err:
        print "Error", err
    finally:
        print "Shutting down"
        reactor.stop()
 
if __name__ == '__main__':
    print_file()
    reactor.run()

In this example, we assumed that async_get_file neeeded an expecific argument, that needs to be retrieved asynchronous too. By doing returnValue(arg), we make anyone who yields get_arg() to receive arg, or raise an exception if an error occour.

A little complicated but, after a while you get used to it!

Any doughs?