Writing asynchronous Python code with Twisted using inlineCallbacks

A few weeks ago, I started using twisted to develop a plugin for synchronization between Stoq and Magento.

Twisted is a really great tool, and it makes it very easy to write asynchronous code. The only problem is that your code has a high probability of becoming Spaghetti code :P. And anyone who knows me knows that I’m crazy when it comes to code organization.

Well, those days, specially after a hint by a work mate, Johan Dahlin, I started to take a look on inlineCallbacks decorator. It’s just beautiful and solve all my problems :).

Since I found the documentation a lot hard to understand, and very few examples on the web, I decided to try to make one of my own. Hope you will enjoy!

To start, consider the following piece of code (using the classic Twisted way):

from twisted.internet import reactor
 
def on_failure(err):
    print "Error:", err
    reactor.stop()
 
def on_success(*args):
    print "Success. Shutting down"
    reactor.stop()
 
def print_file(file_):
    d = async_print_file(file_) # This will return a Deferred
 
    d.addCallback(on_success)
    d.addErrback(on_failure)
 
def get_file():
    d = async_get_file() # This will return a Deferred
 
    d.addCallback(print_file)
    d.addErrback(on_failure)
 
if __name__ == '__main__':
    get_file()
    reactor.run()

On this example, we want to get a file, print it, and then shutdown the application. Yeah, it’s ugly, a little spaghetti (could be a lot more if the code wasn’t a simple example)… but it works.

Just for a fast explanation, the async_*() functions are fictitious functions that will return a Deferred. When it’s fired, it’ll call the function added by addCallback, or, in case of failure, the one added by addErrback. If more than one callback (or errback) is added, when the first one returns, that return value will be passed to the second function, and so on, as a chain of callbacks.

Now, take a look at the following piece of code (utilizing the inlineCallbacks way):

from twisted.internet import defer, reactor
 
@defer.inlineCallbacks
def print_file():
    try:
        # async_get_file still returns a Deferred
        file_ = yield async_get_file() # After yield, it's not a Deferred anymore
        yield async_print_file(file_)
        print "Success."
    except Exception as err:
        print "Error", err
    finally:
        print "Shutting down"
        reactor.stop()
 
if __name__ == '__main__':
    print_file()
    reactor.run()

Beautiful, isn’t it? Well, not really the most beautiful piece of code you will see around the world, but better than the previous example. What does all the magic is the yield statement (without using it, file_ would still be a Deferred)

When the code inside a function decorated by the inlineCallbacks decorator yields a Deferred (in that case, a function that returns a Deferred), the code goes on and the reactor will come back after the Deferred fires. It’s return value will be returned on the yield statement, and, if any errors occurred, the exception will be raised (that’s why I yielded inside a try/except clause).

Note that, because yield is captured by inlineCallbacks, there’s no way to use that function as an iterator generator.

And if we need to call another function decorated by inlineCallbacks? How to get it’s return value, as the return statement won’t work? Well, that’s why there is a function called returnValue. Take a look at this piece of code:

from twisted.internet import defer, reactor
 
@defer.inlineCallbacks
def get_arg():
    retval = yield another_async_func()
    defer.returnValue(retval)
 
@defer.inlineCallbacks
def print_file():
    try:
        arg = yield get_arg()
    except Exception as err:
        arg = None
 
    try:
        file_ = yield async_get_file(arg)
        yield async_print_file(file_)
        print "Success."
    except Exception as err:
        print "Error", err
    finally:
        print "Shutting down"
        reactor.stop()
 
if __name__ == '__main__':
    print_file()
    reactor.run()

In this example, we assumed that async_get_file needed an expecific argument, that needs to be retrieved asynchronous too. By doing returnValue(arg), we make anyone who yields get_arg() to receive arg, or raise an exception if an error occour.

A little complicated but, after a while you get used to it! icon wink Writing asynchronous Python code with Twisted using inlineCallbacks

Any doughs?

12 thoughts on “Writing asynchronous Python code with Twisted using inlineCallbacks”

  1. I agree, a great blog post. I’ve written some Twisted applications in the past in the “traditional” style and always struggled to get it right. And to be able to read/understand what I did when I went back to the program later. The @inlineCallbacks style is much clearer to me, and your post really helped me see how to use it better.
    Thanks!
    Doug

  2. I am a twisted neophyte, so perhaps my observation is wrong, but it would see that the traditional way allows a programmer to mix asynchronous with synchronous code, where as with the inlineCallback every caller has to implement the same generator patter by yield’ing the next call?

    1. Not necessarily. The inlineCallbacks is just a way to receive the Deferred return value when it fires (using yield), instead of having to connect it to another callable (and with lots of callbacks, becoming spaghetti code).

      For example, on first example, you just called get_file(). It just creates the Deferred and connects it’s callbacks. When reactor.run() is called, everything starts to happen.

      Now, look at the last example. You call print_file() from place not implementing the generator. On the first yield, the code returns and them reactor.run() is called. After that, when the Deferred there fires, the code will continue on the print_file function, just like if you connected it there.

      If you don’t implement inlineCallbacks decorator, you can’t use the yield statement. The only thing the yield does inside it is to wait for the Deferred to be fired. If you call a callable implementing inlineCallbacks (from inside one implementing it or not) and not use yield, it would just be a deferred and them you could work in the traditional way with it (connecting callbacks and etc).

      I don’t know if I answered your question. I’m not a “master” of twisted too =P. Just ask if you need more help!

  3. I was going through bunch of defer.inlineCallbacks in buildbot src and came across this
    Thanks a lot. It really helped!

  4. Hi, i feel that i saw you visited my website thus i got here to
    go back the desire?.I’m trying to to find things to enhance my website!I guess its good enough to make use of some of your ideas!!

  5. In fact a good company won’t be able to complete the entire job alone. It is easy to over-accumulate furnishings you actually don’t need over the years. This could assist you in saving a lot of money and time both.

Leave a Reply