Writing asynchronous Python code with Twisted using inlineCallbacks

10Aug/119

Writing asynchronous Python code with Twisted using inlineCallbacks

A few weeks ago, I started using Twisted for developing a plugin for synchronization between Stoq and Magento.

Twisted is really a great tool, and it makes it very easy to write asynchronous code. The only problem is that your code has a high probability of becoming Spaghetti code . And anyone who knows me knows that I'm crazy when it comes to code organization.

Well, those days, specially after a hint by a work mate, Johan Dahlin, I started to take a look on inlineCallbacks decorator. It's just beautiful and solve all my problems .

Since I found the documentation a lot hard to understand, and little examples on the web, I decided to try to make one of my own. Hope you will enjoy!

Now, consider the following piece of code (utilizing the classic Twisted way):

from twisted.internet import reactor
 
def on_failure(err):
    print "Error:", err
    reactor.stop()
 
def on_success(*args):
    print "Success. Shutting down"
    reactor.stop()
 
def print_file(file_):
    d = async_print_file(file_) # This will return a Deferred
 
    d.addCallback(on_success)
    d.addErrback(on_failure)
 
def get_file():
    d = async_get_file() # This will return a Deferred
 
    d.addCallback(print_file)
    d.addErrback(on_failure)
 
if __name__ == '__main__':
    get_file()
    reactor.run()

On this example, we want to get a file, print it, and then shutdown the application. Yeah, it's ugly, a little spaghetti (could be a lot more if the code wasn't a simple example)...but it works.

Just for a fast explanation, the async_*() functions are fictitious functions that will return a Deferred. When it's fired, it'll call the function added by addCallback, or, in case of failure, the one added by addErrback. If more than one callback (or errback) is added, when the first one returns, that return value will be passed to the second function, and so on, as a chain of callbacks.

Now, take a look at the following piece of code (utilizing the inlineCallbacks way):

from twisted.internet import defer, reactor
 
@defer.inlineCallbacks
def print_file():
    try:
        # async_get_file still returns a Deferred
        file_ = yield async_get_file() # After yield, it's not a Deferred anymore
        yield async_print_file(file_)
        print "Success."
    except Exception as err:
        print "Error", err
    finally:
        print "Shutting down"
        reactor.stop()
 
if __name__ == '__main__':
    print_file()
    reactor.run()

Beautiful, isn't it? What does all the magic is the yield statement (without using it, file_ would still be a Deferred)

When the code inside a function decorated by the inlineCallbacks decorator yields a Deferred (in that case, a function that returns a Deferred), the code goes on and the reactor will come back after the Deferred fires. It's return value will be returned on the yield statement, and, if any errors occurred, the exception will be raised (that's why I yielded inside a try/except clause).

Note that, because yield is captured by inlineCallbacks, there's no way to use that function as an iterator generator.

And if we need to call another function decorated by inlineCallbacks? How to get it's return value, as the return statement won't work? Well, that's why there is a function called returnValue. Take a look at the following piece of code:

from twisted.internet import defer, reactor
 
@defer.inlineCallbacks
def get_arg():
    retval = yield another_async_func()
    defer.returnValue(retval)
 
@defer.inlineCallbacks
def print_file():
    try:
        arg = yield get_arg()
    except Exception as err:
        arg = None
 
    try:
        file_ = yield async_get_file(arg)
        yield async_print_file(file_)
        print "Success."
    except Exception as err:
        print "Error", err
    finally:
        print "Shutting down"
        reactor.stop()
 
if __name__ == '__main__':
    print_file()
    reactor.run()

In this example, we assumed that async_get_file neeeded an expecific argument, that needs to be retrieved asynchronous too. By doing returnValue(arg), we make anyone who yields get_arg() to receive arg, or raise an exception if an error occour.

A little complicated but, after a while you get used to it!

Any doughs?

Enjoy this article?

Consider subscribing to our rss feed!

Tagged as: async, asynchronous, python, twisted Leave a comment

Comments (9) Trackbacks (1) ( subscribe to comments on this post )

Tobias
August 15th, 2011 - 15:20

Great blog post! Twisted is sophisticated, and once I got my mind wrapped to the Twisted way, just everything fits. It lacks docs though, which is why nuggets as this post are welcome;)

If you’re interested in WebSockets and Twisted producer/consumer stuff, checkout this http://www.tavendo.de/autobahn/tutorial/producer.html

Or on GitHub https://github.com/oberstet/Autobahn ..

( REPLY )
Doug Farrell
February 29th, 2012 - 11:49

I agree, a great blog post. I’ve written some Twisted applications in the past in the “traditional” style and always struggled to get it right. And to be able to read/understand what I did when I went back to the program later. The @inlineCallbacks style is much clearer to me, and your post really helped me see how to use it better.
Thanks!
Doug

( REPLY )
Rick Copeland
July 28th, 2012 - 11:34

Great post! While I think I still prefer using gevent for my async code, this looks like a welcome alternative to writing callback spaghetti.

( REPLY )
Jon
September 7th, 2012 - 22:17

Great post.

A good YouTube video also explaining this @ http://www.youtube.com/watch?v=7Au2PDYD6Bk

( REPLY )
Leniy
November 12th, 2012 - 00:06

每次看到perl就想到了正则。苦恼

( REPLY )
Lev
January 1st, 2013 - 23:56

I am a twisted neophyte, so perhaps my observation is wrong, but it would see that the traditional way allows a programmer to mix asynchronous with synchronous code, where as with the inlineCallback every caller has to implement the same generator patter by yield’ing the next call?

( REPLY )
- hackedbellini
  January 2nd, 2013 - 12:22
  
  Not necessarily. The inlineCallbacks is just a way to receive the Deferred return value when it fires (using yield), instead of having to connect it to another callable (and with lots of callbacks, becoming spaghetti code).
  
  For example, on first example, you just called get_file(). It just creates the Deferred and connects it’s callbacks. When reactor.run() is called, everything starts to happen.
  
  Now, look at the last example. You call print_file() from place not implementing the generator. On the first yield, the code returns and them reactor.run() is called. After that, when the Deferred there fires, the code will continue on the print_file function, just like if you connected it there.
  
  If you don’t implement inlineCallbacks decorator, you can’t use the yield statement. The only thing the yield does inside it is to wait for the Deferred to be fired. If you call a callable implementing inlineCallbacks (from inside one implementing it or not) and not use yield, it would just be a deferred and them you could work in the traditional way with it (connecting callbacks and etc).
  
  I don’t know if I answered your question. I’m not a “master” of twisted too =P. Just ask if you need more help!
  
  ( REPLY )
Shriram
March 1st, 2013 - 12:06

I was going through bunch of defer.inlineCallbacks in buildbot src and came across this
Thanks a lot. It really helped!

( REPLY )
Dallas2012.com
May 21st, 2013 - 11:44

Hi, i feel that i saw you visited my website thus i got here to
go back the desire?.I’m trying to to find things to enhance my website!I guess its good enough to make use of some of your ideas!!

( REPLY )