Python Closures and Decorators (Pt. 2)

In Python Closures and Decorators (Pt. 1), we looked at sending functions as arguments to other functions, at nesting functinons, and finally we wrapped a function in another function. We’ll begin this part by giving an example implementation on the exercise I gave in part 1:

>>> def print_call(fn):
...   def fn_wrap(*args, **kwargs):
...     print("Calling %s with arguments: \n\targs: %s\n\tkwargs:%s" % (
...            fn.__name__, args, kwargs))
...     retval = fn(*args, **kwargs)
...     print("%s returning '%s'" % (fn.func_name, retval))
...     return retval
...   fn_wrap.func_name = fn.func_name
...   return fn_wrap
...
>>> def greeter(greeting, what='world'):
...     return "%s %s!" % (greeting, what)
...
>>> greeter = print_call(greeter)
>>> greeter("Hi")
Calling greeter with arguments:
 args: ('Hi',)
 kwargs:{}
greeter returning 'Hi world!'
'Hi world!'

>>> greeter("Hi", what="Python")
Calling greeter with arguments:
 args: ('Hi',)
 kwargs:{'what': 'Python'}
greeter returning 'Hi Python!'
'Hi Python!'
>>>

So, this is at least mildly useful, but it’ll get better! You may or may not have heard of closures, and you may have heard any of a large number of defenitions of what a closure is – I won’t go into nitpicking, but just say that a closure is a block of code (for example a function) that captures (or closes over) non-local (free) variables. If this is all gibberish to you, you’re probably in need of a CS refresher, but fear not – I’ll show by example, and the concept is easy enough to understand: a function can reference variables that are defined in the function’s enclosing scope.

For example, take a look at this code:

>>> a = 
>>> def get_a():
...   return a
...
>>> get_a()

>>> a = 3
>>> get_a()
3

As you can see, the function get_a can get the value of a, and will be able to read the updated value. However, there is a limitation – a captured variable cannot be written to:

>>> def set_a(val):
...   a = val
...
>>> set_a(4)
>>> a
3

What happened here? Since a closure cannot write to any captured variables, a = val actually writes to a local variable a that shadows the module-level a that we wanted to write to. To get around this limitation (which may or may not be a good idea), we can use a container type:

>>> class A(object): pass
...
>>> a = A()
>>> a.value = 1
>>> def set_a(val):
...   a.value = val
...
>>> a.value
1
>>> set_a(5)
>>> a.value
5

So, with the knowledge that a function captures variables from it’s enclosing scope, we’re finally approaching something interesting, and we’ll start by implementing a partial. A partial is an instance of a function where you have already filled in some or all of the arguments; let’s say, for example that you have a session with username and password stored, and a function that queries some backend layer which takes different arguments but always require credentials. Instead of passing the credentials manually every time, we can use a partial to pre-fill those values:

>>> #Our 'backend' function
... def get_stuff(user, pw, stuff_id):
...   """Here we would presumably fetch data using the supplied
...   credentials and id"""
...   print("get_stuff called with user: %s, pw: %s, stuff_id: %s" % (
...         user, pw, stuff_id))
>>> def partial(fn, *args, **kwargs):
...   def fn_part(*fn_args, **fn_kwargs):
...     kwargs.update(fn_kwargs)
...     return fn(*args + fn_args, **kwargs)
...   return fn_part
...
>>> my_stuff = partial(get_stuff, 'myuser', 'mypwd')
>>> my_stuff(3)
get_stuff called with user: myuser, pw: mypwd, stuff_id: 3
>>> my_stuff(67)
get_stuff called with user: myuser, pw: mypwd, stuff_id: 67

Partials can be used in numerous places to remove code duplication where a function is called in different places with the same, or almost the same, arguments. Of course, you don’t have to implement it yourself; just do from functools import partial.

Finally, we’ll take a look at function decorators (there may be a post on class decorators in the future). A function decorator is (can be implemented as) a function that takes a function as parameter and returns a new function. Sounds familiar? It should, because we’ve already implemented a working decorator: our print_call function is ready to be used as-is:

>>> @print_call
... def will_be_logged(arg):
...   return arg*5
...
>>> will_be_logged("!")
Calling will_be_logged with arguments:
 args: ('!',)
 kwargs:{}
will_be_logged returning '!!!!!'
'!!!!!'

Using the @-notation is simply a convenient shorthand to doing:

>>> def will_be_logged(arg):
...   return arg*5
...
>>> will_be_logged = print_call(will_be_logged)

But what if we want to be able to parameterize the decorator? In this case, the function used as a decorator will received the arguments, and will be expected to return a function that wraps the decorated function:

>>> def require(role):
...   def wrapper(fn):
...     def new_fn(*args, **kwargs):
...       if not role in kwargs.get('roles', []):
...         print("%s not in %s" % (role, kwargs.get('roles', [])))
...         raise Exception("Unauthorized")
...       return fn(*args, **kwargs)
...     return new_fn
...   return wrapper
...
>>> @require('admin')
... def get_users(**kwargs):
...   return ('Alice', 'Bob')
...
>>> get_users()
admin not in []
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 7, in new_fn
Exception: Unauthorized
>>> get_users(roles=['user', 'editor'])
admin not in ['user', 'editor']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 7, in new_fn
Exception: Unauthorized
>>> get_users(roles=['user', 'admin'])
('Alice', 'Bob')

…and there you have it. You are now ready to write decorators, and perhaps use them to write aspect-oriented Python; adding @cache, @trace, @throttle are all trivial (and before you add @cache, do check functools once more if you’re using Python 3!).