I was reading about introspection in Python 2.7 and came across code similar to what is shown in Figure 1: class Bag. It turns out that it is a very interesting passage of code for me as it uses many of what I would consider intermediate Python techniques. In Part I of this blog post, I will cover some of the interesting Python constructs that crossed my mind when looking at the code. Part II is a short explanation of introspection in Python 2.7.

1 class Bag:
2     def __init__(self, **d):
3         for k,v in d.iteritems():
4             exec("self.%s = %s" % (k,v))

Figure 1: class Bag (figure numbers are only for reference)

This code as written does not execute in Python 3. ROI is in the process of converting our Python courses to Python 3 as Ubuntu 16.04 has Python 3 as the default version. Hopefully, there will be a few blog posts on our experiences.

Part I: Looking at and Using the Passage
Line 3 contains a **d variable. This is a keyword value gather variable. The code below is used to show how it works.

1 #! /usr/bin/env python
2
3 def afun(**d):
4     print "type of d: ", type(d)
5     for key in d.keys():
6         print key, '=', d[key]
7
8 afun(l = 5, m = 'this', n = [1,2,3])

Figure 2: keyword.py

If you execute keyword.py, you see:

$ ./keyword.py
type of d: <type 'dict'>
m = 'this'
l = 5
n = [1, 2, 3]
$

As you can see, **d creates a dictionary.

Here is the class Bag again.

1 class Bag:
2     def __init__(self, **d):
3         for k,v in d.iteritems():
4             exec("self.%s = %s" % (k,v))

Figure 3: class Bag

It is the exec() function of line 4 that is of interest. Why not an eval?

The syntax for eval is:

eval(string, [[global], [local]])

The string is an expression to be parsed. The value of the parsed expression is returned. Example is below.

In [1]: a = eval('5 + 3')
In [2]: print a
8

global is a dictionary which is used as the global namespace. If __builtins__ is not in the dictionary, the current globals are copied in before the string (expression) is parsed. The key/valued pairs defined in globals have precedence. If global is omitted, the global namespace of the script is used.

local is another dictionary with local definitions.

If the local dictionary is omitted, it defaults to the global dictionary. If both dictionaries are omitted, the environment where the eval is called is used.

Below is an example of eval using the global dictionary.

In [1]: x = 100
In [2]: a = eval( 'x + 10', { 'x':20 })
In [3]: print a
30

Notice that the key/value pair in the global dictionary has precedence over a value declared before the eval in the script, data from global namespace.

Here is the exec statement again:

exec("self.%s = %s" % (k,v))

Here I am using the tuple form for exec because it makes for a very easy translation to Python 3 where exec is a function.

The general format for the tuple format is:

exec( expression [, global [, local]])

The expression can be a string, a file containing a script, or a code object. global and local are the same as in eval. As a statement, exec() does not return a value. The changes are made as side effects of the execution of the code in the expression. Below is a simple example.

In [1]: exec("%s = %d" % ("y", 25))
In [2]: print y
25

In the class Bag, the exec statement can add new attributes at the time the object is created. This is the original reason for looking into introspection.

I wanted to bring the Bag class definition into iPython. This blog post explores three different methods of making code in a file (a module) available to a script.

Approach 1:
I could use:

import bag

and then accessing the class Bag as:

a = bag.Bag(l = "'this'", m = 5, n = [1, 2, 3])

And executing:

In [1]: import bag
In [2]: a = bag.Bag( l = "'this'", m = 5, n = [1, 2, 3])
In [3]: a.l
Out[3]: 'this'
In [4]: a.m
Out[4]: 5
In [5]: a.n
Out[5]: [1, 2, 3]

On In [2], notice the "'this'". This is required by the exec statement used in line 4 of Figure 1 above. This will be covered after accessing the module.

Approach 2:
I could use:

from bag import Bag

and then accessing the class Bag as:

a = Bag(l = "'this'", m = 5, n = [1, 2, 3])

and executing:

In [1]: from bag import Bag
In [2]: a = Bag(l = "'this'", m = 5, n = [1, 2, 3])
In [3]: a.l
Out[3]: 'this'
In [4]: a.m
Out[4]: 5
In [5]: a.n
Out[5]: [1, 2, 3]

These two methods are in common use and are working here exactly as expected.

Approach 3:
There is a third way using the built-in execfile().

The execfile() is closer to a C/C++ #include concept. Each time the execfile() function is executed, the file is accessed, parsed, and executed. This is done because the Python interpreter does not have control over the contents of the file and must assume that it has changed.

Using the builtin function execfile():

execfile("./bag.py")

and using class Bag:

a = Bag(l = "'this'", m = 5, n = [1, 2, 3])

and executing:

In [1]: execfile("./bag.py")
In [2]: a = Bag(l = "'this'", m = 5, n = [1, 2, 3])
In [3]: a.l
Out[3]: 'this'
In [4]: a.m
Out[4]: 5
In [5]: a.n
Out[5]: [1, 2, 3]

The function execfile() parses the file each time it is called. This is different from the import statement, which only executes the module (file) the first time it is called, creating a .pyc file which will be used for the other calls.

Figure 4 is a very short program using execfile().

1 #! /usr/bin/env python
2
3 execfile("./bag.py")
4
5 a = Bag(l = "'this'", m = 5, n = [1, 2, 3])
6
7 print a.l
8 print a.m
9 print a.n

Figure 4: exefile.py

Figure 5 shows the directory before execution, execution, and the directory after execution for execfile.py.

$ ls
bag.py       execfile.py keyword.py       module.py
$ ./execfile.py
this
5
[1, 2, 3]
$ ls
bag.py       execfile.py keyword.py       module.py
Arthurs-MacBook-Pro:Python arthur$

Figure 5: Directory listing and execution of exefile.py

Notice that no .pyc file is generated for bag.py.

Figures 6 and 7 show the same sequence for module.py. Notice the .pyc generated by the import statement.

1 #! /usr/bin/env python
2
3 from bag import Bag
4
5 a = Bag(l = "'this'", m = 5, n = [1, 2, 3])
6
7 print a.l
8 print a.m
9 print a.n

Figure 6: module.py

$ ls
bag.py       execfile.py keyword.py       module.py
$ ./module.py
this
5
[1, 2, 3]
$ ls
bag.py       bag.pyc execfile.py keyword.py       module.py

Figure 7: Directory listing and execution of module.py

Using execfile() inside a module also prevents import from creating a .pyc when the module is called. Not usually a problem unless you have initialization code that is designed to only be run once.

Lastly, the first parameter to execfile() is the path to the file to be parsed. There are a second and third parameter available. The second parameter is a dictionary of global variables to be used for the parsing and executing of the file. The third parameter only makes sense when execfile() is used inside a class or function, and is a dictionary of local variables. These work the same in the eval() command.

Part II: Introspection
As it turned out, the tools for introspection are very easy to understand. I really haven’t come across a good use case for using a class like Bag and then introspection to see what the object contains.

In [1]: execfile("bag.py")

bag.py has the definition of the class Bag.

1 class Bag:
2     def __init__(self, **d):
3         for k,v in d.iteritems():
4             exec("self.%s = %s" % (k,v))

In [2]: a = Bag(l = "'this'", m = 5, n = [1, 2, 3])

This creates the object a that is of type Bag.

In [3]: getattr(a, 'l')
Out[3]: 'this'

The syntax is:

getattr(object,
    attribute_name_as_string
    [, default_value])

In [4]: getattr(a, 'Q')
- - - - - - - - - - - - - - - - - - - - - - - - - - - - -
AttributeError          Traceback (most recent call last)
<ipython-input-4-47aea6ca945c> in <module>()
- - - -> 1 getattr(a, 'Q')

AttributeError: Bag instance has no attribute 'Q'

The error can be trapped in a try except block. Using the default value available in getattr is faster.

In [5]: getattr(a, 'Q', "Not found!")
Out[5]: 'Not found!'

Instead of the string, a boolean could have been returned.

In [6]: hasattr(a, 'Q')
Out[6]: False

Returns True if object has attribute, and False otherwise.

In [7]: setattr(a, 'Q', "Added!")

This adds a new attribute. The syntax is:

settattr(object,
    attribute_name_as_string,
    value)

In [8]: getattr(a,'Q')
Out[8]: 'Added!'

Just showing that it was added.

In [9]: setattr(a, 'l', "THIS")

This is done to show that you can modify an attribute of the object.

In [10]: getattr(a, 'l')
Out[10]: 'THIS'

Confirmation that the attribute was changed.

Leave a Reply

Your email address will not be published.