Python Classes: inherit, override, extend

Here is a question that comes up from time to time when people start writing classes in Python: “I defined a derived class but it is not getting the data from the base class. What’s going on?”

Here is a bit of example code that shows this problem:

class A(object):
    def __init__(self):
        self.dataA = 'instance of class A'

class B(A):
    def __init__(self):
        self.dataB = 'instance of class B'

b = B()
print("b.dataA:", b.dataA)

But this apparently trivial code fails:

Traceback (most recent call last):
  File "inherit.py", line 34, in <module>
    print("b.adata:", b.adata)
AttributeError: 'B' object has no attribute 'adata'

Wait, what? I thought the derived class inherited from the base class, yet the variable adata set in the base class does not exist? The short answer is that is how classes work in Python. But the details end up needing a bit of explanation.

Viewing Class Inheritance Details

Before getting to the explanation, it is worth pointing out that you do not have to take an expert’s word for these things, it is pretty easy to just stick some diagnostic output into your script to examine what is happening. In this post I am going to be lazy and just use print(), but it is worth noting it is usually better to use Python’s logging module which gives you far more control of the output and where it goes. But logging is its own topic.

So here we will insert some diagnostics into the simple program:

class A(object):
    print('Setting class variables in A')
    clsdataA = 'class A'

    def __init__(self):
        print('Initializing instance of', self.__class__.__name__)
        self.dataA = 'instance of class A'

    def methA(self):
        return "A method from class A"

class B(A):
    print('Setting class variables in B')
    clsdataB = 'class B'

    def __init__(self):
        print('Initializing instance of', self.__class__.__name__)
        self.dataB = 'instance of class B'

print("Begin examination...")
print("Data from classes:")
print("A.clsdataA:", A.clsdataA)
print("B.clsdataA:", B.clsdataA)

print("Instantiating A as a:")
a = A()
print("Data from instance a:")
print("a.clsdataA:", a.clsdataA)
print("a.dataA:", a.dataA)
print("call methA directly from a:", a.methA())

print("Instantiating B as b:")
b = B()
print("Data from instance b:")
print("b.clsdataB:", b.clsdataB)
print("b.dataB:", b.dataB)
print("b.clsdataA:", b.clsdataA)
print("call methA from b:", b.methA())
print("b.dataA:", b.dataA)

When running this, we see:

Setting class variables in A    <1>
Setting class variables in B
Begin examination...
Data from classes:
A.clsdataA: class A             <2>
B.clsdataA: class A
Instantiating A as a:
Initializing instance of A      <3>
Data from instance a:
a.clsdataA: class A             <4>
a.dataA: instance of class A
call methA directly from a: A method from class A
Instantiating B as b:
Initializing instance of B      <5>
Data from instance b:
b.clsdataB: class B
b.dataB: instance of class B
b.clsdataA: class A             <6>
call methA from b: A method from class A
Traceback (most recent call last):
  File "./override.py", line 41, in <module>
    print("b.dataA:", b.dataA)
AttributeError: 'B' object has no attribute 'dataA'

1. The two assignments of class data happen before anything: a class definition is an executable statement, executed when reached in the file.

2. You can access the class data before any instances are created, as per the previous note.

3. The class A initializer is called when A is instantiated.

4. Data from an instance of the base class is as expected.

5. The class B initializer (only) is called when B is instantiated.

6. The class data from the base class and the method from the base class are inherited as expected.

So now we have a good idea what happened. Instantiating the derived class never called the __init__ method in the base class. That is because inheritance is child-oriented in Python. If the base class defines something and the derived class does not, the reference resolves to the one in the base class. But here we have defined an __init__ in the derived/child class, so the reference resolves to that. Think of a given class instance as a dictionary, where there’s one (and only one) mapping of each name in self to the object that will correspond to that name.

Python has a term for the way things are looked up, the Method Resolution Order (MRO). The MRO documentation is quite detailed because it deals also with complex multiple inheritance questions (something many OO languages just avoid by not allowing multiple inheritance at all), but the situation here is pretty simple: just walk up the tree, until you find a match; if a method is not found in any of the explicit defintions, it will pick the method from the object class, which is the base of all so-called New-Style Classes and so is there implicitly even if not listed. This "walk up" concept is very important and we’ll come back to it.

So after a lot of words, we can see how to "fix" our initial problem: make a call to the base class initializer if we think we need that initialization to happen:

class A(object):
    def __init__(self):
        self.dataA = 'instance of class A'

class B(A):
    def __init__(self):
        A.__init__(self)
        self.dataB = 'instance of class B'

b = B()
print(b.dataA)

After this simple addition, the output is as we might originally have expected, instead of the previous error:

instance of class A

The order matters, it might be worth trying the experiment again with the initializer call and the assignment to self.dataB swapped to see this. It means we can perform steps both before and after the call to the parent’s method, if we need to.

There is a subtlety here: because we are calling class A by name, rather than through an instance, the the __init__ method of A does not get automatically supplied with an instance reference and you would get an error if you did not supply it (specifically, TypeError: __init__() missing 1 required positional argument: 'self').

This behavior is not limited to the __init__ function, any method of the base class can be called, which means the derived class has the flexibility to tailor the behavior it wants: inherit from the base clase without doing anything, override the base class, or "extend" the base class by doing some local work before or after calling the base class method. You can even extend methods from builtin classes - the facility is by no means limited to your own classes.

The Method Resolution Order

If you are interested in the MRO, it can actually just be printed out. For the code above, add this line:

print(B.__mro__)

Which would give this response:

(<class '__main__.B'>, <class '__main__.A'>, <type 'object'>)

As noted earlier, this was a simple case with no surprises, so the order is simple.

Introspection

Python makes it easy to look inside objects to see what they look like. For example, to see data defined in an instance, we can print out the __dict__ attribute of the instance.

print("Dict:", b.__dict__)

Before adding the the call to the base class initializer the dictionary would look like:

Dict: {'dataB': 'instance of class B'}

Afterwards:

Dict: {'dataA': 'instance of class A', 'dataB': 'instance of class B'}

This actually shows more clearly than the previous words the difference between having called up to the base class initialization method and not: in the second case, base’s __init__ method has added an entry to the data dictionary of the child instance.

You can see all the defined symbols (methods and data) in the scope of the instance object by adding this:

print("dir:", dir(b))

The super Method

All of the above is pretty standard stuff about how Python classes work. There can always be surprises when you come from a familiar language to a new one and things look kind of similar but something is subtly different, but that is just part of learning.

Calling to a base class by name, however, may or may not be a good idea. It is very clear what you mean, but it is not very flexible. You hardcode a name; if you later change the definition of the derived class to inherit from some other class, you have to update any calls to the previous base class to update the name. In fact, for Python 3, a class which does not state a base class still has one: object; it is not terribly clear to a reader if you call a method by name in a base class but the base class does not visibly define it. Once you start playing with multiple inheritance then you have to do some work to figure out where the call is going - see the MRO sidebar. This is a potential code maintainability issue.

There is a tool to make that happen cleanly, the built in super function. In Python 3 super is easy to use in simple cases. For our modified short example, replace the explicit call to the initializer for class A with this:

super().__init__()

super uses data from the context of the call to figure out where to delegate it to: the class where the call appears, and the ancestry tree of the instance. So we do not need to supply the self argument in this case, unlike the hardwired call to a class’s init method.

super in Python2 is a little dfferent, look it up if you need to use it there.

The "walk up from the bottom" nature of super means you have to give some thought when designing your classes. From the viewpoint of a given method, calling super does not mean call my immediate parent’s method, it means walk up from self until you find it. Remembering that self written in a method does not mean "me", it means the instance object I was called with, which could very well be an instance of a class that derived from me. So be nice to others who may want to derive from you, don’t build in assumptions that may not work for derived classes.

Somewhat surprisingly, super is a bit controversial in the Python community, there have been blog posts and flame wars on whether it is a good thing or bad thing. Most of that centers around complex inheritance trees, especially multiple inheritance. I see no reason not to use it as described in this post. If things get more complicated, you do need to make sure the arguments line up between caller and callee, and because super doesn’t tell you what the callee is, that is work you have to take care of manually.

Summing Up

If you’ve read this far, it may have occurred that Python classes are very flexible. This may be unlike the rather rigid contstraints of more classical object-oriented languages. Here is a concept to think about:

Classes in Python do not so much inherit, override, extend as they delegate and reuse. You derive from another class in order to reuse its code, which in practice means your dictionary of methods and data is pre-populated with references from the class(es) you are "deriving" from. But you are completely in charge of this relationship. If you define a name that is also defined above you in your inheritance tree, your definition is the one that goes in the dictionary, and you are stating you want to do that work yourself, not delegate it. You can choose, in your implementation, to call out to another implementation for the same name from a base class - and that is not so much extending as augmenting your version with code found elsewhere. And if you do not define it, you are reusing the code for that name as defined elsewhere.

---

Last updated 2018-10-06 17:03:08 MDT

Python Test Driven Development Basics with PyTest¶

Introduction

Not long ago I was chatting with someone about some code he was working on that did something some might consider 'obscure';, and how to be sure it worked correctly. You can of course put in print statements to trace what is going on, then remove them later, but this also sounds like a case for writing a test that confirms the behavior, then coding up the function, mkaing adjustments until the test passes. This roughly, is what is called test-driven development (TDD).

When we chatted about how to do that, the person said they had trouble working with the Python unittest module, and I pointed out I don't much care for it either. One reason is because it forces you to use classes even when it may not feel all that natural to write a class for a particular problem. I wrote up some notes for him on using PyTest, and then dedicded to modify those for somewhat wider sharing.

So here's a vaguely practical example of applying the pattern that led to our discussion, that is how to effectively run a test function several different ways, rather than writing up a function for each permutation of your test.

First, and immediately violating the principles of TDD which say to write the test first, let's write the function to test. Our candidate function tries to return the reverse of its argument, to keep it simple, we will assume the argument is something that can be iterated over, so we can use fancy list slicing (thus we can't reverse a dictionary - but that has no meaning anyway since a dictionary has no order). To show it's working there is also code to try it out if it is called as a program (as opposed to a module). This is the way people used to write tests for a Python module, before test harnesses became widely used: just code up a few checks and stuff them in the "main" part.

def slicerev(collection):
    return collection[::-1]

if __name__ == "__main__":
    print slicerev([1,2,3,4])
    print slicerev((1,2,3,4))
    print slicerev('abcd')

If we run that, we see that all of list, tuple and string did indeed get reversed as we expected:

[4, 3, 2, 1]
(4, 3, 2, 1)
dcba

Using PyTest

Let's take this basic test code and turn it into a separate test using PyTest. Unit tests for a particular function are often named (this is convention) test_{funcname}.py. If it's named this way pytest can find it automatically - runing py.test without arguments lets it hunt for files that begin with 'test'. It's not mandatory to use this naming, you can give the name of the test file as an argument, or use other methods to describe exactly where the tests should be picked up from.

The code can be really simple since this is a contrived example - we're not really systematically "unit testing", we're spot checking. All we have to do is import the function we are going to test (even this is not needed if the test is in the same file as the code being tested, as opposed to a separate file), and then write out our three tests cases, which do nothing but call the function with a known argument, then compare the return with what we expect the result to be.

from reverser import slicerev

def test_slicerev_list():
    output = slicerev([1,2,3,4])
    assert output == [4,3,2,1]

def test_slicerev_tuple():
    output = slicerev((1,2,3,4))
    assert output == (4,3,2,1)

def test_slicerev_string():
    output = slicerev('abcd')
    assert output == 'edcba'

That's really all there is to it.

To make things a little more interesting, I have introduced an error in the test itself: the function checking the reversed string claims it expects 'edcba' instead of 'dcba'. This is done to show what it looks like when PyTest reports a failure.

Let's run it:

$ py.test test_slicerev.py
============================= test session starts ==============================
platform linux2 -- Python 2.7.13, pytest-2.9.2, py-1.4.33, pluggy-0.3.1
rootdir: /home/mats/PyBlog/pytester.d, inifile:
collected 3 items

test_slicerev.py ..F

=================================== FAILURES ===================================
_____________________________ test_slicerev_string _____________________________

    def test_slicerev_string():
        output = slicerev('abcd')
>       assert output == 'edcba'
E       assert 'dcba' == 'edcba'
E         - dcba
E         + edcba
E         ? +

test_slicerev.py:13: AssertionError
====================== 1 failed, 2 passed in 0.01 seconds ======================

A run with that problem fixed is considerably quieter:

$ py.test test_slicerev.py
============================= test session starts ==============================
platform linux2 -- Python 2.7.13, pytest-2.9.2, py-1.4.33, pluggy-0.3.1
rootdir: /home/mats/PyBlog/pytester.d, inifile:
collected 3 items

test_slicerev.py ...

=========================== 3 passed in 0.00 seconds ==========================

PyTest Fixtures

If you think about this for a bit, you notice that the same code is run three times, only the data in the three test functions differs. As mentioned above, this is a very common situation in testing, where you want to try different cases to see how a unit behaves - test the boundary conditions, test invalid data or data types, etc.

PyTest provides a mechanism called a "fixture" - a fixed baseline that can be executed repeatedly, which helps with this situation.

In the first iteration of our tests, we did not need to import "pytest" for it to work when the test is run by PyTest - PyTest wraps the code and the code itself never uses anything from PyTest. However, in our second iteration, we do want something from PyTest namespace - the definition of the decorator we need to turn something into a PyTest fixture, so the import is needed.

Since what we're factoring here is supplying different sets of data, the fixture function 'slicedata' itself is extremely simple: all it does is return the data. The test function has the same two functional statements that each of the test functions had before - call the function under test, then use an assertion to check the result was as expected. In addition to that, the takes the fixture function as an argument, which would not make much sense by itself, but once it is turned into a fixture it does.

We use a decorator to turn 'slicedata' into a fixture - remember Python decorators are a piece of special syntax that helps alter the behavor of a function. The PyTest fixture decorator can take a "params" parameter, which should be something that can be iterated over, the fixture function can then receive the data one at a time. In this case we are going to pass a list of tuples, the first element of each tuple being the data we are going to apply to the test, the second element being the expected value.

We now know the other change we need to make to the test function: the "fixture object" returned by the fixture will be a tuple, so we should unpack the tuple into the pieces we want.

The new code looks like this:

import pytest
from reverser import slicerev

@pytest.fixture(params=[
    ([1,2,3,4], [4,3,2,1]),
    ((1,2,3,4), (4,3,2,1)),
    ('abcd',    'dcba')
    ])
def slicedata(request):
    return request.param

def test_slicerev(slicedata):
    input, expected = slicedata
    output = slicerev(input)
    assert output == expected

Run these tests and we'll see the results are the same as before:

$ py.test test_slicerev_fix.py
============================= test session starts ==============================
platform linux2 -- Python 2.7.13, pytest-2.9.2, py-1.4.33, pluggy-0.3.1
rootdir: /home/mats/PyBlog/pytester.d, inifile:
collected 3 items

test_slicerev_fix.py ...

=========================== 3 passed in 0.00 seconds ===========================

Table Of Contents

• Python Test Driven Development Basics with PyTest
  • Introduction
  • Using PyTest
  • PyTest Fixtures

Related Topics

• Documentation overview

This Page

• Show Source

Quick search

©2017, Mats Wichmann. | Powered by Sphinx 1.5.2 & Alabaster 0.7.9 | Page source

Properties in Python

Encapsulation is an important concept in object-oriented programming. Encapsulation just means that some information is not available directly from outside the encapsulating object, only through specially provided accessor methods. This sounds like a pretty good idea at first glance: the implementation of something is hidden, and is allowed to change without breaking things, because people using this code aren’t able to poke around inside and count on details they’ve discovered, they have to use the accessor methods.

What is a property?

Here’s a contrived example, a fragmentary class (in Java, perhaps) describing a person, where we’ve only shown one piece of data, the age of the person:

class Person
{
    int age;
public:
    Person() : age(0) { }

    int getAge() { return this.age; }
    void setAge(int x) { this.age = x; }
};

Here the instance variable age is private, but we’ve also provided a pair of public methods which give access to the value of age: getAge to read it and setAge to set it. Now if something a lot more complicated needs to happen later to age the getter/setter methods can be updated but it doesn’t affect clients using this code. And some evolution is not not actually an unreasonable expectation - the age of a person isn’t a good candidate for data that looks static, since every year on their birthday it changes. So perhaps someday we’ll improve the class by calculating the age from the current date and the person’s birth date only when it is needed?

A lot of developers don’t like this kind of code; it does accomplish a useful function if you need it, but because you can’t necessarily guess when you’re going to need it, when you’re writing in a language like Java, and to a slightly lesser extent C++, you have to set up the getter/setter methods for all data members that are to look public up front, in anticipation that you might need the accessors later - since the interface has a binary (ABI) nature, you can’t change from a public instance variable to a private one with a getter/setter or you will break programs depending on the older class signature. This bloats the code in anticipation of something that might not actually needed, and no new value has been introduced by all those extra lines. Ah, but no problem, right? My IDE just auto-built those for me…

There’s an aesthetic concern regarding the syntax, also. A Person instance includes that person’s age, but we can’t perform natural operations on that age - if person is an instance, we can’t access person.age or set it, we have to use person.getAge() and person.setAge().

The C# language improves on this by providing properties. C# defines properties thus:

A property is a member that provides a flexible mechanism to read, write, or compute the value of a private field. Properties can be used as if they are public data members, but they are actually special methods called accessors. This enables data to be accessed easily and still helps promote the safety and flexibility of methods.

A simple example looks like this:

class Person
{
    private int age = 0;

    public int Age
    {
        get { return this.age; }
        set { this.age = value; }
    }
}

So if person is an instance of Person, person.Age (but not person.age) can be accessed externally as if it were a variable. That leads to the ability to write the much nicer

person.Age += 1

instead of

person.setAge(person.getAge() + 1)

Properties in Python

Python has properties too, but there is another benefit in Python: as a dynamic language, it does not have the limitation of static languages, you can change the implementation, without causing problems to clients because you’re not dealing with a compiled interface. This means you can define an instance variable first, then evolve it to a property later if needed, and it will not break clients.

Here is a series of examples showing how properties work in Python.

Consider a Vector class that should be able to provide both an angle in radians and an angle in degrees. This provides an excuse to use a getter method - we don’t actually need to store both angles in the instance, and indeed we don’t really want to, because they’re related, and if someone updates one angle, we have a problem because the other one needs to change in sync with it. It’s nicer to store one, and generate the other one on demand - that solves the sync problem.

Using the property function

Python provides the built-in property() function which sets up a property given arguments which describe the methods which implement the property behavior. The arguments are in order are the getter, setter, deleter, and docstring; they’re successively optional so if you pass only one argument to property only a getter is assigned.

import math


class Vector(object):
    def __init__(self, angle_rad):
        self.set_angle_rad(angle_rad)

    def get_angle_rad(self):
        return math.radians(self._angle_deg)

    def set_angle_rad(self, angle_rad):
        self._angle_deg = math.degrees(angle_rad)

    angle = property(get_angle_rad, set_angle_rad)

    def get_angle_deg(self):
        return self._angle_deg

    def set_angle_deg(self, angle_deg):
        self._angle_deg = angle_deg

    angle_deg = property(get_angle_deg, set_angle_deg)

We can do some experiments with this class - in the first set of lines below we create an instance with a starting value and print both angles, then change the first the angle then the angle_deg values to show they’re working in unison. In the final chunk, we ask for some information about the objects in question to illustrate how Python has set this up.

v = Vector(2*math.pi)
print("Rad: {}, Deg: {}".format(v.angle, v.angle_deg))

v.angle = math.pi
print("Rad: {}, Deg: {}".format(v.angle, v.angle_deg))

v.angle_deg = 90.0
print("Rad: {}, Deg: {}".format(v.angle, v.angle_deg))

print(Vector.angle, Vector.angle.getter, Vector.angle.setter)
print(Vector.angle_deg, Vector.angle_deg.getter, Vector.angle_deg.setter)

Here’s the output of one run:

Rad: 6.283185307179586, Deg: 360.0
Rad: 3.141592653589793, Deg: 180.0
Rad: 1.5707963267948966, Deg: 90.0
<property object at 0x7fab853b5f48>
  <built-in method getter of property object at 0x7fab853b5f48>
  <built-in method setter of property object at 0x7fab853b5f48>
<property object at 0x7fab7d3d9818>
  <built-in method getter of property object at 0x7fab7d3d9818>
  <built-in method setter of property object at 0x7fab7d3d9818>

Using the property decorators

Python provides decorators that have the same effect as the the call to the property function. @property is used for the getter, @x.setter for the setter and @x.deleter for the deleter method which would be the third argument to the property function if included (replace x with the method name).

import math


class Vector(object):
    def __init__(self, value):
        self.angle = value

    @property
    def angle(self):
        return math.radians(self._angle_deg)

    @angle.setter
    def angle(self, value):
        self._angle_deg = math.degrees(value)

    @property
    def angle_deg(self):
        return self._angle_deg

    @angle_deg.setter
    def angle_deg(self, value):
        self._angle_deg = value

v = Vector(2*math.pi)
print("Rad: {}, Deg: {}".format(v.angle, v.angle_deg))

v.angle = math.pi
print("Rad: {}, Deg: {}".format(v.angle, v.angle_deg))

v.angle_deg = 90.0
print("Rad: {}, Deg: {}".format(v.angle, v.angle_deg))

print(Vector.angle, Vector.angle.getter, Vector.angle.setter)
print(Vector.angle_deg, Vector.angle_deg.getter, Vector.angle_deg.setter)

And the output of our experiments:

Rad: 6.283185307179586, Deg: 360.0
Rad: 3.141592653589793, Deg: 180.0
Rad: 1.5707963267948966, Deg: 90.0
<property object at 0x7f7ba29b5818>
  <built-in method getter of property object at 0x7f7ba29b5818>
  <built-in method setter of property object at 0x7f7ba29b5818>
<property object at 0x7f7ba29b5868>
  <built-in method getter of property object at 0x7f7ba29b5868>
  <built-in method setter of property object at 0x7f7ba29b5868>

By decorating the angle and angle_deg method pairs, we’ve turned them into properties with getter/setter methods, just like the call to the property function did, but this looks cleaner, you can immediately see what each method is for rather than going hunting to see they’re later part of a property call. Notice that the method names have to be the same for all the parts of the property; for the setter and deleter the decorator also takes the name of the method.

Code Simplification

I don’t particularly like this code, though. We are using a sort of hidden instance variable as the backing field which holds the value, and we’ve served up getter/setter pairs for both public variables. Except there is really no hidden data in Python - starting a name with an underscore is a visual hint that we don’t intend something to be public, but that is all it is, a hint (a leading single underscore only "matters" in imports). That means someone could actually fiddle directly with the backing field _angle_deg, bypassing the getter/setter, if they were so motivated. In the trivial example here, that doesn’t introduce any new problems, but in a setter which does a bunch of validation so you know an invalid value is never stored, it is not ideal. And in fact, that the setter for angle_deg does not do anything special is my other complaint: why implement a getter/setter when there is no need to?

So why not undo the property definition that does not seem needed and just make angle_deg an instance variable, then we don’t need _angle_deg at all. If we find we need to do something "special" with angle_deg later we can always turn it back into a property. Notice in the initializer, we are invoking the property setter, because we assign to angle. As a next refactor, I would probably turn this around and use the radians form as the instance variable to make it all feel more natural. This is the Python flexibility I was referring to at the beginning of this article. Here’s the refactored code, which is now quite a bit shorter:

import math


class Vector(object):
    def __init__(self, value):
        self.angle = value

    @property
    def angle(self):
        return math.radians(self.angle_deg)

    @angle.setter
    def angle(self, value):
        self.angle_deg = math.degrees(value)

v = Vector(2 * math.pi)
print("Rad: {}, Deg: {}".format(v.angle, v.angle_deg))

v.angle = math.pi
print("Rad: {}, Deg: {}".format(v.angle, v.angle_deg))

v.angle_deg = 90.0
print("Rad: {}, Deg: {}".format(v.angle, v.angle_deg))

This works just the same, as we see from the output:

Rad: 6.283185307179586, Deg: 360.0
Rad: 3.141592653589793, Deg: 180.0
Rad: 1.5707963267948966, Deg: 90.0

---

Last updated 2017-02-24 13:16:42 MST