Python 3

For evaluating exercises written using Python 3, we're using the unittest framework.

Hello World

Let's start with a Hello World exercise to explain the basics of evaluation in Python3. Following is the solution file named exercise.py:

print("Hello World")

Evaluation of this simple "Hello World" exercise is also very simple. All we have to do is to create a class extending unittest's TestCase class and declare a test method inside. The test method will get the string printed in stdout and compare it with expected output ("Hello World").

import sys
from unittest import TestCase

class Evaluate(TestCase):
    def test_hello_world(self):
        import exercise  # Imports and runs student's solution
        output = sys.stdout.getvalue()  # Returns output since this function started
        self.assertEqual("Hello World\n", output, "You must print Hello World")

Let's see what we did in the above solution code.

• 1st line, the sys module is imported which will help us get the output of student's code

• 2nd line, TestCase class is imported to define our test class Evaluate and it is defined on the 4th line

• 5th line, our first test function is defined

• 6th line, the exercise module that is implemented by students is imported. Since the solution code is not in a function, it is enough to import the module to run the student's code. We have to import it inside the test function to be able to get the output.

• 7th line, we get the output of exercise module. sys.stdout.getvalue() does exactly what we want here. It will only get the strings written to stdout since the test function has started. So you cannot get any outputs written before the test_hello_world() has begun.

• 8th line asserts whether the output is correct and returns feedback if it's not.

Reloading Modules

When what you want to test lives outside of a function or class definition (like our Hello World example above), you will need to re-import the student's module for each of your test cases.

Let's see an example. Consider a very basic exercise which has a solution as below:

import sys

arg1 = sys.argv[1]
arg2 = sys.argv[2]
print(arg1 + arg2)

This code gets two command line arguments and prints the concatenation of them.

To test this code with various inputs, we will need to import the module several times. The test code will be a bit ugly, however this is the only way we can do such trick in Python. An example test looks like:

import sys
from unittest import TestCase
import importlib # module to import other modules

# This is necessary to import exercise module successfully the first time
sys.argv = ['0', '1', '2'] 
import exercise # Import the module to enable reloading

class Evaluate(TestCase):

    def test_exercise(self):
        sys.argv = ['exercise.py', 'face', 'book']
        importlib.reload(exercise) # re-import the module and run the code again
        self.assertEqual('facebook\n', sys.stdout.getvalue(), "Fails for the arguments 'face' and 'book'")

    def test_exercise_longer_args(self):
        sys.argv = ['exercise.py', 'randomaccess', 'memory']
        importlib.reload(exercise) # re-import the module and run the code again
        self.assertEqual('randomaccessmemory\n', sys.stdout.getvalue(), "Fails for the arguments 'randomaccess' and 'memory'")

Let's see what we did above and why.

• 1st line imports the sys module which will help us provide arguments to student's solution and get output strings.

• 3rd line imports the module that provides functionality for loading and reloading other modules. Because we are not inside a unittest at this point, we can't test the student's code yet.

• 5th and 6th lines are meaningful together. We need to import the module globally to make reloading work. In order to import this module, we have to provide some arguments since the module doesn't check whether there are arguments or not. If we don't fill sys.argv , importing will fail.

• On the first test, 11th line, we add the actual test arguments to sys.argv , then reload the exercise module on 12th line. It will simply re-run the entire module code again and print the new outputs.

• On the second test we do the same thing, pass new arguments and reload the module.

If you try importing the module with import statements in each test function, Python will ignore the second and other imports. And if you try importing and reloading the module in the same test function, sys.stdout.getvalue() will return the output of two imports since the code will run twice.

Thus, you have to import the module globally first, and reload it in each test function to be able to run the code several times in different test functions.

Function Calls

Now we will change the Hello World example a little bit and move the print() statement into a function. The task is still to print "Hello World" but only when the hello_world() function is called.

Following is the sample solution code in exercise.py file:

def hello_world(): 
    print("Hello World")

To check whether the student's function is working properly, we need to import the module and call the function.

The code below is a sample evaluator for hello_world() function.

from unittest import TestCase

class Evaluate(TestCase):
    def test_sum(self):
        import exercise  # Imports student's solution
        exercise.hello_world() # Call student's function
        output = sys.stdout.getvalue()  # Returns output since this function started
        self.assertEqual("Hello World\n", output, "You must print Hello World")

The only difference of this evaluation code from the original is that we explicitly called hello_world() function since importing the module is not enough to run the code anymore.

Variable Declarations

In your evaluator code, you can check whether a variable is defined in the student's code and check its value. Consider the exercise is to declare two variables, namely name and age and assign "Jon Snow" and 29 respectively. A sample solution file (exercise.py) is below:

name = "John Snow"
age = 29

In the test code, we will first check if the variables exist and then compare their values with expected ones. Here is an example test code:

import exercise  # Import student's code
from unittest import TestCase

class Evaluate(TestCase):
    def test_name(self):
        self.assertTrue(hasattr(exercise, "name"), "You must declare 'name'")
        self.assertEqual(exercise.name, "John Snow", "'name' value seems wrong")

    def test_age(self):
        self.assertTrue(hasattr(exercise, "age"), "You must declare 'age'")
        self.assertEqual(exercise.age, 29, "'age' value seems wrong")

Class Declarations

You can test student's object oriented knowledge. Let's say the exercise is asking students to declare a class named Dog and implement a function inside named bark() . The function will print "woof woof" when it's called.

An example solution is given below:

class Dog:
    def bark(self):
        print("woof woof")

Despite the fact that student may submit an empty solution file, we don't need to manually check whether the Dog class is declared. The only thing we need to do is to import it and test it.

import sys
from unittest import TestCase
from exercise import Dog  # Import Dog class only (will fail if not declared)

class Evaluate(TestCase):
    def test_bark(self):
        dog = Dog()
        dog.bark()
        output = sys.stdout.getvalue()
        self.assertEqual("woof woof\n", output, "bark() must pring 'woof woof'")

Testing Student Use of Functions via Mocking

Now we will get into the details of writing unit tests in Python 3.

In a test case where you would like to test the use of a specific function, you will need to mock that function. How can you test use of functions?

Force students to use a specific function

You can mock a function and assert the call count of the function is more than 0.

Ensure the arguments passed to a function are correct

You can test a mocked function arguments with assert_called_once_with() assertion.

Prevent students from using a specific function

You can mock a function and assert the call count of the function is 0.

Change the behaviour of a function

You can change the return value of a mocked function and or completely override it via side_effect property.

Let's see how to use mocking in different situations.

Mocking Builtin Functions

Let's say we want student to call a builtin function in order to solve a problem. We may want to change the behaviour of the builtin function while testing student's code.

Let's say the question is to read a file, people.txt , from disk and parse it. The file contains "first_name | last_name | birthdate" in each line. Students will implement a function named parse() and this will return a list of first names. An example way to solve this problem is given below:

    def parse():
        people_file = open('people.txt', 'r')
        lines = people_file.readlines()
        return [line.split('|')[0] for line in lines]

In the test file, we will have to mock the builtin open() function to test different behaviours.

    from unittest import TestCase, mock
    from exercise import parse  # Import parse function only (will fail if not declared)

    class Evaluate(TestCase):
        @mock.patch('builtins.open')
        def test_parse_single_line(self, mock_open):
            mock_open.return_value.readlines.return_value = ['Ned|Stark|10/10/10']
            result = parse()
            self.assertEqual(['Ned'], result, "Feedback about the mistake")

Let's examine the new concepts in the above evaluation code.

@mock.patch

This decorator mocks a function and gives you the ability to change it's behaviour or track if it's called or not. The mocked function is provided as argument to the test function.

mock_open.return_value.readlines.return_value

This line sets the return value of readlines() function. Since the readlines is actually on the object returned by open() function, we had to first reach the return_value of open.

This evaluation code works with the sample solution code. However, students may use the open() function in different ways and cause the tests fail. For instance, if a student uses read() rather than readlines() on the returned file object, it will return a Mock object since we didn't set return_value for read() and the test will fail although student's solution might be correct. Thus, using mocks in some type of exercises that students can solve the same problem in several ways is not a good practice.

On the other hand, it is a good way to force students to use a specific method or a specific algorithm. In this example, we may intentionally want students to use readlines() method if there is a lecture about it before this quiz. Then, we will have to add one more line to the test code to force the usage of readlines:

self.assertEqual(mock_open.return_value.readlines.call_count, 1, "You must call readlines()")

This ensures that the call count of readlines() is 1.

So the final test file would be something like:

from unittest import TestCase, mock
from exercise import parse  # Import parse function only (will fail if not declared)

class Evaluate(TestCase):
    @mock.patch('builtins.open')
    def test_parse_single_line(self, mock_open):
        mock_open.return_value.readlines.return_value = ['Ned|Stark|10/10/10']
        result = parse()
        self.assertEqual(mock_open.return_value.readlines.call_count, 1, "You must call readlines()")
        self.assertEqual(['Ned'], result, "Feedback about the mistake")

    @mock.patch('builtins.open')
    def test_parse_multiple_lines(self, mock_open):
        mock_open.return_value.readlines.return_value = [
            'Ned|Stark|10/10/10', 'Jon|Snow|11/11/11', 'Cersei|Lannister|10/11/12']
        result = parse()
        self.assertEqual(mock_open.return_value.readlines.call_count, 1, "You must call readlines()")
        self.assertEqual(['Ned', 'Jon', 'Cersei'], result, "Feedback about the mistake")

You can find more information and examples in the official documentation page.

Mocking Module Functions

In this section, we will give an example of mocking class methods and other functions to change the behaviour.

Consider the exercise is about Python exceptions and we want students to call a given function in a try / except block and print the exception message.

A sample solution is given below:

def method_from_instructor():
    # This method is given by instructor
    # Students shouldn't change it
    raise Exception("This is an exception message")

def method_for_students():
    # This method is for students to edit
    try:
        method_from_instructor()
    except Exception as e:
        print(e)

While writing the test for this exercise, our purpose is to make sure students don't just copy / paste and print the exception message. So one way would be mocking method_from_instructor() and raising an exception with a different message.

import sys
from unittest import TestCase, mock

class Evaluate(TestCase):
    @mock.patch('exercise.method_from_instructor')
    def test_method_for_students(self, instructor_mock):
        import exercise
        def mock_method():
            raise Exception("Another random exception message")
        instructor_mock.side_effect = mock_method
        exercise.method_for_students()
        self.assertEqual("Another random exception message\n", sys.stdout.getvalue(),
            "You must print exception message")

This evaluation code overrides the method_from_instructor() and makes sure it raises an exception with a different message. To do that, we basically implemented another function mock_method() and set it as side_effect of method_from_instructor() mock. It means that mock_method() function will be executed when method_from_instructor() is called.

Thus, even if students just copy / paste the exception message and print it, it wouldn't work because we are raising the exception with a different message inside mock_method() .

For further information, we strongly recommend you to read the official mock documentation of Python.

Mocking Class Methods

How would you mock a class method? It is almost the same with mocking module functions. We need to change the example given in above section but the purpose is the same.

Consider that we ask students to implement a method in a class which calls another method in the same class provided by us. We don't want students to be able to edit our method to pass the quiz.

Let's say the exercise is concatenating two strings retrieved from two different methods. The solution is given below:

class Customer:
    def get_full_name(self):
        # This is the function to be edited by the students
        return self.get_first_name() + " " + self.get_last_name()

    def get_first_name(self):
        # This is provided by the instructor
        return "Arya"

    def get_last_name(self):
        # This is provided by the instructor
        return "Stark"

Our purpose in this example is to make sure students don't just copy / paste the given names. So we will mock two methods provided by the instructor and return different names while testing.

import sys
from unittest import TestCase, mock
from exercise import Customer

class Evaluate(TestCase):

    @mock.patch.object(Customer, 'get_first_name')
    @mock.patch.object(Customer, 'get_last_name')
    def test_method_for_students(self, last_name_mock, first_name_mock):
        last_name_mock.return_value = 'Lannister'
        first_name_mock.return_value = 'Tyrion'
        customer = Customer()
        full_name = customer.get_full_name()
        self.assertEqual("Tyrion Lannister", full_name,
            "You must return full name")

What is different here? We used @mock.patch.object instead of @mock.patch to patch the function and passed Customer class as the first argument to the decorator. Besides that, it's only a different version of the example in the previous section.

Mocking User Input

When you want to test code that reads user input, you will need to mock the stdin. Let's see an example:

a = int(input())
b = int(input())
print(a + b)

This code gets two numerical user inputs and prints their sum. To test this code with various inputs, we will need to mock the stdin to provide sample user inputs, separating them with a newline character '\n'. An example test looks like:

import sys
from unittest import TestCase
import io

class Evaluate(TestCase):
    def test_exercise(self):
        sys.stdin = io.StringIO('1\n3')
        import exercise
        output = sys.stdout.getvalue()
        self.assertEqual('4\n', output,
                         'Fails for the arguments 1 and 3')

You can find more details in the official mock documentation of Python.