Up until now, there was no concrete way to perform push-button testing of the full behavior of PySheetGrader and PysheetGrader assignments. Manual testing is cumbersome, time-consuming, error-prone, and not repeatable. The latest release fills this gap by adding a testing mode to PySheetGrader as a way to specify cell-level unit tests in the CheckOrder tabs, and a special grading report format to display the test-mode test results. These unit tests are only run in the test mode and are different from the test cases specified in a cell’s rubric note using the test_cases property. The test_cases property is for grading a student’s assignment based on instructor-specified unit tests to decide on a cell’s correctness, whereas test-mode tests are used for testing the behavior of the grader itself before new releases as well as testing new assignments before deploying them to students.
Running PySheetGrader in test mode
The test mode is enabled by invoking PySheetGrader with the --test-mode or -T options from command line. The key file containing the unit test cases and the submission file containing the data for the test cases are specified as follows, in the usual way:
pysheetgrader TestGraderKey.xlsx TestGraderSub.xlsx --test-mode \
--html-report-output Report.htmlHere TestGraderKey.xlsx is the test key file containing the test-mode test cases and the TestGraderSub.xlsx is the test submission file containing the data. The --html-report-output option asks PySheetGrader to generate a detailed HTML report, which will be stored in the file Report.html.
Specifying test-mode test cases in CheckOrder tab
In the test mode, cell-level test cases are specified in a CheckOrder tab using additional, special columns with headers test-name, expected-result, and failure-message (this formal will be familiar to the reader who has experience with unit testing in software development). These columns are ignored outside the test mode, during regular grading. The column headers are order-independent and case insensitive.
| Test-mode column purpose | Column heading | Content/Value |
| Test name | test-name | text |
| Expected result | expected-result | correct or incorrect |
| Failure message to be added to the report in case of test failure | failure-message | text |
A CheckOrder tab of a test key file containing test-mode unit tests may look like this, with the addition of the extra columns used in test mode:
Note that column headers are case-insensitive. Therefore, for example, Expected-Result is valid as the column header for the expected result column.
The expected result column indicates whether the test expects the evaluation of that cell to be correct or incorrect. The PASS/FAIL status of the test in the produced report will consider what the expected result is to produce a verdict. This allows both correct and incorrect cell solutions in the test submission file to be tested. If no value is specified in this column, PySheetGrader assumes the default value correct. For example, the test named T4 expects the grader to produce a zero score (indicating an incorrect solution) with the given test submission file if the cell A5 is positively graded or a negative score if the cell A5 is negatively graded.
The test-mode grading report
The grading report generated using the test mode has extra columns that indicate the status of each test-mode test case. Failures produce red rows with a FAIL verdict under the Status column. A failure occurs when the test submission file contains a solution that produces an unexpected result for the tested cell. The failure message, if specified, is appended to the status. Passing tests generate green rows with a PASS verdict under the Status column (and no failure message).
We hope that the test mode will allow PySheetGrader itself as well as individual PySheetGrader assignments to be tested more throughly and in a repeatable manner, increasing the confidence of PySheetGrader’s contributors who maintain and extend it as well as instructors who design assignments.
Related Posts
Automatic Grading of Spreadsheet Assignments
Acknowledgment
Aakash Javia implemented the test mode during the Spring 2022 semester.
