Recently, PySheetGrader has added several new features that enhance its functionality and make it even more useful for spreadsheet grading. These new features include a refactored grading algorithm, two new execution modes–log mode and debug mode–for better troubleshooting, the assertion rubric type to support in-cell Python assertions, the addition of an Expected Score field in the grading report, and the ability to evaluate built-in Excel functions using a Python-based Excel formula library in rubric types that rely on relative grading. In this blogpost, we will introduce you to these new features and explain how they can help you grade spreadsheets more efficiently and effectively.
Oh, and also: PySheetGrader is now open-source, available under the BSD 3-Clause License at: https://github.com/cmusv/pysheetgrader-core. If you wish to contribute, please write to hakane@andrew.cmu.edu.
New grading algorithm
PySheetGrader has undergone a significant refactoring of its grading algorithm that uses the Template Method design pattern to minimize duplicated code and ease the maintenance of existing rubric types, as well as to facilitate the addition of new rubric types in the future. This refactoring has made the codebase more modular and understandable, which should benefit project contributors. Additionally, the new algorithm provides more consistent and accurate grading, which will benefit both users and developers. With the new algorithm, PySheetGrader continues to improve its performance, robustness, functionality, and reliability.
Debug mode
Debug mode helps with the testing and troubleshooting of graded spreadsheets by disabling hidden and pre-emptive grading features. It is invoked by adding the -d flag or the --debug flag to the pysheetgrader command. When this mode is activated, hidden and killer cells (specified in the grading instructions in the _CheckOrder tabs) are ignored during grading. In the future, this mode will be improved to also ignore the minimum work requirements and prerequisite cells for better visibility during troubleshooting and testing. Disabling these features provides the instructor with full visibility when debugging the assignment. Here is an example invocation using this mode:
pysheetgrader TestGraderKey.xlsx TestGraderSub.xlsx --debug-mode --html-report-output Report.html
Log mode
Log mode can be invoked by adding the -l or --log-mode flag. This mode provides additional traceback logging, which can be useful in identifying the source of errors or unexpected behaviors in grading specifications and in the grading engine. Here is an example invocation using this mode:
pysheetgrader TestGraderKey.xlsx TestGraderSub.xlsx --log-mode --html-report-output Report.html
Assertions
Assertions are a powerful new relative grading strategy added to PySheetGrader. They extend the existing repertoire of relative grading strategies, which include the check, relative, and relative_f rubric types, with a new rubric type: assertion. The new rubric type allows the instructor to express custom conditions unconstrained by Excel’s limitations. Assertions can can verify complex relationships in submission files, and instructors can use them to determine the partial correctness of solutions and perform sanity checks.
Assertions are written using Python syntax that permit bound variables prefixed with $ to represent cell references. Assertions must express a Boolean condition: they must evaluate to either True or False. They typically contain Python logical operators (e.g., and, or, not) and comparison operators (e.g., ==, !=, <=, <, etc.), and may even include conditional structures such as if/else statements. They are inserted into cells of the key file with an attached rubric note that tells the grader that the cell contains an assertion to be evaluated on the submission file. Assertions work seamlessly with existing
As an example, suppose we have the Python boolean expression $B5 < $C7 or $B5 == $B6 in a given key cell with the following rubric note:
rubric:
score: 1
type: assertionThe expression includes bound variables $B5, $C7, and $B5, which refer to the evaluated values of the corresponding cells in the key file. Suppose in the submission file, the value of cell B5 equals 2020, C7 equals 204, and B6 equals 2020. The given assertion would evaluate to True, the key cell will be graded correct, and the grader will assign the given score to the key cell.
PySheetGrader features–including alternative cells, negative grading, and more–that work with other relative grading strategies work as expected with the assertion rubric type.
Expected Score field
The Expected Score feature is a new optional field in the _CheckOrder tab of the key file. It is used when running PySheetGrader in test mode to include an expected score for each graded cell to compare with the actual score. It helps with debugging the scoring: the instructors can quickly identify discrepancies between the expected scores and the scores generated by the grader on a test submission file. If the expected score does not match the score generated by PySheetGrader, the graded cell will FAIL with a failure message in the test-mode grading report, and the corresponding row in the HTML report will be red.
To use this feature, simply add a field with the name Expected-Score to the _CheckOrder tab of the key file you want to test with a submission file. Then, add the expected score as an integer as shown below. This change is fully backwards compatible, and will not fail columns with no added Expected-Score.
Excel formula library
PySheetGrader now uses a new Python library called formulas that allows the instructors to evaluate built-in Excel functions during relative grading. By setting a new rubric note attribute parse_excel to true in a check, relative, or relative_f, instructors can use this Python library to evaluate common Excel built-in functions that PySheetGrader’s grading engine normally does not recognize. These functions include AVG, SUM, LEN, LN, ROUND, CONCATENATE, COUNTIF, and more. An example rubric note is shown below that sets the parse-excel attribute to use the new feature:
The complete list of supported built-in functions as of this writing is given below, but more can be added after testing to make sure that they work as intended. The list of functions will be gradually extended with each new release as we test them. Many more functions are expected to work out-of-the-box but we haven’t yet added them to the sanctioned list. Any attempt to use a function that is not in the sanctioned list will throw an error.
SUPPORTED_FUNCTIONS = [
'_xlfn.STDEV.S','SQRT', 'OR','IF', 'EXP',
'LN', 'NOT', 'OR', 'ROUND', 'CONCATENATE',
'LEN', 'SUM', 'MAX', 'AVERAGE', 'COUNTIF', 'AND',
]For the most up-to-date list, please check the string array at https://github.com/cmusv/pysheetgrader-core/blob/main/pysheetgrader/formula_parser.py#L19. A new function can be added to the array if needed provided that it has been tested and is known to work.
Conclusions
In this blogpost, we’ve discussed several new features that have been added to PySheetGrader in its most recent release to facilitate troubleshooting and testing, improve relative grading, and support sanity checks for partial credit. Simultaneously, the grading algorithm received an overhaul to make it easier to maintain the code and add new features. The new features make PySheetGrader more powerful and flexible than before, allowing the instructors to create customized grading criteria that are tailored to their specific needs. Stay tuned for further improvements and fixes, which will be introduced in future blogposts.
PySheetGrader is now open-source
We have made PySheetGrader available to the greater community under the BSD 3-Clause License. To access the codebase and try it out, visit: https://github.com/cmusv/pysheetgrader-core. If you wish to contribute to the pysheetgrader-core project, please write to hakane@andrew.cmu.edu.
Related Posts
Automatic Grading of Spreadsheet Assignments
Robust Testing of PySheetGrader and Its Assignments
