It’s ridiculous we don’t have a proper solution that uses Continuous Integration (CI) to grade homeworks hosted on GitHub.
CI, in short, is just a server that runs unit tests on your code every time you commit it. That means instead of you having to run unit tests before your commit your code (or manually run them after you commit them), a little robot checks your code against your unit tests every time you commit the code. That little robot never gets tired, doesn’t need to be fed. In short, these little robots are way better than human Teaching Assistants (TAs) (who in contrast, need to be fed pizza).
Given that I’ll be Head TAing a class with nearly 300 students this fall, I can’t afford enough pizzas to keep my team running. Or maybe I can, but I prefer not to. So why not get CI servers to grade homeworks? That requires two things:
- Getting students on GitHub (and submitting homeworks via GitHub)
- Grading those homeworks on GitHub
I’ll talk about (1) in a later post when I have the system fully set up, but basically it involves playing nice to GitHub and getting a team with lots and lots of private repositories. Then you give each student one private repository (so that they can’t copy. But then again, since they’re all on GitHub, it should be easy to implement a cheat check. That’s for next time).
Let’s say that the student is supposed to write an assignment like this:
This happens in a file called
FibonacciRecursive.java. (Wow surprise).
This is usually one of the first assignments in a data structures class or a last assignment in an intro programming class. Whatever it is, we can be sure of the following:
- You probably want to avoid unit tests, since they make it difficult for you to score the entire assignment, run asynchronously, and produce rather ugly output.
Given those two limitations, you are set back by quite a bit, since most CI servers only run well on those systems. Give it Java with Maven and CircleCI (one of the larger CI services out there that provide free CI) will work like magic. However, give it non standard stuff and it coughs blood. Unfortunately, that bloody puddle is what we have to work with.
Setting up CircleCI
(Side note: I chose CircleCI not TravisCI because Circle allows for private repositories while Travis insists on public).
Without a single unit test command, we have to be a little hacky with CircleCI. Now here’s how CircleCI runs in a nutshell:
- CircleCI reads off
circle.ymlin your repository. If it finds that file, it follows the instructions in that file. Otherwise, it tries to infer your environment (be it some Ruby thingum, Maven for Java, or Grunt for JS).
- It runs the commands in
circle.ymlor the inferred commands.
- Specifically, in the unit test portion, it runs every command in the unit test portion. If all of them produces an exit code of 0 (that means everything’s good), the unit tests pass. If any of them produce a non-0 exit code, unit tests fail, you get an error.
- The results of these get emailed to you.
That sounds cool right? So let’s start appropriating that system for homework grading.
Your directory will end up containing
Here’s how everything works:
- CircleCI reads
circle.ymlwhich tells the server to run
build.shinstead of running standard unit tests
build.shdoes the following
- It tries to compile
- It runs Grader (now compiled as a class). Grader runs a series of tests on FibonacciRecursive, printing out the score that the student receives along each step. Depending on the score, Grader returns with an exit code of either 0 or 1.
- It tries to compile
build.shreturns whatever exit code Grader returns.
- CircleCI consumes that exit code.
And this is how we’ll abuse CircleCI to grade homeworks.
We know that we have to override
circle.yml to not try and run Maven / Ant / freeze up and die when we run our custom Java stuff. What’s simpler than making it run a
That’s all that should go into your
build.sh should just compile every
.java file in your directory and run Grader. So that’s exactly what it does:
Creating a Grader Tool in Java
This should be rather self explanatory – you’re basically making a series of “assertions”, but tabulating scores at the same time.
Pushing to GitHub
We push all this code to a GitHub repository:
Setting up CircleCI
Add your repository on CircleCI after you’ve set up an account.
You’ll see that the build has started.
And you should get the results of
Grader.java being run on
FibonacciRecursive.java as a result of
build.sh being run as told by
Grade On Commit
It shows that we’ve failed the assignment and didn’t get full marks. Well obviously, since we made
Test 5 in the Grader an errorneous test. It should be producing 0 instead of 1. Let’s change that line:
We make this modification, and submit. We see that CircleCI has already seen the new commit and started building it:
Seems like the problem has been fixed! And this happened without us intervening (and no pizzas involved).
Isn’t this awesome? Even awesome-er, CircleCI has a great API. That means you can use this to create a live scoreboard for your students / email notifications every time they submit code.