Jenkins CI with Python and Github

This installation of Jenkins has Nosetests and Pylint with the Github and Git Plugins

This installation of Jenkins has Nosetests and Pylint with the Github and Git Plugins

NC State has a Github Enterprise installation that’s great for working on class projects, especially because it’s possible that some of your team-mates may be students studying remotely. It’s probably a good idea to use a server that will pull the latest changes from your master branches, run tests, and perform linting so that you know how well you’re progressing. Fortunately, all the tools for these are easily accessible. We used:

Installing software

  1. To start with, set up an organization in Github Enterprise for each course team we have and add your team-mates to it.

  2. Create a repository in Github Enterprise.

  3. Install Jenkins on your server.

  4. Install the Git Plugin for Jenkins. Optionally install the Github plugin as well. I didn’t use this for the CI features.

  5. Install the Violations plugin for Jenkins so we can report linting results.

  6. Create an SSH key-pair on your Jenkins server and set up the public key as a deploy key in the settings for your repository.

  7. Set up a web hook on Github Enterprise by going to the Service Hooks tab in the settings for your repository and picking the “Jenkins (Git Plugin)” option. Remember to set the URL to the base URL of your Jenkins installation.

  8. Add the deploy key to Jenkins credentials.

Jenkins Project Settings

  1. Under Source Code Management, select Git and specify the repository URL as specified on the Git Plugin page. This is currently the same URL you’d get if you selected the SSH option on the Github page for your project.

  2. Set the credentials to the key you’ll be using.

  3. In the Build Triggers section, pick the “Build when a change is pushed to Github” and “Poll SCM” options. Set the “Poll SCM” schedule to infrequent (it uses the same format as your crontab).

  4. For your build steps add the following for linting: (the echo command is to account for pylint returning non-zero unless you’re absolutely clean)

    find -iname "*.py" | xargs pylint --rcfile=.pylint -f parseable > pylint.xml || echo "pylint violations"
    

    and for testing, add:

    nosetests src/path/to/test/ --with-xunit
    
  5. Add a post-build option to “Publish JUnit Test Report” and specify **/nosetests.xml as the path.

  6. Add a post-build option to “Report Violations” and specify **/pylint.xml as the path.

  7. Add the Github Project URL if you want Github links.

Using the VCL XML-RPC interface with Python

Apache VCL is a cloud computing platform that NC State uses to provide on-demand access to specific software or environments. It’s neat because you can quickly and easily get a virtual machine to try things out with, or a cluster to experiment with networking. The remote API for VCL is provided via an XML-RPC interface. Python has a nice XML-RPC library called xmlrpclib that makes building clients comically easy.

For instance, here’s a short example, with in-line comments explaining the four lines that need any explanation:

import xmlrpclib
import getpass

# We're connecting over SSL so subclass SafeTransport
class AuthenticatedTransport(xmlrpclib.SafeTransport): 

    def __init__(self, username, password):
        xmlrpclib.SafeTransport.__init__(self)
        self.password = password
        self.username = username

    def send_content(self, connection, request_body):
        # X- headers are user-defined in HTTP
        # These next few are used by VCL
        connection.putheader("X-User", self.username)
        connection.putheader("X-Pass", self.password)
        connection.putheader("X-APIVERSION: 2")
        # That's the end of the VCL-specific headers

        xmlrpclib.SafeTransport.send_content(self, connection, request_body)

username = raw_input("Username: ")
password = getpass.getpass("Password: ")

auth_transport = AuthenticatedTransport(username, password)
server = xmlrpclib.Server("https://vcl.ncsu.edu/scheduling/index.php?mode=xmlrpccall", transport=auth_transport)

try:
    result = server.XMLRPCtest("foo")
except xmlrpclib.ProtocolError as e:
    print "URL: {}".format(e.url)
    print "Headers: {}".format(e.headers)
    print "Error code: {}".format(e.errcode)
    print "Error message: {}".format(e.errmsg)
except xmlrpclib.Fault as f:
    print "Fault Code: {}".format(f.faultCode)
    print "Fault: {}".format(f.faultString)
    print "Args: {}".format(f.args)
    print "Message: {}".format(f.message)
else:
    print result

If all goes well, then you should receive a response of the form:

{'status': 'success', 'message': 'RPC call worked successfully', 'string': 'foo'}

Then you can use the API reference to construct a program that will, for instance, spin up a small cluster of instances, run a student’s code, and then grade it.

Mounting NCSU AFS locations locally on Elementary OS Luna (Ubuntu 12.04)

NC State uses AFS to store student files, course submission lockers, and the course web page. A friend of mine wrote a blog post with instructions on how to mount these locations on Windows. Using that and the MIT CSAIL page on using OpenAFS on Ubuntu, it’s possible to have the your AFS directories show up locally. I am on NCSU’s ResNet internet service, but this should work even from outside.

Installing the required software

First, add the openafs PPA for Ubuntu 12.04 and update your package cache

sudo add-apt-repository ppa:openafs/stable && sudo apt-get update

Then install the packages required

sudo apt-get install openafs-krb5 openafs-client krb5-user module-assistant openafs-modules-dkms

Answer the questions as follows:

Question Answer
AFS Cell this workstation belongs to EOS.NCSU.EDU
Size of AFS Cache in KB 512000

The first answer is then populated to /etc/openafs/ThisCell and the second to /etc/openafs/cacheinfo, in case you want to look it up.

Configuring Kerberos

By default, you won’t be asked to set up the Kerberos realm and other such things and so you’ll have to reconfigure the packages:

sudo dpkg-reconfigure krb5-config openafs-client

Here, answer:

Question Answer
Default Kerberos version 5 Realm EOS.NCSU.EDU

Installing the kernel module

Try to load the openafs kernel module:

sudo modprobe openafs

Restart the OpenAFS client:

sudo service openafs-client restart

Logging in

Authenticate against the Kerberos 5 Realm to receive a ticket-granting ticket:

kinit unityID

Confirm that you have received tickets:

klist

Expect output like:

Ticket cache: FILE:/tmp/krb5cc_105195
Default principal: unityID@EOS.NCSU.EDU

Valid starting    Expires           Service principal
11/02/2013 17:44  11/02/2013 00:24  krbtgt/EOS.NCSU.EDU@EOS.NCSU.EDU

At this point, you should be able to access /afs/eos.ncsu.edu and look at the parts that don’t require authentication. If you want to access your EOS home folder or course submission folders or any other place that you would imagine you require authentication to access, you first have to acquire a Kerberos token by running:

aklog -c unity.ncsu.edu -k EOS.NCSU.EDU

aklog -c eos.ncsu.edu -k EOS.NCSU.EDU

And that’s pretty much it. Your home folder is /afs/unity.ncsu.edu/users/firstLetterOfUnityID/unityID.

Naturally, I didn’t want to have to login each time and I wanted to try out some Python things, so here’s a small script to do the job.

Troubleshooting

  1. If kinit complains that it “Cannot parse principal ‘username'”, then try specifying the fully-qualified username: unityID@EOS.NCSU.EDU. Remember that capital letters are significant.

  2. If you have lots of possible cells that you don’t want to connect to in /afs, try blanking out or commenting lines in /etc/openafs/CellServDB.

  3. You can set up cell aliases, so that it’s easier to navigate to cells you frequently visit. Edit /etc/openafs/CellAlias to do so. Here’s the format:

    cell alias
    eos.ncsu.edu eos
    unity.ncsu.edu unity
    bp.ncsu.edu bp

  4. If you don’t have the openafs kernel module, then you may need to build it. From the CSAIL page, this is easy using module-assistant:

    sudo m-a prepare
    sudo m-a auto-install openafs
    sudo modprobe openafs