Source Code

Introduction

Hello! This is the area to come to if you want to find out how the Public Whip website works. All the "source code" which makes the website is freely available. You can download it, run it on your own computer, and try out your own anlayses and algorithms. The code is licensed under the GNU Affero GPL, a standard open source software license.

At the moment I've only run this on Linux. It will certainly work on other Unixes, and it should run on Windows. You need to install MySQL, Perl and a web server with PHP also. Since so few people have used this code, there are bound to be problems. Don't hesitate to email us for help.

If all this sounds like gobbledygook, then email me and ask for help. I can tell you where to go and what to read to learn how to get it up and running. Or if there is a particular good idea that you want to try out, I might help more directly by implementing it.

Ideas and things to do

You'll find these interesting.

todo.txt - Things we're working on now
ideas.txt - Good ideas, that we'd like to do
ideas-marginal.txt - Ideas we think aren't worthwhile

Roughly how it all works

The separate Parliament Parser project supplies XML files of debates in parliament. These are downloaded to Public Whip by HTTP or using rsync. A Perl script loads the divisions from the XML files into a MySQL database. A combination of Perl and Octave (an open source mathematics package, compatible with Matlab) code perform various calculations on the data to form other database tables. The website is written in PHP and makes pages by querying the MySQL database.

You can skip a whole stage by grabbing the database dumps from the bottom of the raw data page, and loading these into MySQL. Then go straight to "Running the website locally" below.

Getting the source code

Browse - If you're just curious, you can browse the code online. Look at README.txt in the top level for more information.

File download - To use the code, download a snapshot. Go to our Github project page, and click the Downloads link.

From Git - For the live code-as-we-change-it, you can use the version control system Git. Go to our Github project page where you will find the appropriate links to clone the repository.

There is README.txt file with the source code, explaining what is in each directory, and what the various todo and idea list files are.

Setting up Unix-utils, Perl and MySQL

Perl - Under Windows download ActivePerl. Unix-based operating systems will probably have Perl already installed.

You will need to install some extra Perl modules. To install them follow the CPAN instructions (Comprehensive Perl Archive Network) under Unix or PPM instructions (Perl Package Manager) under Windows. When using PPM, if you get errors about Text::Reform then see this message.

The modules you need to make sure you have installed are: Text::ExtractWords, Text::Autoformat, Date::Parse, Getopt::Long, XML::Twig. DBI, DBD-mysql, encode. Tell me if this list is wrong.

You will also need the "ttf-bitstream-vera" package for producting graphs.

For Debian 9, the following installation lines should help:

sudo wget -O /etc/apt/trusted.gpg.d/php.gpg https://packages.sury.org/php/apt.gpg
echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/php.list
sudo apt-get update
apt-get install curl software-properties-common apt-transport-https lsb-release ca-certificates gcc build-essentials expat mariadb apache2 snarf tff-bistream-vera libxml-perl libxml-twig-perl libxml-tidy-perl libxml-simple-perl lib-json-perl libfile-slurp-perl libfile-slurp-unicode-perl libfile-slurp-tiny-perl libperl6-slurp-perl libtext-autoformat-perl libunicode-string-perl libxml-rss-perl php7.2-cli php7.2-common php7.2-curl php7.2-gd php7.2-json php7.2-mbstring php7.2-mysql php7.2-opcache php7.2-readline php7.2-xml

MySQL - Get this database server from MySQL.com. You need version 4.0 or above. After installing it you need to set up the instructions to create a database and a user with privileges to access that database. The database can have any name. You need to set up the database tables, which all begin with pw_. To do this read the instructions at the top of the file create.sql. It contains the SQL commands which create the tables.

Getting the XML files

Download these from the separate Parliament Parse project. The division listings are in the main debate files, but are also available in smaller division only files. To get them do something like this:

rsync --recursive data.theyworkforyou.com::parldata/scrapedxml/divisionsonly/ ~/pwdata/scrapedxml

Loading data into the database

Change to the loader directory. You need memxml2db.pl, which will load the information about MPs into database tables. First you have to tell Perl your MySQL username and password. Copy the file PublicWhip/Config.pm.incvs to PublicWhip/Config.pm and edit it. Now run:

./memxml2db.pl

Next you need the script called load.pl. It loads the divisons from the XML files into the database and does various cached calculations for use on the website. Run it with no parameters to find out its syntax. Now do this:

./load.pl divsxml check calc

While you're doing this all, you probably want to run a tool like the MySQL Control Center to browse the database tables and see what data they are filling up with.

Running the website locally

You need to install a web server with PHP if you would like to run the website locally as a way of viewing your local database. Which web server you use isn't important. I'm using Apache, but you can use PHP with IIS. Download Apache from The Apache Software Foundation and get PHP from php.net.

Configure the web server to serve the files in the website folder from the Public Whip distribution. For Apache, add lines like this to httpd.conf. (This installs Public Whip as the top level site for a domain on your web server, it only runs properly in that configuration, as some URLs are referred to as, for example, /publicwhip.css).

DocumentRoot /home/francis/devel/publicwhip/website/

<Directory /home/francis/devel/publicwhip/website>
    Options Indexes Includes FollowSymLinks MultiViews
    AllowOverride All

    Order allow,deny
    Allow from all
</Directory>

You also need a .htaccess file to say how to handle .php files. Two example ones htaccess-francis and htaccess-pworg are in the website folder. Rename htaccess-francis to .htaccess as a first go - exactly what you need to do depends on the rest of your Apache configuration. When you're done make sure you restart the webserver so it reloads its config files.

Finally you need to tell the PHP scripts about the database. Copy the file config.php.incvs to config.php and edit it with your MySQL settings.

Now browse to http://localhost/.

Vote map (Clustering, Multi-dimensional scaling)

Octave is a mathematics package which we use to perform linear algebra for the MP clustering. Download it from the Octave website. You also need Java to compile and view the clustering applet. Download the latest stable J2SE 1.4 from Sun's Java website.

Now go into the cluster folder. There is a Makefile here which lets you build the clustering data. If you do make test it will export the data from your database as a distance matrix DN.m, then use Octave to perform MDS on this, and create mpcoords.txt containing the coordinates. It will then run the Java applet as an application for you to see the results.

To play with the distance metric which is fed into the clustering, have a look in octavein.pl. To see the formula we use for the calculation, look at mds.m. To alter the set of MPs used for clustering, edit mpquery.pm. You may want to add a "limit 20" to the end of the query to reduce the number of MPs so you can try out different algorithms quicker.

Ask me to make it easier

This stuff isn't in the easiest form for you to use at the moment. Do email us for help, and make suggestions as to how it could be better. I'd love to hear from you if you've got the thing working at all, and what you're using it for if you have.