eduroam in Freshers’ Week: Some graphs

This year, eduroam is an interesting service. Not only has it been running on all-new Fog-based infrastructure since the mid-summer, but eduroam and ResNet Wireless have been merged to form one authenticated wireless network to rule them all with more users than ever before.

We’ve been watching how the servers are performing under load for the first time with great interest. Let’s have a look at some of the numbers. All of these graphs show the time from midnight on Saturday 21st September (UK freshers arrive) to midnight on Monday 30th September (end of weekend after freshers week).

First let’s have a look at the graph of RADIUS authentication requests – scaled to show Access-Request packets received per second (not necessarily authentications per second, as one authentication usually constitutes several Access-Requests).

The gentle swell of users on the weekend of the 28th/29th is mostly undergraduates using eduroam in residences. The taller spikes on weekdays is made up by University staff using eduroam on campus.

radius01 usually serves eduroam users on campus, at Bristol – both Bristol students and visitors from other eduroam institutions. radius02 usually serves Bristol users who are visiting other institutions, while radius03 usually authenticates any user authenticating at eduroam hotspots in Bristol City Council buildings, and certain local hospitals. However the servers have got each other’s backs, and will take on additional roles at will if there’s an outage.

There are a couple of large spikes on the graph of radius02 with matching notches on the graph of radius01. These are times when the servers decided to shuffle around and radius02 temporarily became the primary for campus users, which is by far the largest set of users.

radius01-total-access-challenges
radius02-total-access-challenges
radius03-total-access-challenges

Now for DNS. Pretty self explanatory – these graphs for the two DNS servers show the number of successful queries per second. At peak, times, we serve over 600 lookups per second.

dns5-success dns6-success

This graph shows the number of valid IPv4 DHCP leases currently in the lease file. We don’t currently graph IPv6 leases so there’s no way of knowing how many there are. Adding this is on my to-do list ūüôā

dhcp1-eduroam-totals

Last but not least, this graph shows the number of queries per second for both nodes in the MySQL cluster that powers eduroam, ResNet and related services. It’s used for authentications, logging and infrastructure management. The two nodes are in a cluster and can rearrange themselves at will, but at any one time, there is one master and one slave. By default db1 is usually the master and handles mostly INSERTs and UPDATEs while db2 is usually the slave and handles only SELECTs.

db1-qps db2-qps

More Juniper VPN Client…

As mentioned at the bottom of my previous post, it is possible to connect to the Juniper VPN without using the java gui, you just need the right command line arguments.

Well, I’ve worked out what those are.

./ncsvc -h uobnet.bris.ac.uk -f ./uobnet.crt -r UoB-Users -u ab12345

(replacing ab12345 with your UoB username obviously)

The list of 32bit dependencies is significantly smaller (as it doesn’t involve Java!) and on Ubuntu the client seems to be satisfied with:

sudo apt-get install libc6-i386 lib32z1 lib32nss-mdns

As before, the above has had very little testing at all.  Any reports of what it does on non-ubuntu systems would be very welcome indeed!

Making the Juniper VPN Client work…

As well as the 32bit rpm version of the Linux client, Juniper have a browser based Java applet for all OSes – which also works on Linux.

It’s not as pretty as the windows/mac client, but it can be made to work – and if you rummage around in it enough it contains a non-java executable at its core which may be useful as the basis of a homebrew non-java application.

Caveats:
This blog post describes a way to get connected to the new UoB vpn using a linux client. ¬†It’s not tested enough, or foolproof enough to qualify as “official” documentation. ¬†It’s more a record of “what I did to make it work”.

We’ve still got a fair way to go before we can publish any of this fully alongside the other instructions.

So try it, if it works for you great! Let us know in the comments! ¬†If I’ve got something wrong, or you’ve worked around one of the problems I hit, let us know in the comments!

If it doesn’t work for you we’d like to hear about that too, but asking for help via the bris-linux mailing list or the IRC channel is probably going to be more productive than asking in the comments.

Ubuntu 12.04 and 12.10 (64bit)
The GUI for the vpn client is a 32bit Java application. ¬†Ubuntu doesn’t appear to install java at all as part of the base install. ¬†So if you’re starting from a point where you have no java support at all, it’s fairly straight forward:

 sudo apt-get install icedtea-7-plugin openjdk-7-jre:i386

The icedtea-7-plugin package will pull in the 64bit version of Java as a dependency, and the 64bit version is chosen by preference.  You can change this by running:

sudo update-alternatives --config java

And select the 32bit version from the list.

At this point, you’ve got 32bit java installed and selected as your default version of java. ¬†You also have the icedtea java plugin for 64bit web browsers installed.

Now point your web browser at https://uobnet.bris.ac.uk¬†and sign in using your UoB username and password. ¬†When the page loads, you should see a line which says “Network Connect” – click the Start button.

You’ll probably get a popup warning saying that you need to make sure you’ve got all the required 32bit libraries installed. ¬†You have (we’ve just done that!) so click “Yes” ¬†– you’ll also get a couple of popups asking if you’re sure you trust the certificate the application was signed with

The first time you run Network Connect it will pop up a terminal window and ask you to put in your (local) sudo password. ¬†This is because the underlying vpn widget needs to run as root, and this popup will setuid the binary so you don’t need to sudo every time you connect.

NetworkConnect

With a bit of luck and a following wind, you’ll get an ugly box pop up which says you’re connected!

Fedora/RedHat/CentOS/ScientificLinux
I haven’t tested any of these, but as far as I can tell, doing this would probably work:

sudo yum install java-1.7.0-openjdk.i386 icedtea-web.i386

Then follow the rest of the Ubuntu instructions

I don’t want Java in my Browser!
Neither do I to be honest. I have the plugin disabled most of the time and enable it only when I want to connect to the VPN.

The version of the client which Juniper distribute as an rpm appears to be a packaged version of the java client which you can load from the browser as described above.

As yet, I’ve not had a chance to do any testing of the rpm version. ¬†Details of that will follow in later posts…

I don’t want Java at all!
As far as I can tell, Java is only required for the GUI. It’s a wrapper around a 32bit binary executable which could be called from the command line (or wrapped in a non-java GUI or script I suppose)

I haven’t managed to find the right magic incantation to get it to actually build a VPN tunnel yet, but this is where I’ve got to so far:

  1. Download the java applet by going to https://uobnet.bris.ac.uk/dana-cached/nc/ncLinuxApp.jar If it asks you to log in, do so using your UoB username and password
  2. Unpack the archive with
    unzip ncLinuxApp.jar
  3. That should give you a folder full of files, which we need to go into:
    cd ncLinuxApp
  4. There’s a helper script we need to run first, which grabs the SSL certificate in use by the VPN connection. ¬†Run it like this:
    bash ./getx509certificate.sh uobnet.bris.ac.uk uobnet.crt
  5. The binary which actually builds the tunnel is “ncsvc” which is a 32bit executable. ¬†It appears to require 3 packages on Ubuntu:
    sudo apt-get install libc6-i386 lib32z1 lib32nss-mdns

However, at that point I’m stuck. ¬†It looks very much like you should be able to build a tunnel by making ncsvc setuid and then calling it with the appropriate parameters:

sudo chown root ncsvc
sudo chmod +s ncsvc
...

Unfortunately I can’t make it work

I’ve made it work! ¬†See this followup blog post for details.

I don’t like closed source binaries either..
As Ian has mentioned previously, We’ve not managed to get the network-manager-vpnc client to connect at all. ¬†We’ll probably come back to that after prodding at rpms, but in the mean time, if you manage to make it work please let us know!

Further Reading
Most of the above is information culled from the following links.  They may be helpful if you fancy having a prod at this yourself.

Some of those links contain perl script wrappers for ncsvc, but as I’ve not had a chance to read every line of that code to make sure it’s not malicious in any way I really can’t recommend you use them!

Linux and the UoB VPN

The University of Bristol has had a VPN service for at least 11 years now, and although the hardware has had a few upgrades in that time the service is essentially running in the same form, using the same old technology.

Time has moved on, and the old VPN service needs replacing so that we can bring its support level up to where we want it.

We’ve procured a VPN appliance from Juniper, and this meets the needs of the majority of Windows, OSX, iPhone and Android users (and meets those needs much better than the old service ever has done.)

Unfortunately there are a few “wrinkles” in Juniper’s Linux client support.

Juniper Client
Juniper do distribute a Linux client. ¬†It’s packaged as an RPM suitable for RedHat Enterprise (or derivatives such as CentOS or Scientific Linux) and is 32bit only. Which is fine, if that’s what you’re using!

If you’re using a Debian derived distribution (Debian, Ubuntu, Mint etc) or a 64bit OS, then it’s not so great.

Various workarounds which will get that 32bit RPM installed on a 64bit Ubuntu (or whatever) seem to be available according to google, but we’ve not had the resources available through the summer to test or document them. Work is now tenatively beginning…

Network Manager

Most linux distributions come with Network Manager these days, and on most of those distributions you can get the “network-manager-vpnc” plugin, which claims to be “compatible with various Cisco, Juniper, Netscreen, and Sonicwall IPSec-based VPN gateways”.

This would seem to be the ideal solution (as it’s built in, doesn’t have any 32 bit dependencies etc) unfortunately we haven’t managed to make it work yet. Discussions with Juniper suggest that this may be caused by a bug in the way they talk to our AD, with a possible fix coming out later in the year, but this is yet to be proven.

That’s not great either, so we need some coping strategies…

Plans B, C, etc…
In the short term, we’re going to keep the old VPN service around so that linux clients still have a way to connect in remotely until we can come up with something better.

We would welcome you to try getting the 32-bit client installed on the distribution of your choice. If anyone makes any progress with that before we’ve had a chance to, please let us know! ¬†Any testing we do will initially result in more blog posts here, and eventually some proper documentation (and hopefully some wrapper scripts/installers if we can get that working)

In the longer-term, we’d obviously like Juniper to release 64bit binary packages (and .debs!) We’re in contact with Juniper about their roadmap and release cycle and will be re-evaluating all of the above options as & when more information becomes available…

** STOP PRESS ** We have managed to make a 64bit Ubuntu machine successfully connect to the VPN. Details will be in a follow-up post.