Dev: Continuous Integration with Jenkins in a mixed Linux and Microsoft Environment

Explanation
Jenkins is an open source continuous integration server. It boasts 929 plugins to handle about any sort of bizarre requirement you can throw at a project. In most cases, setting up CI or build servers is a tedious but incredibly important part of any environment.

Why Setup Continuous Integration
If your career has remained in small shops, you may not realize how easy something like this can make your life. If you put a little investment into setting up CI, you can gain quite a lot of time and peace of mind back on your system’s environments.
For me there are two big wins here:

  • First: Continuous Integration setup means you can reasonably disconnect your developers from your deployments. In small shops this in interpreted as a loss of control, however I sell this to developers as a risk management technique. No developer wants to be held accountable for breaking production at 2am. Automate this process, hand the keys over to the people who want to wake up at 2am, and back away slowly from the smoking gun.
  • Second: Continuous Integration is a positive feedback loop for good project maintenance. Once you have nightly builds configured, your developers will quickly learn not to check in broken code, and will use peer pressure to ridicule any developers who “break” a QA environment because they were too lazy to make sure their code built. Also, it’s nice to configure test cases to run before building, so, the “breaks” should be caught. If they’re not caught and something breaks, you know you need more tests.

Jenkins on Linux
In most cases, you’re going to find examples of Jenkins being installed on a linux server and being configured through it’s administrative website. If you’re deploying to all linux servers, life is easy. However, if you have even one server that requires any .Net compilation… well, life is not easy. You need this MSBuild plugin that needs the MSBuild dll. Surprisingly, Microsoft does not actually make a linux distribution of this tool (haha). If you rolled Jenkins on Debian or CentOS, well, you’re in a sticky place where you have to rely on WINE or MONO to hopefully execute a Win DLL. While this is a cute technical challenge, it’s also a waste of time in most cases that adds nothing to your project but hours and maybe a few stack exchange points.

Jenkins on Windows
If you run Jenkins on Windows, then there really are no technical challenges. Deploying to linux and windows systems is now doable with standard plug-ins.
With three plugins you can integrate a Git repository and deploy to linux and windows servers:

  1. Git Plugin: https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin
  2. Publish Over SSH Plugin: https://wiki.jenkins-ci.org/display/JENKINS/Publish+Over+SSH+Plugin
  3. MSBuild Plugin: https://wiki.jenkins-ci.org/display/JENKINS/MSBuild+Plugin

Installation on Windows.
Jenkins is written in Java, so you will need the Java Runtime Executable installed on your server.
JRE: http://www.oracle.com/technetwork/java/javase/downloads/jre7-downloads-1880261.html

You’ll also need to install Git on the server so Jenkins can use it.
Git: http://git-scm.com/downloads

Get the Jenkins Windows Installer: http://jenkins-ci.org/
Outside of the plugins, there’s very few configurations you have to make. Go to Configure Security:
Click “enable security”
select “Jenkins own user database” as realm
select “matrix-based security”
and create an account.
Then add in whatever plugins you need.

There are a few “gotchas” to avoid frustration:

Regarding MSBuild
Jenkins MSBuild plugin requires .NET framework 4.5 to be installed if the Visual studio project is dependent on it (http://msdn.microsoft.com/en-us/windows/hardware/hh852363.aspx)

Regarding Git
The initial path is wrong. In “Manage Jenkins–>Configure System”. Change it to “C:\Program Files (x86)\Git\cmd”
Be careful when you setup your Git credentials on your job. Jenkins will automatically try your credentials without asking over and over. If you typo’d it, your account will get locked out.

Regarding Publish Over SSH
Server Setup is here: Jenkins->Manage Jenkins->Configure System
This is where you configure and add servers. These will populate the server drop down when you are creating a new job.

Summary
Just to restate; the point is not that a Jenkins install on Linux can-NOT handle running the MSBuild.dll through WINE or MONO.
The point is that going through this exercise is not always mission critical and that Jenkins can easily run on a Windows machine and handle deployments to all machine types now without the extra time spent setting up the above.

Setup: MariaDB 10.0 on Azure

Back again with some more notes.

Opinion and Motivation

MS SQL is a popular tool, but in has a big hurdle in cloud adoption: pricing.
The obfuscation of pricing in Azure hurts. When a client asks how much it will cost to run a MS SQL cluster; no one can answer this very well. It’s sort of like asking Azure billing support how much running Sharepoint in Azure would cost (this is an inside joke; most of the Azure billing support reps don’t know about the Office365 cloud).
When you finally get an answer, well, it’s quite expensive. You’re paying for the Azure DB or the VM running SQL 2012; you’re paying for licenses, cores, compute time, huh? Enterprise shops are not immune to budget cuts. If you’re simultaneously being asked to migrate to the cloud and cut expenses… remember this post.

Solution: Use another popular relational database: MySQL
MySQL is the famed open source database.

Problem 1: Oracle bought it. If you don’t get why that’s a problem; you’re probably an executive. Congrats.
Problem 2: The original developer left and built a better version.

New Solution: Use the less popular open source relational database: MariaDB
Never heard of it? That’s okay. This post explains it pretty well:

Okay, so, that’s why, now let’s focus on the how-to.
If you followed my last blog, you’ve got a Virtual Machine in the Azure Cloud running Ubuntu 13.10. So what we’re going to do now is:

  • Install MariaDB 10.0
  • Connect to it via MySQL Workbench from my local desktopn
  • Create a test database
  • Go back to the Azure VM and prove it worked.

Install MariaDB

Here is the official website:

Open Putty and connect to your Azure VM running Ubuntu.
Issue these commands:


sudo apt-get install software-properties-common
sudo apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xcbcb082a1bb943db
sudo add-apt-repository 'deb http://download.nus.edu.sg/mirror/mariadb/repo/10.0/ubuntu saucy main'

That sets up the key and repository. Now you can install it.


sudo apt-get update
sudo apt-get install mariadb-server

This takes you to the following setup screen for MariaDB:
azure-ubuntu-mariadb

Verify MariaDB Works & Nice To Know’s

My source for this is the documentation:

To connect on your Azure VM to your MariaDB installation:


mysql -u root -p -h localhost

Here’s what you should see:
mariadb-login

Great, now let’s just take it a little further so we have something to work with.
Enter the following commands into MariaDB to create a test database and look around.


CREATE DATABASE IF NOT EXISTS test;
SHOW DATABASES;

Okay, so, it’s installed, alive, and working properly.

If you want to connect remotely to your MariaDB/MySQL database, be sure to:
Create an ENDPOINT on the Azure Virtual Machine
Go to your Azure Portal, click on Virtual Machines in the left navigation, and select your test VM.
Go to the EndPoints tab in the main portion of the portal.
At the bottom center, select Add (which infers you want to add an endpoint).
Select MySQL from the Endpoint drop down and it should look like this:
azure-endpoint-mysql

In a near future post I will describe how to connect to your MariaDB via MySQLWorkBench

Setup: Azure, Ubuntu 13.10, Node.js, Express.js

In my last post, I covered how you would setup a Virtual Box virtual machine on a Windows host for a Ubuntu 13.10 client, running as a web server with Node.js.

This post will take it to the next level: the cloud. Microsoft’s Azure cloud specifically. We will spin up a virtual machine in Azure with an Ubuntu image. From what I’ve seen, the Azure team has put a lot of effort into their Ubuntu images in the gallery. They’re pretty nice.

The next step will be to setup an SSH connection using Putty. Putty tends to be the most popular way to connect to remote servers that only have a command line. It’s not very exciting from user interface perspective, but, these are web servers, not gaming machines.

After the Putty install, certificate key generation, and connection to the server, the rest will be the usual install with some additional information I learned to make the process a bit better. Specifically, I’ll focus on setting up Express.js on top of Node.js. Express.js is a server side MVC framework that runs on top of Node.js and makes handling requests just a little bit easier.

Okay, on with the walk through.

The Azure Portal part

Log into your Azure Portal using your Windows Live Login ID.
On the left navigation of the Azure portal, click the Virtual Machines.
azure-virtual-machines

Then in the bottom left, there will be a “+New” button. Click that to begin creating your new virtual machine.
Pick through the choices like in this image:
azure-virtual-machine-gallery

Choose the Ubuntu 13.10:
azure-virtual-machine-gallery-ubuntu

This will take you the first server setup page. Before you can fill this page out though, you need a very important thing: An Azure Compatible Key
So, open a new browser, and download a program that generates SSL keys.

If you want to know how to get an SSL cert for your Windows Azure account, please follow the instructions here:

http://www.windowsazure.com/en-us/documentation/articles/linux-use-ssh-key/

It can be a pain, but eventually you’ll end up with a .pem file that you enter on the Azure VM page.

Fill out the rest of the information: server name, size (small), username, password along with your .pem cert file.
This takes you to the second screen of the VM setup, where you can choose your DNS name, storage account, and regional affinity.
Choose a region closest to you or your client physically

The final screen lets you configure your End Points. You want a web server and you want to control it through Putty, so you need three: HTTP, HTTPS, and SSH. Just select them in the drop down and Azure does the work for you:
azure-virtual-machine-endpoints

Create the server and watch it spin for a few minutes. Now that part is done. Next we will setup Putty. If you are a Windows Developer, you may never have used a Telnet or SSH client before. You may feel like you’ve gone back in history 20,000 years to witness the awesome power of a server without a user interface. And you may be surprised to know that this is exactly why Unix admins made jokes behind your back. Well, now it’s time to put that behind you.

The Putty part

Okay, so the first thing you need is not putty, it’s the puttygen program. It can be downloaded here: http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html.
You might be thinking, why is that URL not something like www.putty.com or some more legit sounding domain. Suffice to say, the developer doesn’t care.

On that page, look for the binaries section and download puttygen and putty
puttygen-download

The Ubuntu part

Okay, so you’ve got your VM, you’ve installed and configured Putty, and you’re connected to the VM. Here’s the command line steps to get a minimal install setup:


sudo apt-get update
sudo apt-get install -y python-software-properties python g++ make
sudo add-apt-repository ppa:chris-lea/node.js
sudo apt-get update
sudo apt-get install g++ curl libssl-dev apache2-utils
sudo apt-get install git-core
sudo apt-get install nodejs
sudo npm install express
sudo nano package.json

Copy these lines into your window, then hit CTRL-X, Y, Enter. This file sets up the dependencies for the project. In this case, we’re just including express. After this, when we do the sudo npm install, that command will run through the package.json file and install any libraries under the dependencies part. If you have them mistyped… well it won’t work.


{
  "name": "a-test-app-no-spaces",
  "description": "spaces are okay here",
  "version": "0.0.1",
  "private": true,
  "dependencies": {
    "express": "3.x"
  }
}

sudo nano server.js


var path   =   require("path");
var fs    =  require("fs");
var http  =  require("http");
var express  =  require("express");
var app  =  express();

//capture logs in the console
app.use(express.logger("dev"));

//serve static files - blank in the quotes means server.js is in the same folder as your HTML files.
app.use(express.static(__dirname + ''));

//404 error
app.use(function(req, res, next) {
  res.send(404, "file not found");
});

//start server
app.listen(80);
console.log("listening on port 80");

sudo npm install
sudo node server.js

So, after all this, you can open a browser to the domain name of your server, and you should see the 404 error message, since you have no HTML files:
azure-express-web-server

And if you are watching your Putty session, you should see the activity logged by Express.js like this:
azure-putty-express

And there you have it.
So now, we’ve covered how to setup this web server in both Virtual Box and Windows Azure.
Thanks for checking in.

© Copyright Duke Hall - Designed by Pexeto