Joomla Jenkins Selenium

I want to start doing Continuous Integration. I used to be doing it when I was at HP and I thought it was a great idea. So I've got myself an Azure VM with an Ubuntu-Jenkins image installed on it. That worked out ok. Now to setup the rest ...

For Web Site testing I'll need Selenium. I used Selenium in the last Jenkins environment I was administering and I'm sing it now on my development machine.

I'll let you know how it goes.

Following this instruction first off.

http://alex.nederlof.com/blog/2012/11/19/installing-selenium-with-jenkins-on-ubuntu/

Got so far, not much use though.

The repository I'm using is Git and I have a remote repository at Github. I'm going to test the new web site I'm working on for Vision Car Rentals. I commit what I've done through eclipse.

To do this I create a new repository under my Github account, calling it VCR. In eclipse I open the Git perspective. I've already setup the eclipse project I'm using for the VCR web site as a Git project. In the Git perspective I configure the remotes for the VCR Git Repository by adding remote I call origin (the default name). This includes putting in the URI off the Git repository web page. Then I commit and push.

So far so good.

Now I'm going to want to clone the repository onto the Jenkins machine (I imagine) somewhere where apache can serve it for Jenkins to invoke some Selenium tests on it.

We're also going to need some tests.

Looks like I need to be sure I have a LAMP (Linux-Apache-MySQL-PHP) environment here on this server so I can run Joomla.

Definitely Linux and Apache.

Running into problems accessing Github from the Jenkins server using ssh.

And that's not all. Not easy enough for me to install MySQL into the bitnami Jenkins stack. Might be easier to install the Jenkins app into the bitnami LAMP stack. 

So I going to scrap the Jenkins VM and setup a LAMP VM (appliance?) instead.

One job I needed to do both times that I should note is that wanting to setup SSH access with PPK authentication during the Azure VM creation means I need a windows compatible PPK public key file to upload. Found Instruction here : http://jrwren.wrenfam.com/blog/2013/09/22/converting-your-existing-ssh-rsa-key-for-use-with-windows-azure/

So far so good. I have a Azure LAMP VM at cwf-ci-server.cloudapp.net. Now to install Jenkins and Joomla.

Well got Jenkins running on the LAMP. I'll try for Joomla next.

Starting to install the bitnami Joomla module. Wants the MySQL root password ... hmm I setup a bitnami LAMP Stack VM, I didn't get to specify the password.

Found the default password 'bitnami'. I think I aught to change that. OK, how to do that? 

I'm used to using phpMyAdmin, how do I get to that?

I have to tunnel through a SSH connection because phpMyAdmin can only accept requests from a local address (127.0.0.1). So, how do I do that?

Fire up a terminal session on my local machine (MacBook Pro) and issue the following command. I've already setup PPK authentication. So the following command does the job of setting up a tunnel:

$ ssh -N -L 8888:127.0.0.1:80 This email address is being protected from spambots. You need JavaScript enabled to view it.;

Now I just need to open a browser to this end of the tunnel and the phpMyAdmin path:

http://127.0.0.1:8888/phpmyadmin

Excellent I can change the password, done. Back to the Joomla module install.

That's weird. I don't remember the install asking for a joomla admin password, just a username and the user's full name, never the password. So when I go to login to the sit'e administrator I don't have a password. Maybe old-timers disease and my memory is slipping. I usually generate and secure store any passwords that I have to supply for new accounts. I'll have to hack the admin password through phpMyAdmin.

Oh bollocks, I'm playing with using my mac's Keychain for generating and storing passwords, comparing it with KeePassX that I normally use. I generated the root password with Safari/iCloud/Keychain and didn't save this password in KeePassX . My memory is bad, hmmm.

Any way I tried this MySql root password as the joomla administrator password (although the username is not root), and it worked. So note that, the default joomla admin password is initially the same as the MySql Root password. I'll be changing that to avoid the confusion. I'll change the Joomla Super User's password to a new one (generated).

I decided to install the site as a "production" site with the security settings bitnami applies in that case. Never worked on a Joomla site like this. Seems I can't install or uninstall anything without ftp. I'm not going to use ftp if I can avoid it.

Getting annoyed with the SSH timeouts. I go away to read up on somethings and I come back to SSH and I have to reconnect. Pain. so I add this to my ~/.ssh/config on my mac:

Host *
 ServerAliveInterval 60

There's no cpanel so i'll try webmin instead.

Webmin's in and it's fine-ish. File managers is good, used it to adjust file and folder permissions. The upload in file manager does not seem to work. Starting a terminal seems problematic.

I will want to restrict access to the endpoints to my IPs. Can be done at the azure level I read, meaning I won't have to touch the Linux firewall. To do this I'll have to use PowerShell it seems (yetch). There's this thing called the Azure CLI. It's not entirely clear whether I can use it to do this PowerShell thing. Installing it on my mac to see what I can do with it. I don't want to have to use my Windows Virtual Box if I can avoid it.

Didn't need PowerShell of Azure CLI to restrict the end points, did it through the Azure management site. Select the CI, the end point, and down the bottom of the page there's Manage ACL link ... easy, done.

Connected Jenkins to my GitHub repository I'm using for the VCR project. Some messing about with OpenSSH public-private keys. Generated a public-private key for jenkins without a passphrase :/ (/var/lib/jenkins/.ssh) and added the public key to the GitHub repository as a "deploy key". Created the Vision Car Rentals project in Jenkins - as a "Freestyle Project". Source code management is Git, Repository URL is the SSH one and added Credential username "git" and specified the location of the private key file explicitly ("From a file on Jenkins Master").

Managed to get the build of the Vision Car Rentals project to trigger from GitHub when a change is pushed to my GitHub hosted repository. I needed to generate an OAuth token for my account in GitHub. I wasn't ready to create a new account in GitHub for Jenkins at this time.

The build creates the website root under the jenkins workspace. I have to get the web server to use this location for it's web root. Actually what I'll do is create a Virtual Server in Apache that has it's document root pointing at the htdocs in the project's workspace.

Something to note, the Vision Car Rentals project that I keep in the Git repo is the parent directory of the Joomla site I created for it. So changes to the Joomla site will be versioned in the Git repository. Thing is I'm not versioning changes to the database though. Something t think about.

Now to setup Apache. I added the webmin module for Apache and configured it to manage the bitnami Apache installation. I setup a virtual server named vcr.test and pointed it's document root at /var/lib/jenkins/workspace/Vision Car Rentals/htdocs.

Under Network Configuration (webmin) - Host Addresses I added hostname vcr.test for 127.0.0.1

I setup ssh tunnelling to let me access a hostname only defined locally on the server.

ssh -f This email address is being protected from spambots. You need JavaScript enabled to view it. -L 8888:vcr.test:80 -N

When I browse to http://127.0.0.1:8888 I see:

Error displaying the error page: Application Instantiation Error: Could not connect to MySQL.

I'll look at the joomla config there and see what db it's trying to use, probably same name as on my Mac. That won't work will it.

I suppose I also need to think about how I put the database into and out of the Git repository. How Jenkins is going to build the test site, if that's the way to do it. Just cloning the repo won't do, but I don't want anything too explicit either.

I could use Akeeba Backup and Unite, but I think I don't think I want to be versioning one single file for the whole site. I want more granularity than that. Do I want to version at the table or database level? When I deploy I usually use Akeeba Backup and Unite anyway. I might be looking at a database drop and rebuild script generated by a tool of some name. I think I should make it independent of the site under test, that would rule out akeeba backup. Looking at Phinx. No Phinx will take too much work. Need something to simply dump the database as an SQL script, and in Jenkins run this SQL script to drop and rebuild the database in the test environment. mysqldump in an ant script will do I think.

I'll have to add Ant build capabilities to my web site project in Eclipse.

Been working on these ant build scripts. Using mysqldump to create the SQL script that will get versioned. Needed to exclude any data in akeeba backup tables, but still create the tables if they don't exist. Also create the database if it doesn't exist. Found pair of template scripts that I'm expanding.

Note that in the mysqldump SQL scripts there are what appear to be comments, these an't comments they are conditional executions. For example the CREATE DATABASE line looks like this

CREATE DATABASE /*!32312 IF NOT EXISTS*/ `joomla_vcr` /*!40100 DEFAULT CHARACTER SET utf8 */;

This means that when executed if the MySQL version is 3.23.12 or higher then the "IF NOT EXISTS" will be included, and if 4.01.00 or higher then DEFAULT CHARACTER SET utf8 will be included in the line. Cool.

Not so cool though when I wanted to only create the akeeba tables if they don't exist. I had to use an ant task, <replace> to replace CREATE TABLE with CREATE TABLE IF NOT EXIST in the sql file.

I had to use mysqldump twice so as to exclude the akeeba data but not the akeeba table creates. The first mysqldump includes data and excludes the akeeba tables. The second mysqldump excludes data but only dumps the akeeba tables. This one is the one I had to run the replace on. Then I append the second to the first.

Wrote a Selenium test using phpunit. Configured local Apache (MAMP) to use the htdocs directory in my Git work area for this project. Got it working. Setup build ant task to build the MySql Joomla database. Setup a shell script to invoke the ant script from the php test case setup.

Commited to Git and pushed up to GitHub.

Wouldn't trigger the build in Jenkins. GitHub could not send the notification through to Jenkins because I had not permitted the IP range used by GitHub access to the Jenkins port 80 endpoint in Azure. Fixed. Should be using https for this. Will need to generate a self signed SSL for my Jenkins  server cwf-ci-server.

Ok build is triggering in Jenkins now when I put a change to the project's GitHub repository. Now to configure the Vision Car Rentals job in Jenkins to do something useful when it's triggered. Run a build (just the db at the moment) and run the tests. The tests will require setting up Selenium and probably a testing ant script. Not ready to use maven for that yet, if that's what I should use for php project like this.

So now I have setup the buld job (Ant) to startup the selenium server. I tried the Selenium plugin but my phpunit test wouldn't connect. I suspect that I'd have to have a Jenkins slave service running. A bit more complicated (maybe) than I want at the moment.

I got the selenium tests to do screen dumps bit I found that he xvfb service didn't give me the screen dump image sizes matching the screen sizes I set. I switched to using the xvnc server instead. The screen dumps match the expected sizes. Still though I'd like to be able to control the firefox window size in the phpunit tests for the responsive tests I'd like to do.

I added to the build so it will deploy to the projects User Acceptance test server. Using Akeeba Backup Pro CLI and Unite. I had to reverse the decison earlier so that the Akeeba Backup database tables data was included in the repository.

Having problems, Joomla cannot write to the backup output directory. Err yeah, it's a permissions thing. Jenkins is rnning using account jenkins in group jenkins, Joomla running under Apache (Bitnami LAMP) is running under the daemon account. Now for some unix sysadmin work. what am I going to do?

I made daemon the primary group for the jenkins account. That seems to work.

Problems now with Search Engine Friendly URLS - the URL rewriting won't work on the CI server.

Spell check as I type is not working in this editor....

Decided to change the end point access for port 80 on the Azure VM. Hey where has my VM gone? I can still access the VM using the host name I set up, but it's no longer listed in my Azure management panel. But there's this other one listed, looks like a clean install of Bitnami-Jenkins ... weird can access it too, looks like I've got two VMs running now. This is a free trial of Azure so there's no tech support ... bugga. Not happy. Giving Azure the flick, they expect me to trust them at this point and pay so I can get basic tech support with what is obviously a bug in their management system.

Got myself a Digital Pacific VM. Cent OS - starting the job of getting this setup now.