Trigger jenkins deployment from github push with ansible

We are looking at a simple and robust way to automate deployment. The following show some of the non-intuitive things necessary to set this up.

Jenkins

  • I recommend using the github oauth plugin for authentication and of course the github plugin to easily clone github repos.
  • use https://wiki.jenkins-ci.org/display/JENKINS/Build+Token+Root+Plugin – because the default endpoint to trigger builds cannot be accessed by anonymous users
  • Do not allow job names with spaces: In the jenkins global settings look for Restrict project naming and enter \S*
  • Set up a jenkins job for the repository that contains your ansible deploy script
  • Set up jenkins like this with the following Dockerfile
FROM jenkins
# if we want to install via apt
USER root
# stuff required by jenkins jobs, inlcuding ansible
RUN apt-get update && apt-get install -y php5-cli php5-curl python-setuptools python-dev build-essential libssl-dev libffi-dev

RUN easy_install pip
RUN pip install virtualenv

USER jenkins

Ansible

This script bootstraps ansible

#!/bin/bash

# exit the bash script if one of the command returns an error code
set -e

# Setup a proper path, I call my virtualenv dir "venv" and
# I've got the virtualenv command installed in /usr/local/bin

echo "setting path variable"
PATH=$WORKSPACE/venv/bin:/usr/local/bin:$PATH

echo "create virtualenv"
if [ ! -d "venv" ]; then
    virtualenv venv
fi

echo "activate virtualenv"
source venv/bin/activate

echo "installing ansible"
pip install ansible boto httplib2

echo "tossing in some secrets"
# use the secret file function in jenkins to fill in these variables
mkdir credentials
echo "move $all_yml, $deploy_key_pem and $server_stage_pem to $(pwd)/credentials"
mv "$all_yml" credentials/
mv "$deploy_key_pem" credentials/
mv "$server_stage_pem" credentials/

chmod 600 -R credentials/*

# if ansible roles are pulled in from other git repos:
git submodule update --recursive

echo "deploying"
./deploy stage 3-deploy-site.yml

Github

  • Set up a webhook. Example: http://jenkins.what.digital/buildByToken/build?job=Job-Name&token=2093....234

Email Gateway Provider credentials / connection test

Here I am using sparkpostmail.com

openssl s_client -starttls smtp -crlf -connect smtp.sparkpostmail.com:587

Type

auth login

Enter the baes64 encoded username that you get as follows:

echo -n "SMTP_Injection" | openssl base64

Do the same for the password.

Full conversation:

250 PIPELINING
auth login
334 VXNlcm5hbWU6
U01UUF9JbmplY3Rpb24=
334 UGFzc3dvcmQ6
ZTU1......MQ==
235 2.0.0 Authed. Go on.

auth plain:

echo -ne "\0User Name\123123123" | base64

Hint: Use swaks for testing and full smtp logging output:

brew install swaks
swaks -server smtp.sparkpostmail.com:587 -tls --auth-user SMTP_Injection --auth-password 123123123 --to mario@what.digital --from mario@what.digital --auth PLAIN

Simple Squid on Docker Setup

Prerequisites

  • Linux Debian Jessie
  • root access

Install Docker

Add the following to /etc/apt/sources.list.d/docker.list

deb https://apt.dockerproject.org/repo debian-jessie main

Install Docker and some required dependencies:

apt-get update
apt-get install apt-transport-https ca-certificates apache2-utils docker-engine
service docker start

Squid

add this to /srv/docker/squid/squid.conf (example squid.conf)

auth_param basic program /usr/lib/squid3/basic_ncsa_auth /etc/squid3/squid_passwd
acl ncsa_users proxy_auth REQUIRED
http_access allow ncsa_users

Create the password file

touch /srv/docker/squid/squid_passwd
chown proxy /srv/docker/squid/squid_passwd
htpasswd /srv/docker/squid/squid_passwd user1

Get the docker image:

docker pull sameersbn/squid:latest

Run the docker container

docker run --name squid -d --restart=always   --publish 3128:3128   \
--volume /srv/docker/squid/squid.conf:/etc/squid3/squid.conf   \
--volume /srv/docker/squid/cache:/var/spool/squid3   \
--volume /srv/docker/squid/squid_passwd:/etc/squid/squid_passwd   \
sameersbn/squid:latest

Checks

Check your proxy setup like this

curl -x your.proxy.com:3128 -U username:password https://what.digital

Sources

Richard Stallman – Saint IGNUcius, Church of Emacs

*I attended Richard Stallman’s talk at ImpactHub Zurich two weeks ago. Yes, he did take off his shoes, and if you can look over a healthy dose of craziness, what he said made somewhat sense.

When Richard Stallman called himself Saint IGNUcius of the Church of Emacs he did so with a smile. He showed off a very fine and subtle sense of humour and seemed to be enjoying every ounce of it. When somebody from the audience asked a provocative question, in a very short time, the Saint was in a furious passion, and went stamping about, and shouting: ‘Shut his mouth’. Literally. *

This is not really my point though. Apart from being very eccentric, Richard Stallman has something important to say.

The software we use on a daily base on our computers, to do our work, should be free. Mind the difference between gratis and free here. Moreover, in answer to that provocative question, if Richard gets his money out of an ATM that’s perfectly ok. Because the ATM is not his and the bank can install the software they want on their machine, for their reasons.

These fine distinctions are important. And they show that what Richard (and GNU) wants is a lot more reasonable and practical than what you might think. Think about all the software we use for our daily work – much of it is in fact free/opensource (think of Chrome, Android OS, the web server that serves this page, or even parts of Mac OS X). And then think about how much headache non-free software has been causing (namely Microsoft software), and how much it has been limiting its users in what they wanted to do.

A trend Richard could not foresee is the Software-as-a-Service revolution. Many applications are being reinvented as cloud services, and the server software is, of course, proprietary. However not even SaaS can invalidate what GNU prescribes.

  1. First of all, the underlying software that runs much of SaaS is mostly free, which cannot be said of proprietary software that runs on personal computers.
  2. Secondly, the trend to move applications from personal computers back onto servers doesn’t mean that those applications cannot be open-source. In fact, open-source SaaS has been a very hot trend for years, just think about the widespread use of opensource Content Management Systems such as WordPress, Typo3, DjangoCMS.
  3. And last but not least, the architecture of modern applications is a lot more open than you might think. Hardly a SaaS provider can do without an open API (Application Program Interface) and many allow third-parties to extend and integrate and these extensions and plugins are open source.

Even if opensource / free software is not the cure to everything it is certainly on the rise and many modern companies such as Google itself often chose to opensource their software!

Website Relaunch Checklist

These days there is not much to do for a website relaunch once the owner is happy with the touch & feel on a stage system. But the following things sure make a difference:

  • Google Analytics: Make sure the old Google Analytics code is running on the new system upon launching. Use Google Tag Manager whenever possible.
  • Favicon: Check in the browser whether a favicon is correctly loaded.
  • Redirects: Make sure, the old URLs are properly redirected to the new ones. Also make sure that the sitemap.xml of the new website is available and still registered with Google Webmaster Tools / Search Console. (for example, in WordPress, use the Redirection plugin). This is important for SEO.
  • SSL: Make sure that your webmaster has access to your SSL certificates and configures them to take advantage of secure HTTP (HTTPS). For all projects at what.digital, new SSL certificates are automatically generated, so you dont have to worry about this point if you relaunch with what.digital.
  • Email Gateway Provider: If you have email forms on your website you want them to deliver form submissions to you in a safe and reliable way. Make sure your webmaster uses a solid email gateway provider (such as mailchimp mandril or sparkpost) and has your sending domain verified and set a valid email address as the sender. If you relaunch with what.digital, we take care of setting up an account for you.
  • If you use Google Maps on your website make sure your webmaster whitelists the new domain in the Google API Console. Otherwise your Google Maps might stop working.
  • SEO: In order to maintain, respectively improve your online visibility, use your CMS’ capabilities to set speaking page urls, titles and descriptions for each page.
  • Domain Name System (DNS): Ask your hosting provider or technical contact to check any secondary domains’ DNS settings and redirects in the web server configuration, to make sure these still redirect correctly to the main domain. One week before the relaunch, make sure to brief your technical contact to reduce the TTL to the lowest possible value (normally 1 minute or 10 minutes), so that the DNS update happens instantaneous, right after the relaunch.

Jenkins Setup on Docker

Install

apt-get install nginx python-certbot-nginx

replace /etc/nginx/sites-available/default by the following:

# Default server configuration
#
server {
   listen 80 default_server;
   listen [::]:80 default_server;
   server_name jenkins.domain.com
   root /tmp;

   location ^~ /.well-known/ {
      # letsencrypt domain validation
      alias /tmp/.well-known/;
   }

   location / {
      proxy_pass http://127.0.0.1:8080;
   }
}

run certbot --authenticator webroot --installer nginx

use /tmp as root if asked.

docker pull jenkins

create /data/jenkins and execute chown 1000 /data/jenkins (1000 is the user id that runs jenkins inside the docker container)

After starting the docker container, you can retrieve the initial password like:

docker-compose exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword

Customize

Create a Dockerfile:

FROM jenkins
# if we want to install via apt
USER root
RUN apt-get update && apt-get install -y php5-cli
USER jenkins

or

FROM jenkins/jenkins:jdk11
# if we want to install via apt
USER root
RUN apt-get update && apt-get install -y \
    libmysqlclient-dev \
    php-cli \
    php-curl \
    python-setuptools \
    python-dev \
    build-essential \
    libssl-dev \
    libffi-dev \
    python3 \
    python3-setuptools \
    python3-dev

RUN easy_install pip
RUN pip install virtualenv

USER jenkins

Your docker-compose.yml

version: '2'

services:
  jenkins:
    build: .
    ports:
      - 8080:8080
      - 50000:50000
    privileged: true
    volumes:
      - /data/jenkins:/var/jenkins_home

Update

docker system prune -a
docker pull jenkins/jenkins:jdk11
docker-compose up --force-recreate --build -d
/etc/init.d/docker restart
docker-compose up -d

(Better) Aldryn Setup Guide for Frontend Developers

Prep

  • ask the Aldryn administrator to add you as a collaborator to Aldryn
  • You will get a signup email.
  • Add your SSH key here: https://control.aldryn.com/account/ssh-keys/ (if you don’t have one, make one. See System Setup for instructions)

System Setup

Mac OS X:

  • Your SSH key should be in your ~/.ssh/ directory and named id_rsa
  • New one: ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

  • Download and install Docker for Mac: https://docs.docker.com/docker-for-mac/

  • On the command line, execute pip
  • if it doesn’t exist run sudo easy_install pip

Linux

  • Your SSH key should be in your ~/.ssh/ directory and named id_rsa
  • New one: ssh-keygen -t rsa -b 4096 -C "your_email@example.com"
  • Install docker: https://docs.docker.com/engine/installation/linux/debian/#

Windows

Traditionally with Docker Toolbox

  • Install https://www.divio.com/en/products/aldryn-desktop-app/ and click on Aldryn Local which will set things up for ya.
  • Update the newly installed VirtualBox to the latest version

Manually with new Docker for Windows

  • this doesn’t work with Win10 Non-Pro edition at the time of writing due to limitations in Docker for Windows or Win10.
  • download and install Docker for Windows: https://docs.docker.com/engine/installation/windows/#/docker-for-windows
  • download and install Python for Windows https://docs.python.org/3/using/windows.html (best choose customize installation, install for all users and check add Python to PATH

Aldryn Setup

  • Runpip install aldryn-client
  • Run aldryn login and enter the access token from your https://control.aldryn.com/account/desktop-app/access-token/
  • aldryn project list and get the slug for your project
  • Go to your preferred project directory and run aldryn project setup project-slug (replace project-slug with the slug you got from aldryn project list) – make sure to select a location within your user home directory otherwise it wont work due to OS restrictions!
  • after the setup you should be able to get the project going by running docker-compose up
  • download the media and database from stage by aldryn project pull db and aldryn project pull media if necessary.

Aldryn – Github Integration

  • As Aldryn runs its own git (no GUI) we normally run our Github repo alongside. Here is how to set up:
  • ask the Github administrator to add you to the project repo
  • Add the remote with git remote add what git@github.com:what-digital/project-repo-slug.git
  • Check out the remote branch with git checkout -b "what/develop" what/develop (Aldryn uses develop instead of stage)
  • You can create your own branches to later create proper pull requests on Github: git checkout -b "your-branch" and then git push --set-upstream what your-branch

All the Android / iOS devices on your MacOSX

Us web developers have great use for simulating the behaviour of mobile websites on our computers. It’s much simpler and more cost-efficient to handle multiple devices on my desktop rather than having these phones lying around on my desk, is it not?

here is how I set this up on my system:

iOS devices:

  • Just install XCode
  • Open XCode and select from the menu XCode -> Open Developer Tools -> Simulator
  • et voila your iOS devices including watch and iPad on your desktop!

Android (Google) devices:

  • install or update homebrew from brew.sh
  • Install the Android SDK with brew install android
  • execute android and install your SDK version and the image (currently 23), as well as Google Play Services and Intel HAXM
  • execute android avd to start the AVD manager, create an AVD with the Google APIs Intel Atom x68_64 CPU, and choose the skin with dynamic hardware control, activate use host GPU
  • Download the Google Apps from here: http://opengapps.org/
  • unzip the .tar.xz inside the .zip file with tar -xvf ./*.tar.xz
  • adb install ./insert/path/from/your/download/location/to/the/extracted/Chrome.apk
  • et voila you have a running Android device on your desktop!

Note: In case you have Docker or VirtualBox running – these must be shut down before the emulator can run.

A tweaked link and button plugin for DjangoCMS (Alternative to djangocms-link)

I have tweaked the standard DjangoCMS link plugin a bit. My version can be found here: https://github.com/what-digital/djangocms-link

Standard DjangoCMS Link/Button Plugin: https://github.com/divio/djangocms-link

The new version features some advantages:

  • Both button and link are a elements that can be easily styled
  • Made use of the new PageField that features a better page selection for links to DjangoCMS pages
  • Introduced the possibility to link to a file
  • Introduced the possibility to add a class to the a element for custom button styling
  • Bugfix: Inside the link plugin, other content elements can now be added (avoid other links though!) instead of using the link name as anchor text.
  • The plugin also unregisters the Aldryn Bootstrap3 Button/Link element.