If you are working in a company that wants to keep its continuous integration (CI) server internally, probably, like us, you are using Jenkins. It's robust, versatile and not at all complex to use.

TL;DR

However, it lacks job management usability at scale. When you have to deal with more than 10 jobs, it starts to be a boring and error prone task. What about using the same strategy that new CI tools are using? Like Travis for instance. Wouldn’t be fantastic if we had a file in our code repository that Jenkins could recognise and execute jobs accordingly?

At Mindera we use Jenkins like that. Using an open-source tool called Jenkins Job Builder (JBB) and by ‘eating our own dog food’, we configure our jobs with a simple YAML file, that sits next to the project code in the version control. This enables us to manage job configurations not only for CI but also for CD (continuous delivery) with just a YAML file.

The immediate advantages are:

1. No more point and click configuration
2. Auditable and saved configurations changes
3. Job configuration
4. A few lines of configuration sets up a full continuous delivery pipeline
5. Config reuse with help of JJB templates and macros

Context

At Mindera we use Jenkins. We use it because it’s open-source and has an enormous versatility related with the huge number of plugins. We could use others. But for some reason they don’t fit exactly with what we want. The CI’s services in the cloud, like Travis, have ‘the small big problem’ of being paid but also the issue that our code/artifacts are somewhere in a place we don’t control.

So we use Jenkins. But Jenkins is not perfect. The big issue that we found when using Jenkins massively is the difficulty to maintain hundreds of jobs in a clean way. Imagine the work of updating, using the Jenkins web UI, 287 Jenkins jobs to start using Java 8 instead of 6, or change the path of their SCM from Github to Bitbucket.

Well, fortunately, there are several solutions for these boring tasks. For example:

1. Jenkins Job Builder (JJB)
2. Job DSL plugin
3. Configuration Slicing Plugin
4. Configuration management tools (Like Chef, Puppet, Ansible, etc… )

All these alternatives have strong advantages but we had to pick one. We chose JJB for its simplicity and template/macros power.

JJB

Jenkins job builder is also an open-source project. It enables the configuration of Jenkins Jobs, using YAML or JSON, that are then converted to XML and pushed to Jenkins.

JJB example configuration in YAML:
job: name: job-name project-type: freestyle description: 'Do not edit this job through the web!' quiet-period: 5 node: slave-with-java logrotate: daysToKeep: 3 numToKeep: 20 scm:

  - git:
      url: https://example.com/project.git
builders:
  - shell: mvn install
publishers:
  - email:
      ecipients: some@email.com

Old days

When we first adopted JJB we used the one-repo-for-all approach. I mean, every project has a YAML configuration file with its own name and that file was stored in a big repo with all JJB YAML files. Something like this: ├── dev │ ├──jenkins.conf │ ├──macros.yml │ ├──utils.yml │ ├──project_1.yml │ ├──project_2.yml │ ├──project_3.yml │ ├──(....) │ └──project_38.yml ├── prod │ ├──jenkins.conf │ ├──macros.yml │ ├──utils.yml │ ├──project_1.yml │ ├──project_2.yml │ ├──project_3.yml │ ├──(....) │ └──project_38.yml └── test_dev.sh

Using JJB this way makes us earn lifetime. We used a shared macros file where we added all templates and macros that can be used by the projects. Despite this having worked for us, we got a big mess in version control history. Also, the developers continued to not have ownership of those jobs configurations. This is because those weren’t on their projects but in a common repository.

Travis

In the rare case you don’t know what Travis (or other similar services) is, you should Google it. Travis is a CI cloud service that is a free for open-source project stored in Github. Travis has a lot of fantastic features but the one that I will talk about is the .travis.yml file. To be able to start using Travis in your Github project, you just need to grant Travis permissions and add a simple file like this on your project root: language: java

install: mvn install -DskipTests=true

jdk:

• openjdk7

notifications: email:

- some@email.com

Very easy right? Now every time that some push is made to Github, Travis will run maven install skipping the tests, using openjdk7 and emailing success of failure notifications for some@email.com. No web UI intervention. Just a human-readable file making all the configuration. Just like JJB.

Jenkins + JJB + Travis

So, why not replicate the process to Jenkins? Let’s add a configuration file to the project directory (let’s call it “jobs.yaml”) file, with the CI tasks and even the CD tasks. Now we just need Jenkins to be aware of it. How?

Our strategy is very simple. Each project (let’s say we have a “Project_x” project), beyond their jobs for building, testing, and deploying, has also a “jjb” job (that we call “Project_x-jjb”). This job is responsible for reading the YAML file and (re)loading the configuration of the jobs. Then it triggers the first job of the pipeline (in our case the “project_X-build” job.). From then on, the jobs triggers the next job in the line in order to do the project pipeline.

Because an image is worth more than a thousand words…

Do we need to create the Project_x-jjb by ourselves? Of course not! That’s the magic. We have a project called “meta-jjb-root” having its own job and repo containing macros and templates for JJB but also three crucial files for all this process:

1. builder.sh: responsible for updating the macros and templates in the Jenkins Master so it can be used by all the projects-jjb jobs, execute the meta_repositories_jobscreator.py script and run Jenkins job builder against the output of ‘meta_repositories_jobscreator.py’.
2. meta_repositories_jobscreator.py: generates a YAML file with the job configuration for each repository (project) listed in repository_list so we can automatically create the projects-jjb jobs. repository_list.yml: when you want your project to use this magic, after adding the jobs.yaml to your project, you need to add an entry in this list with your SCM repository link.

To make this more beautiful we have some tricks:

The project_x-jjb job will only execute if there is any change on the jobs.yaml file

1. Every other jobs for project_x should not execute if there is any change on the jobs.yaml file
2. We can also create views in the same way! Adding the file views.yaml and using the open-source project jenkins-view-builder.

Try it yourself!

Finally, tired of theory, you probably want to see this in practice. You can test it by yourself by cloning the following projects:

• cfgmgmtcamp_meta-jjb-root - Where you can find the builder.sh, meta_repositories_jobcreator.py and repository_list.yml files that I referenced before. There is also a Vagrantfile, so you just have to vagrant up and you have a Jenkins supporting ‘Jenkins a la Travis’ process running in ubuntu.
• cfgmgmtcamp_jenkins_a_la_travis where there is the folder .jenkins where you can find the jobs.yaml and views.yaml files.