An Open-Source

Chef Cookbook CI/CD Implementation

Using Jenkins Pipelines

Steffen Gebert (@StGebert)

Config Management Camp, Gent, 06.02.2017
Agenda
Context

Motivation

Jenkins Pipelines (in general)

Chef CI/CD using Jenkins Pipelines

2
About Me
Researcher / PhD Student
(Software-based Networks)
2011 - 2016

Core Team Member

2010 - 2013
Server Admin Team Member
since 2011

Co-Organizer DevOps Meetup Wrzburg

since 2016

3
PHP-based Content Management System

Picture by Benjamin Kott 4

 TYPO3 Open Source Project
Established open-source
community since 1997 Thanks for partially
funding my work and
covering travel costs!
TYPO3 Association as non-profit
organization

Server Admin Team responsible

for project infrastructure *

Picture by Stefan Busemann

* listed on opensourceinfra.org
5
Picture by Daniel Ptzinger: http://www.typo3-media.com/blog/article/typo3-wallpaper-for-wide-screen.html
Based on image by Mike Swanson: http://interfacelift.com/wallpaper/details.php?id=1402
 Chef at TYPO3
Started with Chef in 2011
Started with librarian-chef (monolithic repo)
Took years to get rid of monolithic repo

Currently operating 45 Debian nodes

Usual tooling: berkshelf, test-kitchen
berkshelf-api-based /universe (connecting to Chef Server 11*)

Small team of volunteers, varying degree of Chef knowledge

* because of laziness 6
 Demo Time!

Picture by annca/ pixabay:

7
https://pixabay.com/en/popcorn-cinema-ticket-film-1433327/
Working with Chef
(and knife, and berks, and..)
 We have
workflow problems!

Picture by tpsdave / pixabay:

9
https://pixabay.com/en/firefighters-fire-flames-outside-115800/
10
11
 12
https://blog.twitter.com/2017/the-infrastructure-behind-twitter-scale
Related Work
zts/cooking-with-jenkins
Provides resource to create single Jenkins freestyle jobs per cookbook
Executes pre-defined rake tasks
chef-solutions/pipeline
Reads cookbooks from a Berksfile enclosed in main chef-repo
Creates single Jenkins freestyle job per cookbook
Chef Automate
Commercial
Little known to me, what's going on under the hood

13
Survey
Whos Chef cookbooks (Puppet modules/etc) are tested in CI?

Who has an automated release process?

14
Jenkins Pipelines

15
Jenkins Pipelines

16
17
18
Introduction to

Jenkins Pipelines
Jenkins Pipeline DSL
Groovy DSL sh "make"
Provides steps, like sh sh "make install"
Defined in the Jenkinsfile
Can use Groovy/Java logic (Chef users like that idea, right?)

Pipeline plugin (workflow-aggregator)

Formerly called "Workflow"
Open-sourced one year ago
Whole bunch of plugins (10+)

20
Jenkins Jobs as Code?
# Job DSL example
Jenkins Job Builder job('my-project-main') {
Python / YAML based scm {
From OpenStack git('https://github.com/...')
}
triggers {
scm('H/15 * * * *')
Job DSL plugin (job-dsl) }
Also Groovy DSL publishers {
Supports many, many plugins downstream('my-project-unit')
}
Creates only single jobs, not pipelines
}

21
 Cookbook Pipeline?

sh 'berks install'
sh 'foodcritic'
sh 'kitchen test'
sh 'berks upload'
Picture by BarnImages / pixabay:
22
https://pixabay.com/en/sushi-food-japanese-fish-seafood-789820/
Stages
Allow visual grouping
stage('lint') {
sh 'foodcritic'
sh 'cookstyle'
}
stage('resolve dependencies') {
sh 'berks install'
}
stage('test-kitchen') {
sh 'kitchen test'
}
stage('upload') {
sh 'berks upload'
}
23
Nodes
Allocates a Jenkins executor (master/slave)
node {
stage('lint') { Pipeline
sh '..'
Stage Stage Stage
}
stage('resolve') { Step Step Step
sh '..'
Step Step Step
}
}

Optionally with label

Executors (node):
node('chefdk') {
Agent 1
stage('lint') {..} Master
stage('resolve') {..} Agent 2
} 24
Ready, Set, Go!

25
Multibranch Jobs
Scans repo for branches containing
Jenkinsfile
Automatically creates (deletes) jobs

Works with plain Git

Works better with GitHub / Bitbucket (via API calls)

26
More Steps to Come..
Jenkins Plugins can contribute DSL steps

27
 Global Variables
Environment variables: env.BRANCH_NAME
env.BUILD_URL

Information about current build: currentBuild.displayName = "v1.2.3"

CurrentBuild.rawBuild.getCause()
currentBuild.result = 'FAILED'

Parameterized build*: params.myparam

params.deployEnv

* well-hidden feature 28
see https://st-g.de/2016/12/parametrized-jenkins-pipelines
 Now Copy & Paste?

Picture by aitroff / pixabay:

29
https://pixabay.com/en/stormtrooper-star-wars-lego-storm-1343772/
 Pipeline Shared Libraries
Additional code to be used by Jenkinsfiles
Loaded from SCM repo

Inclusion via
@Library annotation in Jenkinsfile
Configuration of enclosing folder
Jenkins global configuration*

* then library code is "trusted" 30

Configuring Global Pipeline Library
Jenkinsfile:
@Library('chefci')
org.example.Pipeline ...

Load implicitly would load it

Don't play
well together
without @Library
Use non-default branch/tag
@Library('chefci@testing')

31
 Pipeline Shared Libraries
(root)
+- src # Groovy source files the magic
| +- org
| +- foo actually code
| +- Bar.groovy # for org.foo.Bar class
|
+- vars
| +- foo.groovy # for global 'foo' variable
feel like
| +- foo.txt # help for 'foo' variable functions
|
+- resources # resource files
| +- org static files
| +- foo
| +- bar.json # static helper data for org.foo.Bar
32
https://jenkins.io/doc/book/pipeline/shared-libraries/
Global Variables
Can store global state (for the current build)
Can behave like steps
Useful to simplify pipeline code
Hide implementation details from users

# vars/deployTo.groovy # Jenkinsfile
def call(def environment) { node {
echo "Starting deployment to ${environment}" sh "make test"
withCredentials(..) { deployTo "production"
sh "rsync avzh . ${environment}.example.com" }
}
}

33
More Magic: Global Library Classes (src/)
Groovy classes implementing arbitrary logic
Make use of pipeline DSL steps
Use other Jenkins functions
Import and use Java libraries

How we implement our pipeline

34
Scripted vs. Declarative Pipelines
# Declarative Jenkinsfile
pipeline {
Scripted pipelines agent label:'has-docker', dockerfile: tru
environment {
Just (imperative) Groovy code GIT_COMMITTER_NAME = "jenkins"
The original implementation }
The approach used here stages {
stage("Build") {
steps { sh 'mvn clean install' }
}
Declarative pipelines stage("Archive"){
// ..
Hit the 1.0 release last Friday }
Can be validated prior to execution }
Ease some tasks, i.e., failure handling post {
always {
Visual Pipeline Editor plugin deleteDir()
}
success { 35
mail to:"me@example.com", subject:"SUC
Jenkins Pipeline Summary
Groovy DSL for specifying pipelines as code

Ideally stored within the tested repo

Extract shared functionality to library repos

Pipeline execution triggered on push event

36
An Open-Source Chef Cookbook CI/CD Implementation
Using Jenkins Pipelines

jenkins-chefci
 Forbidden Commands
$ berks upload
$ knife cookbook upload
$ knife data bag from file
$ knife data bag delete
$ knife environment from file
$ knife environment delete
$ knife role from file
$ knife role delete

$ git pull
$ git commit
$ git push

Allowed Commands
Pictures by PublicDomainPictures & serrano / pixabay: 38
https://pixabay.com/en/costume-demon-devil-board-female-15747/
https://pixabay.com/en/bebe-girl-child-child-portrait-1237704/
Meet the jenkins-chefci cookbook
Sets up a Chef CI/CD infrastructure using Jenkins
Sets up Jenkins master
Installs ChefDK, configures credentials
Configures job that scans a GitHub organization
Configures our shared pipeline library
https://github.com/TYPO3-infrastructure/jenkins-pipeline-global-library-chefci/

Add this Jenkinsfile to your cookbooks:

# Jenkinsfile
@Library('chefci') _
org.typo3.chefci.v2.cookbook.CookbookPipeline.builder(this, steps)
.buildDefaultPipeline().execute()
39
Meet the Shared Pipeline Library
Implements pipelines for
Cookbook testing, versioning & upload
[WIP] Chef-repo's data bags, environments, roles

Contains couple of Groovy classes

Some helper classes
*Pipeline as entry points from Jenkinsfile
AbstractStage as base class for.. Stages

Yes, we're at v2 already J

40
 Executed Stages & Steps
Linting
Foodcritic
Cookstyle
Build
Berkshelf install (endpoints for Supermarket & Chef Server)
Berksfile.lock in Git for top-level cookbooks
Acceptance
Test-Kitchen (using kitchen-docker)
Publish
Version Bump (thor-scmversion, based on user feedback)
Berkshelf upload (to Chef Server) 41
Picture by Kaz/ pixabay:
https://pixabay.com/en/hands-hand-raised-hands-raised-220163/
42
 Test-Kitchen
If there is no .kitchen.docker.yml, put default one
Use Jenkins' parallel step

node {
createKitchenYaml()
expected result,
stash "cookbook" not hard-coded
}

parallel(
"essentials-debian-86": { node {
unstash "cookbook"
sh "kitchen test --destroy always essentials-debian-86" }},
"essentials-ubuntu-1604": { node { .. } },
"full-debian-86": { node { .. } },
"full-ubuntu-1604": { node { .. } } 43
)
Test-Kitchen (2)
Parallel instances automatically derived from kitchen status

Log files of failed instances are archived

Less chaos than complete Jenkins log
Traditional UI does not clearly show failed parallel branch

Not limited to kitchen-docker

44
* cookbook configures this via API 45
46
47
48
The Art of Cookbook Versioning
Let's agree on SemVer
I do not agree with manual changes to metadata.rb

49
 Demo Time!

Picture by annca/ pixabay:

50
https://pixabay.com/en/popcorn-cinema-ticket-film-1433327/
Versioning
Based on thor-scmversion
Made by RiotGames for their cookbooks
Uses Git tags to derive current version
Version treated as "label", not part of the source code
# metadata.rb
name "example-cookbook"
version IO.read(File.join(File.dirname(__FILE__), 'VERSION')) rescue '0.0.1'

Jenkins pushes only Git tags, no code changes

51
Versioning (2)
Publish stage notifies via Slack

Except commit message subject

indicates version increase
Do this #patch
Add this #minor
Break this #major

52
"Main Chef-Repo"
Not covered here, but in site-chefcitypo3org cookbook
Freestyle job, defined via JobDSL
Uses script to parse git diff and
upload changes using knife

53
Additional Notes
Manually configured J organization-level webhook triggers
immediate builds
Everything else is done automatically, no "manual handholding"

54
Using jenkins-chefci
Ease of Use
Checkout cookbook from github.com/TYPO3-cookbooks/jenkins-chefci
Increase GitHub API rate limit
export JENKINS_GITHUB_LOGIN=johndoe WARN: allows commit
export JENKINS_GITHUB_TOKEN=123456supersecure
status update
Allow to steal your Chef credentials
export JENKINS_COPY_CHEF_CREDENTIALS=1 WARN: allows knife/berks
Run test-kitchen to access Chef Server
kitchen converge full-ubuntu-1604

56
(Potential) TODOs for you
Write your wrapper cookbook around jenkins-chefci
Point it to your organization
Add authentication, e.g., GitHub OAuth?
Fork the global library
Adjust pipeline to your needs
You don't have to agree with our (current) implementation

57
Outlook / TODOs
Documentation / blog posts
Move more stuff from site-chefcitypo3org to jenkins-chefci
chefdk() global function to run in chef/chefdk Docker container
Store Chef private key as Jenkins credential
Test using multiple chef-client version
Use JobDSL for organization-folder setup (#966)
Trigger downstream cookbooks
Read out dependencies and subscribe to successful pipeline runs
Paves the road to Policyfiles
Use Jenkins slaves
Collect chef-client deprecation warnings
58
Conclusion
Jenkins Pipelines allow to define pipelines as code
Groovy-based DSL allows programming of pipelines
Can definitively become complex, i.e., debugging
Jenkins Pipelines for Chef cookbook CI/CD
Running at TYPO3 since May 2016 (site-chefcitypo3org cookbook)
Public instance at https://chef-ci.typo3.org, open source from day 1
Warning: Many cookbooks still use v1 pipeline (git-flow)
jenkins-chefci cookbook as reusable implementation
Makes setup accessible to broader audience
No dependencies to other TYPO3 cookbooks
59
Further Reading
TYPO3's Chef CI:
https://chef-ci.typo3.org
TYPO3-cookbooks on GitHub:
https://github.com/TYPO3-cookbooks
TYPO3's Shared Global Library:
https://github.com/TYPO3-infrastructure/jenkins-pipeline-global-library-chefci/

Pipeline Tutorial:
https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md
Getting started with pipelines:
https://jenkins.io/pipeline/getting-started-pipelines/
Step documentation:
https://jenkins.io/doc/pipeline/steps/
Pipeline shared libraries:
https://jenkins.io/doc/book/pipeline/shared-libraries/
Notifications (Mail, Slack, etc.) in Scripted Pipelines:
https://jenkins.io/blog/2016/07/18/pipline-notifications/
Declarative Pipelines
https://jenkins.io/blog/2016/12/19/declarative-pipeline-beta/
https://jenkins.io/blog/2017/02/03/declarative-pipeline-ga/
60