Jenkins is a leading open source automation platform for CI/CD orchestration tool to deliver and deploy our applications, workflows, and automate tasks to systematize the software development lifecycle. Jenkins provides hundreds of plugins to support building, deploying and automating any project that enhance and extend Jenkins capabilities and integrate Jenkins with other software like Artifactory, Github, P4/Perforce, Slack, Splunk, Jira etc.

— Why Jenkins?

Jenkins facilitate timely, scheduled updates to our Applications, and furthermore, enabling and supporting tasks like System Tests/Code Coverage, Monitoring Systems, Automation Tests etc.

Jenkins Best Practices:

1. Utilize RBAC, this is a requirement for projects with mandatory NDAs
2. Backup Jenkins Home
3. Avoid resource collisions by using Locks and Reservations
4. Create scalable Jenkins Pipelines with Shared Libraries (multibranch pipelines)
5. Tests, include System Testing as part of our pipelines
6. Upgrade Jenkins

— Why upgrade Jenkins?

Keep Jenkins up-to date as the current versions will have the latest security patches and fixes.

— How to upgrade Jenkins?

1. Backup Jenkins Home (i.e. /Users/dalevu/.jenkins on a Mac (/opt/homebrew/Cellar/jenkins-lts/2.414.3/libexec/jenkins.war), For enterprises, we usually utilize a filesystem like for example ~/jenkins-data/test/jenkins)
2. Backup/Rename jenkins.war
3. Download the new Jenkins version. Release Notes
4. Put Jenkins into maintenance mode by clicking “prepare for shutdown” from Manage Jenkins (we will need to be an admin). Quiet Restart is recommended before a maintenance if we have this plugin
5. Stop Jenkins
   • Run command, brew services stop jenkins-lts
   • Or we can also kill the jenkins service (gracefully)
     • Run command, ps -ef|grep jenkins
     • Run command, kill -1 <pid of Jenkins>
6. Copy the new Jenkins version, jenkins.war
7. Start Jenkins
   • Run command, brew services start jenkins-lts
   • Or we can also execute
     • Run command, java -jar jenkins.war –httpPort=<our port> (we may have to specify running Jenkins in the background, in which case, run command, nohup java -jar jenkins.war –httpPort=<our port> &)

Notes to consider:

• Using Nginx in front of Jenkins which allows us greater throughput/threads and also enable us to block project access until maintenances are completed.
  • For example:
    • location ~ /(<path>) { deny all; return 404; }
• Update via RPM (e.g. like with Cloudbees Jenkins – cloudbees-core-cm)
  • Red Hat / CentOS
    • Beginning in CloudBees Jenkins Platform version 2.319.1.5, the epel-release package is required to upgrade CloudBees Jenkins Platform using an RPM, however some CentOS distributions do not include it. For those CentOS distributions, run the following command before you begin the installation procedure: 
    • CAP – Auto upgrade/downgrade plugins – turned on
    • Quiet Restart enabled
    • Put Jenkins into maintenance mode (Prepare for Shutdown)
    • Systemctl Stop Jenkins (cloudless-core-cm)
    • sudo yum install epel-release.
    • sudo wget -O /etc/yum.repos.d/cloudbees-core-cm.repo https://downloads.cloudbees.com/cloudbees-core/traditional/client-master/rolling/rpm/cloudbees-core-cm.repo
    • sudo rpm –import https://downloads.cloudbees.com/cloudbees-core/traditional/client-master/rolling/rpm/cloudbees.com.key
    • yum install cloudbees-core-cm
    • We may need to update the symlink (possibly, most likely not)
    • Systemctl Start Jenkins (cloudless-core-cm)
    • Quiet Restart removed 
    • Cancel Prepare for Shutdown

— How-Tos?

• Developing a Pipeline
• Using a jenkinsfile
• Scheduling jobs
• Log forwarding to Splunk
• Restoring Jobs
• Backing up Job configurations
• Agents (i.e. On-prem, Openstack, AWS, Linux, Windows, Kubernetes)
• Using Ephemeral agents
• Cancelling hung jobs
• Re-set build numbers
• Remove lockouts-anyone can do anything
• Release Locks
• Agent Pools – using labels
• etc. etc.