“Where is my new feature?”
“The team said it was easy!”
“It’s been months, and I still don’t have it in production. Why is it taking so long?”
Have you had this conversation with your team, only to get a bunch of excuses about the build tools, governance, environment setup delays, and other unforeseen issues? If so, this post is for you. The good news is that you can help get things back on track and see improvements in time to market. Through process analysis of your current build and deploy process you can improve the throughput of all your development teams.
Process analysis is not a new concept.
In the 80s automotive manufacturers focused on improving quality to compete with the Japanese brands that were gaining market share. Systems like TPM, ISO, and others were introduced to provide guidelines for documenting processes for clarity, consistency, and quality. One such technique is called Single Minute Exchange of Dies or SMED.
SMED includes five steps focused on breaking down a process and improving the downtime it takes to perform a task.
- Measure each stage of the process from the start of the job to completion.
- Classify the work into two categories: internal or external.
- Internal steps occur during a shutdown or stoppage of work
- External steps are taken while things are running
- Convert internal steps to external steps where possible.
- Simplify and improve internal steps.
- Reduce the amount of external work.
By applying SMED to your development process you can identify and reduce the time needed for certain steps.
SMED at work
Begin by outlining all the steps in your process at a high level. Start where you believe you can make the most significant impact, or map out the entire process—from when a product manager first drafts new requirements.
In the following example, we’ll focus on the build and deploy process, which includes two main steps: build and deploy. In step 1, we start with the build process, breaking it down into smaller actions until each one can no longer be reasonably divided.
The process starts when the developer commits a change to the code repository. No time is required for this step since the process is not initiated until the step is complete. Your steps and analysis for the build process may look something like this:
- Commit polling: max wait 30s
- CI Build scheduling: 20s
- CI Quiet time wait: 60s
- CI Agent acquire: 90s
- Branch checkout: 30s
- Compile: 17s
- Unit testing: 4m 23s
- Report collecting: 12s
- Archiving: 45s
- Checkmarx scan: 13m43s
- Blackduck scan: 23m9s
- Sonarqube scan: 10m33s
- Integration/E2E testing setup: 5m53s
- Integration/E2E testing: 18m3s
- Integration/E2E testing cleanup: 20s
- Initiate deploy: 5s
Total build time: 1h 21m 13m
This simple set of data can be charted using any chart type, line graph, bar graph, pie chart, etc. For this example, it is easily seen that our 3 largest tasks in the build are Blackduck scanning, Integration/E2E testing, and our Checkmarx scan.
Next steps: Separate internal and external steps and externalize internal steps.
Elimination is the simplest option, but each step is required by the company’s governance policy and the team’s standard operating procedures. Instead, we could consider deferring or gating steps. For example, every time code is committed, not all steps need to be run immediately—they can be externalized. Several steps can act as gates to move changes to the next environment. We could also perform integration and end-to-end testing after deployment to the development environment, removing the need to set up and clean up a dedicated testing environment. This change reduces build time by over 24 minutes, though it adds 18 minutes to deploy time.
Another idea is to eliminate the automatic deploy initiation, making deployment a manual process that only uses the latest or configured build archive. This approach separates build and deploy functions, reducing the number of deployments needed. By doing this, we adopt the 'build once, deploy multiple times' principle from DevOps. An added benefit is that if deployment issues arise, we can attempt redeployment without rebuilding the entire application.
Next Step: Improve remaining internal steps
We should ensure that our process halts if any scan step detects a failure, then arrange the steps in order of runtime efficiency. If we move the Sonarqube step to before the Checkmarx scan we can reduce the feedback loop time. Leaving the expensive scans only when our faster scans have passed. We might also look at running the scans in parallel to reduce the wait time. That would save an additional 23 minutes.
After these changes our process would look like this:
- Commit polling: max wait 30s
- CI Build scheduling: 20s
- CI Quiet time wait: 60s
- CI Agent acquire: 90s
- Branch checkout: 30s
- Compile: 17s
- Unit testing: 4m 23s
- Report collecting: 12s
- Archiving: 45s
- Checkmarx / Blackduck / Sonarqube parallel scans: 23m9s
Total build time: 32m 56s
We've reduced build time by over an hour through a few process improvements.
Multiply this by the number of team members and the number of times a day the building process is executed, and you can see how much time is being saved. If we have a team of 4 that checks in code on an average of twice a day, that’s 8 hours of savings every day.
Final Step: Improve external steps
Now that the deployment process has been externalized, we can repeat these steps to improve performance.
Final Thoughts
The SMED process offers significant benefits for DevOps teams aiming to reduce delays. By breaking down tasks into smaller, manageable components, SMED enables teams to identify and eliminate bottlenecks, streamline handoffs, and reduce setup times. This approach improves deployment efficiency by isolating steps that can run in parallel, automating where possible, and simplifying workflows. Ultimately, SMED helps DevOps teams minimize downtime, speed up build and deploy cycles, and enhance overall agility, allowing them to deliver updates and fixes to end-users faster and more reliably.
Try applying SMED techniques to some of your processes and see how much time and money you can save your company.
Discover more strategies to accelerate your team’s workflow by emailing us at contactus@callibrity.com and schedule a time to connect.