Software development in a commercial context is usually done for one of the following reasons (or a mix of them of course):
- Legal or regulatory requirement
- Enabling factor for the business model
- Increase of efficiency
All those reasons have a substantial impact on the profitability or even the survival of the business. The latter can be true not only mid- to long-term. It is therefore critical to have the software delivery process under control. In particular all possible measures need to be in place to avoid software bugs.
While some bugs are just annoying, in a critical place it is possible that they wipe out a large and otherwise successful company rather quickly. Back in 2012 exactly this happened to Knight Capital Group. It took only one hour and the loss was USD 440 million.
While it should not be your only means, CI/CD can be a powerful factor to help with software delivery. This article looks at a few aspects how this works. Also, where I think it is helpful, I will touch upon adjacent topics to give additional information.
Why should I care?
Ultimately moving towards CI/CD is more a change management project than a technical one. As with so many things, the technical challenges, while existing, are usually smaller than the non-technical ones. You, as a developer, can therefore shrug and say that it’s none of your business. Or you you grab the chance to demonstrate your value and especially future potential. Because, let’s face it, the world is moving towards CI/CD. The benefits are simply way too big to ignore it.
If you as a developer don’t want to find yourself on the loosing side of the IT job market sooner or later, you must act now. Pure coding capabilities are already partially replaced AI and that trend will continue. No, I absolutely don’t join those that say AI will completely take over this. But it already handles things like boiler plate code. Like robots have changed manufacturing, and ERP software transformed the back-office, the requirements for software developers will also change.
If you are currently coding on the level of copy-pasting snippets from Stack Overflow and other places, you are already at risk. On the other hand the demand for people who understand the business and can translate business requirements into IT solutions, will continue to thrive. Therefore this article will not be about technical details, it will cover an even more important part: How to help the organization move into the right direction.
Yes, there are developers who are solely interested in the technical side and don’t want to be bothered with “business” aspects. Folks, this is a severely career-limiting attitude. It is the business that pays your salary and therefore calls the shots. Whether you like it or not, understanding their needs and thinking will help your career much(!) more than yet another Git training.
What is CI/CD all about?
For the context of this writing I will focus on the outcome side of CI/CD. In a nutshell, and this is leaving out quite some details, the result of applying CI/CD will be the ability to deliver better software faster. Yes, you read correctly. While many people think that those two aspects are fundamentally in conflict, they are not. You might now say: But haven’t we all seen that a team can either take shortcuts, using the saved time to deliver features faster, or spend more time on quality? Yes, we have. But that is a symptom and not the root cause.
What people have actually witnessed, is a competition for a scarce resource (i.e. time) when using a vastly inferior approach to software delivery. It is a bit like building a house with no power tools but purely manual labor (like let’s say in 1925), and without proper drawings. So things often have to be redone a couple of times. In addition we have the future owner, who only cares about the appearance and knows nothing about statics or engineering in general. This person is constantly breathing down the necks of the workers and questioning their decisions, with a lot of time used up for those effectively useless discussions.
In such an scenario it is obvious that meeting a deadline that was unrealistic right from the start, means to compromise on features or quality. Yet, one crucial aspect is still missing to make the comparison complete. It is the fact that the to-be-built house is not properly thought through and no plans exist. Instead the future owner constantly throws in new ideas and something that was critical last week shall now be removed altogether.
And most if not all of the workers have never built such a house. They need to figure out things while they are going. So sometimes it takes 10 times longer, and sometimes they need to cover up the fact that something is not possible at all. Because if they told the owner, they would be fired immediately.
What CI/CD does is introduce some lab, where people can quickly build small components for the house. To the outside they comply with requirements like e.g. size. Internally they are built with the current understanding, but it is possible to change the internal structure without changing the size. So if a requirement changes, based on an improved understanding of the overall requirements, the impact (or shall we say blast radius?) is comparatively limited. I know the comparison is not 100% accurate. But I think it helps to understand the challenges we often face in software development.
So, again, from a business perspective CI/CD is an approach that acknowledges the fact that we are working with a lot of uncertainty and moving parts. Instead of denying reality, we embrace it and by that everybody wins.
Continuous Delivery (CD)
Let’s start with Continuous Delivery (CD) because it is the result, while Continuous Integration (CI) can be seen as a prerequisite.
Continuous Delivery (CD) is the ability to always deliver working code.
In other words, if the customer (can of course also be an internal one) comes to you and asks “When can you deploy the latest features that are complete?”, the answer is by definition always “now”. Of course that leaves out other steps like internal communication, training, etc. that are likely necessary. But as far as you are concerned, you are good to go. Not only every now and then by happy accident, but all the time.
This is in contrast to a development team that can only ship a new version of their software after an explicit phase of consolidating loose ends into something that works. I have been “fortunate” enough to witness a few projects that worked like this. It always meant that at least 2-3 weeks’ notice was necessary for a deployment. During that time everybody was panicking and somehow trying to make things work.
Tough luck if component A had already implemented the new interface that had to be finished only in 2 months, while others were in different stages of doing so. You will loose the first 3 days (or more) by just trying to sort out things like that. Then a lot of long hours are needed, and usually there will still be some bugs left. Because you needed all the time to integrate the loose ends and fix at least the worst bugs. With no time left for proper testing, what do people expect?
This nightmare scenario begs the question: Why would anyone want to do it like that and not be able to deploy all the time?
Because there was a time when most projects were not able to do it any other way. Most importantly, we simply didn’t have the required tooling. Let me give you my personal highlights from a specific project:
- No TEST environment. Who needs more than DEV and PROD? Before virtualization, bringing in an additional server would usually cost between 10k and 50k EUR/USD (incl. company-internal uplift of 50-200% on top of vendor price) and take between 1-6 months with all the approvals, installation in the data center, firewall configuration, OS installation etc.
- No VCS (Version Control System). This was before Subversion, and Git was not even thought about yet. CVS was not supported by the proprietary IDE and even if it was: Have you ever worked with it? I’d rather do manual file copy than use CVS ever again in my life.
- No CI server invented yet.
- No automation of any kind for build, unit testing, or deployment.
In the mid to late 2000s these supporting factors slowly came into being. And one of the most important ones were CI servers. They automated the build of the code from the VCS, where Subversion as the successor for CVS was a huge improvement. Early versions were still a bit flaky, compared to the current state. Still it was finally a reasonably stable and easy to use tool.
Continuous Integration (CI)
So, like before for CD, what is the core purpose of CI?
Continuous Integration (CI) is the approach to constantly make sure that code changes from a multitude of developers do not break anything (regression).
This is also the reason why long-lived feature branches are fundamentally in conflict with CI. If you insist to work like that, you can of course do this. But please be aware that you are by definition not doing Continuous Integration anymore. Because you are not continuously integrating your changes, but keep them in isolation in your branch. What you are doing is running a CI server, but without doing CI.
There are basically two reasons I have seen, why people are doing this. On the one hand they are copying the approach of large open source projects like the Linux kernel. Depending on the circumstances that may indeed work well. It is just not CI. But given how different the organizational side is for a corporate in-house project, I am tempted to think that it is not the best possible solution.
The other big reason for using branches is that people are afraid of trunk-based development (TBD). I fully understand that it looks daunting at first. But so many people are good at it now and they all started as rookies (including of course me). So this is mainly a training and cultural issue. If I am working in a context where asking for helping or making a mistake can be a career-limiting move, I don’t blame people, if they resist TBD.
The transformation
Any business transformation is not only about the actual changes, but to a large extent about the communication side. People have reservations against change, some more so than others. Because there is always a risk for themselves and their position. So formally introducing CI/CD is really a change management project. That means getting a lot of people on board across many different levels of the organization.
If your organization has so far not used CI/CD for software delivery, people will come up with various reasons, why CI/CD is either not necessary or even does not fit. And I can understand why they do. Because it is innovation and that is always something difficult to introduce into an organization. Plus it will usually hurt MBOs. Why would I want to cut off my arm, metaphorically speaking?
My personal journey
So let me take you on a very personal journey. My very own journey with CI (and later CD) that started back in 2008. I was basically forced into the subject by a prospect. They had a large internal development team with a lot of really competent folks. Their “arsenal” include CI, MDA (Model-Driven Architecture), AOP (Aspect-Oriented Programming), and some others things that were the “latest shit” then. Of course, when we pitched our software, they wanted to see how it fits into all this.
Well, it didn’t do too well in terms of official out-of-the-box support. So I was tasked to figure out how to convince the prospect that we could still somehow fit in. Because they otherwise really liked our product. It turned out that the product was well suited, because it was working with files instead of storing development artifacts into a database. All that was needed was a bit of additional tooling or at least good documentation.
I then developed a bit of custom tooling and was overwhelmed. Not by my tooling (too quick and dirty), but how much CI improved the overall development experience. Yes, there was a lot of rough edges but still. Ever since I have been a strong proponent of CI and all too often irritated people. You don’t necessarily make friends by lobbying for a new idea (more on that below).
Having done CI for so long it surprises me that a whole 16 years later we still are where we are as an industry. Yes, that is not a unique issue. Knowledge management has been around for well over 25 years and not improved much in corporate reality. But is that really a good excuse?
Part of the problem is certainly that the average professional experience of software developers is about 5 years. There is only so much you can learn in a given amount of time. And what is more important? Being able to deliver the latest feature or lobby for CI?
Combine this with a culture that does not tolerate mistakes or admitting that you don’t know something. Your only option is to deny relevance. In other words: You will say that CI does not help in your particular situation, because it is really so very different from the rest of the world.
What does not work
Many people try to introduce new ways of working by lobbying for them. They create nice slide decks (which takes a lot of time) and show them to their boss. They explain the benefits and want the boss to be excited about it and change things tomorrow. Of course that does not work.
As a manager I am in a very different position from that of an individual contributor. My value to the organization is determined by how well I deliver what is expected of me from “above”, rather than going on some personal “crusade”. In addition, and all too often there is a conflict here, a significant part of my salary depends on meeting certain goals. Those goals have been determined (i.e. made up in a quick-and-dirty way more often than not) many months ago. Since then half the org-chart has been turned upside down and we have a new “strategy” (whatever that means).
So what do I do as a manager? Assuming that it was not the Peter-principle that brought me to my job, I really like the fact that one of my directs shows this kind of initiative. That is a capability you don’t find very often. Still, the last thing I need is yet another moving target to juggle. I will therefore commend the direct for his or her initiative and the innovative way of thinking. And then I will bury the whole thing quietly. Yes, I will need to manage the direct’s disappointment. But that is the smaller issue.
What worked for me
My first attempt to change a certain procedure (it wasn’t related to CI at all) on the team level ended up more or less like in the section above. At the time I did not realize the underlying mechanisms. But the outcome was still very clear. So I decided to change my approach.
It is a really simple one. Don’t ask for permission but choose a small area under your own control, where you can implement what you think makes sense. Of course it should be something truly suitable. That means at least two things: 1) If your idea is not as brilliant as you thought, make sure to limit the blast radius. 2) Look for something where the benefit is as obvious as possible.
That way, if everything works out, people will instantly recognize the value. For you that may not be so different from the slide deck mentioned before. But managers are bombarded with pretty slides that all promise lots of things. Yet most of it is pure BS. How should they know the difference?
Demonstrating value that was created with very little additional effort, really makes you stand out. It shows business acumen and entrepreneurial thinking. And your manager can take credit for it ;-). The latter is a somewhat dicey point, but it can make sense for you to share the recognition. This gets you an ally and that is never a bad thing.
So far on the fundamentals. Let’s now look at the CI/CD side.
Starting CI/CD in your organization
Your best bet for introducing CI/CD is a two-step approach. It needs to bring together two conflicting aspects. Because on the one hand organizations only change fundamental ways to operate if the benefit is big enough. A mere 5% improvement is not sufficient. Or as someone once said: We can easily get 5-10% more output by yelling at people.
On the other hand organizations are risk-averse. And even more so, the higher you go in the hierarchy. Simply because people usually made their career by avoiding mistakes, not by being innovative. So forget the idea to go over your boss’ head. It will backfire on all levels.
You need to demonstrate two things at the same time: Considerable value and manageable risk. That sounds contradictory. But only if you want to do everything at the same time. Don’t do that and instead separate these aspects.
Proving that it works
As the first step we look at the risk management side. That can be a small official project, or something more “unofficial”. Whatever works best under the circumstances. The specific steps depend on your organization, industry, and a lot of other factors incl. externals ones that are completely outside your control. If you are uncertain here, get help from others. This can of course be a paid consulting engagement and JahnTech has considerable experience here.
But you can also start by getting in touch with people on suitable social media or the Internet in general. As an example, I spend quite some time on a public forum about an open source storage system and provide advice for free. Or you can read a good book. I can whole-heartedly recommend the one that more or less invented Continuous Delivery (CD) as a discipline. It is the classic “Continuous Delivery” by Jezz Humble and David Farley.
Then there is online trainings (incl. the upcoming one from JahnTech). They have the benefit to be curated (compared to e.g. YouTube) and you can choose between a lot of offerings. Also learning platforms like Udemy and Skillshare have many courses on the topic and they are usually pretty cheap. I must admit, though, that the curation can be somewhat limited on those platforms. I have seen courses (on another topic) where the freely available section (that was supposed to support the sales) was so bad, I wouldn’t even have watched it on YouTube.
And last but not least you can find a lot on YouTube. Just make sure to verify the content and do not rely on a single opinion. Some YouTube videos are more about entertainment then providing accurate information. Plus there is always the context. Some things are more or less a “universal truth”, while others only apply to a very specific set of circumstances. I have found that those circumstances tend to be omitted. For CI/CD my impression is that the majority is more a click-through demo, but without much conceptual background or information why you actually want to do something (or not). Of course, your experience can be very different.
Whatever your approach, don’t try to completely re-invent the wheel. You have more than enough to do looking at the specifics of your organization and project (those are different levels and you are well-advised to keep them that way).
Explaining the value (also for people)
The second step is of course the truly difficult one. So far we primarily looked at technical things for the risk management side. Now comes the value and convincing people of it. The critical part that many technical people miss is that we need to put a lot of focus on what a change means for individuals. While few will admit it, that is usually the main issue to overcome.
Of course the official side is the impact for the organization. And that must be done first. You will find a lot of content freely on the Internet about how organizations like Amazon deploy tens of thousands of times per day and how that saves them millions. In terms of aspiration that is a good statement. But if your organization is just starting its journey, it can backfire. Because the by far number one argument against CI/CD is “we are different and it therefore doesn’t work for us”.
Today you mainly get that argument relative to companies like Amazon, Meta (Facebook), Netflix, etc. People nowadays think that CI/CD is only for the “big boys”. That is funny. Because 10 years ago it was just the other way around. One of the large organizations known for their introduction of CI/CD was HP for the firmware of their laser printers. Details can be found in the book “Leading the Transformation: Applying Agile and DevOps Principles at Scale” by Gary Gruver and Tommy Mouser. The general opinion then was that CI/CD was only for smaller teams with less complicated code. While laser printer firmware may not sound exciting to you, its complexity is remarkable. Just think about the hard real-time requirements. If the toner powder comes a tenth of a second too late, the printout is for the trash can. There are many more aspects, but that alone nicely illustrates it for me.
Today a lot of scientific research exists on the benefits of true agile and CI/CD. As a book I can recommend “Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations” from Nicole Forsgren, Jez Humble, and Gene Kim. Also, the “State of DevOps” report is worth looking at. I have heard rumors that newer versions have come to be more of marketing material. But any older version will still cover the core points well enough.
So, once the organizational side has been addressed, we need to look at the individuals affected. I cannot give a lot of advice here, because this will always be highly specific to your situation. But in general, after you explained the value for the organization, you need to sell(!) it to the people one by one. I have had good experience with asking people for their MBOs in general. As long as the specific numbers are kept out, most people I met were rather open here. At the end of the day you offer to help them explore the opportunity for a better target achievement and thus get more money or even a promotion.
Still, it is a difficult undertaking. So don’t be too disappointed if it doesn’t work the first time. You are in good company, in that hundreds or even thousands of people have gone through the same frustration. The important thing is not to give up. Try again and again. Each time you will be better. Play around with approaches, exchange your experience with others, and in doing that also extend your network. People will notice what you are doing, even if they say “no” a few times.
This is also what my last point in this section will be about. If, despite all your hard work and good intentions, the answer is “no”, you have to make a tough choice. What are you going to do about this “refusal”? One way to look at it, is what you want to prioritize: The result or the relationship?
Result means that you are not willing to accept the “no”. That alone is not (yet) a problem. But it becomes a huge one, as soon as it means to go up against the person who said no. And that is what happens from most actions you can take. Asking the same person again and again, makes you look like a stubborn child. Much worse would be to decide that the whole things needs to be escalated. Nobody likes it, if their directs go to their superior. In many cases you can expect serious retaliation here.
So at the very least I would recommend to not argue too hard against the “no”, once you learn about it. Instead you can say something like “I must confess that I hadn’t expected this. Can you perhaps give me some more details?” That additional information can then be used to think things over. Perhaps you realize that there was simply a misunderstanding. Or you hadn’t been aware of a crucial aspect. Or …
If at all possible, please try very hard not to damage the relationship with this person. It will come back and bite you in so many ways. And yes, I have made that mistake early on in my professional live. With only the best of intentions, I had more or less thrown someone under the proverbial bus. I quickly realized it, but the damage was done. And while we probably all make that mistake sooner or later, we should just as well try to avoid it.
In closing
The length of the article tells you that this is very interesting topic to me. Yet due to the highly individual nature of the challenge at hand, it can only go so deep. If you have a specific situation and would like me to write about it, please get in touch. As in all my posts and articles the details will be anonymized and/or obfuscated, so that confidentiality is ensured.
The most important thing with CI/CD is to get started today. There are so many free resources available, that there is really no excuse to stall this ;-). And again, I am more than happy to receive questions and comments. Each situation is different and I am eager to learn from your challenges.
If you want me to write about other aspects of this topic, please leave a comment or send an email to info@jahntech.com. The same applies if you want to talk how we at JahnTech can help you with your project.
© 2024 by JahnTech GmbH and/or Christoph Jahn. No unauthorized use or distribution permitted.
