Yesterday (Thursday, September 24th 2015) we were host to the fourth BredaPHP meetup
. This article is a re-cap of the first presentation that was given by me;(you'll find the slides for this presentation at the end of this article). I talked about the challenges we are facing within our Development and Operations teams. I also talked about how we - as a company - are moving forward, the choices that we are making and for what reasons.
Introduction into CM Development challenges
I want to share with our vision on future web development, the how, why and which technologies and ideas we are currently implementing. One important project that we are constantly improving is our branding website application which consists of cm.com
. It is during this project that we learned that development time decreased rapidly when we started using tools such Composer
These tools are meant to make our development lives easier and besides the obvious upside of these tools there is another thing we noticed right after we started using these tools. Because our development time decreased there were other delays that were revealed to us. For example, delays in testing (which - for the biggest part - we still do manually unfortunately), delays in deployment, and delays in network and server configuration. So first we tackled the time issue from a Developers point of view, new tools were introduced with which a Developer can create software more quickly and now we are ready to tackle the next set of challenges, two of those challenges I want to share with you. One is the Monolithic
vs Micro Services
approach, the second is called the 'Continuous Delivery
Monolithic vs Micro Services approach
Several teams within CM Telecom already adopted the Micro Services mindset, it's only recently that the rest of our Developers also switched mindsets about the way we develop (web) applications and moved towards the Micro Services approach.
We currently have multiple complex web applications with a lot of functionalities which are depended on each other, the so-called Monolithic approach where all the logic resides in a single application. For this we used to use the MVC principle.
It turnes out that in a company where there are both PHP
Developers this approach is not very effective. For us it would be more effective to think in terms of Services and how these Services interact with each other but not in a way that they are also technically depended on each other. It needs to be possible that a .NET Developer and a PHP Developer can work together on the same project, something that was not possible before we started thinking in Services. In this process we also decoupled the frontend from the backend, basically the frontend is now an application on itself, for this we chose Angular
We think a Monolithic application architecture isn’t inherently flawed, it’s just out of date. It was designed before smart phones and other mobile devices existed, when applications only needed to interact with one kind of entity (the web browser running on the desktop).
We took the Micro Services mindset to our frontend as well, we have a number of applications that are available for our clients and although we take great care in order to create similar looking applications, sometimes there are minor differences. So we have created a universal header that can be included in every project, that should be the starting point for every application. We also created our own online style guide which contains a set of generic HTML building blocks which every Developer can re-use.
Introduction 'Continuous Delivery' principle
Because for testing we mainly relay on manual testing we - as Developers - are depended on someone else, in our case that would be a product owner. This product owner may or may not be constantly available on the fly. For a deployment you might be depended on someone from the IT department to configure a specific firewall access rule or to update a package on a production server or whatever. There are examples where a deployment was held for several days because of one of these reasons and that - of course - may never happen. These problems have always been around but mostly because of the newly introduced development tools which resulted in a significant decrease in development time they became more and more obvious.
It is therefore we started thinking about how can we cut out the middle man (the Product Owner and System Engineer) and how can we create a flow in which a Developer should be able to do everything by himself?
, CTO at Amazon, he put it quite nicely, he said; "You build it, you run it
" ('You' being the software Developer).
What he meant by that is that the Developer should be responsible for the software he creates - always. Our solution to this problem is twofold.
Continuous Delivery / Continuous Integration
Firstly; We want to solve the testing and deployment part with the 'Continuous Delivery' principle. Continuous Delivery is an approach in which Developers write software in cycles and ensure that the software can be reliably released at any time. The bottom line is that we want to automate and improve the process of software delivery. To accomplish this we need automated tests and continuous integration to allow software to be packaged and deployed automatically onto our test environments.
These improvements can actually be summarised by the following bullets;
- Accelerated Time to Market: If you can deliver software faster to your customers you are potentially one step ahead of the competition.
- Building the Right Product: Frequent releases let the development teams obtain user feedback more quickly. This lets them work on only the useful features.
- Improved Productivity and Efficiency: If you only focus on the useful features you will save time on development as well as testers and Developers.
- Reliable Releases: The risks associated with a release have significantly decreased, and the release process has become more reliable.
- Improved Product Quality: The number of open bugs and production incidents will decrease big time.
- Improved Customer Satisfaction: A higher level of customer satisfaction is achieved.
Continuous Delivery / DevOps
Secondly we want to make it possible for a Developer - through a set of tools - that he can maintain different environments and also he should be able to tackle any firewall issues by himself. This mindset has been dubbed 'DevOps
' in 2008 by the development community.
Basically DevOps is a development method about communication, collaboration, automation and cooperation between Software Engineers and System Engineers. A Developer literally can't go without a System Engineers and vice versa. This principle is best explained by the following diagram shown on the right;
Our main reason to go down this road is;
- The demand for an increased rate of production releases, we want to be able create more smaller releases in order to deliver software faster and
- We want to increase focus on test automation and configuration management tools.
We want to create a system where a Developer can create a new server preferably with just one click on a button but he also needs to be able to define which packages needs to be installed on this new server and while he's at it also setup the correct firewall permissions. He needs to be able to do this without the help of others. Too often development is delayed because of the right permissions aren't set or the fact that someone within Operations is unavailable to fix these missing permissions on the fly.
That being said, this whole new approach towards Continuous Delivery is relatively new for us so we did a lot searching and browsing the Internet and it turns out that there are a lot opinions out there, there are a lot advocates and adversaries on this topic, the thing that interested me most was the fact that a Developer is often looked at as someone with a specific skill set in a world where technology is evolving at an incredible rate, new tools, insights and programming practices are being invented practically every day. It's almost undoable for one person to keep up with all the latest technologies in the world of development.
When we finish implementing this Continuous Delivery principle we will expect every Developer to go the extra mile when it comes down to mastering and adapting this new mindset. Admitted, the Developer will have an extra set of tools to help him implement this new way of working but whatever tools he is using, he still has to get familiar with a whole new area of expertise.
The in-depth familiarity with Operations will be part of his 'new job' description.
One could argue that it is impossible for one man to have extensive in-depth knowledge of both Development and Operations. I know the tale of DevOps is about sharing responsibility and cooperation but still, Developers nowadays need to have knowledge about more than one area of expertise.
Another downside of Continuous Delivery could be that Developers get more responsibilities but there will be less overview. For example a server can be created with the push of a button, which means there will be less reflection on one's actions. Before DevOps you would go up to a guy in Operations and ask him for a new server. You will then have a conversation about this new server in order to explain its purpose and the new server will be setup for you just the way you need it to be. In the future this will no longer be necessary and is exactly what I mean by 'more responsibility and less oversight on one's actions'.
Also with automated testing there will be less manual testing, of course this is exactly what you want but a test is only as good as the Developer who writes the test (and the code itself), of course this could also be said for the product owner, the point is that again here is 'less overview of what is happening.
If have tried to explain some of the arguments from both advocates and adversaries on the 'Continuous Delivery' topic, both have valid arguments but for us it is clear; the advantages far outweigh the disadvantages and to be clear; this doesn't mean we are trying to create a clear separation between Development and Operations, to the contrary, we are trying to create a more co-operative relationship between Development and Operations. As I said we as Developers are implementing features that interact with each other but are not depended on each other, we want to accomplish the same thing between Operations and Development, a co-operative relationship in stead of a depended relationship.
CM Telecom is an ever growing company and we are always on the lookout for new Developers who are looking for a new challenge. CM Telecom is constantly looking and searching for that dot on the horizon. In terms of Development that's no different. With our DevOps mindset we are always trying to push ourselves to the next level and we are in need of people to help us reach that next level. As a developer you will be a part of a group of people who will never stop challenging each other in order to learn and become better at what they do. If you are interested don't hesitate to reach out or go and have a look at our vacancies