5 Hard Lessons From Microservices Development

Learn about challenges in tools, training, and methodology for microservices development learned the hard way, so you don't have to.

Microservices-based development is happening all around the industry; more than 70% are trying development of microservice-based software. Microservices simplify integration of the businesses, processes, technology, and people by breaking down the big-bang monolith problem to a smaller set that can be handled independently. However, it also comes with the problem of managing relations between these smaller sets. We used to manage fewer independent units, so there was less operation and planning effort. We need different processes, tools, training, methodology, and teams to ease microservices development.

Our Microservices-Based Project 

We have been developing a highly complex project on microservice architecture, where we import gigs of observation data every day and build statistical models to predict future demand. End users may interact to influence the statistical model and prediction methods. Users may analyze the demand by simulating the impact. There are around 50+ bounded contexts with 100+ independent deployment units communicating over REST and messaging. 200+ process instances are needed to run the whole system. We started this project from scratch with almost no practical experience on microservices and we faced lots of issues in project planning, training, testing, quality management, deployment, and operations.

Learnings 

I am sharing the top five lessons from my experience that helped us overcome those problems.

1. Align Development Methodology and Project Planning

Agile development methodology is considered best for microservice development, but only if aligned well. Monolithic development has one deliverable and one process pipeline, but here, we have multiple deliverables, so unless we align the process pipeline for each deliverable, we won't be able to achieve the desired effectiveness of microservice development.

We also faced project planning issues, as we could not plan the product and user stories well, which can produce independent products, and we could apply the process pipeline. For seven sprints, we could not demonstrate the business value to the end user, as our product workflow was ready only after that. We used to have very big user stories, which sometimes go beyond multiple sprints and impact the number of microservices.

Consider the following aspects of project planning:

1. Run parallel sprint pipelines for Requirement Definition, Architecture, Development, DevOps, and Infrastructure. Have a Scrum of Scrum for common concerns and integration points.
2. Keep few initial sprints for Architecture and DevOps, and start the Development sprint only after the first stable version of the architecture and DevOps is setup.
3. Architectural PoCs and Decision tasks should be planned a couple of sprints before the actual development script.
4. Define metrics for each sprint to measure project quality quantitatively.
5. Clearly call out architectural changes in the backlog and prioritize them well. Consider their adaptation efforts based on the current size of the project and impact on microservices.
6. Have an infrastructure resource (expert, software, hardware, or tool) plan.
7. Configuration management.
8. Include agile training in the project induction.
9. Include multiple sprint artifact dependencies in the Definition of Ready (DoR) and Definition of Done.
10. Train the product owner and project planner to plan the scrums for requirement definition, architecture, etc such that they fulfill the DoR.
11. Have smaller user stories, making sure the stories selected in a sprint are really of the unit size which will impact very few deployment unit.
12. If a new microservice is getting added in a particular sprint, then consider the effort for CI/CD, Infrastructure, DevOps.

2. Define an Infrastructure Management Strategy

In a monolithic world, infrastructure management is not that critical in the start of a project, so infra-related tasks may get delayed until stable deliveries start coming, but, in microservices development, the deployment units are small; thus, they start coming early, and the number of deployment units is also high, so a strong infrastructure management strategy is needed.

We delayed defining the infrastructure management strategy and faced a lot of issues getting to know the appropriate capacity of the infrastructure and getting it on time. We had not tracked the deployment/uses of infra components well, which caused a delay in adapting the infra, and we ended up having less knowledge of the infrastructure. We had to put lot of effort into streamlining the infra components in the middle of the project, and that had a lot of side effects on the functional scope getting implemented.

Infrastructure here includes cross-cutting components, supporting tools, and hardware/software needed for running the system. Things like service registry, discovery, API management, configurations, tracing, log management, monitoring, and service health checks may need separate tools. Consider at least the following in infrastructure management:

1. Capacity planning – Do capacity planning from the start of the project, and then review/adjust it periodically.
2. Get the required infrastructure (software/hardware/tools) ahead of time and test them well before the team adopts them.
3. Define a Hardware/Software/Service onboarding plan which covers details of the tools in different physical environments, like development testing, QA testing, performance testing, staging, UAT, Prod, etc.
4. Consider multiple extended development testing/integration environments, as multiple developers need to test their artifacts, and their development machine may not be capable of holding required services.
5. Onboard an infrastructure management expert to accelerate the project setup.
6. Define a deployment strategy and plan its implementation in the early stages of the project. Don’t go for intermediate deployment methodology. If you want to go for Docker and Kubernetes-based deployment, then do it from the start of the project — don’t wait and delay its implementation.
7. Define access management and resource provisioning policies.
8. Have automated, proactive monitoring on your infrastructure.
9. Track infrastructure development in parallel to the project scope.

3. Define Microservices-Based Architecture and Its Evolutions

Microservices can be developed and deployed independently, but in the end, it is hard to maintain standards and practices throughout development across the services. A base architecture that needs to be followed by the microservices and then let the architecture evolve may help here.

We had defined a very basic architecture with a core platform covering logging, boot, and a few common aspects. However, we considered lot of things to come in evolutions such as messaging, database, caching, folder structures, compression/decompression, etc. and it resulted in the platform being changed heavily in parallel to the functional scope in microservices. We had not given enough time to the core platform before jumping to the functional scope sprints.

Consider the following in the base architecture, and implement it well before the functional scope implementation. Don’t rely too much on the statement “Learn from the system and then improvise.” Define the architecture in advance, stay ahead of the situation, and gain knowledge as soon as possible.

1. Define a core platform covering cross-cutting concerns and abstractions. The core platform may cover logging, tracing, boot, compression/decompression, encryption/decryption, common aspects, interceptors, request filters, configurations, exceptions, etc. Abstractions of messaging, caching, and database may also be included in the platform.
2. Microservice structure – Define a folder and code structure with naming conversions. Don’t delay it. Late introduction will cost a lot.
3. Build a mechanism for CI/CD – Define a CD strategy, even for the local QA environment, to avoid facing issues directly in the UAT/pre-UAT environment.
4. Define an architecture change strategy – how architecture changes will be delivered and how they will be adapted.
5. A version strategy for Source Code, API, Builds, Configurations, and documents.
6. Keep validating the design against NFRs.
7. Define a Test Architecture to cover the testing strategy.
8. Document the module architecture with clearly defined bounded contexts and data isolations.

4. Team Management

The microservice world needs a different mindset than the monolithic one. Each microservice may be considered independent, so developers of different microservices are independent. It brings a different kind of challenge: we want our developers to manage code consistency across units, follow the same coding standards, and build on the top of the core platform, and at the same time, we want them not to trust other microservices' code, as it was developed by some other company’s developer.

Consider the following in your team management:

1. Define the responsibility of “Configuration Management” to a few team members who are responsible for maintaining the configuration and dependencies information. They are more of an information aggregator, but can be considered a source of truth when it comes to configuration.
2. Define a “Contract Management” team consisting of developers/architects who are responsible for defining the interaction between microservices.
3. Assign module owners and teams based on bounded context. They are responsible for everything related to their assigned module, and informing the “Configuration Management” team of public concerns.
4. Team seating may be considered module-wise; developers should talk to each other only via contract, otherwise they are a completely separate team. If any change is needed in the contract, then it should come via “Contract Management.”
5. Define the DevOps team from development team. One may rotate people so everybody gets the knowledge of Ops.
6. Encourage multi-skilling in the team.
7. Self-motivated team.
8. Continuous Training Programs.     

5. Keep Sharing the Knowledge

Microservices are evolving day by day, and a lot of new tools and concepts are being introduced. Teams need to be up to date; due to microservice architecture, you may change the technology stack of a microservice if needed. Since teams are independent, we need to keep sharing the learning and knowledge across teams.

We faced issues where the same/similar issues were being replicated by different teams and they tried to fix them in different ways. Teams faced issues in understanding bounded context, data isolationss etc.

Consider the following:

1. Educate teams on domain-driven design, bounded context, data isolation, integration patterns, event design, continuous deployment, etc.
2. Create a learning database where each team may submit entries in the sprint retrospection.
3. Train teams to follow unit testing, mock, and integration testing. Most of the time, the definition of a “unit” is misunderstood by developers. “Integration testing” is given the lowest priority. It must be followed; if taken correctly, it should be the simplest thing to adhere to.
4. Share knowledge of performance engineering — for example:
   1. Don’t over loop
   2. Use cache efficiently
   3. Use RabbitMQ messaging as a flow, not as data storage
   4. Concurrent consumer and publishers
   5. Database partitioning and clustering
   6. Do not repeat  

Conclusion

Microservices are being adopted at a good pace and things are getting more mature with time. I have shared a few of the hard lessons that we experienced in our microservice-based project. I hope this will be beneficial for your project to avoid mistakes.

This was originally posted on https://dzone.com/articles/5-hard-learnings-from-micro-services-development