Embibe purpose and balance in life

I just finished the book "Life's Amazing Secrets" by Gaur Gopal Das. It covers the topic of balance and purpose in life nicely. Few of the concepts from the book I can very well relate to my current organization's focus on cultivation and collaboration culture.

It's now more than 6 months of my journey with Thoughtworks, an organization which started as a social experiment more than 25 years ago and believes in balancing 3 pillars in life; sustainability, excellence, and purpose. It makes the organization as a living entity, and wherever there is life, it has to face the changes and challenges, it's on us how we overcome and excel the art of changing.

Here are my takeaways from the book, which help us grow through life, build social capital, and help the people-centric organization grow.

• Live by the principle of gratitude, which allows us to see the positivity and believe in the trust-first approach. Start making gratitude logs.
• Our attitude towards life affects our social image, the speaking sensitivity is the key while giving or taking feedback. Life can never be perfect, too much corrective feedback may spoil the relationships if one doesn't know the art. Feedback without correct motive and purpose has no meaning.
• Think higher purpose, look beyond the situations and practice forgiveness but we must maintain social decorum of rules and regulations of society/community or organization.
• Uplift relationship by giving and take; exchange of thoughts, values, and beliefs
• Be your own hero, every time you compete with yourself, you will be better than before. Promote the culture of healthy competition. Each one is different and not comparable.
• Go on a journey of introspection, know yourself and find the purpose of life.

• Develop good character by belief, actions, and conduct. A good character has the ability to change lives. We have 2 ears and 1 mouth, so give them a chance in that proportion; Listen before you speak and choose to respond (not react).
• The book presents a great ideology of the ice-cream, the candle, and the oxygen mask, to guide us a journey from selfish to selfless. A view from family first to serving the nation, and how serving others can be Joy of life.
• It also talks about the role of spirituality in our life.

Overall it is a great book, and I recommend it.

Learnings from blockchain based development

Blockchain or DLT(Distributed Ledger Technology) is getting good traction in the IT world these days. Earlier this technology was being mostly explored by the banks and other finance-related institutions in the form of cryptocurrency such as Bitcoin, Etherium. Now, it is getting explored for other use cases for building distributed applications. Blockchain technology comes with a decentralized and immutable data-structure which maintains connected block of information. Each block is connected using hash of previous block, and every new block to the chain is validated (mined) before adding and replicating it. This post is about my learnings and challenges while building an enterprise blockchain platform based on  Quorum blockchain technology.

Blockchain based enterprise platform

We have built a commodity trading platform which matches the trades from different parties and store related trade data on the smart contracts. Trade operations are private to the counterparties, however packaging and delivery operations are performed private to all the blockchain network partners. Delivery operation involve multiple delivery partners and approval processes, which are safely recorded in the blockchain environment. The platform act as a trusted single source of truth, and at the same time keeping data distributed in the client's autonomy.  

The platform built on Quorum, which is an Ethereum based blockchain technology, supporting private transactions on top of etherium. A React based GUI is backed by the API layer of Spring Boot + Web3J. Smart contracts are written in Solidity. Read more here to start on the similar technology stack and know the blockchain. 

Following diagram represents the reference architecture : 

Blockchain Development Learnings

Blockchain comes with a lot of promises to safely perform distributed operation on the peer to peer network. It keeps data secure and make them immutable, so that data written on blockchain can be trusted as the source of truth. However it comes with its on challenges, we have learned a lot while solving the challenges and making the platform production ready. These learnings are based on the technology we used, however it can be co-related any of the similar DLT solutions.   

Product, Business and Technology Alignment

Business owners, analysts, products drives needs to understand the technology and its limitation, it will help in proposing well aligned solution supported by the technology. 

• Not for real-time system -  Blockchain technology support eventual consistency, which means data (block) will be added/available eventually on the network nodes, but may not be available in a real-time. It is an asynchronous system. Products/Application built using this technology may not be a real time system, where end user expect immediate impact of the operation. We may end up building lot of overhead to make real-time end-user interfaces, by implementing polling, web-sockets, time-outs, event-bus on smart-contract events.  The ideal end user interface would be have a request pipeline  where user can see status of all its requests. Once a request is successful then only the user will expect the impact. 
• There is no rollback on the blockchain due to its immutability, so atomicity across transaction is not available implicitly. It is a good practice to keep the operation as small as possible and design the end interface accordingly.        
• Blockchain’s transaction order is guaranteed, thus more of a sequential system, if a user is submitting multiple requests then they will go in sequentially to the chain. 
• (Private Blockchain) Each user/network node has its own smart contract data for which the node has participated into the transaction on the start contract, so adding new node/user to the contact would not see the past data available on the contract. It may have implication on business exceptions. 
• Backward compatibility is anti pattern on blockchain. It would be better if we may bind business features to the contract/release versions, new features will be only available in the new contract, otherwise implementing backward compatibility takes huge effort.         

Architecture and Technology Learnings 

• Think multi-tenancy from the start - Blockchain system is mostly about operations/transactions between multiple parties and multiple organisations. Most enterprise platform have multiple layers of applications such as end-user client layer (web, mobile, etc), API layers, authentications and authorisations, indexes, integrations with other system. It is wise think of multi-tenancy across the layers from the start of the project, and design the product accordingly, instead of introducing it at the later stage.
• Security does not come on its own - Blockchain based system generally introduce another layer of integration between API and data layer. We still need to protect all the critical point at multiple network nodes owned by different participants. So defining security process and practices are even more important and complex for blockchain based solution than that of the classic web application. Make sure these practices are followed by all the participant's node.  
• A hybrid solution -  We might need to keep data index/cache (SQL or NoSQL) to meet the need multiple user reads, as reading every time from smart-contract may not meet the performance criteria. Index/cache building module may subscribe to the smart-contract transaction event, and build data index/cache to serve as read copy of block chain data. Each participant's node will have its own index and  end-users interface of that participant organisation. It will definitely add complexity to maintain the index, but until blockchain technology becomes as fast as read cache, we have this option to go Hybrid way (blockchain + read indexes). we can replay smart-contract transaction event as and when required, it helps in rebuilding indexes/cache in case we lost the index.
• Carefully add business validation in the smart-contract - we may add business data validation in the smart-contract itself but it has transaction-cost (performance cost), more complex validation may lead to out of gas issues, also getting detailed exception from the contract is not fully supported. If we are maintaining indexes then, we may do validation before making the contract transitions by reading from indexes but it is also not a proof solution, as there might be race conditions. Idea way is perform business validation before and in case of race condition occurs,  just override/ignore the operation. Choose the approach wisely based on the business impact.
• Fear of losing transactions - What if we could not process a transaction due to any error? Depending on the case, we either need to wait for the problem to be fixed and halt all the further events or just log the error and proceed. As mentioned in the previous section choose to override/ignore wisely, it may have business impact.
• Smart contract limitation - Solidity has limitation on number of parameters in a method, stack depth, string handling, contract size, gas limits. So design contract methods accordingly. 
• Tracing - Think of application tracing from web/mobile <-> api server <-> blockchain <-> indexes from the start. Each request should be traceable. Quorum doesn’t support any tracing/message header kind of mechanism to send co-relation identifier, however we may link the application a trace id to the transaction hash. 
• Configurability/Master Data Management  -  Every enterprise system needs some master data, which can be used across the organisations, since participant's nodes are decentralised, so this data need to be synchronized across the nodes. Device out a mechanism to replicate this data across the nodes. Any inconsistency may result in failure of transactions. 
• Smart Contract Backward compatibility - This is the most complex part of contract. We need to write a wrapper contract on the older contract and delegate call to old one. Note that, we cannot access private data of existing contract, so we need to implement all the private data in the wrapper contract and conditional delegate call to old contract. Managing event listeners for different versions are also complex, we may need to keep multiple versions in event listeners in the code base to support multiple versions of the contracts. We also need to make sure all the participants are on the same version of contracts. It would be better if we bound business features with version, in that case any new transaction operation may not be available on the old contract.
• Scaling the API/Client Layer - Since blockchain process the transactions is a sequential manners, so scaling api/client layer is complex, we need to implement some locking mechanism to avoid same transaction getting performed from multiple instance of API layer. Deployment is another challenge here. 
• Feature toggle on contract side is quite complex

Testing Learnings

• Solidity unit tests takes lot of gas, as the functionality grows, so it is better to use JS API based tests for contract unit tests.
• Define contract migration testing approach, and it's automation up front. 
• Automate environment preparation as it need multiple tenants nodes to interact to test the functionality.
• Api integration tests needs poll the other participants’s api to see the expected impact, it increases the api test development times as well as execution time. 
• Define approach for test automation around index/cache rebuilding
• Version based feature toggles in tests artefacts are complex to maintain.  
• Automation of meta-data management, across the nodes 
• Testing of CICD scripts, as the scripts grow so complex with time, and impact of any issue is huge. 

Continuous Integration and Deployment

• Define blockchain node deployment strategy upfront (private/public cloud), you may not  change it once it is deployment without loosing data. 
• Securely storing secrets on cloud.
• Have access management reviews. 
• Analyze impact of secret rotation on blockchain node setup, automate the process.
• Backup and restore strategy for blockchain
• Making sure all the node are on same version of blockchain and software artifact
• High availability is a challenge when new deployment has to happen, as we need to stop a service completely before deploying new version to avoid corrupt indexes/blockchain due to intermediate state of operations (multi-transaction operation).

Project Management 

• Blockchain is still a growing technology, it is better to have a dedicated sub-team which takes care of blockchain research related work, define best practices for the project and industry.
• Dedicated infra team is needed manage and improve CICD environment.   
• Security testing resources, and audits. 
• Legal point of view on GDPR and storing data in Blockchain. 
• Coordination with participant’s infra team
• Support SLA with each participants. 

Conclusion 

Blockchain/DLT is a growing technology, and a lot is yet to be done in the private blockchain space. I have presented the learnings and challenges that we have faced during the development of a blockchain based enterprise platform. Hope it will help you in avoiding some of the challenges that we have already faced.

Software Development as a Design Process

The book "Agile IT Organization Design - For digital transformation and continuous delivery" by Sriram Narayan explains software development as a design process. Following statement from the book means a lot.

Software development is a design process, and the production environment is the factory where product takes place. IT OPS work to keep the factory running..

It questions the way we traditionally think of operation support, maintenance, production bugs, rework, their costs and budgets and the focus in software development life.
Thinking software development as a design process helps the organisations, focus on today's need of agile development. Here are some thought points, when anything before production deployment is considered as software product design, and the "software product" is getting produced from the production environment (factory):

• Dev + OPS - A product design which doesn't fit in the process of factory can not be produced by the factory, in other words A factory which do have sufficient resources needed to produce given design can't produce the product. The product designer, and factory operators needs to be in the close collaborated team, to improvise the design, and do production.
• Continuous Delivery - There is no point of thinking about factory after the product design is done, we may not be able to build the factory optimally. Think of the production factory while design the product. Have continuous feedback in place by continuous integration and delivery.
• Value driven and outcome oriented teams - A factory can produce the product on a shop floor/production line with checks, balances and integration at each step. It implies to the software development as well, value driven projects, and outcome oriented teams are more helpful in making product successful over the plan driven projects and activity orientated teams. The book covers it very well.
• Sorter cycle time - Software product design has no value until it is produced, and reached to the people. Sooner we produce, better we get the value - so cycle time for product design must be as small as possible. Luckily in software development we have tools to simulate factory environment/test labs, and try to produce the models well before the actual production.
• Measure the product quality more than the design quality - It is important to measure product quality than that of the product design (software before prod environment). So metrics which can really measure product quality (software running in production environment) such as velocity in term of "Value". Measuring burn-up/burn-down, bugs counts, defect density, code review quality are all product design metrics, it may not matter how good these numbers are if software is not producing value on the production environment.

Conclusion

The book covers an agile organization design with a great breadth and depth on structural, cultural, operational, political and physical aspects. We can relate many of these aspects while thinking the software development as a design process, must read the book.

Request Tracing Using Nginx and Spring Boot

Most web applications are hosted behind a load balancer or web-server such as Nginx/HTTPD, which intercepts all the requests and directs dynamic content requests to the application server, such as Tomcat. Correlating requests traversing from the front-end server to the backend servers are general requirements. In this post, we will discuss tracing the request in the simplest way in an Nginx and Spring Boot-based application without using an external heavyweight library like Slueth.

Assign an Identifier to Each Request Coming Into Nginx

Nginx keeps request identifier for HTTP request in a variable $request_id, which is a 32 haxadecimal characters string. $request_idcan be passed to further downstream with request headers. Following configuration passes the $request_id as X-Request-ID HTTP request header to the application server.

server {

    listen 80;

    location / {

        proxy_pass http://apiserver;

        proxy_set_header X-Request-ID $request_id;

    }

}

Log the Request Identifier in Front-end Access Logs

Include the $request_id in the log format of the Nginx configuration file as follows.

log_format req_id_log '$remote_addr - $remote_user [$time_local] $request_id "$request" '

                 '$status $body_bytes_sent "$http_referer" "$http_user_agent" '

                 '"$http_x_forwarded_for"';

access_log  /dev/stdout req_id_log;

It will print the access logs in the following format:

172.13.0.1 - - [28/Sep/2018:05:28:26 +0000] 7f75b81cec7ae4a1d70411fefe5a6ace "GET /v0/status HTTP/1.1" 200 184 "http://localhost:80/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36" "-"

Intercept the HTTP Request on the Application Server

Every HTTP request coming into the application server will now have the header X-Request-ID, which can be intercepted either in the interceptor or servlet filter. From there, it can be logged along with every log we print in the logs, as follows.

Define a Request Filter Using MDC

Define the following request filter, which reads the request header and puts it in the MDC (read about MDC here). It basically keeps the values in the thread local.

...

@Component

public class RegLogger implements Filter {

  @Override

  public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain) throws IOException, ServletException {

        try {

            MDC.put("X-Request-ID", ((HttpServletRequest)servletRequest).getHeader("X-Request-ID"));

            filterChain.doFilter(servletRequest, servletResponse);

        } finally {

            MDC.clear();

        }

    }

  @Override

  public void init(FilterConfig filterConfig) throws ServletException {}

  @Override

  public void destroy() {}

}

Configure the Logger to Print the Request id

I have used logback, which can read MDC variables in the %X{variable_name} pattern. Update the logback pattern as follows:

<?xml version="1.0" encoding="UTF-8"?>

<configuration>    

  <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">        

    <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">

      <Pattern>%d{HH:mm:ss.SSS} [%thread] %X{X-Request-ID} %-5level %logger{36} - %msg%n</Pattern>

    </encoder>    

  </appender>    

  <root level="info">        

    <appender-ref ref="STDOUT" />    

  </root>

</configuration>

It will print the following logs:

17:28:26.011 [http-nio-8090-exec-7] 7f75b81cec7ae4a1d70411fefe5a6ace INFO c.d.e.controllers.StatusController - received request

This way, you can see the request id in all the logs from the origin to the end. We can configure ELK to aggregate the logs and trace them in a nice user interface to help with troubleshooting.

Get the Generated Request id Back in the Response

Nginx can also add the request id in the response header with the following configuration:

server {

    listen 80;

    add_header X-Request-ID $request_id; # add to response header

    ....

As you see here:

Conclusion

We have explained a simple way to assign a unique identifier to each request, then trace the request end-to-end. You can get more details here.