Accept the change

Change is inevitable and constant; it is a part of our lives. Sooner or later, we have to accept the change. 13 years ago, I joined a small IT organization of around 100 people, as a developer. I have grown with the organization, a journey that has seen it grow to 40 times its original size. I really thank the organization for giving me the opportunity and transforming me into what I am today. Now, the time has come to accept a change, and start a new journey. I am writing this post to share my point of view about when to accept the change.

When to Change the Organization?

The intention behind this post is not to answer this question, but to address what you get and what you lose when you stay too long in the same organization. I have also taken inputs from many senior folks who joined us or left us after a long stay. When to change is really an individual's choice. If you are aware of the pros and cons of staying at a company for a long time, then you may probably work on the cons and stay relevant even longer, or don't change at all.

What Do You Get?

When you join a small software organization and stay for a while, say more than 10 years, and grow with the organization's pace, these are some factors which may keep you attached to the same organization:

You Are Tried and Tested

You become a tried and tested resource of the organization; many stakeholders may want to take you in their group. You don't need to constantly prove yourself to get into an opportunity in the organization.

You Know the Beats

Every organization grows in its own style and nurtures its own culture. You know the beats of the organization, you know how things are done in a certain way, you know the history. In time, you become a consultant for many things in the organization. People believe in your inputs, and you become an encyclopedia of organizational information. Since you have participated in almost everything, you see yourself in every part of an organization, be it hiring, training, process or the system.

Breadth and Depth

You may get a chance to work in different departments and a chance to set up groups and lead multiple initiatives. This gives you a chance to gain the breadth and depth, and a rich experience to lead the organization.

Attractive Role and Appreciations

You may be working on really attracting talent at for a subordinate role with less experience in terms of years. You may be recognized and appreciated well.

Emotional Connect

After many years, your relationship with the organization is no longer just professional, it becomes an emotional one. The organization becomes a family, and you will have a lot of friends.

Comfort Zone

You are in your comfort zone and can work very confidently in the same zone.

Stability

The industry may see you as a stable resource that doesn't change too often.

What Do You Lose?

Staying too long will accumulate following, and you may keep building on top of these unknowingly. You may work to overcome these and stay prepared to go even longer on the journey.

Lack of Diversity

Since you have worked in the same organization for such a long period, you start thinking that certain things can happen only in a certain way. These certain things could be dealing with processes, people, resources, and even technology solutions. You know them well, which worked well and which did not for you in that organization. Thinking beyond your area of easy imagination does not come easily. It may start blocking your path and in time may become frustrating.

Resistant to The Change

After staying so long, you become resistant to the change in the processes, people and the system. It becomes hard to accept the change, if that is not according to you.

Working Out of Your Comfort Zone Becomes Difficult

It becomes difficult to work beyond your comfort zone, and you may start running away from situations which take you out of your comfort zone. Risk-taking capabilities may also be reduced, as you worked in a really safe and secure zone for long. You may not have learned to deal with the failures, and have not been exploited and exposed well since long. We learn well when we get exposed from time to time.

You Are Taken for Granted

You may be taken for granted by your peers of similar experience and by management; they know all of your flaws, as you have grown in front of them. They may use you the way they want and wherever they want. It will be hard to change the perception of people around you. You may also be a target for the newcomers, as they don't really know the history and achievements that you might have.

Energy and Satisfaction

After so many years, you may not have the energy to fight against changes, which may kill the culture you are addicted to. Remember that change is necessary, to scale the organization, and to sustain and to go beyond the barriers, but you might not be aligned with the changes. You may not be satisfied with the situations and with what you get. In such a situation, you start creating a negative environment, which is not good for both you and your organization.

Confidence in Industry

As the years progress, jobs become limited at your level. The market outside may not be ready to accept you, in the way you are, and you will need to adapt. If you stay up-to-date and refresh your skills from time to time, you gain a lot of confidence, and that can help in your current job as well.

Conclusion

I have shared the pros and cons of a long stay in the same IT organization; it may help you in identifying the losses you are accumulating year by year after certain years. You may plan to overcome the losses and stay useful to the same organization you are in, or take the next step.

Forces behind IOT

Everybody in the industry is talking aloud for the IOT. It is always discussed in terms of billions of devices, their grows in coming years, changing business models, and adaptation to market etc. Connecting the things become so prevalent that every company want to expand there. Most companies are expanding themselves to build IOT centric products and services.  Neither the “Thing” nor the “Internet” is new, but there is so much hype of connecting the ordinary things to internet.  Who is actually making IOT, the IOT?

There are many industrial applications, which uses sensors and actuators, and they have been around a long, they were controllable remotely. So connected things were already there and the technology is enhanced with time. IOT is not just about connecting the ordinary things and make them accessible remotely, but it is much more than that.

The “Connected Things” become the “Internet of Things” when they are connected to people and their processes. Connected Things produce hugeamount of data which can be monitored and visualized remotely, but in the “Internet of Things” era this data is analyzed to generate useful insights and build patterns,  predict the situations, prescribe the solutions and instruct the ordinary thing so that the connected thing becomes a self-optimizing thing. The “Internet of Thing” is a package of :

• The things with capability to sense and act.
• Communication protocols to connect the things.
• Platform to collect, analyze and store the data. Predict the instructs the things.
• Network to connect all the things with platform.
• Strong visualization over the collected data, and interactive user interface monitor the things. 

The connected things of past to getting converted to the Internet of Things due to following supportive forces:

1. Reliance on Software - Usage of software is getting increased in making of the hardware better. Software plays critical role lot of mission critical applications
2. Technology commoditization – Technology is reaching in the hand of common people. We are living with it, our basics devices such as phones, watches, tv, washing machines, shoes etc are getting smarter. Technology is becoming affordable. Hardware is getting cheaper due to mass production, process enhancement.
3. Connection everywhere - Network speed and coverage is increasing. The world is getting more connected to the internet every day. Internet is getting cheaper and accessible. Communication overhead is reducing by better protocols, thus data access speed is increasing.
4. More data and more science – Necessity is the mother of invention, thus the focus of data analytics and science is increasing as more data is getting generated. Data storage capacity is also increasing with time.
5. Rapid development – Companies are investing in frameworks and platforms to build, deploy, and productionize the software faster. Cloud services are getting cheaper and easily accessible
6. Increase revenue by cost reduction - Enterprises are targeting to reduce cost by predictive maintenance, and self-optimizing machines.
7. Stay in the race - Predictions of IOT, everybody want to stay in the race due to the predictions of IOT in 2020 or so.

Following factors are obstacles in the IOT growth :

1. Lack of standards – Current IOT development is more of consensus based standards. Set of companies come together and define their own standard, and it is leading to interoperability problem.
2. Security tradeoff – IOT devices are smaller, less capable and may not perform well when high security standards are required.  
3. Investment – Lot of companies are talking about IOT but very few are really investing, most companies playing safe by wait and watch.
4. Huge ecosystem – IOT’s technical eco-system is huge, it involves everything from hardware, network, protocol, software, data science, data storage, mobility, IOT platform, cloud infrastructure, maintenance, integration services. It is really difficult for companies to build such as huge eco-system. Companies need to partner with specialists and share their customers and revenue.
5. Integration to existing infrastructure – Integrating IOT tech stack to traditional communication infrastructure is challenging , for eg. HTTPS, SSL, SOAP, RDBMS, SFTP for CRM, ERP, ECM portal.

Conclusion:

IOT is not new, it has been there for long, “Connected Things”/M2M of the past is today’s IOT with new set of communication protocol, improved hardware, better data collection, analytics capability with nice visualization to monitor and interact with IOT devices. There are lot of forces to keep IOT growing in future, but there are some obstacles also, which need to be crossed by the IOT providers.

Choose ORM carefully

Introduction

We have recently migrated source code from Hibernate ORM to JDBC (Spring JDBC template) based implementation. Performance has been improved 10 times. This post describes the use-case, bench-marking and the migration steps. 

Use case

A tree structure in database is getting populated from a deep file system (directory structure) having around 75000 nodes. Each node (directory) contains text files, which get parsed based on business rules and then populate the database ( BRANCHs representing a node,  tables referring to branch, tree_nodes).  The database tables was well mapped to Hibernate JPA entries and on saving Branch object, its relevant entities were automatically getting saved. Whole operation was performed in recursive manner by traversing over directory tree. Each table’s primary key is auto generated from a separate sequence.

As per development level performance testing, it was estimated that initial tree will take 30 hours to load whole tree. This was not acceptable, as UAT cannot be started without this migration. (Env : Oracle 11g, JBoss 6, JDK 6, Hibernate, Spring 3)

Bench-marking: Hibernate vs Spring JDBC

Data model

Create table and sequence

CREATE SEQUENCE customer_ID_SEQ START WITH 1 INCREMENT BY 1;

create table customer (id bigint not null, name varchar(200), primary key (id));

Model

@Entity

public class Customer {

  @Id

  @GeneratedValue(strategy = GenerationType.SEQUENCE,

  generator = "CUSTOMER_ID_SEQ")

  @SequenceGenerator(name =  "CUSTOMER_ID_SEQ",

  sequenceName =  "CUSTOMER_ID_SEQ", allocationSize = 1)

  private long id;

  private String name; // application assigned

  @Column(name = "NAME")

  …

  …

Approaches

1.      Approach 1 : Existing

@Transactional

  public void bulkPersist(List<Customer> entities) {

    for (Customer entity : entities) {

      em.persist(entity);

    }

 }

2.      Approach 2 : Batch size - Set hibernate batch-size property and flushing after  a batch same.

@Transactional

  public void bulkPersist(List<Customer> entities) {

    int i = 0;

    for (Customer entity : entities) {

      em.persist(entity);

      i++;

      if (i % batchSize == 0) {

        flush();

        clear();

      }

    }

}

3.      Approach 3 : JDBC template

@Transactional

 public void bulkPersist (final List<Customer> entities) {

 template.batchUpdate("insert into customer (id, name) values (CUSTOMER_ID_SEQ.nextval, ?)", new    

       BatchPreparedStatementSetter() {

       

        @Override

        public void setValues(PreparedStatement ps, int i) throws SQLException {

                  ps.setString(1, entities.get(i).getName());

         

        }

       

        @Override

        public int getBatchSize() {

                return entities.size();

        }

        });

 }

Result

Approach 3 is coming out 10 time faster than that of Approach 1. Approach 2 has also improved the performance but lessor then that of Approach 3;

Migration

As per benchmarking, Spring JDBC template’s ‘batchUpdate’ is the fastest as compared to hibernate based approaches. Now while migrating code base from hibernate to JDBC, biggest issue was to resolve the generated ID referred in other queries. 

Since ID is getting generated by a sequence and Spring JDBC template’s ‘batchUpdate’ do not have provision to fetch generated IDs. Hibernate was doing this automatically by updating ID field of the entity object.

 So solve this, we had fetched bulk ids from the sequence, in a single query:

List<Integer> ids = template.queryForList("select customer_id_seq.nextval from (select level from dual connect by LEVEL <=" + 

                                                                    entities.size() +")", Integer.class);

     

Set ids in entity object while iterating BatchPreparedStatementSetter.setValues method

    public void setValues(PreparedStatement ps, int i) throws SQLException {

              Customer customer = entities.get(i);

               customer.setId(ids.get(i));

              ps.setInt(1, customer.getId());

              ps.setString(2, customer.getName());

    }

This way entity object get populated in the same way as it is done in hibernate, without any special iteration/processing. Once entity object is populated, all the dependent batches can be fired so that entity.getId returns the correct value.

We were able to migrate Hibernate based codebase to Spring JDBC with minimal changes in source code. To save JVM memory, we have also implemented batching over it. Processed only 2000 nodes at a time.

After porting codebase to Spring JDBC Template, we measured performance for the tree load, on local database and it was coming out around 3 hrs. We run the same on UAT environment, and the whole tree get loaded in 1 hour.

Client Vs Server Side Validation

I am writing this post in reply to the questions asked in an Technical  Workshop presented by me, “What is best way of validation, client side or server side or both?”, “Why to duplicate the validations at client and server side?”.

What is validation?

Come on, I am sure you know this.

Ok what is Client-side Validation?

Simple, the validations which are done at client side. Usually done by java scripts.

And Server-side Validation?

Obliviously, the validation which are done at server side are the server-side validations. Server side code do this.

Which one should I use? - Server-side Validation Or Client-side Validation Or Both

Yes this the topic of discussion. I think, it does not matter which one is best or which one is worst, the server-side validations are must. Client side validation are in the scope of client, he can disable the javascripts or bypass them by seeing the javascript code ( most of the browser now come with javascript debugging support with watches or so).

Ok it means server side validation is best, so we should not use client-side validation?

Well, I haven’t said this. You should consider having client-side validation as well. It is useful to increase the usability of the web application, improve the user experience by speeding up the interaction, instead of waiting for the server side validation and seeing the error only after filling up so many information.

So I think both should be used, first perform client side validation to improve user experience and then once a form is submitted to server, validate there as well to check the correctness of the input.


Nowadays, AJAX based validation is also used to provide better user experience with the power of server-side validation, but I still say server side validation is must even after this.

Software Effectiveness Vs Software Efficiency

I define software effectiveness as, doing the objective effectively, I mean correctly. Efficiency can be defined as, using the resources optimally where resources could be memory, CPU, time, files, connections, databases etc.

From my experience, in most(should I call many) of the software projects, efficiency/performance is not much accentuated during the system design and earlier phases(requirement and estimation) as compared to the emphasis given in later phases, coding and testing and mostly in maintenance.

Stressing on efficiency, during the early SDLC phases, can eliminate lot of problems. If we consider the efficiency late say in coding phase, then we probably able to develop an optimal system in which 90% of the code using just 1% of CPU time in peak load, but 10% of code is using 99% of CPU. If we worry about the performance only after the system is built then we are in the worst situation, we probably could not reach 90% optimal code level.

Coming back to effectiveness, usually this is the most emphasized topic in all the SDLC phases, we always try to make the system understandable, testable, and maintainable.

Note that, efficiency is generally against the code quality measures that were considered to improve effectiveness, more efficient code is usually more difficult to understand, hard to maintain, sometime very hard to test.

So based on project, we should benchmark/strengthen the SDLC process to balance-out the efficiency and effectiveness in each phases. Keep it in mind that there are always some modules where efficiency is more concerned than the understandability, maintainability. We may change our mind setup when understanding a highly efficient module that this will require more effort to understand, maintain and test.

This post is not meant to allow the developer writing less effective code, saying that you are writing efficient code so; it will be less understandable, not testable and not maintainable. Use better design pattern, prepare approach and discuss it, design the efficient module very carefully.