How to make a Software David?
When people work together in projects, lots of human dynamics come into play.
There are Engineers who are very bright and can immediately roll up their sleeves to sling code. And there are those who are slow to catch up but can come out with deeper questions that can make the structure fall apart. Apart from these, there are those who are just willing to stay by the side giving hand where it is needed.
In one of the consulting engagements I had to restructure a legacy application into a REST based architecture. It was a very critical app being used across the globe for over a decade. It had become a ball of mud.
I was asked to come up with an architecture after the team with a lead architect have captured a lot of the user stories and were kind of circling on a possible architecture.
The domain of that App was image processing and I had no idea of it.
I began to understand the user stories. The characters involved, their needs on the system, the reactions of the system to their inputs from a purely black box perspective. This took me a long time. I was not working full time. But whenever I was thinking about it, a new question will pop up in my head. I will immediately shoot it to the domain expert and he will lead me to another winding path through the maze.
At some point, I began to get the vision of the blocks, their broad functions, their interactions.
I wrote the first cut draft that had the problem I see as what we want to solve, the picture and a high level explanation of how things will happen across the blocks in the picture which is the architecture.
I had a meeting with the architect and went through the picture. The legacy software was based on messaging. There were lot of events coming from the hardware, from the UI, movement of images of GB size and complex algorithms at play.
The architect had all these details in his head. He was visibly unhappy with the high level picture I presented which lacked any of these details.
I did not show any details, because I had not yet known the details of the ‘legacy’ system as to how it was implemented. I had taken a ‘top down’ approach and proposed a brand new solution purely going by the user stories.
This was a conflict. The architect thought that the picture I showed was too high level and wouldn’t tie in with the existing code. I told him we will fill in the blanks in subsequent iterations. But he was not convinced. I could see it because at one point in time the discussions flared up to a point where he called it ‘idiotic’ or ‘foolish’ or something. He later apologized for his temper while we were at the coffee station.
However, the project manager who was there along with the management line somehow felt that the existing thinking on rearchitecting the legacy software was not working and fresh thinking was needed.
Over the next few months I remember working with the domain expert and other members of the team to expand on the details within the broad outline of an architecture.
Every week I was presenting the new iteration with the details to the entire team. The team was more into supporting the existing software and had little experience in developing new products from scratch.
They were extremely doubtful to see a new structure coming up in place of the home they were living in. They had apprehensions and used to question every tiny aspect of what they were working on as to how it will work in the new sketch.
At times it was frustrating as I felt the team was antagonistic to a new idea. They were in their comfort zone working on the existing system and hated to move onto something brand new. Probably this was making them to expect extending and enhancing the existing code rather than thinking fresh. This wasn’t going anywhere because the software became hard to use, hard to modify with new use cases coming up.
However, I was very thankful for all those discussions. It is impossible to understand a software by reading the code all by myself. Every challenge posed by the team was opening a new question on the sketch I had drawn. When you get into the details, the call flows, the movement of data, the messages passed across systems, the events being handled, the resource constraints, performance constraints, it is like zooming into the original broader sketch of components and interactions to expand it to contain all possible details.
The idea is the beauty and simplicity should not go away when you detail them. These two parameters are extremely important as they allow others to understand the architecture and further refine it. Also if you have reached a point of simplicity, it is ready to scale in the future to take in new requirements without much alteration.
The success of an architecture is when the Engineers can understand why it was done the way it was done and they can make improvements to it as new needs arise or existing needs change.
Software becomes complex because of the complex human needs. What you begin as a ‘hello world’ can soon turn into a monster tormenting you when you have to make a small change.
It takes a lot of absorbing the problem to arrive at something that keeps things simple and manageable. It is true with everything that humans are involved with. Like organizations that start with a simple search algorithm becoming a behemoth of unmanageable complexity.
Unlike a building, Software has to be built for change. They cannot be rigid and serve only a single purpose what was conceived in the beginning. They are not like movies that has a plot that cannot be altered after its release. A successful software is always bombarded by needs from the users. They need new movies every week.
Well I digressed a bit here. After six months or so, I presented the final architecture, an extremely detailed document running close to a hundred pages ready to be engineered and having answers to all the questions known at that time and some futuristic use cases.
I was pretty happy with the outcome not just because I could see it working on paper, but because the Engineers who are my customers understood it and were willing to take it forward.
The key learnings are
Software architecture is not just about the systems, components, algorithms or technologies. They are very important. But ultimately it is about how they come together to handle the use cases that a team of Engineers can understand the reasons for doing it in a certain way and can expand on it as a base for future needs. That is when the architecture begins to have a life of its own.
Software Architecture is not just about the image of a super architect like Michelangelo doing his David. It is about shaping a David with viewpoints of those who will use it, those who will build it and those who have knowledge and experience around it.