Thursday, 10 March 2011

Why do software development projects for large customers take unproportionally longer time than for smaller customers

Occasionally, I meet other developers who tell me that they can't understand how we could have spent the amount of money and time that we did on a project. Often they will even say things like; "My small company could have developed this in half the time". So, why is it that solutions for large organizations takes so much more time to develop than a similar type of solution for a small company? 
Here's what I think; Time and resources are NOT a linear function of the number of features to be developed. And the reason for this? In one word I like to blame it on complexity.

Lets take a look at the definition of complexity: In most dictionaries you'll find that complexity tends to be used to characterize something being intricate and compounded. Ok, so to have an intricate and compounded project you have to have many things together. A complex project therefore must consist of many things. This seems logical. Large companies have more stuff, right? Stuff? yes more people, more projects, more sub systems, more interfacing systems, more history, more data, more legacy code, more policies, more legal requirements, more constraints, more processes, more methodologies, more guidelines, more internal and external stakeholders and so on. All these things together form a complex project. Changes to one part of the project will often have consequences for other parts. In the following sections I'll try to delve into what complexity consist of, bare with me:  

The uncertainty factor
Projects for large customers have more stuff. No one can have a full overview of all the stuff and all the details of all the stuff. Naturally this leads to situations where things are just unknown or uncertain. When the number of things we have to deal with gets over a certain size it gets hard for us manage them. Often more people are brought in to deal with the mess. Also a risk management [1] consultant might be hired to tackle the uncertainty by calculating risks. Computers are great at managing large amount of data. The human brain, on the other hand, tends to overlook, forget, make faulty assumptions, bias etc. This naturally leads to errors and wasted time. To fix the uncertainty factor we will usually try to seek information to make sense of the mess. This search for just-in-time knowledge will be overlapping. Several people will try to dig for the same information over time and the same information is prepared over and over for various purposes with slight adjustments and the information prepared are often not found again for others to reuse. And when the information is found, the reader might even misinterpret the information or apply it to a different situation where the knowledge don't apply. These are typical problems that the field of Knowledge Management has tried to solve for a couple of decades now.

Uncertainty also causes problems later in the project. Humans are very bad a comprehending complex systems so they are bound to under estimate. This under estimation leads to schedules being broken later in the process which again leads to less time to do the job right. So you end up with less quality which again later can cause big problems and lots of wasted time.   

The dependency factor
As we have seen complexity is being intricate and compounded. This implies there are relations between the stuff. Stuff are dependent on other stuff. An example of a not so complex project is a one man programming shop doing a small project for a small customer. He would have relatively few relations to other people and between the things he is working on.

We can use graph theory and network complexity as examples in this regard. When the number of nodes increases, the number of possible connections increases at a higher rate. This means that the people involved will spend more and more time on communication and dealing with more and more stuff.


According to triangular polygonal numbers using the formula n(n-1)/2; if you have 3 nodes the maximum number of relations between these nodes is 3. If you have 4 nodes the number of relations can be 6. For 8 nodes there are 28 connections and so on.

In many cases there is not a direct relation and communication needs to take several node hops which obviously take more time. This also increases the possibility of communication error so the project takes even longer time.

Are there really more stuff to deals with on projects for large customers?
Below is a listing of what I have experienced during my work as a freelancer, and as a consultant for everything from small companies to large multinational corporations.

Processes
  • Large companies usually have higher demands on quality assurance.
  • Large customers are more likely to have a number of parallel projects where internal resources have to be coordinated between projects. Often you will have to wait for some other project.
  • Large companies often have rigid processes with decision gates and formal project planning steps that just takes time. Decisions need to be approved on many levels.
  • You are probably more people and because many are dependent on you, you can not just sit wherever you like and do the project. Less work situation flexibility usually results in less effective work as you have to spend more time on travel instead. 
  • In large organizations there's usually more beurocracy, which means more waiting if you need to order something, if you need access rights etc etc. You might have internal rules preventing you from fixing your computer when it breaks etc.  
  • There are usually more meetings in large organizations as there are more people involved that wants to have their say. A two hour meeting with 10 persons equals 20 hours.  
  • You are likely to have more guidelines, compliance or regulations on your process so you need to spend more time reporting and writing documentation. 
  • You often can't choose the tools, methods, frameworks you like. You have to learn and reuse what is already in use. This makes your process less lean.
  • Working with large customers you are more likely to also work on several projects for the same customer. When working on several projects at once, a lot of time will be wasted because of all the  chaos that comes with doing parallel work.   
People
  • For large customers there's a bigger chance that there are specialists or experts to deal with certain parts of the project. When these persons quit or leave for another customer, project or even new job it takes time to replace them.
  • There are probably more people involved and there might be some misunderstandings when something is communicated verbally and new people who come on the project don't get the same piece of information.
  • For large customers there is a bigger chance that some of the new people on the project is or will get less motivated and slows down progress or create tension in the group. The new guys have less knowledge about the project and often feel they have less to contribute. Often the developers who have been on the team from the start will get an "I'll just fix it my self" mentality where he or she just solves the problem him self because he knows that it will take considerably longer time for the new person who don't know the system just as much. This again leads to the situation where the new person feels left out or less worth. 
  • For large companies there are usually more stakeholders and end users involved. The chance is bigger that relevant stakeholder and/or users do not get to give feedback or input to the development process. This often cause problems later and leads to longer development time and possibly even conflicts.
  • As a side effect of the project taking more time, you probably need more developers to deliver the project on time. Everyone can't be super programmers. The project often has to "pick from the bottom of the barrel" and so there will be novice developers that are less effective.
  • More people also mean there's greater chance that someone will become a problem. This can be because they don't fit into the group, because they simply didn't want to be on the project, because they had other expectations to what they were going to work on etc. 
  • More people needed often results in more staffing problems. Some people might feel that they are just thrown on the project.
Politics
  • For larger customers there might be collaboration problems or unhealthy competition between departments, suppliers, contractors, partners etc. 
  • The larger the organization, the more layers of management. The more layers of management, the harder it is for bad news to get to the top. No one wants to be the one to communicate the bad news.
  • When there's many departments involved you will often get into a situation where you have to find out how to please everyone. You don't want to have opposition to you project.    
  • You might even meet managers that have their own personal agenda, and you will have to deal with what comes along with that. 
  • It's more likely that you are required by IT politics to use large frameworks or platforms that make the development more difficult than it has to.
  • Large organizations usually have more requirements on branding.
Technical
  • For larger customers there's usually more infrastructure, more servers etc. This means more time goes into moving files here and there for testing etc.
  • The build and test environments are usually larger because of higher degrees of QA and Application Lifecycle Management requirements. Managing access to these environments takes more time.
  • There are usually more technologies involved and you can't know them all so you have to spend your work day google how to do this and that.
  • You are more likely to integrate your solution with some legacy system or third party product. This requires the team to learn new API's.
  • There's probably more systems and infrastructure involved in your work process so when one sub system is down the entire team might be impeded.  
  • With more server infrastructure comes more security management. 
Communication
  • For large customers there are usually more people involved so there's more waiting on getting in touch with other people. Maybe you need a confirmation on some issue so you wait for others to get around or wait for them to pick up the phone. Of course you don’t sit waiting doing nothing until you get the answer but this takes your concentration away and it makes your day more chaotic as you have a bunch of thing to do and follow up on.
  • There's going to be more coordination needed. Coordination between core team developers, other developers, between developers and designers, testers, managers, product owners etc. 
  • When there are more people, you are more often distracted. Maybe someone needs you to do something. This again takes your attention away and once attention is lost you go for coffee, start chatting or read the online news paper etc.
  • You probably have to work with external agencies or contractors. Either because they want to have the best people or because they are not able to find the required skills within either your or their own organization. So they outsource parts of the project. Then, you are left having to communicate with the other remote contractor. Teams not co-located have more communication and trust problems [2] and communication is of course much slower.   
Code
  • Large organizations usually have more history and therefore more legacy code. Writing code is very fast when there is little code. As the code base gets bigger things slow down. Rewriting and refactoring takes more time. More time is needed to architect and engineer the code. 
  • It's more likely that code is shared between projects. When requirements change it is usually more difficult to make changes if the code is reused on several projects. If these projects are not co-located it can become a nightmare to coordinate changes on shared code.  
  • Because the organization has more history it will probably have more code. As the code base gets larger, compilation takes longer time. 
  • More code makes it more difficult for developers to get an overview, so new features or sub systems might not get implemented in the best way. Often this code will need an overhaul later when it has escalated into some kind of a problem. 
  • When the code base is large and there are many developers there is a tendency for developers to be afraid to change the code of other developers. Especially if there are no unit tests. Developers fear it might have consequences they don't know of or they are afraid other developers don't agree on the code change etc. They might even feel a pressure to just get the job done so they don't do any refactoring and just adds more code making it less readable. 
  • When there are more developers involved not everyone will have the same feeling of responsibility. Some developers will avoid doing to big changes to the code because they don't want to be the one messing up the system if something doesn't work as planned.
  • As the system gets larger and time goes by there's more places and chances to put in even more code. You get the requirement creep [3], where stakeholders push for more features.
  • Large organizations usually have more users so more work need to be done at scaling the system either through code or by adding more server infrastructure. 
  • Because there are more users you have to develop fall back solutions and cater for more type of users.
  • With more users you need to do more stress testing and performance tuning.

Time leads to more time
  • As the project drags out there will be more opportunities to make changes. Suddenly someone decides that the system should be based on a completely different platform because of some top management strategic decision. 
  • If the project goes on for a long time some of the project participants will probably get tired or bored. This naturally slows down the pace.

Conclusion
Projects for large customers take more time simply because the project will be more complex.