At first inspection the two words don’t seem to be in conflict. It seems that your software development project can be in control our out of control, and your status data be honest or dishonest. It looks like four valid combinations. With a little thought you realize that you can’t have true control with dishonesty, so there are really only three valid combinations. You can have control with honesty, lack of control with honesty, and lack of control with dishonesty.
But I submit that the effort to control a software development project will lead to dishonesty. When I say this, I don’t mean it will lead to outright lying. I do mean it will lead to behaviors that have exactly the same impact on your project. If your information isn’t accurate, you’ll have the same problems as if your team had been intentionally lying to you. You just won’t have grounds to dismiss them.
Here’s my explanation for why this happens. Initially you create a project plan with some rough estimates to allow the company to make a funding decision. Perhaps your company is enlightened enough to know that you will have better estimates after analysis, so they make a preliminary funding decision and you return with an updated project plan after the analysis is complete. At this point, if the company approves the plan, they expect to get the anticipated return on investment. That means you need to deliver the full project on time and within budget.
Now the headaches begin. There’s a lot of inherent uncertainty and a lot of research that’s required to get your product built. You can build the research into the project plan, but all you can do with the uncertainties is add management reserve (schedule padding) and contingency reserve (budget padding) to your plan. Even if your management team is enlightened enough to let you keep these reserves, you still may not have requested enough.
What happens when you start to burn through your reserve? You get creative. You all start creating optimistic scenarios. You believe in the project and don’t want it cancelled, so you project things you aren’t sure about. You hide the uncertainty and the problems and focus on the successes. This isn’t because you or your team want to lie, but because you want to succeed. You know if management starts thinking control is slipping they will either take over (adding new risks) or shut you down. The focus on control leads to information hiding.
How do you avoid this trap? You have to shift your paradigm. You shift from focusing on control to focusing on accurate information that can allow you to make accurate, timely decisions. And within the team you need to change your definition of success from delivering the product to delivering the truth. This means breaking your project down into short iterations of work that achieve something measurable, and accepting the truth those measurements present. In short, you implement an Agile methodology.
Let’s look at a real world example. Suppose you decide to create a new tool for viewing marketing data that relies on modifying your underlying data model to support map-reduce searches. You don’t know if your data model can be modified in this way, but you pitch your project on the benefits and get funding approval.
In a traditional model, you would schedule your dependencies and start working. The completed model and map-reduce search functionality aren’t needed until about the fourth month of work. When you start to realize that deadline might be missed you become creative on ways the project can continue while the issues with the data model are resolved. If those issues can’t be resolved, you might waste six months of work before scrapping the project. This would be a failure.
In an Agile model you would recognize the risk of the map-reduce assumption and establish your first few iterations around the tasks needed to prove or disprove the feasibility of the assumption. If the issues can’t be worked out, you know within six weeks. Management can redirect your team to a more productive project. This would be a success.
In neither case did you deliver the project, but we can call the second approach successful because the entire company learned something important and resources were not wasted on code that would be thrown away. In the first approach we learned the same lesson, but we also wasted months writing other parts of the system that we will never use.
I’ll concede it is possible to teach a team to be unafraid of sharing bad news on a long project that is facing serious issues. I’ll even concede that you can teach a team to let go of an exciting project after months of work without feeling like a failure. But both will be major efforts that go against human nature. If you take the Agile approach you don’t need to ask people to work against their natures. The end result will be a much better return on your software development investment. A side benefit will be a higher morale as you succeed much more often.