Lessons learnt in managing software projects
This post is a slightly modified version of a report published for my Kelburn automated parking project.
Introduction
This report discusses my personal contribution to scope and time management management in the context of our Engineering group project. Our project aims to automate the car-parking across Kelburn campus, reducing administrative load and associated permit printing costs. It involves building a License Plate Recognition system, and a web interface for end users and administrators to interact with.
We followed the Xanpan development methodology, employing techniques such as iterations (sprints), code reviews and peer programming, retrospectives, stand-ups and Kanban boards. I actively contributed to these processes. Likewise, I can confidently say that the management of this project so far has been a success, allowing for a high quality technical demonstration to be made.
However, this project has experienced multiple issues that have affected our ability to satisfy our client’s requirements and our intended deliverables.
The first main issue is that there was a tendency to overestimate our ability to handle the project scope. In the planning stages of the project, myself and the team were presented with a large number of features and use cases that the client wanted. I took on a large number of these features, not realising that the scope was too complex to feasibly complete in the project this year.
Furthermore, because of an inadequate understanding of the project requirements, I, along with other team members, went down the wrong road in design - we developed requirements for a system that could track individual carparking slots within a carpark. To be able to have this fine grained monitoring we would need extra hardware and a much more complicated software system to support it. I realised that a simpler system - one that only monitors the state of carparking zones - would accomplish the clients original goals of automation. This is reflected in section 1.3 of our Architecture Document (not available to the public!), which also discusses further scope reductions such as reducing the number of gates to manage.
The second main issue was that a lack of long term planning and time management in the early stages of the project slowed the development and understanding of the projects overarching goals. My team spent the first seven weeks of the project solely gathering requirements, which I took part of. Although it fell roughly in line with the client’s initial roadmap, I noticed that because no technical spike or walking skeleton had even partially been developed, I effectively had achieved nothing. A further side effect, and evidence of our lack of planning, was that the requirements and architecture documents were submitted very close to and over the due date, respectively. They both required significant work close to the deadline to finish.
Moreover, although I planned out an iteration schedule for the team, the iterations were only used for short-term planning, and so their use in a long-term context was limited. By only planning for two weeks in advance, I, along with my team members, were effectively developing blind.
The further sections of this report analyse these issues, and attempt to detail why these issues occurred, along with how I changed my management style in response to these issues.
Analysis
Scope Management
The primary issue with scope management was that we overestimated our ability to fulfil the given requirements, and as a result set our scope too high.
Delving further, it became apparent that a lack of strict direction on the client’s behalf could have contributed to the overestimated scope. In early meetings, the client presented a list of essential features, but beyond that left it up to us to decide what was most important to implement. Although we decided on a list of use cases shown in our requirements document which the client was happy with, both parties were unaware that the scope was not technically feasible at this point.
Likewise, there was no formal scope validation process, relying simply on “looks good to me” verbal statements. The requirements document was intended to be a collation of the project’s technical and non-technical requirements which feed into determining the scope. However, no formal validation of the scope was carried out - the document was an overly verbose collection of non-cohesive requirements, randomly collected as we learnt about the project. I went against my team in this regard as my focus on the document was to try to restrain the scope by composing it into a number of use cases, although the use cases were too broad and numerous to substantiate a well defined scope with clear limits of what is and what is not applicable for this project. Moreover, because of a discrepancy in knowledge amongst team members, sections of the document directly contradicted others parts, which is further indication that the scope of the project was not well understood.
This lack of applicable technical experience was reflected in the quality of the requirements document. Because some team members didn’t understand the technical implications of implementing the project governed by our requirements, there was a tendency to overextend. I noticed this and made some attempts to reduce scope. An example of this was a request by the client to incorporate LED signage for showing parking availability - I shut this down early as I understood the additional hardware and software components would be simply too much to manage.
Thus, I realised that management of scope is about the top-down organisation: breaking up the problem into smaller workable chunks that can be worked on, and prioritising the goals or user stories that were most important. To remedy these scope issues, one action I aided in implementing with another team member was to introduce fully defined, documented user stories. These were assigned to Gitlab Epics and encompassed relevant development tasks. I realised that this was the “glue” that brought the overarching issues together with the development work - we had the top layer and bottom layer of the issue hierarchy, but not the middle. Defining user stories served to bring all layers together to form a cohesive whole.
Further management actions that I implemented was introducing retrospectives in our team - as well as helping with time management (section B), retrospectives allowed us to reflect on our recent development iteration, and judge if scope creep had occurred. A good example of this was sprint 5 retrospective.
Time Management
Overall, I believe my personal time management is a reflection of our teams management. Early stages of the project had little to no time planning, but over time this improved to the point where we accurately know where we are in the project cycle, and where we should be. As discussed in the introduction, lack of time management and long term planning in the project’s early stages slowed the development and understanding of the projects overarching goals, causing issues such as late deadlines, low accomplishments with little evidence of progress, and developing blindly week to week. I believe that there were multiple reasons for this.
Firstly, as discussed in the previous section, a lack of appropriate knowledge of the scope and technical work required to fulfil the scope contributed to our lack of planning. It is hard to plan something that you don’t understand - by not being aware of what is required to fulfil a goal, it is hard to plan the steps that need to be taken to accomplish it. For me, I personally put off the “hard thinking” - and as a result it meant that I chose to not manage this part of the project.
Secondly, I left submission of assessment documents late and made last minute changes. These were both a glaring indication that we hadn’t been managing our time correctly. Part of this could have been due to a lack of prioritisation in terms of our course work - the assessed documents were treated as assignments, not as useful documents that are beneficial to the success of the project. Thus, as is common with regular assignments, I didn’t manage the time that it would take to complete the documentation and as a result the quality, and hence the usefulness to our project, suffered.
Thirdly, there was an air of uncertainty about the future of the project after this year. Whilst this uncertainty was a scope management issue, it affected our ability to manage time. This project was originally intended to span multiple years, and be further developed by future ENGR301/302 students. As a result, how I felt (and, how I imagine the rest of the team felt) was that there wasn’t a pressing need to manage our time to develop to a deadline, as the work would be handed over to future developers in whatever state it would be in at the end of the year. This obviously was wrong and should have been planned for a working product from the start - we are expected to deliver something to the client.
When I realised this, I completely changed the time management in the project. Me and another team member, with the user stories we created, designed a roadmap with the help of the client Roadmaps are discussed in Xanpan as a good way of planning long term goals, factoring in variability [xanpan book c 10.4]. The roadmaps gave context to the iterative development cycles that we had already established. Before, we were developing week-to-week, and had put off long term thinking. With a roadmap we could contextualise the sprints.
Along with breaking our work into iteration cycles, one further thing I did was to title sprints in relation to the roadmap. Things gave context to the sprint in relation to the project life-cycle and helped the team understand the bigger picture. Moreover, each sprint was given a list of deliverables that we compared against in the retrospective.
In Xanpan, taking work into an iteration does not mean that the team is committed to doing the work. We followed this approach, and used leftover work to plan future iterations. I found that this became a good way to estimate work forecast velocity, and to see how much work both myself and the team is capable of achieving.
Nonetheless, my time management of personal tasks was very good. In 70 hours, along with other foundational work I completed a fully working backend system for our application that was unit tested and integrated with the rest of our software product. Comparing this to my previous summer’s internship, I had a similar project using the same tools - and although I had industry professional support it still took me well over 80 hours of full time work to build a product that has the same level of quality as my current one. This shows that I managed my time efficiently to be able to achieve project goals that align with the teams roadmap.
Conclusion
It is impossible to find a single cause for the issues that have slowed down the development of the project. Rather, lack of adequate management across the different areas of the PMBOK [pmbok book, c 4,5] framework have contributed in different ways to slowing the understanding of goals, and the development of the project. This section concludes the analysis by making balanced recommendations for strategies to improve our management of this years project.
For scope management, it is absolutely imperative that the scope is well defined before significant development work is undertaken on the project. Good scope definition is crucial because it defines a baseline for performance measurement and project control [xanpan p. 141].
In our case, as we are already too far into development, the scope should be strictly managed to prevent scope creep [pmbok p. 313]. This cannot be done unless the scope is clearly defined in the first place. Moreover, formal processes need to be developed for soliciting and monitoring changes to the scope. Examples of formal processes are change request forms. For any significant change in the requirements of the finished product, a formal document describing the change, its effects, risks and benefits could be required before being considered. Although in some ways this goes against the Agile Manifesto because it slows adoption of changing requirements, a large project needs a formal process in place so that potentially damaging variables are controlled. For my project, I need to control feature requests. Uncertainty should decrease as a function of time, and by controlling what features are required to satisfy the basic need - automation of carparks - I can manage this uncertainty.
There are lots of alternative frames of thinking when it comes to controlling scope. Some practises completely set a static scope after the initial requirements gathering phase. I believe this can be beneficial in some ways because it constrains the project and allows the overarching issues to be broken down into tasks without worrying about change. Because we are an Agile team, we need to be adaptable to change, and the requirements can and will change throughout the year.
Secondly, it is important to not underestimate the complexity of a given project. You can often predict the success or failure of a project by looking at whether similar projects have been successful in the past. A big takeaway for me is that I should learn to understand the complexity of the project im working on, and not take on jobs that I am not confident on being able to complete. When I am not confident, I should do research and make preparatory action (such as conducting development spikes) before committing to implementing it. Our system is very complex because of the large number of external systems that integrate with it. Moreover, external integration is something that I have not conducted research onto yet. Therefore, a good management practise to implement in the future is to break down incoming scope changes or feature requests so that its complexity can be better understood.
On the time management side, the most important thing I learnt was to actually manage our time. The process from choosing not to manage our time beyond trivial iterations, to making roadmaps and milestones for planning long term work was a big learning experience for me. By definition, a project has an end date, and for my purposes, that end date is at the end of this year - not whenever the future developers are finished.
An important takeaway from this is understanding that time management is a multi-step process. The PMBOK framework recommends 7 steps to creating an efficient time plan [pmbok p. 165] - planning schedule management, defining activities, sequencing activities, estimating activity resource, estimating activity durations, and developing the project schedule. I personally believe that although this is incredibly valid and useful information to build a time plan from, because I have already planned my time by using a cut down version of this process, I should focus on iterating on my time plan to make it more effective, accurate and relevant to our project. My cut-down plan involved defining the activities, estimating the time spent on activities and then generating a schedule off of that. This is evident from the projects architecture document, user story issues, and roadmap, respectively.
For the future, I should incorporate proper time management from the beginning, and not cut corners. This includes following the 7 step process outlined in the PMBOK framework [pmbok p. 166]. At the same time, I should take care to not spend too much time trying to plan the time management, as there becomes a certain point where the level of management doesn’t significantly increase the tangible benefit.
Another takeaway in terms of time management is to treat the assessment items not as assignments but as relevant and useful project management tools that can help us in the delivery of our project. This course, and ENGR302, puts the assessments in at strategic times and frame them in a way that could be extremely beneficial to us, not just as a way to grade a student. Now that I understand this, I should put more stake into managing my time to do a better job on these assessments, as not only will it improve my grade in the course, but improve the quality of my final product.
In conclusion, scope and time are two very important management areas for any project. This report has provided an analysis into some of the issues associated both scope and time management, and has provided some insight into learnings that we can apply into the future.