Tag Archives: Sentinel

By David Bicknell

It’s not often that you get an insight into the workings of Agile development within a public sector  IT project.

So the Inspector General’s report into the Sentinel IT project at the FBI that I mentioned a couple of days ago offers a rare and unique picture into how the sprints, story points etc are progressing. This will not be new to Agile exponents – but the detail below may be of interest to those unfamiliar with Agile’s processes.

Transition to an Agile Development Approach

The report’s discussion of Agile within the Sentinel project says this:

“Agile software development is not a set of tools or a single methodology, but an approach that leverages close collaboration between representatives of system users, system developers, and testers to deliver functionality in a compressed timeframe and on a continuous basis. The delivery of working software is the primary measure of progress, and satisfying customers through the delivery of valuable software is treated as the highest priority during development.

“While an Agile methodology can be implemented in a variety of ways, the FBI is implementing a variation called Scrum, an iterative methodology which breaks the development effort into increments called sprints, each of which the FBI decided would last 2 weeks.

“At the conclusion of each sprint, User Stories – functions that a system user would typically perform – along with Architecture Stories – qualities that define the system software architecture and configuration – are planned and completed, and it is the successful completion of these stories that is measured as progress for the project.”

Development Progress

“As of August 26, 2011, the FBI had completed 22 of 24 planned sprints. Under the Scrum approach, a project’s progress and amount of work remaining is measured using a burndown chart, which depicts how factors such as the rate at which a development team completes work (a team’s velocity) and changes in a project’s scope affect its likelihood of staying on schedule and within budget over time.

“This information can be used by project management and project stakeholders to estimate the duration of the project or the amount of work that can be completed within an identified amount of time.

“During the first 22 sprints (Sprint 0 through Sprint 21), the FBI had completed 1,545 of the 3,093 story points (1,548 remaining) that it identified at the beginning of the project, or about 50 percent.  As of December 2, 2011, the FBI reported that it had completed 28 of 33 planned sprints ….It had also completed 2,345 story points  – 748 remained to be completed.”

Velocity

The Report says this of the Agile team’s velocity:

“According to FBI officials, after five sprints have been completed, the velocity, or rate at which an Agile team completes story points, can be used to project the completion rate of future work. During Sprints 5 through 21, the Sentinel team’s average velocity was 80 story points per sprint.

“During our review, we estimated that if the team’s velocity remained at 80 story points per sprint, the FBI would complete about 55 percent of the intended functionality by the end of the project’s originally planned 24 sprints on September 23, 2011. At that rate of development we estimated that Sentinel will be completed in June 2012.

“On September 6, 2011, the FBI CIO stated that the FBI had added six development sprints to Sentinel’s development schedule and that the FBI then planned to end development on December 16, 2011, after 30 sprints. After development ended, the FBI planned to test Sentinel for about 6 weeks and then deploy the system to all users in January 2012. During the additional development sprints, the FBI planned to finish the functionality work that it previously planned to complete by September 23, 2011.

“Based on the average velocity of 80 story points per sprint, and the number of remaining story points to be completed (1,548) we estimated that the FBI would complete about 71 percent of the intended functionality by the end of the project’s 30 development sprints on December 16, 2011.

“On December 1, 2011, the FBI again extended the schedule for the completion of Sentinel. The CTO stated that the FBI had added four development sprints to Sentinel’s development schedule and that the FBI now plans to end development in February 2012, after 34 sprints. After development, the FBI plans to test Sentinel for about 12 weeks and then deploy the system to all users in May 2012. During this testing period, the FBI plans to test Sentinel’s hardware and execute a test of all major Sentinel functionality that will involve personnel from across the FBI.

“Also in December 2011, after the FBI received a copy of our draft report, the FBI reported to us that during Sprints 5 through 28 it had completed 2,167 story points, an average of 90 story points per sprint – 10 more story points than its average rate as of September 2011.

“Based on this average velocity and the number of remaining story points to be completed (748) during the final 5 sprints under this plan, the Sentinel team must increase its average velocity to approximately 150 story points per sprint.

“However, the six sprints between the end of development and deployment – during which the FBI will test Sentinel – could also have story points assigned to them that the FBI is not accounting for at this time, and as a result the total number of story points to complete the project could increase. Without including such an increase, the FBI would need to average about 68 story points per sprint over the total 11 sprints remaining before the planned May 2012 deployment.”

Sentinel Agile Development Approach

The report’s Appendix says this about the FBI’s approach to its Agile development for Sentinel:

“In October 2010 the FBI identified a total of 670 stories for the Sentinel Product Backlog, or the compilation of all of the project’s stories. The FBI has mapped the Product Backlog to each of the requirements in Sentinel’s Systems Requirements Specification (SRS), which serves as an important control to ensure that the backlog, and the stories it contains, cover all of Sentinel’s requirements. The FBI also assigned weighted amounts, or “story points,” to each story in the Product Backlog based on the difficulty of the work associated with each story. The FBI assigned a total of 3,093 story points to its 670 stories in the Sentinel Product Backlog.”

The Report’s Conclusion

Although it appears that the FBI has made good progress with its Agile development, adopting Agile may not be enough to get the project exactly on track, with some testing issues and hardware problems discussed in the report.

“It is too early to judge whether the FBI’s Agile development of Sentinel will meet its newly revised budget and completion goals and the needs of FBI agents and analysts.

“While the Sentinel Advisory Group responded positively to the version of Sentinel it tested, results from wider testing were not as positive. Also, none of the Agile-developed Sentinel has been deployed to all users to give them the ability to enter actual case data and assist FBI agents and analysts in more efficiently performing their jobs.

“Despite the FBI’s self-reported progress in developing Sentinel, we are concerned that the FBI is not documenting that the functionality developed during each sprint has met the FBI’s acceptance criteria. Our concerns about the lack of transparency of Sentinel’s progress are magnified by the apparent lack of comprehensive and timely system testing.

“Our concerns about the lack of transparency also extend to Sentinel’s cooperation with internal and external oversight entities, to which Sentinel did not provide the necessary system documentation for them to perform their critical oversight and reporting functions.

“We believe that this issue could be resolved, at least in part, with a revision to the FBI’s Life Cycle Management Directive to include standards for Agile development methodologies.

“….Sentinel experienced significant performance problems during the Sentinel Functional Exercise. The FBI attributed these performance problems to either the system architecture or the computer hardware.

“According to the FBI, subsequent operational testing confirmed the inadequacy of the legacy hardware and the requirement to significantly expand the infrastructure before the system could be deployed to all users. In November 2011, the FBI requested that Lockheed Martin provide a cost proposal for this additional hardware.”

The FBI’s Response

In its response to the report, the FBI says:

 “….we are mindful of the short delay we have recently encountered under our new” Agile” approach. The Sentinel development schedule has recently been extended by two months (from December 2011 to February 2012), and the FBI-wide deployment is now scheduled for May 2012, as described in this Report.

“This modest extension is due primarily to the need to implement a standard  five-year “refresh” of computer hardware, so the Sentinel software will provide the required functionality as intended. Indeed, you have determined that, given the pace at which the program has proceeded under the Agile approach over the time period you reviewed, your estimate for completion is essentially the same – June 2012.

“We have one concern with the current draft of the Report. We request that you note that the hardware we are acquiring for the refresh, which is being purchased using fiscal year 2012 operations and maintenance funds, is separate from the development activities being carried out by the Agile team under the development budget.

“The refresh is part of the normal and expected operations and maintenance activities of the FBI, and such a refresh is a common maintenance activity where hardware has reached its expected replacement threshold. We do not agree that the FBI is using operations and maintenance funds for the development of Sentinel…and we ask that you make this revision.”