Tag Archives: Sentinel

CIO behind FBI’s Agile-developed Sentinel IT project to leave his post

By David Bicknell

The US CIO behind one of the world’s highest profile public sector Agile IT projects is to leave his post and return to the private sector.

Chad Fulgham, CIO at the FBI will leave next month having overseen the creation of the FBI’s Sentinel case management system. Sentinel replaces the FBI’s outdated Automated Case Support system, with the hope that it will transform the way the FBI does business by moving it from a primarily paper-based case management system to an electronic work flow-based management system of record with enhanced data sharing capabilities.

“When I was hired as the CIO, it was understood Sentinel was going to be one of my top priorities,” said Fulgham. “Today, I can tell you the software coding is done, the new hardware is in place, and it has been quite impressive during initial performance testing. We have trained hundreds of FBI special agents and employees, and it will have a lasting impact on this organisation.”

In a press release announcing Fulgham’s departure, the FBI said that “using a progressive Agile software development methodology, partnering with industry, and employing an aggressive deployment schedule, Sentinel is scheduled to be implemented in summer of 2012.”

The US Inspector General recently issued a report into the use of Agile in the Sentinel project. You can read the report here

The US magazine Information Week has also covered the story

Lifting the lid on Agile within a public sector IT project

Advertisements

Lifting the lid on Agile development within a public sector IT project

By David Bicknell

It’s not often that you get an insight into the workings of Agile development within a public sector  IT project.

So the Inspector General’s report into the Sentinel IT project at the FBI that I mentioned a couple of days ago offers a rare and unique picture into how the sprints, story points etc are progressing. This will not be new to Agile exponents – but the detail below may be of interest to those unfamiliar with Agile’s processes.

Transition to an Agile Development Approach

The report’s discussion of Agile within the Sentinel project says this:

“Agile software development is not a set of tools or a single methodology, but an approach that leverages close collaboration between representatives of system users, system developers, and testers to deliver functionality in a compressed timeframe and on a continuous basis. The delivery of working software is the primary measure of progress, and satisfying customers through the delivery of valuable software is treated as the highest priority during development.

“While an Agile methodology can be implemented in a variety of ways, the FBI is implementing a variation called Scrum, an iterative methodology which breaks the development effort into increments called sprints, each of which the FBI decided would last 2 weeks.

“At the conclusion of each sprint, User Stories – functions that a system user would typically perform – along with Architecture Stories – qualities that define the system software architecture and configuration – are planned and completed, and it is the successful completion of these stories that is measured as progress for the project.”

Development Progress

“As of August 26, 2011, the FBI had completed 22 of 24 planned sprints. Under the Scrum approach, a project’s progress and amount of work remaining is measured using a burndown chart, which depicts how factors such as the rate at which a development team completes work (a team’s velocity) and changes in a project’s scope affect its likelihood of staying on schedule and within budget over time.

“This information can be used by project management and project stakeholders to estimate the duration of the project or the amount of work that can be completed within an identified amount of time.

“During the first 22 sprints (Sprint 0 through Sprint 21), the FBI had completed 1,545 of the 3,093 story points (1,548 remaining) that it identified at the beginning of the project, or about 50 percent.  As of December 2, 2011, the FBI reported that it had completed 28 of 33 planned sprints ….It had also completed 2,345 story points  – 748 remained to be completed.”

Velocity

The Report says this of the Agile team’s velocity:

“According to FBI officials, after five sprints have been completed, the velocity, or rate at which an Agile team completes story points, can be used to project the completion rate of future work. During Sprints 5 through 21, the Sentinel team’s average velocity was 80 story points per sprint.

“During our review, we estimated that if the team’s velocity remained at 80 story points per sprint, the FBI would complete about 55 percent of the intended functionality by the end of the project’s originally planned 24 sprints on September 23, 2011. At that rate of development we estimated that Sentinel will be completed in June 2012.

“On September 6, 2011, the FBI CIO stated that the FBI had added six development sprints to Sentinel’s development schedule and that the FBI then planned to end development on December 16, 2011, after 30 sprints. After development ended, the FBI planned to test Sentinel for about 6 weeks and then deploy the system to all users in January 2012. During the additional development sprints, the FBI planned to finish the functionality work that it previously planned to complete by September 23, 2011.

“Based on the average velocity of 80 story points per sprint, and the number of remaining story points to be completed (1,548) we estimated that the FBI would complete about 71 percent of the intended functionality by the end of the project’s 30 development sprints on December 16, 2011.

“On December 1, 2011, the FBI again extended the schedule for the completion of Sentinel. The CTO stated that the FBI had added four development sprints to Sentinel’s development schedule and that the FBI now plans to end development in February 2012, after 34 sprints. After development, the FBI plans to test Sentinel for about 12 weeks and then deploy the system to all users in May 2012. During this testing period, the FBI plans to test Sentinel’s hardware and execute a test of all major Sentinel functionality that will involve personnel from across the FBI.

“Also in December 2011, after the FBI received a copy of our draft report, the FBI reported to us that during Sprints 5 through 28 it had completed 2,167 story points, an average of 90 story points per sprint – 10 more story points than its average rate as of September 2011.

“Based on this average velocity and the number of remaining story points to be completed (748) during the final 5 sprints under this plan, the Sentinel team must increase its average velocity to approximately 150 story points per sprint.

“However, the six sprints between the end of development and deployment – during which the FBI will test Sentinel – could also have story points assigned to them that the FBI is not accounting for at this time, and as a result the total number of story points to complete the project could increase. Without including such an increase, the FBI would need to average about 68 story points per sprint over the total 11 sprints remaining before the planned May 2012 deployment.”

Sentinel Agile Development Approach

The report’s Appendix says this about the FBI’s approach to its Agile development for Sentinel:

“In October 2010 the FBI identified a total of 670 stories for the Sentinel Product Backlog, or the compilation of all of the project’s stories. The FBI has mapped the Product Backlog to each of the requirements in Sentinel’s Systems Requirements Specification (SRS), which serves as an important control to ensure that the backlog, and the stories it contains, cover all of Sentinel’s requirements. The FBI also assigned weighted amounts, or “story points,” to each story in the Product Backlog based on the difficulty of the work associated with each story. The FBI assigned a total of 3,093 story points to its 670 stories in the Sentinel Product Backlog.”

The Report’s Conclusion

Although it appears that the FBI has made good progress with its Agile development, adopting Agile may not be enough to get the project exactly on track, with some testing issues and hardware problems discussed in the report.

“It is too early to judge whether the FBI’s Agile development of Sentinel will meet its newly revised budget and completion goals and the needs of FBI agents and analysts.

“While the Sentinel Advisory Group responded positively to the version of Sentinel it tested, results from wider testing were not as positive. Also, none of the Agile-developed Sentinel has been deployed to all users to give them the ability to enter actual case data and assist FBI agents and analysts in more efficiently performing their jobs.

“Despite the FBI’s self-reported progress in developing Sentinel, we are concerned that the FBI is not documenting that the functionality developed during each sprint has met the FBI’s acceptance criteria. Our concerns about the lack of transparency of Sentinel’s progress are magnified by the apparent lack of comprehensive and timely system testing.

“Our concerns about the lack of transparency also extend to Sentinel’s cooperation with internal and external oversight entities, to which Sentinel did not provide the necessary system documentation for them to perform their critical oversight and reporting functions.

“We believe that this issue could be resolved, at least in part, with a revision to the FBI’s Life Cycle Management Directive to include standards for Agile development methodologies.

“….Sentinel experienced significant performance problems during the Sentinel Functional Exercise. The FBI attributed these performance problems to either the system architecture or the computer hardware.

“According to the FBI, subsequent operational testing confirmed the inadequacy of the legacy hardware and the requirement to significantly expand the infrastructure before the system could be deployed to all users. In November 2011, the FBI requested that Lockheed Martin provide a cost proposal for this additional hardware.”

The FBI’s Response

In its response to the report, the FBI says:

 “….we are mindful of the short delay we have recently encountered under our new” Agile” approach. The Sentinel development schedule has recently been extended by two months (from December 2011 to February 2012), and the FBI-wide deployment is now scheduled for May 2012, as described in this Report.

“This modest extension is due primarily to the need to implement a standard  five-year “refresh” of computer hardware, so the Sentinel software will provide the required functionality as intended. Indeed, you have determined that, given the pace at which the program has proceeded under the Agile approach over the time period you reviewed, your estimate for completion is essentially the same – June 2012.

“We have one concern with the current draft of the Report. We request that you note that the hardware we are acquiring for the refresh, which is being purchased using fiscal year 2012 operations and maintenance funds, is separate from the development activities being carried out by the Agile team under the development budget.

“The refresh is part of the normal and expected operations and maintenance activities of the FBI, and such a refresh is a common maintenance activity where hardware has reached its expected replacement threshold. We do not agree that the FBI is using operations and maintenance funds for the development of Sentinel…and we ask that you make this revision.”

Agile approach ‘reduces risk’ but offers no delivery guarantee for FBI as Sentinel project slips again

By David Bicknell

A recent story by the US magazine InformationWeek Government has cast some doubt over whether a move to Agile development will necessarily bring IT projects in on time and to budget, despite the best intentions. Sometimes other technology factors, such as legacy hardware, come into play.

The report says an FBI project to develop a digital case-management system to replace outdated, paper-based processes has been further delayed, despite a move to use Agile development to hasten the project’s completion.

The system, dubbed Sentinel, is now due to go live in May, eight months later than the FBI planned when it embarked on the Agile development plan.

It follows a number of delays going back to 2006 in the plans to build a replacement for the FBI’s 17-year-old Automated Case Support system, which is used by agents and analysts to manage their cases.

 In 2006, Information Week Government reports, the FBI awarded Lockheed Martin a $305 million contract to lead development of Sentinel, but took back control of the project in September 2010 amid delays and cost overruns.

Then, the FBI said it planned to finish Sentinel within 12 months using Agile development. But that worked has slipped (the FBI had earlier pushed Sentinel’s deployment from September 2011 to January 2012), and a four-hour test of the system in October resulted in two outages, according to a report by  the Inspector General released in December which contains details of the Agile development’s sprints’ progress.

The FBI said the glitches  were down to overburdened legacy computer hardware and said the hardware will need to be upgraded to support Sentinel’s use across the agency, according to the Inspector General.

The report’s conclusion says:

The FBI’s transition to an Agile development approach has reduced the risk that Sentinel will either exceed its budget or fail to deliver the expected functionality by reducing the rate at which the FBI is spending money on Sentinel and by instituting a more direct approach to the FBI’s monitoring of the development of Sentinel.

!When we provided our initial draft of this report to the FBI in October 2011, we expressed concern that the rate at which the FBI was developing Sentinel’s functionality indicated the project was at risk of falling behind the FBI’s then planned January 2012 deployment date.

“In December 2011, after we completed our fieldwork for this report and after we provided the FBI with a revised draft report, FBI officials told us that the FBI extended the Sentinel deployment date to May 2012. While we have not had the opportunity to fully review the FBI’s plan to meet these revised completion dates, we continue to believe it will be challenging for the FBI to meet this latest goal for deploying Sentinel to all FBI users in this timeframe.

“It is too early to judge whether the FBI’s Agile development of Sentinel will meet its newly revised budget and completion goals and the needs of FBI agents and analysts. While the Sentinel Advisory Group responded positively to the version of Sentinel it tested, results from wider testing were not as positive. Also, none of the Agile-developed Sentinel has been deployed to all users to give them the ability to enter actual case data and assist FBI agents and analysts in more efficiently performing their jobs.

“Despite the FBI’s self-reported progress in developing Sentinel, we are concerned that the FBI is not documenting that the functionality developed during each sprint has met the FBI’s acceptance criteria.

“Our concerns about the lack of transparency of Sentinel’s progress are magnified by the apparent lack of comprehensive and timely system testing. Our concerns about the lack of transparency also extend to Sentinel’s cooperation with internal and external oversight entities, to which Sentinel did not provide the necessary system documentation for them to perform their critical oversight and reporting functions. We believe that this issue could be resolved, at least in part, with a revision to the FBI’s Life Cycle Management Directive to include standards for Agile development methodologies.”

Responding to the report the FBI said:

“The Federal Bureau of Investigation (FBI) appreciates the opportunity to review and respond to your draft report entitled, “Status of the Federal Bureau of Investigation’s Implementation of the Sentinel Project.”

“We are pleased with your conclusion that, by adopting an Agile development approach, the FBI bas “reduced its rate of spending on Sentinel” and instituted a more “direct approach to monitoring the development of the system’s functionality.”

“Indeed, as the FBI’s figures included in this Report demonstrate, while we have expended only 52% oftbe Agile development budget of$32.6 million. as of December 6 we had completed 88% of the required system functionality. The percentage of functionality completed has further increased during the time that has passed since your report was last updated.

“This accomplishment is significant. In mid-2010, the FBI charted a new course for completing the remaining two phases oftbe Sentinel program using an Agile development approach, which represented a substantial departure from its prior development activities. As a result, you concluded in this Report that the FBI is “expending significantly fewer dollars per month than it had in Phases 1 and 2 for the project.”

“In sum, we agree with your conclusion that the FBI’s transition to an Agile development approach has “reduced the risk that Sentinel will either exceed its budget or fail to deliver the expected functionality.” As you note, “at this point in time, the FBI does not foresee exceeding the $451 million budget to complete the Sentinel project.”

“With that in mind, we are mindful of the short delay we have recently encountered under our new” Agile” approach. The Sentinel development schedule has recently been extended by two months (from December 2011 to February 2012), and the FBI-wide deployment is now scheduled for May 2012, as described in this Report.

“This modest extension is due primarily to the need to implement a standard five-year “refresh” of computer hardware, so the Sentinel software will provide the required functionality as intended. Indeed, you have determined that, given the pace at which the program has proceeded under the Agile approach over the time period you reviewed, your estimate for completion is essentially the same – June 2012.”

Inspectorate General report

Agile for Universal Credit a good choice