| Kessel Run | |
| Native Name: | Air Force Life Cycle Management Center Detachment 12 |
| Preceding6: | --> |
| Superseding6: | --> |
| Headquarters: | Hanscom Air Force Base |
| Motto: | Code. Deploy. Win. |
| Minister8 Name: | --> |
| Deputyminister8 Name: | --> |
| Chief1 Name: | Colonel Richard Lopez |
| Chief1 Position: | Senior Materiel Leader |
| Chief9 Name: | --> |
| Parent Agency: | Air Force Life Cycle Management Center |
| Child25 Agency: | --> |
| Keydocument6: | --> |
Kessel Run, formally Air Force Life Cycle Management Center Detachment 12, is a United States Air Force software development division, based in Hanscom Air Force Base and Boston, Massachusetts. It was founded in 2017 by the Defense Innovation Unit in response to the need to modernize legacy Air Force software.
In October 2016, Eric Schmidt, former CEO of Google, was leading a group touring the Combined Air Operations Center at Al Udeid Air Base in Qatar in his role as inaugural chairman of the Defense Innovation Board.[1] The CAOC at Al Udeid oversees air force operations for over 20 countries, and at the time was engaged in the War in Iraq against the Islamic State.[1]
One of the Air Operations Center tasks was planning daily aerial refueling operations to support combat missions. This was done off the main CAOC hall, in a windowless room with a whiteboard bearing magnetic pucks and plastic laminated cards, and physically measuring distance on the board to determine how long planes could stay in the air.[1] The resulting data was manually entered into an Excel spreadsheet known as "the Gonkulator" by a person called "the Gonker".[1] When a VBScript run on the spreadsheet confirmed the data was correct, another person, called "the Thumper", manually typed in the result into a Master Air Attack Planning Toolkit, which helped generate the Air Tasking Order back at Shaw Air Force Base in South Carolina.[1] Yet another person watched to verify against retyping errors. The process took eight to 12 hours each day for three or eight people.[1] [2] If there was a change needed, the process needed to restart.[2] When Schmidt and the Defense Information Board asked whether the base had access to more modern software to automate this process, the answer was: "Yes, but it doesn't work."[1]
Air Operations Center software had been in use mostly unchanged since the 1990s.[1] From 2006 to 2011, Lockheed Martin worked on the concept of modernizing it, but did not take up the project. A Northrop Grumman project to modernize AOC software was commissioned in 2013 for $374 million for development, and $3.5 billion for lifetime maintenance. By 2016, when the Defense Innovation Board was touring Al Udeid, nothing had been delivered. The development price eventually grew to $745 million, and was three years behind schedule, with an estimated launch date of December 2019, when the project was eventually cancelled in July 2017.[3]
Also touring the Al Udeid CAOC with Schmidt in October 2016 was Raj Shah, tech entrepreneur and managing partner of DIUx, the Defense Innovation Unit Experimental.[1] Shah was a former Air Force fighter pilot who had first hand experience of the importance of regular aerial refueling.[4] That same night, he called lieutenant colonel Enrique Oti who was head of Air Force programs for DIUx in Silicon Valley.[1] They arranged an unusual partnership between Air Force developers and Pivotal Software, and got it approved by General Jeffrey Harrigian, commanding United States Air Forces Central Command.[1] Rather than rewrite the entire AOC software suite, they would just do the tanker whiteboard.[1] In December 2016, coders and program managers were visiting Al Udeid to talk to users. By April 2017, the aerial refueling tanker application, named Jigsaw, was in use at the CAOC; four months from start to production, when a more average Defense Department software operation took as much as three to five years.[1]
The speed of development was credited to the Agile development process, an iterative, adaptive approach, that doesn't try to get the entire solution in the first release. The first version of an application is intentionally only about a 60% solution, which is then changed and improved via rapid iterations as users give feedback.[5] Though this agile development is fairly basic in the modern software industry, it was unusual for the Defense Department.[6] The development team would later adopt the hashtag #AgileAF - the AF, they assure, stands for Air Force.[1]
The total cost of Jigsaw was reported at $1.5 million (Captain Bryon Kroger, Chief Operating Officer of the project), to $2.2 million (Shah),[1] "chump change" (Harrigian). With it, tanker planning was not only faster, taking two to three hours for a single person, but was more reliable, so two to three fewer tankers were scrambled each day.[1] [2] Each scramble had a cost in fuel and maintenance of about $250,000 each.[1] [4] Jigsaw saved 350,000 pounds of fuel a week.[7] Its development costs were recouped in the first week.[1]
In April 2017, after delivering Jigsaw, Oti, Kroger, and others, got approval to form an official Air Force software development team at the Air Force Life Cycle Management Center named Project Kessel Run.[4] [8] [9] The name "Kessel Run" came from a line in the 1977 science fiction film Star Wars, spoken by smuggler Han Solo, bragging about the speed of his ship, the Millennium Falcon.[4] It represented the project's intent to "smuggle" new software development capability into the Air Force and use it to set new software development speed records.
By March 2018, Kessel Run was approximately 70 airmen, partnering with Pivotal Software, the US Air Force Academy and United States Air Forces Central to deliver nearly half a dozen software tools.[9] Isaac Taylor, chief technology officer of DIUx, called them a "rebel alliance",[9] to continue the Star Wars metaphor. In contrast, Kessel Run Chief Product Officer Adam Furtado declared he was not a Star Wars fan, and had quit watching the film at the scene with the eponymous "Kessel Run" line.
In May 7, 2018, the Kessel Run Experimentation Lab set up at a WeWork shared facility in central Boston. It was modeled after Pivotal Labs training locations in Cambridge, San Francisco, and Washington.[10] It was managed from the AFLCMC at Hanscom Air Force Base, which believed the innovation advantages in coworking and creativity would outweigh the hassles of distance and security.[10] The Lab initially had space for 90 engineers, but planned for 300 within a year.[11] Many were on temporary assignment from other Air Force bases; yet others would be sent off to Pivotal Labs offices across the country for training in modern software techniques.[10] [12] [13] The Kessel Run motto on the wall was "Code. Deploy. Win.", a play on the Air Force's motto, "Fly. Fight. Win.".[14]
On January 2, 2019, the Kessel Run Experimentation Laboratory moved to a different location in a Boston skyscraper.[15] Kessel Run's budget since mid 2017 had grown to approximately $140 million, including workspace and personnel, and the operational software produced claimed savings of $13 million and 1,100 man-hours per month.[15]
On May 8, 2019, Kessel Run formally became Air Force Life Cycle Management Center Detachment 12, commanded by Colonel Oti, who had been effectively leading it for just under a year.[16] It had nearly 700 airmen, government civilians, and contractors.[16] Oti was replaced as commander by Colonel Brian Beachkofski on April 15, 2020, with the ceremony held over Zoom teleconference.[17] On June 27, 2022, Beachkofski was replaced as commander by Colonel Richard Lopez, who took the title of senior materiel leader.[18] Lopez had previously been the director of the LevelUP Code Works software factory inspired by Kessel Run.[18] [19]
By autumn 2018, Senator Elizabeth Warren was asking in Congress how to replicate Kessel Run's success throughout the Defense Department, and Air Force Chief Technology Officer Frank Konieczny was predicting that every future acquisition would have "something that looks like Kessel Run".[20] Yet the project faced problems from its agile, reactive structure, as the team could not tell Congress what products it would be working on a year or two in the future.[13]
The March 2019 Defense Innovation Board report on software acquisitions included a chapter on Kessel Run subtitled "The Future of Defense Acquisitions Is #AgileAF".[21] In September and October 2019, Kessel Run received multiple awards: the General Larry O. Spencer Innovation award; the Theodore von Kármán award for modernizing software for the F-35; and the inaugural Defense Acquisition Software Innovation Team award.[22] [23] A 2019 editorial in Defense One said that Kessel Run was "widely seen as the gold standard of military tech done right ... also the most hyped military program office in operation today".[24]
Not all reactions were positive. A 2019 anonymous survey of KREL application users found that some applications did not meet user needs, and that success metrics, documentation, and responsiveness to user feedback could all be lacking.[25] A 2020 Harvard Kennedy School project found and tried to address internal discontent among Kessel Run staff with emerging bureaucracy and increasing technical complexity.[26] The Air Force's Deputy Chief Information Officer, Lauren Knausenberger, acknowledged in 2020 that Kessel Run was having growing pains, but said that was a result of its success.[27]
Kessel Run inspired multiple agile software development teams across the Air Force and military. They were called "software factories".[28] The original definition of software factory was a set of software tools to write and automatically build, test, and document applications; the Chief Information Officer of the Department of Defense slightly redefined that to be a software assembly plant that automated the develop, build, test, release and deliver phases, but in each case to support agile software development practices.[29]
Kobayashi Maru, formally Space C2, or Space Command and Control, in California, was the second such software factory, in August 2018.[30] It was named after an impossible test in the Star Trek science fiction universe.[31] Just as Kessel Run came from an effort to replace an outdated system by getting around bureaucratic rules, Kobayashi Maru intended to update the software of the troubled Joint Mission System (jointly run with the United States Space Force) for space command and control and situational awareness.[32] [33]
BESPIN - an acronym for Business and Enterprise Systems Product Innovation, but also the name of a planet in the Star Wars universe - was the third Air Force software factory, launched in early 2019 in Montgomery, Alabama, to create apps for maintenance crew chiefs, aircrew readiness, and ammunition crews.[30] [31] Space Camp, in Colorado, and Section31, in California, spun off of Kobayashi Maru.[31] LevelUP, in Texas, was a joint cyber operations system for the Unified Platform, connecting the Army, Marines, and United States Cyber Command, debuting in April 2019.[31] By September 2021, there were 17 Air Force software factories across the country.[34]
Software factories weren't limited to the Air Force. The Navy was inspired by Kessel Run to stand up its first software factory, The Forge, in Riverdale, Maryland, in March 2021.[35] The Army Software Factory debuted in April 2021 in Austin Community College in Texas, as part of the United States Army Futures Command.[36] [37] In February 2022, Deputy Defense Secretary Kathleen Hicks wrote a DOD Software Modernization Strategy memo encouraging increased used of software factories throughout the Defense Department; at the time there were 29.[38] By April 2022 the United States Coast Guard was planning a software factory based on the Air Force model.[39] The Marine Corps Software Factory was co-located with the Army Software Factory as a three year test project in Austin in March 2023.[40]
Jigsaw, the 2017 aerial refueling planning application that started Kessel Run, was bought and used by NATO in multiple countries in 2020 and 2021.[41] [42]
The team's second and third projects after Jigsaw were Chainsaw and Raven, applications for assembling and communicating target information.[1] Chainsaw was in operation by November 2017, consolidated many programs into one, and cut the process for dynamic targeting from an hour or two to minutes. Raven, for target development management, cut 12 hours of work down to three or four, and was ready in early 2018.[43]
Starting in late 2018, Kessel Run joined the task of fixing the troubled software for the maintenance of the F-35 fighter jet, called ALIS, for Autonomic Logistic Information System.[44] ALIS was a 17 year old proprietary system full of bugs and data gaps.[45] Maintainers had to keep separate databases because they could not rely on ALIS data.[46] The project to fix ALIS, including Kessel Run, Pivotal, and Lockheed Martin, its original creators, was called Mad Hatter, named by the developers.[44] It officially started in October 2018, but took until January 2019 before developers could write code, while the Air Force negotiated with Lockheed Martin as to what parts of the proprietary ALIS system the government could be able to reach.[44] The Mad Hatter suite of eight programs was tested and favorably evaluated by F-35 aircraft maintainers in March 2020.[47] In July 2020, it was renamed to Torque, and adapted for maintenance of the F-22 stealth jet and CV-22 tiltrotor aircraft,[48] then the C-130J turboprop in January 2021.[49] Meanwhile, on the F-35 itself, between 2020 and 2022 ALIS was gradually replaced by ODIN, the Operational Data Integrated Network, "leveraging" the software practices of Kessel Run, but built by Lockheed Martin.[50] [51]
In 2021, Kessel Run began deploying the initial version of KRADOS, the Kessel Run All Domain Operations Suite meant to replace the Theater Battle Management Core Systems that created air tasking orders throughout AOCs all over the world.[52] The first AOC to use the suite operationally was again the 609th Air Operations Center at Al Udeid, in May 2021, after using the Beta version since December 2020.[53] KRADOS linked together nine applications through cloud-based data, including the latest version of Jigsaw, Kessel Run's first application for tanker planning, and Slapshot, for planning the rest of the air missions and building the Master Air Attack Plan.[54] In 2017, Lockheed Martin had received a $38 million contract to maintain the older TBMCS, but the 609th kept finding problems, so turned to Kessel Run, which delivered the beta version of KRADOS three weeks after receiving the request in November 2020.[55] By August 2022, the 603rd AOC in Ramstein, Germany, employed elements of KRADOS for visualization, though it was not considered mature enough to create air tasking and airspace control orders.[56] [57] In January 2023, the 609th replaced the TBMCS with KRADOS entirely.[58]
The Command and Control Incident Management Emergency Response Application (C2IMERA), is a real time Air Force base resource management tool. It used a different development model: the coding was done by software company Leidos, and the program management by Kessel Run.[59] In August 2019, Moody Air Force Base used the software to monitor and prepare for Hurricane Dorian, though it was not originally intended for this purpose.[60] C2IMERA was also used for the August 2021 evacuation of civilians from Afghanistan in Operation Allies Refuge.[61] [62] It was ordered deployed across all Air Combat Command installations in September 2021.[59] [63] In August 2023, Air Mobility Command joined Air Combat Command in designating C2IMERA as their standard installation command and control tool.[64]
Slapshot, the air mission flow organizer part of KRADOS, was also used for the Operation Allies Refuge Afghan evacuation along with C2IMERA.[65] [66] At that time, KRADOS had known issues with scaling; it couldn't handle many simultaneous operations, which was exactly what it was being asked to do. On August 24, 2021, at 2 am Boston time, the Slapshot server crashed. Over the next 12 hours, Kessel Run developers restarted servers, shifted United States Central Command resources to improve performance, fixed database errors, and added new features to improve load times, so the evacuation on the other side of the world could continue.[67]
Bowcaster, named after a Star Wars weapon, is a chaos engineering tool and playbook that intentionally creates failures in processes to strengthen them.[68] [69] Kessel Run developed it, and shares it with other government agencies, initially with the Navy Black Pearl software factory in 2021.[70] [71] In March 2022 Kessel Run and the General Services Administration's Technology Transformation Services used it to check that the Cloud.gov website could handle 100 million users per hour.[72] [73]