I see what you are saying. Over time, I really imagine this will give evolve into some kind of AI based system to make sure no false alarms & the complexity to cover all possible scenarios to make it effective deterent for adversary. Might gain some situational awareness as humans will keep feeding it data and adding complexity.
When reading into the gist, skynet popped into the head. Sorry for the deflection.
Ya'll Nibbiars there are already papers written on it's.
A Stable Nuclear Future? The Impact of Autonomous Systems and Artificial Intelligence
December 2019
Michael C. Horowitz, Paul Scharre, and Alexander Velez-Green1
Abstract: The potential for advances in information-age technologies to undermine nuclear
deterrence and influence the potential for nuclear escalation represents a critical question for
international politics. One challenge is that uncertainty about the trajectory of technologies such as autonomous systems and artificial intelligence (AI) makes assessments difficult. This paper evaluates the relative impact of autonomous systems and artificial intelligence in three areas: nuclear command and control, nuclear delivery platforms and vehicles, and conventional applications of autonomous systems with consequences for nuclear stability. We argue that countries may be more likely to use risky forms of autonomy when they fear that their second-strike capabilities will be undermined. Additionally, the potential deployment of uninhabited, autonomous nuclear delivery platforms and vehicles could raise the prospect for accidents and miscalculation. Conventional military applications of autonomous systems could simultaneously influence nuclear force postures and first-strike stability in previously unanticipated ways. In particular, the need to fight at machine speed and the cognitive risk introduced by automation bias could increase the risk of unintended escalation. Finally, used properly, there should be many applications of more autonomous systems in nuclear operations that can increase reliability, reduce the risk of accidents, and buy more time for decision-makers in a crisis.
Introduction
Nuclear weapons are arguably the single most significant weapon system invented in modern
history, meaning uncertainty about the viability of nuclear deterrence in the 21st century
constitutes one of the most important security risks facing the world. This uncertainty is both a product and source of increased tensions in nuclear dyads worldwide. The proliferation of conventional military technologies, such as hypersonic weapons, could further undermine deterrence by potentially undermining traditional modes of escalation management, and as a consequence, nuclear stability. The impact of autonomous systems and artificial intelligence (AI) for nuclear stability remains understudied, however. In early 2017, Klaus Schwab of the World Economic Forum argued that the world is on the cusp of a Fourth Industrial Revolution, wherein several technologies but most prominently AI –could reshape global affairs. Many defense experts around the world share Schwab’s recognition of the potentially transformative effects of AI. The most prominent statements about the impact of AI on warfare, however, tend to be extreme. Elon Musk, for instance, has vocally contended that AI run amok could risk World War III. This overheated rhetoric masks the way that advances in automation, autonomous systems, and AI may actually influence warfare, especially in the vital areas of nuclear deterrence and warfighting. The intersection of nuclear stability and artificial intelligence thus raises critical issues for the study of international politics. Relative peace between nuclear-armed states in the 20th century arguably relied in part on mutually assured destruction (MAD). The MAD prevails when each side recognizes that both it and its opponent have an assured nuclear second strike capability, or that either side can impose unacceptable damage on the other in retaliation against a nuclear attack.
The Threat of mutual destruction ultimately led both the United States and the Soviet Union to deprioritize the role of preemption in their nuclear war plans. Furthermore, as Albert Wohlstetter found, the threat of mutual destruction “offer[ed] every inducement to both powers to reduce the chance of accidental war.” While there are no known instances of accidental war, there are historical examples of unintended escalation, either in pre conflict crises or once a conflict is underway. Accidental escalation is when a state unintentionally commits an escalatory act (i.e. due to technical malfunction, human error, or incomplete control over military forces).The Inadvertent escalation can also occur, whereby a state unknowingly commits an escalatory act (i.e., an intentional act that unknowingly crossing an adversary’s red line). The Accidents have increased tensions between countries on numerous occasions, but have not led to escalation. The Nuclear-armed states have expended vast resources to minimize the risk of unintentional escalation, knowing that it could lead to catastrophe should it occur. The Automation may complicate the risks of escalation, deliberate or unintended, in a number of ways. Automation has improved safety and reliability in other settings, from nuclear power plants to commercial airliners. Used properly, many applications of automation in nuclear operations could increase reliability, reduce the risk of accidents, and buy more time for decision-makers in a crisis. Automation can help ensure that information is quickly processed, national leaders’ desires are swiftly and efficiently conveyed, and launch orders are faithfully executed. On the other hand, poor applications of automation could render nuclear early warning or command-and-control (C2) systems more opaque to users, leading to human-machine interaction failures. Human users could fall victim to automation bias, for example, surrendering their judgment to the system in a crisis. Automation is often brittle and lacks the flexibility humans have to react to events in their broader context.
The states most likely to be willing to tolerate these risks for the perceived capability gains would be those that have significant concerns about the viability of their second strike deterrents. Thus, the more a country fears that, in a world without using autonomous systems, its ability to retaliate to a nuclear strike would be at risk, the more attractive autonomous systems may appear. Uninhabited nuclear delivery platforms could undermine nuclear surety, as they could be hacked or slip out of control, potentially leading to accidental or inadvertent escalation. Automated systems could end up reducing decision-maker flexibility by narrowing options, hampering attempts to manage escalation. These dynamics suggest that autonomous systems could influence the potential for nuclear escalation in three ways. First, while many aspects of the nuclear enterprise are already automated in many countries, from early warning and command and control to missile targeting, as autonomous systems improve, states may elect to automate new portions of the early warning and C2 processes to improve both performance and security. From a security standpoint, for instance, increased automation in nuclear early warning may allow operators to identify threats more rapidly in a complex environment. Likewise, automation may help to ensure the dissemination of launch orders in a timely manner in a degraded communications environment. States may also automate – or threaten to automate – nuclear launch procedures in the belief that doing so would provide them with a coercive advantage over adversaries. Second, as military robotics advance, nuclear powers could deploy uninhabited nuclear delivery platforms for a variety of reasons. For instance, a state might deploy nuclear-armed long endurance uninhabited aerial vehicles (UAVs) in the belief that doing so would provide additional nuclear signaling or strike options. They might also look to uninhabited nuclear delivery platforms to bolster their secure second-strike capabilities. Nuclear delivery vehicles such as torpedoes capable of autonomously countering enemy defenses or selecting targets might be seen to do likewise. Alternatively, a government might choose to automate its nuclear forces so that a small number of trusted agents can maintain control. This might could be especially attractive for a nuclear-armed regime that fears a coup or other forms of interference by its nation’s military elite.
The Third, the increased automation of conventional military systems might influence nuclear
stability in direct and indirect ways.16 It may enable or more likely yet, be seen to enable improved counterforce operations by technologically-advanced states. The ineffectiveness of
counterforce operations and hence the survivability of second strike deterrents presently
hinges in large part on the difficulty of finding and tracking adversary nuclear launch platforms
(mobile missiles or submarines) long enough for ordnance to be delivered. Machine learning
algorithms and other applications of artificial intelligence could, in principle, improve states’
abilities to collect and sift through large amounts of data in order to locate and track such targets, though it is important to recognize limitations to any developments given the real-time
requirements for a disarming strike. Likewise, military autonomy could enable the deployment of conventional autonomous systems designed to shadow and/or attack nuclear-armed submarines. Furthermore, if automation gives (or is perceived to give) one side in a competitive dyad a significant conventional military advantage, the weaker side may feel compelled to rely more heavily on nuclear weapons for deterrence and warfighting. These issues surrounding the potential impacts of artificial intelligence are magnified by uncertainty about the trajectory of technological developments. This article first proceeds by clarifying what autonomous systems are and clarifying often-tricky definitional issues surrounding artificial intelligence. It then lays out some key theoretical expectations. Second, the article explores the impact of autonomous systems on early warning and nuclear command and control, as well as intelligence, surveillance, and reconnaissance (ISR) relevant for nuclear systems, in the context of recent research. Third, the article discusses the potential for uninhabited nuclear delivery platforms and vehicles featuring new kinds of automation.
Fourth, the article describes the way conventional autonomous systems could both directly and indirectly influence nuclear stability. Finally, the article concludes by assessing the net likely impact of autonomous systems on nuclear stability and describing potential pathways for future research. The analysis argues that the impact of autonomous systems could depend on the specific application – both where automation falls in the nuclear enterprise but also how it is implemented in terms of design, human-machine interfaces, training, and operator culture.
Autonomous Systems and Artificial Intelligence
The field of artificial intelligence, which dates back to the 1950s, has seen tremendous growth in recent years. Much of these recent gains have come from “deep learning,” a machine learning technique that uses deep (multi-layer) neural networks. Deep learning is relatively new and, while a powerful technique, has certain insecurities. Deep neural networks are vulnerable to a form of spoofing attack that uses adversarial data to fool the network into misidentifying false data with high confidence. This vulnerability is prevalent across neural networks in wide use today. While adversarial training can somewhat mitigate these risks, there is currently no known solution to this vulnerability. Additionally, machine learning is vulnerable to “data poisoning” techniques that manipulate the data used to train a machine learning system, thus causing it to learn the wrong thing. Finally, artificial intelligence systems today, including those that do not use deep learning, have a set of safety challenges broadly referred to as “the control problem.” Under certain conditions, for example, artificial intelligence tools can learn in unexpected and counterintuitive ways that may not be consistent with their users or designer’s intent. These machine learning tools are powerful and are being used in novel experimental applications. But, given these vulnerabilities, they are not sufficiently mature to operate independent of human control for high-risk tasks, such as nuclear operations. These vulnerabilities are fairly well-understood in AI circles and among technical experts within the U.S. defense community. As a result, while AI and machine learning tools are already being incorporated into a variety of commercial applications, it seems unlikely that risk-averse government bureaucracies will be at the forefront of adoption, particularly for high-risk applications such as nuclear operations. However, older “first wave” AI systems that employ rule-based decision-making logic have been used in automated and autonomous systems for decades, including in nuclear operations. These expert AI systems use handcrafted knowledge from humans to create a structured set of if-then rules to determine the appropriate action in a given setting. Automated systems of this type are widely used, including in high-consequence operations such as commercial airline autopilots and automation in nuclear power plant operations. Rule-based expert AI systems can often improve reliability and performance when used in predictable settings. However, because such systems can only follow the rules they’ve been given, they often perform poorly in novel situations or unpredictable environments. In this paper, unless otherwise specified, we generally use the terms automated or autonomous system to refer to “first wave” expert AI systems that perform various tasks on their own, sometimes under human supervision (supervised autonomy) and sometimes absent human supervision for a period of time (full autonomy).
To Trust or Not to Trust: Autonomous Systems
Automation has been used in high-risk applications for decades, in both civilian and military
capacities. Nuclear power plants, commercial airlines, and private space ventures, for instance, all use automation to perform complex operations. Automation also serves niche roles in nuclear operations, including in early warning, targeting, launch control, delivery platforms, and delivery vehicles. Each of these applications, however, relies on mature technology and often retains human control over decision-making prior to the launch of the delivery vehicle.Questions about adopting autonomous systems require potential adopters to grapple with how to balance the risk that humans will not trust machines to operate effectively against the risk that humans will trust machines too much. Trust gaps occur when people do not trust machines to do the work of people, even if the machine outperforms humans in benchmark tasks. This can lead to an unwillingness to deploy or properly use systems. It can also lead to a preference for using human-controlled systems. Dietvorst, Simmons, and Massey show that when humans have to choose between using human forecasters or algorithms to make assessments about the future, they trust humans even when they can see an algorithm outperform humans. Moreover, when algorithms make mistakes, humans are faster to lose confidence in their effectiveness than when they see humans make mistakes.
There is contested evidence that a trust gap exists when it comes to military remotely-piloted
aircraft (i.e., drones). Surveying Ground Tactical Air Controllers (GTACs) about their preference
for inhabited aircraft versus drones for close air support, Schneider and MacDonald find that
GTACs tended to prefer inhabited aircraft. They argue that GTACs believed that pilots in
inhabited aircraft had more “skin in the game” and were thus more likely to perform effectively
(even though there was no evidence that that was the case). Their results are controversial –
other military personnel argue that their trust gap findings do not reflect the reality on the
ground, where military personnel are learning to trust remotely-piloted systems Regardless of
whether a trust gap exists in the GTAC community, the theoretical point has relevance for thinking about potential end users of autonomous systems. For applications of artificial intelligence, the alternative to a trust gap is automation bias. While humans are slow to trust information from a “machine,” such as data from radar, research demonstrates that once they believe in the effectiveness of the system, they become more willing to surrender judgment, even when there is evidence that the machine may be incorrect in a given situation. For example, in flight simulation experiments, participants given very good, but not perfect, automated flight systems tend to make more mistakes. Participants became more likely
to miss problems unless explicitly prompted by the autonomous system; they also tend to trust the autonomous system over their own judgment even when their training suggested the plane might be at risk (e.g. errors of both omission and commission). The Automation bias, whereby humans effectively surrender judgment to machines, therefore represents one important risk from automation. For example, Army investigators found that automation bias was a factor in the 2003 Patriot fratricides, in which Army Patriot air and missile defense operators shot down two friendly aircraft during the opening stages of the Iraq War. In
both instances, humans were “in the loop” and retained final decision authority for firing, but
operators nevertheless trusted the (incorrect) signals they were receiving from their automated radar systems. Army investigators found that automation bias was pervasive throughout the Patriot community at the time, which had a culture of “trusting the system without question.” According to Army researchers, Patriot operators exhibited an “unwarranted and uncritical trust in automation. In essence, control responsibility [was] ceded to the machine.” In addition to the problems of trust gap and automation bias, human-machine interaction failures can manifest in other ways. The opacity of complex machines can be a hurdle to operators fully understanding them, and this can lead to accidents. As systems increase in complexity, human users may not fully comprehend how automated systems will behave in response to certain inputs coming either from the environment or from human operators themselves. This complexity and breakdown in human-machine integration appears to have been a factor in the 2016 fatal accident involving a Tesla car on autopilot and the 2009 crash of Air France Flight 447, which killed everyone onboard. In both cases, the human users failed to understand how their respective automated systems would respond to their environments, leading them to take actions that, had they known otherwise, they would likely not have taken. A breakdown in human-machine integration can have disastrous consequences even if human users retain manual control over the system, as they did in the case of the Air France 447 crash. Finally, automated systems can pose risks because of their complexity, tight-coupling, and ability to take actions at machine speed. Complex automated systems are generally powered by software, making them potentially vulnerable to bugs and hacking. As one example, a 2007 software malfunction caused eight F-22 fighter jets to lose navigation, fuel subsystems, and some communications when they crossed the international dateline.35 Rigorous software testing can reduce error rates, but error-free software is not realistic outside of extremely narrow applications.36 Software vulnerabilities can also leave the door open for hackers. Security researchers have demonstrated the ability to remotely hack automobiles, for example, disabling or taking control of critical driving functions such as steering and brakes. Autonomous systems are also vulnerable to so-called “normal accidents” arising from the interaction of components of complex systems. While these risks are not unique to automation – normal accidents occur inmanual systems automation can increase the “tight coupling” between components. Tight coupling is a condition in which one action in a system directly causes another action with little “slack” in the system – time, visibility, and opportunity for human intervention to manage unanticipated events. Automation can increase the coupling between components and, moreover, accelerate the pace of actions to machine speed. The “brittleness” inherent to emerging forms of automation, along with omnipresent risk of automation bias, human-machine interaction failures, and unanticipated machine behavior, all potentially limit the roles that automation can safely fill.
