Yesterday Iranian media announced that Iranian defense forces had captured an unmanned underwater vehicle (UUV) that looks a lot like an Anduril Dive-LD. Anduril calls the Dive-LD an AUV, an autonomous underwater vehicle. Department of Defense Directive 3000.09, Autonomy in Weapon Systems, covers the employment, at a conceptual level, of autonomous systems. This was an update from a 2012 directive and focused overwhelmingly on autonomy and the application of force. The Dive-LD capture is actually useful for helping us look at how autonomy and contested signals environments generate friction that goes well beyond the imagining of the drafters of this directive and how we have to reconsider both autonomy and control in future war.
Anduril claims the Dive-LD can operate for up to ten days at depths of up to 6,000 meters and can be used for intelligence preparation of the operational environment (IPOE), intelligence, surveillance, and reconnaissance (ISR), mine warfare, and precision placement. In other words, it is an attritable system that can be used to do the types of jobs that the Pentagon does not want sailors and Marines risk their lives to accomplish. In that sense, the loss of the Dive-LD should be a very small deal. However, there is a larger issue of how the U.S. defines autonomous weapons and autonomous systems more broadly because it highlights some of the hardest problems in autonomy thar are not included in the DOD’s 3000.09.
Reuters reports that an unnamed Pentagon spokesperson confirmed the loss of the UUV and that it did not have any sensitive data or equipment onboard. Anduril also confirmed the loss of the Dive-LD but said that it was designed to work where “loss of the vehicle is an expected possibility.” That is correct and, as I pointed out above, it is better to lose a vehicle than a servicemember. This incident brings up some ideas about how to design a system to resist capture because next time, and we have to imagine there will be a next time based on the response so far, the system that malfunctions may have more sensitive information or collection devices onboard. At a minimum, the capture of a complete system allows Iran, and, if it chooses, its friends in Beijing and Moscow, the chance to see inside one of the U.S.’s operational systems and start probing for weaknesses.
I think this is important because the news that the Dive-LD malfunctioned, and that this is why Iran was able to capture it, brings up potential issues with the employment of autonomous systems. Department of Defense Directive 3000.09, Autonomy in Weapon Systems, says, “USD(P), USD(A&S), and VCJCS will verify that … system safety, anti-tamper mechanisms, cyber survivability, operational resilience, and cybersecurity capabilities have been implemented.” The directive helpfully defines a failure as “an actual or perceived degradation or loss of intended functionality or inability of the system to perform as intended or designed.” The Dive-LD had a failure that exists outside of the 3000.09 framework and should serve as a point to reconsider how we think about autonomy, weapon or no.
How should we think about autonomy
Despite all of the anthropomorphizing of autonomous weapons, one thing remains clear: they are not people. They do not have hopes and dreams. They really do not have desires, even if they use the language of desire when they communicate. They “seek” only to meet the parameters of the missions we have given them because we designed them that way. I think of autonomous weapons, because they are interacting in the physical world, as distinct from the controls I would extend to agentic AI. Still, the fact that our weapons have no desires, no thoughts of their own, and no intent means that humans must be responsible for the actions of autonomous weapons. Military commanders own the risks and the outcomes of final decisions because autonomous weapons cannot be held accountable, but military commanders have to determine where, if anywhere, they are willing to delegate decisions to machines before communications are lost. At some point, a machine will make final tactical decisions in real time, the question is will we be prepared for it?
In an ideal world, the world of Directive 3000.09, autonomous weapons have bounded autonomy. This is the basic framework of hard boundaries around autonomous weapons’ actions. The directive lays out a human-on-the-loop form of autonomy that bounded autonomy describes. The human on the loop would be accountable. These are hard limits on what an autonomous system can do and when it must choose to take no action. For me, the boundaries to autonomy would exist around risk and how long-lasting or irreversible the actions the autonomous weapon would take are. This would mean that low-risk missions, i.e. seafloor mapping, would be fully autonomous, and the autonomous system would be able to make nearly all decisions outside of changing its mission parameters. On the opposite end of this would be high-risk missions, such as targeting reconnaissance with highly sensitive signals intelligence devices onboard near an enemy location or attacks against military targets in areas trafficked by civilians and commercial vessels. In this case, nearly all decisions would need a human over the top, on the loop, ensuring that the risk the weapon is proposing is in line with the operational intent of the commander.
Does this framework actually survive a highly contested signals environment with enemy forces using jamming, spoofing, electronic warfare detection, and deception to stop all forms of communication? It does only if the military is comfortable with a machine having bounded autonomy and retreating when its human on the loop is unable to approve a high-risk strike or operation that it identifies as necessary to meet its given objectives.
On edge autonomy means delegation
Prior to the Dive-LD capture, an area to which I had not applied much thought was the parameters around when and how an autonomous system initiates a destruct sequence — not self-destruct, as it does not have a self — zeroizes, or otherwise renders the vehicle inert before an enemy can capture it. This is a useful area to consider, as it presents many of the same dilemmas as targeting enemy systems or personnel, with the benefit of limiting those dilemmas to our side’s equipment. In practice, an autonomous system deciding to destruct would require that system to recognize a mechanical failure or recognize that someone not tied to its command was about to capture it and take actions to protect not itself but the mission it was sent to carry out. Would this require human-on-the-loop intervention for it to abandon its mission?
The inner workings of that machine logic as an unknown vessel approaches might look like:
Unknown vessel approaching; unable to identify
Unable to start propulsion
Probability of capture or interception is imminent
What if I destroy myself unnecessarily?
What is compromised if I am captured?
Am I authorized to make this decision without a human?
You could program an autonomous vessel so that if it loses propulsion, is out of communication, and is approached by an unknown vessel it would zeroize or scuttle itself. This is also a reaction an enemy could exploit to destroy swaths of autonomous vessels without having to fire a shot. In that case, a safety would become a vulnerability. How would you distinguish between an adversary exploit and this failsafe signal? Directive 3000.09 hinted at this when it talked about the need for adaptability.
The full definition of “failure” in Directive 3000.09 is:
An actual or perceived degradation or loss of intended functionality or inability of the system to perform as intended or designed. Failure can result from a number of causes, including, but not limited to, human error, faulty human-machine interaction, malfunctions, communications degradation, software coding errors, enemy cyber-attacks or infiltration into the industrial supply chain, jamming, spoofing, decoys, other enemy countermeasures or actions, or unanticipated situations on the battlefield. For the purposes of this issuance, minimizing the probability and consequences of failure means reducing the probability and consequences of unintended engagements to acceptable levels while meeting mission objectives and does not mean achieving the lowest possible level of risk by never engaging targets.
Failure can result from a number of causes, including warfare and all the friction inherent in it, but we are only trying to minimize risk to an acceptable level while meeting mission objectives. In the highly contested environment of degraded or denied communications, jamming, spoofing, and human error, this means empowering autonomous weapons to make the final decision, or it means saying that there is no level to which we can minimize unintended consequences that the U.S. can accept.
Autonomous final decisions can still be bound with controls. Again, bounded autonomy, is still the constraint. We can establish rules on geographic areas of operation. We can employ risk thresholds that I laid out above. We can deny whole categories of targets and actions, or we can set time mechanisms to recall autonomous systems. This is still bounded autonomy, without contemporaneous supervision.
The gap between doctrine, directives, and performance in combat is a yawning chasm. Combat operations live in the fog and friction of human error, human decisions, and the acceptance of unknown or ill-defined risks. The inclusion of autonomous weapons is not making this easier or cleaner. We have to consider what we are willing to delegate, how we think about contested communications, and what limits we want in place. We need to do this now, when we have time not only to consider those actions but also to develop frameworks and controls around them, rather than trying to blindly adapt to a future conflict when we might not have the luxury of distance, as we do now.
I think it might be a good idea to build in kill switches that activate automatically if a system hasn’t called home in a certain period of time. Could be days, weeks, or months depending on the mission but there should always be a limit. Call it the Bladerunner rule.