The Chatbot Did It

It does it every day

“Norman”, the psychopath AI trained on Reddit posts in 2018 by MIT scientists (who are also credited for the image), via Business Insider.

A report from Haaretz (which I got at second hand on the radio, in an interview with Haaretz journalist Alison Kaplan Sommer) specifying the nature of the “mistake” that killed the seven World Central Kitchen workers as they drove away from a delivery on a route approved by IDF, summarized in the headline:

IDF Drone Bombed World Central Kitchen Aid Convoy Three Times, Targeting Armed Hamas Member Who Wasn't There

The problem, we’re told, wasn’t that they hit the wrong cars, “unintentionally” as Netanyahu said. They knew which cars they were hitting. The problem was that they believed one of the passengers was a Hamas combatant, and according to the risk-benefit calculus IDF applies for every bombing mission, the benefit of getting that one enemy outweighed the risks of attacking a party of humanitarian aid workers rescuing people from the famine induced by Israel’s obstruction.

---

Except that the target, the possible Hamas fighter they were looking for, wasn’t with the group, he was still back in the warehouse. They’d been misinformed.

But’s that not the only mistake. If he had been in one of the cars, would that have made the killing of the WCK staff worthwhile? I mean, really? What kind of monster would make such a monstrous calculation?

I think we know the answer to that, from a report of last November in the Israeli newsmagazine +972, which I wrote about here:

[A] reason for the large number of targets, and the extensive harm to civilian life in Gaza, is the widespread use of a system called “Habsora” (“The Gospel”), which is largely built on artificial intelligence and can “generate” targets almost automatically at a rate that far exceeds what was previously possible. This AI system, as described by a former intelligence officer, essentially facilitates a “mass assassination factory.”

Not exactly a chatbot, but a largely automated decision procedure, “trained” on and prompted with the criteria IDF uses in its relatively rigorous targeting rules for determining whether the risks of civilian harm are “proportionate” to the benefits of a particular strike, making life-and-death choices at a phenomenally productive rate. The kind of thing you can easily imagine Robert S. McNamara or Donald Rumsfeld deploying with delight, a very businessman way of thinking, or avoiding thinking, with a technology that didn’t exist in their times.

And prone, as AI generally is, to bizarre mistakes, because it doesn’t in fact know how to think, only how to simulate (with amazing skill, it can’t be denied) a certain idea of what thinking would look like. This is a big factor in why the Gaza war has been so extraordinarily violent and destructive, and this must be why it’s already killed more than 200 humanitarian aid workers, even when they’d shared their coordinates with Israeli forces, but some 175 of them to date have been among the 30,000 Palestinian employees of the UNWRA, of whom Israel believes 12, that’s 0.04%, may have been involved in the October 7 attacks, so they don’t get as much consideration.

Not that I want to minimize the horror of this killing; but I want to look at it in a particular way, as I was saying yesterday, in reference to Netanyahu’s non-denial denial (“Unfortunately, in the past day, there was a tragic event in which our forces unintentionally harmed non-combatants in the Gaza strip”) that anyone did anything wrong:

See the work he puts into anonymizing the victims when we know who they were. There are dozens of “tragic events” every day in Gaza, where “non-combatants” get “harmed”. Honor the heroes of WCK by remembering the people they were working to save from starvation.

Generally, I think they use the term “unintentionally” in a highly specialized way: they’re aware of the statistical likelihood that the operation is going to kill a particular quantity of noncombatants that they have decided is an acceptable price to pay (not for them to pay, for the community they’re bombing), “proportional”, for the presumed combatant they do intend to kill (who may or may not actually be a combatant and may not even be on the site); they don’t intend to kill anybody else in particular. That is, they have an idea of approximately how many people they are going to kill, but they don’t know who, so you can’t say any of the deaths that actually occur are “intentional”.

But it seems to me it would be more accurate to say they just don’t give a fuck who dies. The lack of intentionality in that sense isn’t an index of how much they care (“the most moral army in the world”) but of how much they don’t. And then sometimes they do know, as certainly should have been the case this time, though the prime minister tries to make it sound as if they didn’t.