Operation Early Bird: How Logically Pioneered Live Fact Checking

On September 29, at nine p.m. Eastern Daylight Time, Logically’s fact checkers sat at their respective desks, their drinks of choice at their side and ninety minutes of quiet focus stretching before them. The evening was an auspicious one: the team was about to attempt something that had never been done before, using technology built over months specifically for the task ahead. 

For Priyam Nayak, nine p.m. in Eastern Daylight Time was six a.m. in India: a perfect hour for coffee. Ilma Hasan was also experiencing six  a.m., but she opted for real chai and not a certain highstreet chain’s chai, which she tells me is “disgraceful.” For Devika Khandelwal, it was two a.m. in London. Nevertheless, she threw caution to the wind and opted for her first coffee in 10 years. For Kristina Gildejeva, our researcher based in Berlin, three a.m. was the right time for maté, a caffeine-rich drink prized by Berlin hipsters.

OSINT expert Joe Ondrak opted for something stronger. “I had my first energy drink since I stopped being a BMX and mountain bike hoodlum,” senior researcher and reformed BMX and mountain bike hoodlum Ondrak tells me, before adding: “And I also snacked on chocolate malted milk biscuits. The absolute quiet king of biscuits.”

Joe Ondrak's desk

Occasions that call for late-night caffeine tend to be memorable. On September 29, the memorable occasion in question was the televised debate between U.S. President Donald Trump and former Vice President Joe Biden. 

Some weeks prior, Logically had made the decision to get involved in the coverage. But while many outlets were verifying claims in exclusively-human-powered live blogs, we had plans for something more automated. 

“We wanted to detect the claims in real-time and verify them in real-time,” Dr. Anil Bandhakavi, Logically’s head of data science, says. “But human fact checking is intellectually demanding, laborious, and time-consuming. Without the right automation tools, it is impossible to fact check a political event such as the presidential debate in real time.” This is a problem Logically has been invested in solving for some time, and the technology was ready in time to be debuted for the first presidential debate.

The night of the first presidential debate was not only the night our sparkly new AI made its debut, it was a world first: the first live fact checking project ever undertaken.  Thanks to advances made by our data scientists in natural language processing, information retrieval, and speech-to-text and speech recognition, when Trump, Biden, or the moderator said something, their speech was automatically transcribed into text. From this transcription, the AI extracted claims and compared them with our existing fact check library. All our fact checkers had to do was sit there with their coffee, chai, maté, or non-branded energy drink, and watch the presidential debate unfold. 

Kristina's workspace and bottle of maté

At least, that was how it was supposed to work, if the President of the United States hadn’t turned out to be a bit of an annoying guy. "Trump was constantly interrupting Biden,"  Kristina says. "It was not easy."

"The speech to text pipeline definitely struggled," Devika agrees.

Anil puts it in more technical terms: "The AI worked well in transcribing the speech to text but struggled to get the speaker attribution correct. Also, the additional speaker detection models that were custom built to recognize the speakers in debates did not give accurate results."

In other words, some of the claims that the AI attributed to Biden actually should have been attributed to Trump, and some of the claims that the AI attributed to Trump actually should have been attributed to Biden. Unsurprisingly, this was a bit of a problem.

"But we were prepared for this," Devika explains. "Months of preparation, research, and reading gave us an idea of both Trump's and Biden's stances and policies. This made it easy for us to change attributions if they were incorrect."

In the end, for the first debate, some of the claims had to be manually extracted. Joe, who had planned to spend the evening on the lookout for trolling campaigns on social media, had to untangle some of the transcriptions. The beauty of AI however, is that these initial bumps in the road smooth out into a learning curve. The data from the first debate was fed back into the model to recognise speakers more accurately - in fact, if the two candidates hadn’t spoken over each other so much on that first night, our model wouldn’t be as accurate as it is now.

As it happened, our AI wasn't alone in getting flustered on the evening of 29 September. After what must have been the hundredth interruption, Biden lost his temper, and the moderator Chris Wallace wound up sounding like a broken record. "Mr. President,” Wallace said, “it's an, it's an open discussion, please. You...Let the vice president answer…Mr. President, let him answer...Mr. President, please stop.”

Afterwards, CNN's Jake Tapper said what we were all thinking: "That was a hot mess inside a dumpster fire, inside a train wreck."

Devika's desk: two computers, one coffee, one Snickers.

Traditionally, fact checking relies on human intelligence and human intelligence alone. When I was a fact checker—at a business unrelated to Logically—I was a fresh-faced intern. Every day, some editor or other would give me a printout of an article, and it was my job to underline each assertion in the article, ticking the ones that were true and highlighting the ones that were false. After this, I would walk back over to the relevant editor, and try to explain to them that high-fructose-corn-syrup was not actually the same thing as fructose, just sort of hoping that they would care enough to tweak the copy.

Unlike my former place of work, and much to my personal relief, Logically doesn't rely on the persuasive power of interns. Instead, it relies on the right automation tools coupled with the brains of real, human people. With Logically’s use of extended intelligence—as we have taken to calling it—it is possible to fact check a political event such as the presidential debate in real time.

How the in-app live fact checking looked

"That last presidential debate was definitely the best of the three,” Kristina says, “because Trump and Biden had their microphones switched off." The AI had improved too: the new data from the previous debates had allowed for improvement in the model, and there were more complete fact checks ready to be automatched by the app so that when users submitted claims, many answers were available in seconds.

Ilma agrees: “In the end, it worked great. The claim extraction worked. It all worked.”

It did work. When President Donald Trump said that he had paid “millions of dollars” in federal income tax in 2017, thanks to our new AI, we could quickly tell the world that this claim was simply not true—not when the words left his lips  and not when people were sitting on the sofas of television sets debating it in what, I might suggest, was a rather pointless fashion. 

Similarly, when former Vice President Joe Biden said his economic plan would create 7 million more jobs than Trump’s had, thanks to our new AI, we could say that predictions are never foolproof, but yes, Moody’s Analytics did say that. 

In the end, during the presidential and vice presidential debates of Autumn 2020, claims that would normally take hours to judge took just 15 minutes, 62 claims were dealt with in real time, and our turnaround time for fact checks decreased by around 70 percent. In other words. It was a success, and because of this success, Logically is in the process of formalizing the live fact checking technology so that we can fact check any future elections that come our way.

Natural light makes Ishaana's workspace ideal

There are moments in life that don’t call for late night caffeine, but are nevertheless memorable. After the debates were over, Logically’s team of fact checkers sat back in their respective chairs. Relieved that everything had gone to plan, they each opted for their celebratory beverage of choice. 

Devika went for a glass of Rioja — cheap, sharp, and to-the-point. Ilma opted for whiskey on the rocks, and Kristina went for Riesling. Priyam, who is almost teetotal, had her annual sip of rosé before tucking into generous helpings of roshogolla—a candy that Wikipedia tells me can be served hot, cold, or at room temperature.

As for Joe, he reassures me that he hasn’t imbibed a single energy drink since the debates, nor is he planning to do so in the near future. When asked how he feels about them now, he is less than enthusiastic. “I'm sure they probably do ungodly things to your insides,” he says. “I prefer Chocomel. If you haven’t had it, it’s  a game changer in the world of chocolate milkshake.” 

I thank Joe for his advice, and opt to drink a glass of the chocolate-flavored milk that night. I find it thoroughly enjoyable, though hesitate to label it game changing. The glass of chocolate milk is not as revolutionary as I had hoped it to be, not nearly as revolutionary as Logically’s in-app live fact checking—the game changer we all need.