top of page

The Editor’s Guide to Fact-Checking with AI

Updated: Dec 5, 2023

I used to say, “ChatGPT is not a fact-checker.” Period. Full stop. But not anymore. ChatGPT now has the capability to cite its sources and browse the live web so it can use current information that you can verify. Plus, it can gather the information you need—and only the information you need—which will save you heaps of time tracking down sources one by one.


This article demonstrates how to use ChatGPT for research and fact-checking and explores Perplexity AI, another highly useful AI tool for research.




Testing ChatGPT for Fact-Checking

To test ChatGPT’s research capabilities, I prompted it to give information about Ada Lovelace, an early pioneer of computer science. Remembering ChatGPT works best when given specific instructions, I wrote this prompt:


Please provide a two-paragraph biography of Ada Lovelace with citations in the text. Ensure sources and information are reputable, current, and accurate. Include Encyclopedia Britannica as a source. Do not use Wikipedia.


The reason I instructed it to use Encyclopedia Britannica as a source is because I’ve been waiting for ChatGPT to specifically be able to access Encyclopedia Britannica as a benchmark of its ability to seek out factual, quality, trustworthy information. Similarly, I asked it to avoid using Wikipedia to test this same ability but through eschewing less-reliable sources.


The Result

ChatGPT supplied the two-paragraph bio, as requested, with sources cited in the text, shown through superscript numbers. The sources it cited included Encyclopedia Britannica, Biography, the San Diego Supercomputer Center at the University of California, San Diego, and ThoughtCo. As requested, it did not use Wikipedia. I checked all of the links, and they were all valid and current.


Here’s a video showing the process and result.



Here’s a screenshot of the ChatGPT-created bio.


Ada Lovelace, born Augusta Ada Byron on December 10, 1815, in Middlesex (now part of London), England, was the only legitimate child of the renowned poet Lord Byron and Annabella Milbanke Byron. From a young age, Ada showed a profound aptitude for mathematics, a path encouraged by her mother to steer her away from her father's "dangerous" poetic disposition. Her mother, Annabella, emphasized the study of music, French, and mathematics in Ada's education. At the age of 17, Ada's life took a decisive turn when she met the inventor Charles Babbage in 1833, with whom she began a substantial correspondence on mathematics, logic, and various subjects​1​​2​​3​.  Lovelace is celebrated as the world's first computer programmer due to her work on Babbage's early mechanical general-purpose computer, the Analytical Engine. In 1843, she developed a program for the Analytical Engine, demonstrating that it could be used not only for calculations but also to create any content, including music or art, given the right inputs and algorithm. This insight laid the foundation for the field of computer programming. Furthermore, Ada Lovelace's contributions extend to the concept of an operating system or software, embodying the idea that a computing machine could execute a program from a set of instructions. Her legacy continues to inspire, with Ada Lovelace Day, celebrated on the second Tuesday of October, recognizing the significant contributions of women in science and technology​4​​5​​6​.

Did ChatGPT work perfectly every time? No. When I prompted it to provide full citations and links to the sources it used, it included dead links and struggled to follow the requested style. Before I specifically selected the “Browse with Bing” option in GPT-4 (the paid version), it sometimes fell into the hallucination trap of filling out a format that looks correct but does not include correct information (incorrect text prediction).

For now, making sure ChatGPT is accessing the live internet (via Browse with Bing) and using specific prompts to guide its research are essential to having the AI return the factual information you can use.



How to Use ChatGPT for Researching and Fact-Checking

To use ChatGPT to research, you can prompt it to supply information about the topic. Remember to be specific in your prompt. This includes telling ChatGPT how much information you want. If you want a quick overview, as with the first Ada Lovelace example, you could instruct ChatGPT to give you a two- or three-paragraph overview. If you want to dive deep into a topic and are looking for a lot of information, you could ask it to be comprehensive and tell it the subject areas it should include (if you know).


For In-Depth Researching

To do more in-depth research on Ada Lovelace, I used this prompt:


Please give a comprehensive biography of Ada Lovelace. Include information about her childhood and family, her personal life, her scientific research, her legacy, and other relevant information. Cite your sources. Include credible information. Do not use Wikipedia.


Here’s the response from ChatGPT:


Ada Lovelace: Comprehensive Biography Childhood and Family Born: Augusta Ada Byron on December 10, 1815, in London, England​1​. Parents: The only legitimate child of the poet Lord George Gordon Byron and Lady Anne Isabella Milbanke Byron​2​. Education: Her mother, Lady Byron, herself mathematically trained, insisted Ada study mathematics—a rare education for women at the time​3​. Personal Life Marriage: In 1835, Ada married William King, who later became the Earl of Lovelace, thereby Ada became the Countess of Lovelace​4​. Children: Ada had three children​4​. Nickname: Given by her father as his "Princess of Parallelograms" due to her mother's mathematical background​3​. Scientific Research Meeting Babbage: Ada met Charles Babbage in 1833 and was captivated by his work on the Analytical Engine​5​. Programming: She is credited with creating the first algorithm intended for processing on a machine, thus considered the first computer programmer​6​. Visionary Work: Ada's notes on Babbage's Analytical Engine included ideas that machines could manipulate symbols and create music, showing a vision of computing beyond mere calculation​5​. Legacy Recognition: Ada Lovelace Day, celebrated on the second Tuesday in October, honors women in science and technology​6​. Prophet of the Computer Age: She was the first to articulate the potential for computers beyond mathematics, seeing the possibility of a machine that could manipulate symbols according to rules, marking a fundamental transition from calculation to computation​5​.

It cited six sources, all credible, including Encyclopedia Britannica and the Computer History Museum. It did not use Wikipedia. And, it covered the requested subtopics. From here, I could ask a follow-up prompt to get more information about one of the details, or I could check out one of the sources to learn more.


For Checking Single Facts

If you are fact-checking an author or client’s writing about, say, Ada Lovelace and need to check one detail, you could try a prompt like this:


In a sentence, answer the following question: Was Ada Lovelace involved in the creation of the Analytical Engine? Cite your sources. Use multiple, credible sources.


Here’s how ChatGPT responded:


Yes, Ada Lovelace was involved in the creation of the Analytical Engine, for which she is celebrated as the first computer programmer, having developed a program for Charles Babbage's early computing prototype in 1843

It cited again Encyclopedia Britannica. This time it also cited IEEE Annals of the History of Computing. But, likely because I did not specifically instruct it to avoid it, it did include Wikipedia. Even so, this short response has given me two credible sources I can use to verify the information it has provided.


For a real-world example of using ChatGPT to track down a tricky fact, check out editor Kristen Tate's blog post, "How to Use AI Tools for Fact-Checking."


For Checking Multiple Facts at Once

If you’re fact-checking a passage or short article about a specific topic, where you know you’ll need to verify a number of related facts, you could incorporate ChatGPT into your workflow by prompting it to create a list of top facts about it.


Here’s the prompt I used in the Ada Lovelace experiment:


Please create a bullet-point list of top facts regarding Ada Lovelace's connection to the Analytical Engine. Cite your sources. Ensure sources are reputable, current, and accurate.


Here’s its response:


Here's a bullet-point list of key facts about Ada Lovelace's connection to the Analytical Engine, with citations from reputable sources:  Ada Lovelace is renowned for her detailed account of Charles Babbage's Analytical Engine, which is acknowledged as an early model for a steam-powered programmable computer, though it was never actually constructed​1​.  Lovelace was an English mathematician and is often celebrated as the first computer programmer because she developed a program for the Analytical Engine in 1843​2​.  Between 1842 and 1843, she expanded a translation with her own comprehensive "Notes," which were three times longer than the original document by Luigi Menabrea. These notes are considered the first detailed description of a digital computer's design and operation​3​.  Her interest in the Analytical Engine began after meeting Charles Babbage in June 1833, facilitated by their mutual acquaintance and her tutor, Mary Somerville​4​.  The work Lovelace did on the Analytical Engine included what we would now call a computer program, intended to calculate Bernoulli numbers, which was part of her translation of Menabrea's paper on the engine​5​.  These points illustrate Lovelace's pivotal role in conceptualizing the operations and potential of early computing devices.

This time it cited the Mathematical Association of America, Oxford University, Patterns data science journal, and, yes, Wikipedia.


This is like a hyperspeed Google search, allowing you to check multiple details very quickly, cross-referencing facts between sources.


Easy-to-Use Option: Perplexity AI

While ChatGPT’s wide-ranging capabilities make it an editorial Swiss Army knife, fact-checkers and researchers may benefit from exploring Perplexity AI. This tool uses both the Claude and GPT AI models and the internet to create curated responses.


It works faster than ChatGPT. But, like ChatGPT, the paid version provides quite a bit better response (six sources with the free version vs. sixteen with the paid “copilot” enabled).


To test Perplexity AI, I used the same Ada Lovelace prompt as with the first experiment. Note that even though instructed not to use Wikipedia, Perplexity AI still did.


Here’s a video of Perplexity AI at work. Prepare to be amazed.


My New Motto

Now my motto about using ChatGPT to fact-check is the old adage: trust, but verify. While ChatGPT and Perplexity AI can reply with content that leans much farther toward Trust on the Trust-O-Meter, like any good fact-checker, it’s still important to double-check.


Key Insights & Takeaways

ChatGPT's Enhanced Capabilities:

  • When provided with specific and detailed prompts, ChatGPT can effectively pull information from reputable sources, including Encyclopedia Britannica.

  • ChatGPT did not use Wikipedia when instructed not to, indicating it can follow directions to exclude certain sources.

  • The tool provided valid and current links, demonstrating its ability to access and cite credible information from the live web.

Limitations and Issues

  • ChatGPT sometimes included dead links and had difficulty adhering to requested citation styles.

  • The AI occasionally produced “hallucinations,” or incorrect predictions, when not using the live internet browsing feature.

Perplexity AI's Performance:

  • It operates faster than ChatGPT and can provide more sources when using the paid version.

  • Perplexity AI still cited Wikipedia even when instructed not to, revealing a potential issue with source filtering.

Reliability of AI for Fact-Checking:

  • Both AI tools showed a high degree of compliance with user instructions, but they are not foolproof.

  • It’s vital to verify the information provided by AI and maintain a "trust but verify" approach.

  • The tests underscore the importance of using specific prompts and live internet features to guide AI tools towards more accurate and useful results for research and fact-checking.



Erin Servais is an editor, educator, and community builder. Committed to helping editors improve their work and lives, she serves on the board of directors for ACES: The Society for Editing, is the president of the Editors Tea Club, and offers editor coaching services. Erin has presented about editing, entrepreneurship, and Artificial Intelligence for the Professional Editors Network, Editors Canada, the Northwest Editors Guild, and ACES.


Her current focus is training editors to uplevel their skills using Artificial Intelligence. She teaches the only comprehensive ChatGPT for Editors course on the market. Erin always tells ChatGPT please and thank you, just in case.


Email Erin: Erin@aiforeditors.com



262 views

Comentários


bottom of page