Skip to main content
NIH Intramural Research Program, Our Research Changes Lives

Navigation controls

  • Search
  • Menu

Social follow links

  • Podcast
  • Instagram
  • Twitter
  • YouTube
  • LinkedIn

Main navigation

  • About Us
    • What Is the IRP?
    • History
    • Honors
      • Nobel Prize
      • Lasker Award
      • Breakthrough Prize
      • Presidential Early Career Award for Scientists and Engineers (PECASE)
      • Presidential Medal of Freedom
      • National Medal of Science
      • Searle Scholars
      • The National Academy of Sciences
      • The National Academy of Medicine
      • The National Academy of Engineering
      • The American Academy of Arts and Sciences
      • National Medal of Technology & Innovation
      • Samuel J. Heyman Service to America Medals
      • Crafoord Prize
      • Fellows of the Royal Society
      • Canada Gairdner Awards
    • Organization & Leadership
    • Our Programs
      • NCI
      • NEI
      • NHGRI
      • NHLBI
      • NIA
      • NIAAA
      • NIAID
      • NIAMS
      • NIBIB
      • NICHD
      • NIDA
      • NIDCD
      • NIDCR
      • NIDDK
      • NIEHS
      • NIMH
      • NIMHD
      • NINDS
      • NINR
      • NLM
      • CC
      • NCATS
      • NCCIH
    • Research Campus Locations
    • Contact Information
  • Our Research
    • Scientific Focus Areas
      • Biomedical Engineering & Biophysics
      • Cancer Biology
      • Cell Biology
      • Chemical Biology
      • Chromosome Biology
      • Clinical Research
      • Computational Biology
      • Developmental Biology
      • Epidemiology
      • Genetics & Genomics
      • Health Disparities
      • Immunology
      • Microbiology & Infectious Diseases
      • Molecular Biology & Biochemistry
      • Molecular Pharmacology
      • Neuroscience
      • RNA Biology
      • Social & Behavioral Sciences
      • Stem Cell Biology
      • Structural Biology
      • Systems Biology
      • Virology
    • Principal Investigators
      • View by Investigator Name
      • View by Scientific Focus Area
    • Accomplishments
      • View All Accomplishments by Date
      • View All Health Topics
      • The Body
      • Health & Wellness
      • Conditions & Diseases
      • Procedures
    • Accelerating Science
      • Investing in Cutting-Edge Animal Models
      • Creating Cell-Based Therapies
      • Advancing Computational and Structural Biology
      • Combating Drug Resistance
      • Developing Novel Imaging Techniques
      • Charting the Pathways of Inflammation
      • Zooming in on the Microbiome
      • Uncovering New Opportunities for Natural Products
      • Stimulating Neuroscience Research
      • Pursuing Precision Medicine
      • Unlocking the Potential of RNA Biology and Therapeutics
      • Producing Novel Vaccines
    • Research in Action
      • View All Stories
      • Too Much of a Good Thing
      • Turning Face Perception on Its Head
      • Safeguarding a Second Chance at Life
      • A Biological Betrayal
    • Trans-IRP Research Resources
      • Supercomputing
    • IRP Review Process
    • Commercializing Inventions
  • NIH Clinical Center
    • Clinical Center Facilities
    • Advancing Translational Science
    • Clinical Trials
      • Get Involved with Clinical Research
      • Physician Resources
  • News & Events
    • In the News
    • I am Intramural Blog
    • Speaking of Science Podcast
    • SciBites Video Shorts
    • The NIH Catalyst Newsletter
    • Events
  • Careers
    • Faculty-Level Scientific Careers
    • Trans-NIH Scientific Recruitments
      • Stadtman Tenure-Track Investigators
        • Science, the Stadtman Way
        • Earl Stadtman Investigator Frequently Asked Questions
      • Lasker Clinical Research Scholars
      • Independent Research Scholar
    • Scientific & Clinical Careers
    • Administrative Careers
  • Research Training
    • Program Information
    • Training Opportunities
    • NIH Work/Life Resources
The NIH Catalyst: A Publication About NIH Intramural Research

National Institutes of Health • Office of the Director | Volume 20 Issue 3 • May–June 2012

Crowdsourcing

And Virtual Colonoscopies

BY LAURA STEPHENSON CARTER

When it comes to interpreting the results of virtual colonoscopies, radiologists “have a hard time taking the advice of computer aids,” said senior investigator Ronald Summers, chief of the NIH Clinical Center’s Clinical Image Processing Service. Computer-aided-detection (CAD) technology, which he helped develop for colorectal cancer screening, is supposed to make the radiologist’s job easier. CAD is more effective than humans at finding tiny dings on the scan that represent polyps, but it identifies mock polyps, too—residual fecal matter clinging to the colon wall, thick folds in the colon, or even artifacts such as surgical clips or hip prostheses. Even though radiologists should be able to tell the difference between true and false readings, they often ignore CAD’s true findings.

Zhuoshi Wei, CC

Reconstructed CT images create a 3D model of the colon.

Summers wanted to reduce what he suspected were perceptual errors involving the CAD system, but he needed to be able to observe its users in action. Radiologists, however, are too busy to participate in such observer-performance experiments. Then he stumbled on a way to get participants: crowdsourcing, also known as distributed human intelligence. It’s a method for recruiting large numbers of anonymous laypeople, or knowledge workers (KWs), to perform simple tasks distributed over the Internet. Summers reported his findings in a recent issue of Radiology (Radiology 262:824–833, 2012).

“This was one of the most fascinating submissions and papers that we’ve [ever] had,” Radiology editor Herbert Kressel said in a podcast interview with Summers.

Colon cancer, the second leading cause of cancer death in the United States, can be prevented by removing colorectal polyps before they become malignant. Polyps can be detected by either a conventional colonoscopy, in which a colonoscope—a long, flexible, lighted tube with a tiny video camera attached—is inserted into the rectum and guided into the large intestine; or a virtual colonoscopy [computed tomography (CT) colonography], in which a CT scanner creates two- and three-dimensional images of the colon. Both methods require the patient to drink a bowel-clearing preparation the day before to be sure the colon is empty.

After obtaining approval from the Office of Human Subjects Research, Summers developed a project using crowdsourcing to assess how well KWs could correctly identify polyps on CT colonographies. He hired an Internet-based crowdsourcing service, which recruited 228 KWs. The KWs were given just one to two minutes of training to recognize features of polyp candidates in CT colonographic images. The CT data—from 24 randomly selected anonymized patients who had at least one polyp of six millimeters or larger—were analyzed by CAD software, and true polyps were confirmed by conventional colonoscopies.

Summers was surprised that when the results were combined, the minimally trained KWs did about as well as a highly trained CAD system—about 85 percent. (The KWs were only shown 11 candidates for training, the CAD system, more than 2,000.) He even did two trials, four weeks apart, and got the same results.

Zhuoshi Wei, CC

The arrows point to a polyp detected in a CT colonography.

Summers will use the insights about how people perceive images to improve CAD systems and develop training programs to help medical personnel more accurately interpret CT colonographic images. He is already doing follow-up crowdsourcing studies to test how well people identify polyps when they can scroll through images as well as see polyp candidates from all angles.

Crowdsourcing studies have “the potential to change how these perception experiments in medical imaging are conducted,” said Summers. “I think the outcomes will be better CAD systems.”

To listen to the podcast of Summers’s interview with Radiology editor Herbert Kressel, visit http://radiology.rsna.org/content/suppl/2012/02/23/radiol.11110938.DC2

This page was last updated on Monday, May 2, 2022

  • Issue Overview
  • Features
    • Piecing Together the Autism Puzzle
    • Crowdsourcing
    • Eyes on the Goal
    • Collins: How Can We Do Better?
    • Tenacious Tara
    • Bike Commuting
  • Departments
    • From the Deputy Director for Intramural Research
    • News You Can Use
    • NIH in History
    • Research Briefs
    • Colleagues: Recently Tenured
    • The Training Page
    • The SIG Beat
    • Announcements
    • Laboratory Confessions
  • Issue Contents
  • Download this issue as a PDF

Catalyst menu

  • Current Issue
  • Previous Issues
  • About The NIH Catalyst
  • Contact The NIH Catalyst
  • Share Your Story
  • NIH Abbreviations

Catalyst links

  • Follow The NIH Catalyst

Subscribe Today!

Subscribe to The NIH Catalyst Newsletter and receive email updates.

Subscribe

Get IRP Updates

Subscribe

  • Email
  • Print
  • Share Twitter Facebook LinkedIn

Main navigation

  • About Us
    • What Is the IRP?
    • History
    • Honors
    • Organization & Leadership
    • Our Programs
    • Research Campus Locations
    • Contact Information
  • Our Research
    • Scientific Focus Areas
    • Principal Investigators
    • Accomplishments
    • Accelerating Science
    • Research in Action
    • Trans-IRP Research Resources
    • IRP Review Process
    • Commercializing Inventions
  • NIH Clinical Center
    • Clinical Center Facilities
    • Advancing Translational Science
    • Clinical Trials
  • News & Events
    • In the News
    • I am Intramural Blog
    • Speaking of Science Podcast
    • SciBites Video Shorts
    • The NIH Catalyst Newsletter
    • Events
  • Careers
    • Faculty-Level Scientific Careers
    • Trans-NIH Scientific Recruitments
    • Scientific & Clinical Careers
    • Administrative Careers
  • Research Training
    • Program Information
    • Training Opportunities
    • NIH Work/Life Resources
  • Department of Health and Human Services
  • National Institutes of Health
  • USA.gov

Footer

  • Home
  • Contact Us
  • IRP Brand Materials
  • HHS Vulnerability Disclosure
  • Web Policies & Notices
  • Site Map
  • Search