• The results are in!

     

  • Congratulations to the Winners at the Annual Meeting !

  • The Winners!

    Normal versus Abnormal category

    1. USC - Vishal Patel

    2 ex aequo. UCI - Peter Chang
    2 ex aequo. Wake Forest - Ryan Godwin
    3. UTSW - Sahil Nalawade

    Detection of Acute Ischemic Stroke

    1. USC - Vishal Patel

    2. UTSW - Sahil Nalawade

    Detection of Acute Hemorrhage

    1. USC - Vishal Patel

    2 ex aequo. UCI - Peter Chang
    2 ex aequo. UTSW - Sahil Nalawade

    3. Wake Forest - Ryan Godwin

    Detection of TBI

    1. USC - Vishal Patel

    2. UTSW - Sahil Nalawade

    Detection of Mass Effect

    1. UCI - Peter Chang

    2. USC - Vishal Patel

    3. UTSW - Sahil Nalawade

    Characterization of Degree of Emergency

    1. USC - Vishal Patel

    2. UTSW - Sahil Nalawade

  • ASFNR Ai Challenge

    Go head-to-head with your AI Algorithm in San Francisco, Nov 3-5, 2019

  • The Challenge

    We are challenging you with a unique, carefully curated, gold-standard, test dataset of non-contrast head CTs from the emergency room!

     

    Ready?

    Prime-time

    Want Neuroradiologists to use your AI algorithm? Come to the ASFNR Annual Meeting in November in San Francisco and convince them using their own dataset!

    Set?

    Our Dataset

    • 1,200 deidentified studies

    • From 4 international contributing institutions (Lausanne, Maryland, UC Irvine, Wake Forest) and different types of CT scanners (GE, Siemens, Philips and Toshiba)

    • Encompassing a broad variety of clinically relevant pathology: trauma, hemorrhage, stroke, hydrocephalus, mass effect, etc

     

    *Our dataset is for testing only, not for development. For development, please use any publicly available datasets such as: http://headctstudy.qure.ai/dataset, https://www.kaggle.com/felipekitamura/head-ct-hemorrhage. You are of course welcome to use any dataset that you have access to.

    Go!

    Winners announced at the ASFNR Annual Meeting

    We will identify the three best AI algorithms for each of the following:

    Normal/abnormal for age

    Stroke

    Intracranial hemorrhage

    Trauma

    Mass Effect

    Degree of Urgency

  • The Labels

    Our labels:

    • All head CT scans were labeled by 5 CAQ neuroradiologists who all previously completed a two-year neuroradiology fellowship training program

    • Consensus among the neuroradiologists was > 90% (any disagreements were resolved by consensus review)

    • Each CT scan was labeled in three categories:

    1. Normal or Abnormal for Age

    Presence or absence of intracranial pathology with consideration to patient age.

    0 = Normal for age: normal examination with/without incidental findings including:

    • Microvascular ischemic changes
    • Brain volume loss (appropriate for age)
    • Incidental calcification (basal ganglia, etc)
    • Sinus mucosal thickening
    • TMJ degenerative changes
    • Upper C-spine degenerative changes (not severe)

    1 = Abnormal for age:

    Any intracranial abnormality, especially those explicitly defined in categories below.

    Sinus air-fluid levels

    2. Category of Abnormality (list as many as you want)

    1. Acute ischemic stroke (early ischemic sign such as dense artery sign, sulcal effacement, loss of the gray white differentiation)
    2. Subacute ischemic stroke (hypodensity)
    3. Acute hemorrhage (or acute on chronic hemorrhage - need to see hyperdense blood)
    4. TBI
    5. Ventriculomegaly
    6. Mass effect: any of midline shift, sulcal effacement, cisternal effacement, herniation, etc
    7. Brain mass: including both intra-axial and extra-axial lesions; benign or malignant
    8. Bone lesion (other than fracture, which would go into TBI)
    9. Sinus, face, temporal bone or mastoid disease or abnormality
    10. Other

    3. Degree of Urgency

    1. Emergent finding - intervention immediately needed (for instance large acute ischemic stroke or large intracranial hemorrhage, or severe ventriculomegaly or severe mass effect)
    2. Critical finding - referring clinician should be called urgently - within an hour (for instance subacute stroke, small stroke or small hemorrhage, or TBI, or mild mass effect) 
    3. Actionable finding - significant but not urgent finding - one hour onward (for instance brain tumors or mild ventriculomegaly, need another imaging study)
    4. Incidental finding - minor abnormality (for instance small meningioma, classic arachnoid cyst)
    5. Normal for age
  • The Process

    • Register for the ASFNR AI challenge below.

    • On August 15 or after you register, you will be sent a link to 100 deidentified cases with labels so that you can familiarize yourself with the format of our dataset.

    • On September 15, you will be sent another link to our 1,200 deidentified cases without labels.

    • The deadline for submission of your results is: 11:59PM Pacific Time on October 15, 2019. You will need to submit your results using this document (link), to the following email.

    • We will perform the analysis centrally to assess accuracy of your submission and share the overall accuracy of your submission with you.

    • We are happy to perform the accuracy analysis for you up to 5 times, with final deadline for all submissions on or before October 15.

    The Analyses

    • You can compete in one, several or all of the several categories (normal/abnormal for age; category of abnormality; degree of urgency). You may compete in as many or as few categories as you wish.

    • For annotations with only two classes (e.g., normal/abnormal for age), we will calculate true positives, true negatives, false positives and false negatives.

    • For annotations with several classes (e.g, type of disease), we will calculate the percent of correct and incorrect classifications for each category and overall.

    • For annotations that are ordinal (e.g, level of urgency), we will calculate the number of undercalls or overcalls for each category and overall.

    The Winners

    • Using the overall accuracy results, we will identify the three top-performing AI algorithms for each of the following:

    Normal/abnormal for age

    Stroke

    Intracranial hemorrhage

    Trauma

    Mass Effect

    Degree of Urgency

    • We will post the three winners for each category on our website for all ASFNR members to see.

    • The three winners for each category will also be announced on Monday November 4, 2019 at 1:15pm during the AI challenge session at the 2019 Annual ASFNR meeting in San Francisco.

    • The top two winners for the normal/abnormal for age category, for the degree of urgency category and the best performing algorithm for the type of disease (total of 6 presenters) will be offered to present their algorithm during that session (8 minutes for each presenter).