American Society of Functional Neuroradiology
Ai Challenge
Congratulations to the Winners at the 2019 Annual Meeting !
The Winners!
Normal versus Abnormal category
1. USC - Vishal Patel
2 ex aequo. UCI - Peter Chang
2 ex aequo. Wake Forest - Ryan Godwin
3. UTSW - Sahil Nalawade
Detection of Acute Ischemic Stroke
1. USC - Vishal Patel
2. UTSW - Sahil Nalawade
Detection of Acute Hemorrhage
1. USC - Vishal Patel
2 ex aequo. UCI - Peter Chang
2 ex aequo. UTSW - Sahil Nalawade
3. Wake Forest - Ryan Godwin
Detection of TBI
1. USC - Vishal Patel
2. UTSW - Sahil Nalawade
Detection of Mass Effect
1. UCI - Peter Chang
2. USC - Vishal Patel
3. UTSW - Sahil Nalawade
Characterization of Degree of Emergency
1. USC - Vishal Patel
2. UTSW - Sahil Nalawade
ASFNR Ai Challenge
Go head-to-head with your AI Algorithm in San Francisco, Nov 3-5, 2019
The Challenge
We are challenging you with a unique, carefully curated, gold-standard, test dataset of non-contrast head CTs from the emergency room!
Ready?
Prime-time
Want Neuroradiologists to use your AI algorithm? Come to the ASFNR Annual Meeting in November in San Francisco and convince them using their own dataset!
Set?
Our Dataset
• 1,200 deidentified studies
• From 4 international contributing institutions (Lausanne, Maryland, UC Irvine, Wake Forest) and different types of CT scanners (GE, Siemens, Philips and Toshiba)
• Encompassing a broad variety of clinically relevant pathology: trauma, hemorrhage, stroke, hydrocephalus, mass effect, etc
*Our dataset is for testing only, not for development. For development, please use any publicly available datasets such as: http://headctstudy.qure.ai/dataset, https://www.kaggle.com/felipekitamura/head-ct-hemorrhage. You are of course welcome to use any dataset that you have access to.
Go!
Winners announced at the ASFNR Annual Meeting
We will identify the three best AI algorithms for each of the following:
Normal/abnormal for age
Stroke
Intracranial hemorrhage
Trauma
Mass Effect
Degree of Urgency
The Labels
Our labels:
• All head CT scans were labeled by 5 CAQ neuroradiologists who all previously completed a two-year neuroradiology fellowship training program
• Consensus among the neuroradiologists was > 90% (any disagreements were resolved by consensus review)
• Each CT scan was labeled in three categories:
1. Normal or Abnormal for Age
Presence or absence of intracranial pathology with consideration to patient age.
0 = Normal for age: normal examination with/without incidental findings including:
1 = Abnormal for age:
Any intracranial abnormality, especially those explicitly defined in categories below.
Sinus air-fluid levels
2. Category of Abnormality (list as many as you want)
3. Degree of Urgency
The Process
• Register for the ASFNR AI challenge below.
• On August 15 or after you register, you will be sent a link to 100 deidentified cases with labels so that you can familiarize yourself with the format of our dataset.
• On September 15, you will be sent another link to our 1,200 deidentified cases without labels.
• The deadline for submission of your results is: 11:59PM Pacific Time on October 15, 2019. You will need to submit your results using this document (link), to the following email.
• We will perform the analysis centrally to assess accuracy of your submission and share the overall accuracy of your submission with you.
• We are happy to perform the accuracy analysis for you up to 5 times, with final deadline for all submissions on or before October 15.
The Analyses
• You can compete in one, several or all of the several categories (normal/abnormal for age; category of abnormality; degree of urgency). You may compete in as many or as few categories as you wish.
• For annotations with only two classes (e.g., normal/abnormal for age), we will calculate true positives, true negatives, false positives and false negatives.
• For annotations with several classes (e.g, type of disease), we will calculate the percent of correct and incorrect classifications for each category and overall.
• For annotations that are ordinal (e.g, level of urgency), we will calculate the number of undercalls or overcalls for each category and overall.
The Winners
• Using the overall accuracy results, we will identify the three top-performing AI algorithms for each of the following:
Normal/abnormal for age
Stroke
Intracranial hemorrhage
Trauma
Mass Effect
Degree of Urgency
• We will post the three winners for each category on our website for all ASFNR members to see.
• The three winners for each category will also be announced on Monday November 4, 2019 at 1:15pm during the AI challenge session at the 2019 Annual ASFNR meeting in San Francisco.
• The top two winners for the normal/abnormal for age category, for the degree of urgency category and the best performing algorithm for the type of disease (total of 6 presenters) will be offered to present their algorithm during that session (8 minutes for each presenter).
© 2019 All Rights Reserved. American Society of Functional Neuroradiology