The HHS Ignite Accelerator: How We Chose the Spring 2016 Finalists

The results are in! We have selected the finalists for the Spring 2016 HHS Ignite Accelerator, the Department's internal innovation startup program.

In the spirit of transparency and knowledge sharing, we are publishing our methodology for how we selected these projects. This blog post mirrors previous posts for previous Rounds of Ignite. Here is the link to our methodology for Summer 2015.

For this upcoming 5th Round (AKA "Spring 2016"), of this cyclical accelerator program, we accepted proposals between October 1 and November 6, 2015. From that pool of proposals, we have narrowed things down to a group of finalists. This post is meant to document that methodology.

We received 82 proposals

Each submitted project idea identified a project lead, and we asked for that person's Agency (or as we call it here in HHS, that person's Operational Division [OpDiv]). Here's how the 82 submitted proposals break down, by the project lead's OpDiv:

ACF = 7
ACL = 0 *
AHRQ = 1
CDC = 17
CMS = 12
FDA = 12
HRSA = 6
IHS = 6
NIH = 16
OS = 4
SAMHSA = 1

A couple other data points:

• Previous Rounds saw submission numbers of 65, 72, 74, and 42, respectively. So this is the most yet.
• [*] One staff member from ACL was on one of the proposals although they weren't the project lead on any proposal. That means that every Agency of HHS was represented.
• There were 5 proposals that included individuals from outside of HHS.
• The Total number of individuals on all proposals was 252. Teams could have up to 3 individuals (plus a project sponsor) on their project team. This is a lower limit than in past Rounds, where the cap on team members was set at 5.
• HRSA is currently and simultaneously running their own internal accelerator pilot which they're calling the HRSA Idea Spring. (FYI - It's awesome.) If you add the submissions to the HRSA Idea Spring to those of HHS Ignite Accelerator, then there were actually over 100 project ideas submitted. WOW.

These 82 proposals were scored by 25 Reviewers

The Reviewers were comprised of previous Ignite team members:

• Lewissa Swanson, OS/OASH
• Michael Johansson, CDC
• Dan Stowell, CDC
• Bethany Applebaum, HRSA
• Amy Wiatr-Rodriguez, ACL
• Jon Langmead, CMS
• Shana Freno, SAMHSA
• Dan Duplantier, OS/ASPE
• Katerina Horska, OS/IOS
• Melody Stevens, CDC
• Leigh Willis, CDC
• Dan Baden, CDC
• Cynthia Vinson, NIH
• Vinay Pai, NIH
• Sabrina Matoff-Stepp, HRSA
• Michelle Davis, OS/OASH

As well as folks from the IDEA Lab family:

• Andy Burton, HHS IDEA Lab (part time detail from NIH)
• Betsy Hsu, HHS IDEA Lab
• Corinne Fukayama, HHS IDEA Lab
• Damon Davis, HHS IDEA Lab
• Malini Sekhar, HHS IDEA Lab
• Mark Naggar, HHS IDEA Lab
• Serena Parks, ACF (formerly of the IDEA Lab. Serena, we miss you)
• Katrina Jones, HRSA Idea Spring
• Candace Webb, HRSA Idea Spring

Each proposal was scored 5 times

We broke our 15 Reviewers into 3 panels and then randomly distributed the 42 proposals across the panels. This means that each proposal was scored 5 times. We used the average of the 5 scores to make a finals score that we used in our analysis.

Each Reviewer received the same instructions and guidance for how to score. Now even with that said, certainly some reviewers were "nicer" and others were "meaner." This means that some panels were "nicer" and others were "meaner". We, of course, don't want that to influence who was selected as finalists, so we used Z-scores to help normalize the scores. See more about that below in the section called: "There Were Four Ways A Project Idea Could Advance"

We asked each individual to self-identify if they should recuse themselves. There was one recusal for one proposal.

Based upon defined criteria

Each proposal was scored on a 0-100 range based upon our communicated criteria:

• The project's alignment to the Office, Agency mission [20 points]
• The proposal's explanation of the process, product, or system to be addressed. [60 points]
• How well the proposed solution aligns with the communicated problem. [20 points]

After scoring, reviewers were asked on a binary scale: do you think this proposal should be considered to become a finalist? Each Reviewer was also asked to provide brief comments on the proposal that help justify their score.

There Were Four Ways A Project Idea Could Advance

The following were ways in which a proposal was able to advance:

• The top 20 z-scores overall
• Top 25% z-score from each OpDiv (minimum of 2 per OpDiv)
• Review Panel unanimously votes to advance it (regardless of score)
• Wildcards picked by IDEA Lab staff

As a refresher on statistical methods: Z-scores are ways to normalize across groups. You can imagine how one of our Panels was "nicer" than another panel. We'd hate for a project to get a lower score simply because of the makeup of their panel. Z-scores, also called "standard scores" help prevent that from happening. Here's the wikipedia article on z-scores if you'd like to dive deeper.

Some proposals fell into more than one category. For example, a proposal may have been both a top scoring proposal and also top in it's OpDiv. Using the first three rules, 31 proposals emerged.

We believe in rules and algorithms to ensure fairness, but we also hesitate to over-rely on them at the risk of losing the touch of human judgement. So we always have a Wildcard category. Wildcards were picked by two staff from the IDEA Lab (specifically, Read Holman and Will Yang). We went through every proposal and picked proposals that we felt eligible for a wild card slot. Reviewers were able to lobby for the ones they liked. From the pool of eligible project ideas, 16 were selected as wildcards.

We Selected 47 Project Ideas as Ignite Finalists

Through the above criteria, a little more than half of the project ideas submitted were identified as Finalists. For the sake of space and to keep the methodology and process information separate from the announcement, we posted information on those finalists on a separate blog post.

Continue Reading: From The HHS Ignite Accelerator: The 47 Finalists for Spring 2016!