Choosing the Ignite Summer 2015 Finalists: Our Selection Methodology and Reflections

We accepted proposals for the Summer 2015 class of our HHS Ignite Accelerator between March 1 and March 31, 2015. From that pool of proposals, we have narrowed things down to a group of finalists. This post is meant to document that methodology.

We received 42 proposals

Each proposal has a designated project lead. Here's how the 42 proposals break down: ACF = 2 CMS = 7 CDC = 4 FDA = 8 IHS = 5 NIH = 9 OS = 7 A couple other data points:

• One staff member from HRSA was on one of the proposals although they weren't the project lead on any proposal.
• There were 7 proposals that had non-HHS individuals involved.
• The Total number of individuals on all proposals was 143.

These proposals were scored by 15 Reviewers

The Reviewers were comprised of previous (and current) Ignite team members:

• Jody McLean, Centers for Disease Control and Prevention
• Sabrina Matoff-Stepp, Health Resource Services Administration
• Bethany Applebaum, Health Resource Services Administration
• Marliza Rivera, Indian Health Service
• Amy Wiatr-Rodriguez, Administration for Community Living
• David Berli, Administration for Children & Families
• Heather Swope, Administration for Children & Families
• David Hale, National Institutes of Health

As well as folks from the IDEA Lab:

• Kate Appel
• Ross Bowling
• Corinne Fukayama
• Lelia Gessner
• Mark Naggar
• Serena Parks
• Malini Sekhar

Each proposal was scored 5 times

We broke our 15 Reviewers into 3 panels and then randomly distributed the 42 proposals across the panels. This means that each proposal was scored 5 times, the average of which was the proposals finals score that we used in our analysis. Each Reviewer received the same instructions and guidance for how to score. Now even with that said, certainly some reviewers were "nicer" and others were "meaner". This means that some panels were "nicer" and others were "meaner". We of course don't want that to influence who was selected as finalists, so we used Z-scores to help normalize the scores. See more about that below in the section called: " X " We asked each individual to self-identify if they should recuse themselves. There were no recusals.

Based upon defined criteria

Each proposal was scored on a 0-100 range based upon our communicated criteria:

• The project's importance to the Office, Agency and/or Department [20 points]
• The potential impact of the proposed solution. [40 points]
• The proposal's understanding and explanation of the problem that needs to be solved. [20 points]
• The proposal's understanding of the customers that the project serves. [20 points]

Each Reviewer was also asked to provide brief comments on the proposal that help justify their score. We will be releasing the scores and reviewer comments for both the sake of transparency and in order to provide feedback to those that submitted a proposal. But of course we'd like to do that anonymously, to both the person submitting the proposal as well as the Reviewer. We haven't had a chance to bring all of that together yet. We plan on releasing this information in mid-May when we announce the selected teams.

Identifying the Top Proposals

The following were ways in which a proposal was able to advance:

• The top 12 z-scores overall
• The top 3 z-scores from each Panel
• The top 2 z-scores from each OpDiv
• Wildcards picked by IDEA Lab staff

Some proposals of course fell into more than one category. For example, a proposal may have been both the top in a Panel and also the top in their OpDiv. Using the first three rules, 20 proposals jumped out. A few staff from the IDEA Lab (specifically, Read Holman, Will Yang, and Greg Downing), then went through every proposal and picked a few proposals that we felt eligible for a wild card slot. We believe in rules and algorithms to ensure fairness, but we also hesitate to over-rely on them at the risk of losing the touch of human judgement. We've run a few rounds of Ignite now, and have a pretty good sense (we think) as to the types of problems that Ignite can help address. From the pool of eligible Wildcard proposals, six wild cards were selected.

The 26 Finalists for the Summer 2015 class

Here are the 26 finalists identified through the above-mentioned process. These projects were also highlighted in our blog post - HHS Ignite Accelerator: The Finalists for the Summer 2015 Class.

• OMHA Adjudication Knowledge (OAK) Resource, OS
• NIH Travel Service Center, NIH
• NIH Collaboration Connector (NC2), NIH
• ASPR Advancing the Women/Peace/Security Agenda, OS
• TEAK: A Tool for Enriching Agency Knowledge, NIH
• Collection of Retail Meats for NARMS, FDA
• Instituting RFID Into Property Management System, FDA
• CMS FOIA and Correspondence Online Request Entry, CMS
• Automation of Summary Statement Generation, NIH
• Appification of Research-tested Interventions, NIH
• Inspect, Don't Expect: Standards Evaluation, CMS
• Clear-Aborate: Improve CMS Communications, CMS
• Real-Time Tracking System for Work Effort & Tasks, NIH
• Web-based Analytics for OpenFDA Using R and Shiny, FDA
• CollabraDiv, ACF
• Global Staffing Accelerator, OS
• Quality Improvement Strategies for Nursing Homes, CMS
• Igniting Population Health Collaboratives, OS
• Development of Territorial Virtual Resource Center, OS
• vacciNATION, OS
• GovShare: A Source Code Repository, CDC
• Rural Clinical Simulation & Education Exchange, IHS
• A virtual pharmacy for "self-administered" drugs, CMS
• Implementing Systematic Review in ATSDR's profiles, CDC
• HHS Anti-Trafficking Case Management Portal, ACF
• Whiteriver Rural Medicine Fellowship, IHS

General Observations

We didn't receive as many proposals this round. In the previous three Rounds of Ignite, we received 65, 74, and 72 proposals respectively. We're not sure exactly why the drop in number, but a couple hypotheses:

• Not a lot of time has passed since the last application window. This is the first time we've run 2 Ignite classes back-to-back; in the past there has been multiple months between the end of one class and the call for proposals for the next. However, we're currently still wrapping up the Winter 2015 team (see this webpage for details on their Shark Tank!).

• We blended the Ignite marketing with other programs that the IDEA Lab is running. It's a busy time for us and we needed to communicate out about opportunities with our Entrepreneurs-in-Residence program and our HHS Innovates Awards.

• Just generally, we've now been around a while. We think there are a LOT of problems out there still to be solved, MANY potential project ideas waiting to be submitted. In the first couple Rounds, we had an exciting and attentive applicant pool, tailored (we think) to the early-adopter types that are more. However, now that we've been around for four Rounds, we're needing to find new ways to reach new people.

But even with the fewer number of proposals submitted, the quality was quite high. This is true not just in the types of projects submitted but also in the writing itself. It's very difficult to communicate complex project ideas using simple language, especially with the low character limits that our proposal essays allowed for. We think that alone is something worth celebrating. Clearly a lot of time and energy were put into these! It was still very tough to select down to 26 finalists. In fact, our original goal was to select only 20 finalists, but we felt we needed to hear from more as their proposals were well written and they convinced us that we should learn more about their project idea.

Next Steps

The 26 Finalists will each get 25 minute interviews with members of the IDEA Lab. These interviews will look beyond the substance of their proposal, beyond even the idea itself. Ignite is an investment not just in projects but in people. Thus these interviews are, well, interviews. In many ways, we see selection into Ignite as a hiring process. Thus we look at team dynamics, we look at the skill-sets across the team members, we look for leadership from the Project Lead, we ask about competing priorities. The interviewers are, in panels of 3: Read Holman (that's me), Will Yang, Julie Schneider, Sandeep Patel, Jamie Elliott, Juliana Cyril. We're greatly looking forward to talking with them during the week of April 27th! From these 26, we will have the seemingly impossible and always incredibly difficult task of identifying 12 teams into the Summer 2015 class of our HHS Ignite Accelerator.