The craft of incentive prize design Appendix A: Advanced prize design guidance

Take a deeper dive into tactical guidance on how to link outcomes with challenge design.

Appendix A

This appendix provides additional guidance on how to link outcomes with challenge design, including tactical considerations for each of the six outcomes. Designers are more frequently structuring prizes for multiple outcomes, which requires blending of design elements and recognizing the trade-offs between the elements and the outcomes themselves. The US Department of Labor’s Equal Pay App Challenge is a good illustration of a multiple-outcome prize. It encouraged the development of a web application and seeks to raise awareness about differing levels of pay between men and women. This challenge offered an interesting mix of incentives: a grand prize of five scholarships, an immersive program for digital entrepreneurs, and three other recognition prizes—conversation with an eminent social enterprise leader, nonprofit adoption of the app, and an accelerator program to launch the app publicly. This mix of incentives drew app developers to the challenge but also raised public interest in this issue. The challenge is a part of a larger portfolio approach that the US government is pursuing to raise awareness about the pay gap through legislation, executive orders, and task forces.

Our Challenge.gov data analysis revealed a pattern of challenges that seek a combination of outcomes across the two dimensions discussed in this report—developing ideas, technologies, products, or services and engaging people, organizations, and communities. More specifically, designers often pair attracting new ideas with raising awareness and developing prototypes and pilots with mobilizing action. When designers create these pairings, they often focus on evaluation criteria and prize structure as the most important design elements.

Attract new ideas: Solicit concepts and techniques

Common pitfalls: Poorly structured problem statements, lack of planning, and solicitation of non-workable solutions are common pitfalls associated with the design of challenges focused on attracting new ideas. Designers should adopt the mentality that the solutions generated are the first step in a series of portfolio prizes and other tools for driving innovation that will advance in maturity and complexity toward stimulating markets.

Design elements Design strategic considerations Tactical guidance
Resources Guard against the development of overly narrow problem statements. Use external expertise to design problem statements that will lead to broadly workable solutions.
  • Create an advisory board of potential end users from the private sector, trade associations, philanthropies, academia, etc., and solicit input on how to design the challenge in order to generate desirable solutions.
  • Test the problem statement with the advisory board through targeted interviews and ideation sessions to generate a list of likely responses based upon variations of the problem statement.
Evaluation Structure format of participant submissions for ease of evaluation. Given the low barriers to entry for the submission of new ideas, work to ensure that the evaluation period is not so lengthy that it deteriorates the experience of the participants.
  • Use word limits and structured response templates to guide participants toward desired solutions and simplify the evaluation process. For example, a challenge focused on generating slogans should be limited to 25 characters while a challenge for a technical proposal should include an example submission on the current industry benchmark.1
  • Dual outcome guidance with raising awareness: Use public voting to engage a broader audience beyond the competitor community. This approach does involve a trade-off, because the final ideas may be of lower quality without vetting by better-informed judges.2
Motivators Plan in advance for future rounds of the challenge, which will focus on outcomes of increasing complexity. Create excitement around the problem by guiding the formation of a vibrant community of participants.
  • Incorporate mechanisms into design that encourage and reward participant interaction and collaboration. This can include leveraging a platform with dedicated collaboration space, using rules to mandate cross-fertilization at certain points in the challenge, and including evaluation criteria that reward the organic formation/combination of teams with similar solutions.
  • Create motivators for future rounds of the challenge to prevent participant fatigue. Consider the broader cost of the challenge to participants (that is, resources invested in the challenge prevent investment in other areas) and develop motivators that will benefit the broader goals of the participant pool. As an example, mentoring will benefit the broader capabilities of a participant in comparison to a minor increase in purse size.
Structure Develop mechanisms prior to the launch of the challenge to support participants in revising ideas if initial submissions do not meet expectations.
  • Hold working sessions with the advisory board to evaluate the diversity and maturity of solutions throughout the submission period. Use this information to provide guidance for other participants and restructure eligibility requirements (that is, participant expertise and/or experiences) for future rounds.
  • Dual outcome guidance with raising awareness: Use a multi-round or mini-challenge approach to allow open engagement and exposure to the topic in early rounds and down-selecting the best ideas through the later and final rounds. This will allow designers to reach participants most likely to provide high-quality ideas while also expanding engagement across a broader community. For example, NASA’s Zero Robotics Video Challenge uses open eligibility in its first round to capture ideas for a video that promotes the student robotics challenge. In later phases, these ideas are pitched and the winners receive $500 to turn their ideas into videos that help raise awareness for the larger robotics challenge.3
Communications Focus on building a strong brand around the challenge from the outset. A strong brand will increase the size and diversity of the participant community and the value of recognition to winners.
  • Determine the types of media valued by the target audience (that is, traditional press, social media, etc.) through interviews with potential participants. Use this information to target marketing efforts in order to build a brand around the challenge and create broad public awareness.

Additional examples:

Build prototypes and launch pilots: Produce, test, and improve models

Common pitfalls: Relying on the purse—even a large one—as a sole means of motivating participants is a common pitfall of designers seeking to build prototypes or launch pilots. There should be significant emphasis on complementary motivators (for example, the recognition, networking opportunities, and investment counseling) that will encourage participants to put their own capital at risk. Designers should recognize that they will need to study their potential participants to understand their constraints, opportunities, and impact on outcomes. Designers should also consider customizing communications to particular participant communities.

Design elements Design strategic considerations Tactical guidance
Resources Build an understanding of the landscape of potential challenge participants, in order to appropriately shape purse size and problem statements.Develop detailed plans for the use of testing facilities early in the design process to fully understand the impact on the cost, length, and fairness of evaluating solutions.
  • Conduct landscape analysis prior to completing detailed design. The analysis should consist of an economic and technical assessment:
    • Economic assessment: Identify and assess potential participants and their likely fixed and variable costs for developing a solution through interviews and financial modeling. Structure a sufficiently sized purse and additional incentives based upon the forecasted economics. If the challenge entails high fixed costs, the purse should be large enough to justify the investment required to build a prototype and reduce participant risks.4
    • Technical assessment: Identify state-of-the-art prototypes and pilots related to the problem statement. Interview the developers of these prototypes and pilots to determine the technical challenges in achieving the desired outcome. Refine the problem statement based upon the identified challenges to create realistic goals for the challenge.5
  • For challenges that require the physical testing and demonstration of prototypes, designers must consider the cost, logistics, and impact of testing facilities on design. Considerations include testing location, validation protocols, cost/length of test, safety, and acts of God.
Evaluation Focus on identifying and rewarding both the best technical solution and the solution with the best commercialization prospects. They are not always the same and both are necessary for long-term success of the prize.
  • Expand the impact of the challenge by including a requirement for the submission of a scaling plan for the prototype or pilot in addition to the technical design. The plan should identify the requirements for bringing the prototype or pilot to full-scale production. Use this tactic to identify the most viable long-term commercial solutions in addition to the best technical solutions.
  • Dual outcome guidance with mobilizing action: It is critical to maintain rigorous, quantitative evaluation standards for these prototypes. Balance the inclusion of more qualitative criteria (for example, those rewarding teaming, which will be important for commercialization) to link the evaluation of submissions to the goal of mobilizing action.
Motivators Vary motivators based upon the community of participants. Designers should ensure that they build an understanding of the potential participants and adjust motivators as necessary.
  • Tailor the motivators to the target community of solvers. For example, networking with the venture capital community to provide funding to bring prototypes to market is ideal for start-ups and entrepreneurial participants. In contrast, academic participants are likely best motivated through grants, publicity, and conference networking opportunities. Designers should be prepared to provide additional motivators (beyond increasing the purse) throughout the registration process, if the community of participants is smaller than anticipated or varies significantly from their projected participants. The added motivators can drive additional excitement around the prize.
Structure Structure the challenge around the technological maturity of the desired outcome. Less mature models will require additional challenge rounds and development time.
  • Design the structure of the challenge based upon the maturity of the technology of the prototype or pilot, including research and development, small-scale proof-of-concept, or commercial prototype. Each stage of technological maturity requires different rules and evaluation criteria. For example, a research and development prototype will likely require an extended multi-stage challenge to mature the associated technology to the desired outcome. In contrast, a challenge focused on the development of a small-scale, proof-of-concept prototype as an outcome will likely consist of fewer phases but focus more heavily on meeting objective performance criteria at lower cost.
  • Dual outcome guidance with mobilizing action: Multiple rounds or mini-challenges can encourage competitors to work with one another and lead to the development of better prototypes. Breaking the challenge into rounds can provide opportunities for judge feedback that can improve participants’ skills. Build participant communities by inspiring both challenge (for example, leaderboards) and collaboration (for example, teaming). Striking the right balance is important so that participants continue to work together after the challenge concludes.
Communications Develop a targeted and extended communication strategy that consists of multiple channels and outreach methods. Length of the communications strategy is longer than challenges focused on attracting ideas and must sustain excitement.
  • Sequence communications to recruit potential participants, share information with participants, and keep the broader community engaged throughout the challenge. Due to the extended length of the challenge, leverage diverse channels to drive momentum and build engagement and anticipation throughout the challenge. Designers should also develop metrics early to assess the effectiveness of their communication strategy within their target participant community. If the communication strategy is not successful, designers can supplement it with personal appeals to specific participants identified during the landscape analysis.

Additional examples:

Stimulate markets: Create and scale new markets

Common pitfalls: Designers working on challenges to stimulate markets should have a clear understanding of market gaps or failures, and what would motivate new or existing market actors to fill or overcome them. Without understanding how these markets work, designers risk incenting participants to engage in market behaviors that are unrealistic, unprofitable, and unscalable.

Design elements Design strategic considerations Tactical guidance
Resources Engage a broad community of external experts through an advisory board to design a challenge focused on addressing specific challenges preventing market development or growth.
  • Expand advisory board composition from that developed for challenges focused on attracting new ideas. Since challenges seeking to stimulate markets frequently address market failures, unrecognized market requirements, or transformational technologies, it is important to understand operational demands from producers, requirements from regulators, and global consumer requirements. These considerations should be included in problem statement design. Representation from each of these stakeholder groups is advised.
Evaluation Create mechanisms to avoid potential conflicts of interest between sponsors and participants and limit gaming of the rules and evaluation criteria from participants.
  • Engage the broader public in the design of the challenge to generate interest in the broader problem/challenge.
  • Establish an independent evaluation board. This group of experts should be distinct from the advisory board and focus solely on vetting evaluation criteria for potential flaws and assessing submissions. The independence of the evaluation board from the advisory board and potential participants is critical so that unbiased feedback may be provided.6
  • Consider holding a public comment period on the draft rules and evaluation criteria to identify potential issues before the challenge begins. This approach may be used to create early excitement from potential participants and the broader public.
Motivators Create motivators for the participants, judges, and experts due to the heavy cost and time investment for all.
  • Similar to challenges focused on developing models as an outcome, a detailed understanding of participant cost structure is required in order to determine the appropriate size of the purse to defray the investment costs and risks for the participants. In contrast to the aforementioned challenges on this spectrum, the purse may need to be larger than simply the costs of the participants. The purse must also be large enough to attract broad public attention and create demand for the solutions.
  • Due to the complexity of challenges focused on this outcome, numerous industry experts are likely needed at different points in time during the challenge. In order to defray additional costs for this expertise, reward market experts by offering “no-cost or low-cost” sponsorship opportunities, exclusive access to participants, and public recognition at different points in the challenge process (for example, launch, judging, and award).
  • Structure the post-award phase of the challenge focused on scaling the winning solution. Use additional funding mechanisms and partnerships to motivate participants to continue to refine promising solutions and maintain broad participant interaction following award.
Structure Establish a structure that permits iterative feedback between designers and participants. Reward participants for successfully achieving technical and economic milestones to maintain interest and reduce risk.
  • Use multiple rounds or stages to support the scaling of the product and actual market testing. Provide milestone payments or advanced market commitments for achieving specific technical proficiency or sales targets. Both the market testing and payments will keep participants engaged, celebrate successes, and demonstrate impact.7
Communications Hire expertise needed to drive a successful public relations campaign and create an appealing narrative around the challenge to gain public interest.
  • Engage public relations experts to design marketing messaging and create a grand narrative around the challenge. Focus the narrative more broadly than the actual desired outcome to appeal to the general public and create demand and interest around the outcome.
  • Use stories about the participants to amplify sustained marketing communications. Video highlights of the first and second rounds or participant testimonials can provide opportunities to build the grand narrative of the challenge. Public tracking of progress through social media feeds or other mechanisms can sustain media interest after the excitement of the launch has concluded.

Additional examples:

Raise awareness: Enhance exposure and educate on an issue

Common pitfalls: In a crowded media environment, designers seeking to use challenges to raise awareness about an issue face the difficulty of customizing their messages and getting them to the target audience. Often, these designers fall into the trap of merely targeting the broadest possible audience in the hope that their target audience will somehow catch on. It is critical for designers to appropriately segment the audience for their challenge and build campaigns specifically related to the media consumed by that audience. Additionally, designers ignore post-award communications at their own peril, as they may represent the greatest opportunity to achieve the desired increase in awareness.

Design elements Design strategic considerations Tactical guidance
Resources Build communications and marketing capabilities into administration staff core capabilities.
  • Prioritize efforts on identifying design and administration staff with expertise leading marketing campaigns, crafting targeted messaging, and community outreach and organization. In particular, identification of resources with experience evaluating the impact of messaging on the target audience is critical.
Evaluation Focus evaluation criteria on selecting participants that aid in increasing problem awareness rather those that a solely deliver the best or most-refined solution.
  • Increase the number of winners to gain broad exposure and expand the incentives to participate. This can be accomplished without an increase in the purse by expanding the number of “recognized” winners in different categories. While not every winner will receive a monetary prize, this approach will help to engage more participants.
  • Dual outcome guidance with attracting new ideas: Use public voting to engage a broader audience beyond the competitor community. This approach does involve a trade-off, because the final ideas may be of lower quality without vetting by better-informed judges.8
Motivators Develop understanding of the non-economic incentives that drive the target audience participation.
  • Build a profile of the target audience and compare the impact of formal marketing campaigns and a challenge on that audience. This analysis can include discussions with public relations firms, measurement of benefits to running campaigns, and comparisons of the differences in costs across marketing channels.
Structure Pair challenge outcomes with an additional target outcome to maximize reach and impact.
  • Structure the challenge to include additional outcomes or as part of a larger group of challenges. Break the problem into multiple topics, including those concerning further engagement with individuals, organizations, and communities.9
  • Dual outcome guidance with attracting new ideas: Use a multi-round or mini-challenge approach to allow open engagement and exposure to the topic in early rounds and down-selecting the best ideas through the later and final rounds. This will allow designers to reach the competitors most likely to provide high-quality ideas while also expanding engagement across a broader community. For example, NASA’s Zero Robotics Video Challenge uses open eligibility in its first round to capture ideas for a video that promotes the student robotics challenge. In later phases, these ideas are pitched and the winners receive $500 to turn their idea into videos that help raise awareness for the larger robotics challenge.10
Communications Create a multi-channel marketing campaign to account for crowded media markets.
  • Create a plan for publicizing the challenge and the results within the targeted audience to amplify understanding of the problem. This will involve communications through a number of platforms and across partner networks to account for the different methods in which the target audience accesses and internalizes information. Extend marketing beyond a designated website or targeted email communications to other forums including social media and print media campaigns. Designers must also account for regional and international differences in communications and media markets.

Additional examples:

Mobilize action: Spark engagement and build skills

Common pitfalls: Designers should be careful not to believe that recruiting participants into a challenge is sufficient to mobilize action. Getting participants and larger audiences to act typically requires facilitating the formation of new communities. Designers also need the credibility to incent participants to act in new ways. For this, branding and clear messaging are critical.

Design elements Design strategic considerations Tactical guidance
Resources Capitalize on the energy of existing movements, initiatives, and partners to supplement challenge infrastructure.
  • Leverage infrastructure from established communities with a focus aligned with the target outcome, such as conferences and community initiatives. Designers can reduce cost and improve their understanding of target participants by engaging with leaders of initiatives that complement the problem.
  • Select and engage partners that can increase the level and depth of interaction with the communities of participants before, during, and after the challenge. Specifically, identifying and engaging partners with prior success in mobilizing action in communities similar to the target participants is advantageous.
Evaluation Create definitive measures of progress to determine success of the challenge and provide opportunities to revise and improve future challenges.
  • Develop metrics that record progress for each phase of the prize challenge. Metrics must extend beyond participant counts and include ways of measuring the sustainability of relationships or the number of new entrants. Metrics will vary by challenge but core items should include new entrants within target communities of interest and include an assessment of activity/action following the completion of the challenge.
  • Dual outcome guidance with building prototypes or launching pilots: It is critical to maintain rigorous, objective evaluation standards for submitted solutions. Balance the inclusion of more subjective criteria (for example, those rewarding teaming which will be important for commercialization) to link the evaluation of challenge submissions to the goal of mobilizing action.
Motivators Incorporate a high degree of competitor collaboration while recognizing the trade-off between encouraging teams and maintaining challenge.11Incorporate incentives that will build the skills of participants (for example, expert coaching, speaking opportunities, etc.).
  • Use existing challenge participants to recruit new ones during the signup period. Embed recruitment of new participants in the scoring system. Trust that competitors will assist in developing and extended community that will last well after the challenge.
  • Engage independent coaches to serve as a team resource during participant progression throughout the course of the challenge. Assist participants in developing skillsets to mobilize others that are tangentially connected to the challenge after the challenge has completed. Many prize designers use judges to provide the same coaching opportunities, but making coaches separate from the evaluation process may provide added benefits, such as allowing participants to be more candid because they know they are not being judged.
Structure Structure forums for meaningful personal interaction. Mobilizing action requires trust and commitment that may be best suited for direct contact. Structure forums for meaningful personal interaction. Mobilizing action requires trust and commitment that may be best suited for direct contact.
  • Identify a set of possible locations for participants to interact. In-person conferences or meetings can create networking opportunities that also provide incentives for participants to enter the challenge. For example, the US Army’s Federal Virtual Challenge showcased the competitors at an in-person conference where the winners were not only crowned, but also able to network with other colleagues in their field.12
  • Dual outcome guidance with building prototypes or launching pilots: Multiple rounds or mini-challenges can encourage competitors to work with one another and lead to the development of better prototypes. Breaking the challenge into rounds can provide opportunities for judge feedback that can improve competitors’ skills. Build competitor communities by inspiring both challenge (for example, leaderboards) and collaboration (for example, teaming). Striking the right balance is important so that competitors continue to work together after the challenge concludes.
Communications Create an environment to generate a dialogue between participants and the broader community.
  • Develop communications that elicit responses from participants to encourage dialogue around a problem. Unlike challenges focused on ideas, products, or services as an outcome, mobilizing action requires communication between designers, participants, and the public to advance the discussion on the target issue rather than simply relaying information to participants and announcing the winners to the public. For example, designers can send out a weekly question through social media that allows participants to broadcast their progress or provide thoughts and feedback to administrators.

Additional examples:

Inspire transformation: Organize for sustained change

Common pitfalls: Inspiring transformation requires scaling and institutionalizing behavioral change. This can often be achieved through centralized coordination and a top-down approach. On the other hand, transformation can also be achieved through decentralized or grassroots action. Effective designers are aware of both means, do not conflate them, and are intentional about which elements they use to evoke change.

Design elements Design strategic considerations Tactical guidance
Resources Select partners with significant public recognition and the ability to capture attention on a broad scale.
  • Engage partners with strong brands to raise the challenge’s profile and reach a larger audience. Select partners with missions or heavy investment/perspective on the desired transformation. Leverage these partners to engage the targeted participants through their existing networks. Use the combined reach of your partners to create interest at the regional level, national level, etc., by highlighting that the problem is significant enough to bring together a group of preeminent partners.
Evaluation Develop meaningful measures to act as the new basis for discussion and progress around the problem.
  • Use the challenge as an opportunity to refine and define metrics for the entire community and issue area. These metrics can set expectations and encourage sustained behavior change. The challenge can act as the forum for setting a de facto standard on how the issue should be monitored and addressed going forward.
  • Develop measures of success to help communities understand the scope and impact of the challenge. This context can provide a starting point for future marketing and participant interactions throughout the course of the challenge.13
Motivators Engage neutrally viewed surrogates or spokespeople to promote the challenge. Look to reduce potentially divisive politics around the problem being addressed and focus on transforming behavior.
  • Use surrogates and other community leaders to promote the brand of the challenge, broadcast desired outcomes, and motivate participants. Communications from these individuals may be viewed more impartially than messages communicated in an official capacity from the administrators. Also, the use of surrogates or community leaders as the face of the challenge may reduce the politics or emotions surrounding the problem and enable the engagement of a broader audience.
Structure Develop a reoccurring challenge to maximize impact by continuing broad dialogue around the transformation outcome and through progressively more competitive evaluation criteria.
  • Due to the complexity of transformative outcomes, a recurring challenge can allow competitors to continue momentum and understand how their work is moving the field forward. For longer challenges, the design should be restructured to prevent community burnout and sponsor fatigue.14
  • After the first challenge, evaluate which aspects of the challenge to maintain for future challenges and determine potential areas for revision. Change the evaluation criteria to keep it interesting and perhaps more competitive. Test these potential changes with the past participants to understand if new approaches will energize and resonate with those competing.
Communications Plan on a sustained marketing effort that includes traditional and non-traditional marketing tactics.
  • Determine the community you want to reach and the behaviors you want to change. The marketing effort should revolve around this community for a sustained period of time. In order to prevent messaging fatigue with the target community, identify potential viral or guerilla marketing tactics to vary the delivery and impact of the messaging.

Additional examples:

Read the full report on The craft of incentive prize design.

DUP_819_SponsorLogos