2016 State of Bug Bounty - Bugcrowd [PDF]

THE STATE OF BUG BOUNTY Bugcrowd’s second annual report on the current state of the bug bounty economy JUNE 2 0 16

TABLE OF CONTENTS Introduction 3 WHAT EXACTLY IS A BUG BOUNTY?

Executive Summary 5 About the Data Set 6 BUGCROWD PLATFORM DATA PUBLIC DATA SOURCES SURVEY DATA IN THIS REPORT

Market Adoption 8 ACCESSIBILITY OF BUG BOUNTIES PROGRAM GROWTH OVER TIME INDUSTRY DIVERSIFICATION ENTERPRISE ENTERING THE MARKET

Submissions and Vulnerabilities 13 VULNERABILITY RATING TAXONOMY VULNERABILITIES BY CRITICALITY VULNERABILITIES BY TYPE

Bounty Payouts 16 DEFENSIVE VULNERABILITY PRICING MODEL

Researchers 17 AGE AND EDUCATION REGIONAL RESEARCHER ACTIVITY REGIONAL RESEARCHER QUALITY BUG TYPES AND SPECIALIZATIONS RESEARCHER ENGAGEMENT

Conclusion 22

©BUGCROWD INC.

STATE OF BUG BOUNTY REPORT 2015 2

INTRODUCTION What we’re witnessing right now is the maturation of a model that will fundamentally change the way we approach the security, trust and safety of the Internet. Bug bounty programs are moving from the realm of novelty towards becoming best practice. They provide an opportunity to level the cybersecurity playing field, strengthening the security of products as well as cultivating a mutually rewarding relationship with the security researcher community. While bug bounty programs have been used for over 20 years, widespread adoption by enterprise organizations has just begun to take off within the last few. Developing, deploying, and managing secure products presents a massive challenge to all Internet-dependant organizations in 2016. The pressure on short time-to-market continues to increase, and attackers are upping their intensity and resourcefulness to capitalize on security vulnerabilities. Product owners must grow and evolve their vulnerability assessment and identification processes to match their adversaries and keep their users safe. Our second annual State of Bug Bounty Report provides an inside look into the economics and emerging trends of bug bounties, with data collected from Bugcrowd’s platform and other sources throughout 2016. This report is published on a yearly basis for CISOs and other security decision makers to provide a transparent look at the evolving bug bounty market. In this report, you’ll learn more about the bug bounty ecosystem, the researcher workforce, and how modern organizations are tackling their application security challenges with bug bounties.

©BUGCROWD INC.

THE FIRST BUG BOUNTY The first bug bounty program was started at Netscape in late 1995 to find bugs in Netscape’s Navigator 2.0 Internet Browser. The idea of this program was to incentivize the security research community to provide feedback on the Netscape Navigator 2.0 by providing cash rewards to anyone who found bugs in their software. Although the program is noted as one of Netscape’s biggest successes, the bug bounty model did not spread quickly among other software companies. Read the full history of bug bounties >

STATE OF BUG BOUNTY 2016 3

What Exactly is a Bug Bounty? In the past year, the term “bug bounty” has become more well known and widely publicized through popular programs such as Tesla Motors’ car hacking program launched mid 2015, and “Hack the Pentagon.” This uptick in interest is portrayed below.

DEFINING ‘BUG BOUNTY’ A bug bounty is most simply defined as “an incentivized, results-focused program that encourages security researchers to report security issues to the sponsoring organization.” Learn more about how it works > 2005

2007

2009

2011

2013

2015

Figure 1: Google search keyword trends by interest from 2004 depicts an all time peak interest at the beginning of 2016.

As bug bounties have gained traction and evolved to achieve organizations’ security assessment goals, additional variables have been introduced to the basic model. As a business, and for the purposes of the State of Bug Bounty Report, we use the term ‘bug bounty’ more holistically, encompassing programs that can be further classified into the below categories. The majority of today’s bug bounty programs are scoped to web and mobile application targets, although there are several high profile examples of programs run on IoT devices and cars, such as Tesla Motor’s program and General Motor’s program. Other bounties focus on traditional, installable software, including Microsoft’s Bug Bounty program and Google’s Vulnerability Reward Program (VRP). P R O G R A M T Y P E + G OA L

VISIBILITY

INCENTIVE

SCOPE

Vulnerability Disclosure Programs: The primary objective of these programs is to ensure there is a single, public, well-defined channel for security issues.

Public

Recognition (i.e. public leaderboard)

Generally broad, accepting anything that could be considered a security risk

Public Bug Bounty Programs: The organization running the bounty typically interacts directly with researchers to incentivize them to submit vulnerabilities.

Public

Cash, swag, misc. (i.e. airline miles)

Slightly less broad, anything that could be considered a security risk and requires a fix

High cash incentive

Typically more specific scope or focus to encourage testing on a particular aspect of an attack surface - can be either time-boxed, or on an ongoing basis

Private Programs: A more exclusive and more highly incentivized program, often run via a crowdsourcing platform vendor that provides submission vetting and program management.

Private

Differences in the type of program, incentives, time frames, and exclusivity all affect the results of a program. In this report we will address these variables in terms of the various success metrics used by the market.

©BUGCROWD INC.

RESEARCHERS


EXECUTIVE SUMMARY Bugcrowd’s second annual State of Bug Bounty report provides comprehensive data from organizations running bug bounty programs, researchers participating in them, vulnerabilities discovered and rewards, with a specific focus on trends over the past year. Here are some of those top trends...

1 2 3 4 5 6

Public bounties are just the beginning Organizations looking to reap the benefits of a traditional public bug bounty program are utilizing private, on-demand and ongoing, bounty programs more and more. 63% of all programs launched have been private. Jump to this finding.

Bug bounties move beyond just technology companies In nearly 300 programs run, our customer base has diversified from mostly tech companies, to now over 25% of programs launched by more traditional verticals such as Financial Services + Banking. Jump to this finding.

Average priority of submissions increases across all programs We saw an overall increase in average priority per vulnerability, up from what we reported in our last report, with regional differences in average priority. Jump to this finding.

XSS continues to dominate The most commonly discovered vulnerability is still Cross-site Scripting (XSS), which represents over 66% of categorized vulnerabilities disclosed, followed by Cross-site Request Forgery (CSRF). Jump to this finding.

Payouts are on the rise Related to the increased severity of vulnerability submissions, the all time average bug reward on Bugcrowd’s platform has risen from $200.81 in our first annual report, to $294.70, an increase of 47%. Jump to this finding.

“Super hunters” emerge Earning hundreds of thousands of dollars from bug bounties alone, a tier of ‘super hunters’ is emerging, often getting attention from organizations’ security team recruiting efforts. Jump to this finding.

©BUGCROWD INC.


ABOUT THE DATA SET Our inaugural State of Bug Bounty report, released mid-2015, included data from the programs run during an 18-month period between January 1, 2013 and June 30, 2015. This report adds to that data, including figures from programs run from January 1, 2013 to March 31, 2016. This data is analyzed with a specific focus on trends over the last year. As one of the largest sources of vulnerability submission and bug bounty data, we aim to present a novel and impactful view of the current state of crowdsourced security testing, including data from external sources when relevant and necessary.

2015 STATE OF BUG BOUNTY REPORT Our inaugural report launched in June 2015 was the first of its kind and gave a brief overview of the bug bounty economy, including the beginning stages of the bug bounty economy evolution. View the full 2015 report >

Bugcrowd Platform Data Bugcrowd platform data includes program data gathered from January 1, 2013 through March 31, 2016:

• • • •

286 total programs, 64% private and 36% public 54,114 total submissions $2,054,721 paid out across 6,803 paid submissions and additional payments 26,782 researchers as of March 31, 2016

Public Data Sources Since the origin of Bugcrowd, we’ve maintained a list of all public bug bounty programs as a service to researchers. We utilize this list and related data in this report to generate and estimate some figures related to overall market size. We’ve also gathered and estimated other public statistics based on various sources of open source intelligence, as well as data from high profile public bounty programs:

• • • • •

THE LIST Started in 2012 as an open source list of all companies with known public responsible disclosure programs and policies, ‘The List’ now has over 600 entries contributed by the community and maintained by Bugcrowd. View the list >

Google’s Vulnerability Reward Program (VRP) Yahoo’s Bounty Program Facebook’s Bounty Program Microsoft’s Bounty Program Mozilla / Firefox Bounty Program

©BUGCROWD INC.

RESEARCHERS


Survey Data in this Report This year we’ve included survey data from security researchers, as well as organizations engaging in bug bounty programs, through surveys carried out from the end of 2015 to March 2016.

R E SE ARCH E R SU RVEY DATA At the beginning of 2016 we conducted a survey on a subset of our crowd, receiving responses from approximately 500 researchers with experience in all three types of the aforementioned bug bounty programs, and from 51 different countries. This survey data provides context for the growing security research community as well as insight into the potential growth and global sustainability of this economy. Readers should keep in mind that this data is primarily demographic, and is but a subset of Bugcrowd’s community.

39.23% India

62.98% Private Researchers

11.79% United States

37.02% Public Researchers

4.76%

Philippines

4.08%

Pakistan

2.72%

United Kingdom

2.04%

Netherlands

2.04%

Italy

2.04%

Germany

2.04%

Egypt

2.04%

Russia

27.44% Other

Figure 2: Bugcrowd researchers who responded to the 2016 State of Bug Bounty survey fit into one of two categories; researchers participating in solely public programs and researchers who are invited to private programs.

Figure 3: Countries of origin represented in the 2016 State of Bug Bounty researcher survey are representative of the crowd as a whole, with respondents hailing primarily from India and the United States.

COMPANY S URVEY DATA We also surveyed organizations currently engaging in bug bounties, both with Bugcrowd and independent of Bugcrowd, as well as organizations who are not currently utilizing bug bounty programs. In total, we received survey responses from over 600 security professionals from every major industry and company size.

27.55% 5,000 Employees+

43.55% Technology

26.82% 501 to 5000 Employees

8.20%

Finance

18.98% 51 to 500 Employees

8.01%

Professional Services

24.82% 1 to 50 Employees

5.27%

Healthcare

5.08%

Government

4.88%

Education

4.49%

Consumer

3.71%

IT & Security

3.13%

Non-profit

2.54%

Manufacturing

1.82%

Did Not Disclose

11.13% Other

Figure 4: Companies represented in the 2016 State of Bug Bounty survey by number of employees, representing a specific subset of the market.

Figure 5: Companies represented in the 2016 State of Bug Bounty survey by industry, representing a specific subset of the market.

We use this survey data to understand the market in three main areas: the value of bug bounties, challenges in implementing bug bounty programs, and potential for future growth. We find this data is representative of the broader market, with a wide range of organizations and researchers involved.

©BUGCROWD INC.

RESEARCHERS


MARKET ADOPTION Bug bounties, in their traditional form, were originally uncapped “blank check” affairs, introduced by technology giants such as Facebook, Google, Yahoo and a few others. These giants activated marketplaces exchanging security vulnerabilities for cash. The organizations who started this market have spent over $13 million on bug bounty payouts through the beginning of 2016.

BUG BOUNTY GIANTS Today, organizations are able to access the benefits of bug bounties without a blank check, and thus they are being adopted by all types of organizations; from startups to enterprises, and from virtually every industry. Most notably in the past year, organizations such as United Airlines, the United States Department of Defense, Tesla Motors and General Motors started bug bounty programs, garnering attention from worldwide press outlets.

The top paying bug bounty programs are Google VRP, with over $6M paid out over since 2010, Facebook, with $4.3M paid out since 2013, Yahoo, with $1.6M paid out since 2013, Mozilla and Firefox programs having paid out nearly $1M to date since 2010, and Microsoft with over $500K paid out over all their programs.

As they become more accessible, more organizations are starting bug bounty programs, and many more are reviewing the prospect of introducing them.

Read more about these programs >

Accessibility of Bug Bounties From survey data, 37% of respondents work within companies who have either run, or are currently running a responsible disclosure or bug bounty program. Of those programs, over half offered rewards, including swag, while the rest offered recognition only. From those respondents, the top three points of perceived value of bug bounties are the diversity in skill sets and methods used by hackers, the volume of bug hunters testing their applications and the pay for results model. Creative Testing Methods

Volume of Testers

Results Based Rewards Model

Positive Marketing Impact

0%

10%

20%

30%

40%

50%

60%

70%

Figure 6: Top perceived value of bug bounties from survey respondents working in organizations running a bug bounty program, currently or previously.

Additional responses about the value of bug bounties included building a relationship with the security researcher community, positive external marketing, and internal education tools. When asked which roadblocks or apprehensions their organization overcame in order to start a program, the top response was ‘skepticism around quality of results.’ However, when asked what about the quality of the program results compared to other, more traditional methods, 63% stated that the results from crowdsourced programs were better or the same, while just 4% stated that the results were worse. Additionally, 64% reported that they would spend additional or equal resources on their program, while 4% reported they would spend less.

©BUGCROWD INC.

RESEARCHERS


S:

Accessibility of Bug Bounties (cont.) The bug bounty economy is growing rapidly, and yet it still has a long way to go as proven by recent research stating that 94% of companies on the Forbes 2000 list do not currently have a vulnerability disclosure or bug bounty program. Our findings show that most organizations have a fairly comprehensive suite of application testing methods, as expected in a modern security organization. The below graph shows what application security efforts are utilized by companies who are not running a bounty program, 63% of total respondents. Penetration testing Vulnerability scanning Static analysis Application security training Compliance reviews/audits Code review Threat modeling Other 0%

10%

20%

30%

40%

50%

60%

70%

80%

Figure 7: Top application security efforts being carried out by companies represented in survey data.

Of that same data set, 18% stated that their organization is currently reviewing launching a bug bounty program, and of the remaining that are not currently considering it at present, 28% were optimistic about their organization considering it in the future. Additionally, of those reviewing starting a bug bounty program within their organization, the top roadblock was budgeting (71.93%). Of those not currently reviewing a bug bounty program, the top reasons were uncertainty of where to begin (36.78%), internal bureaucracy (34.48%) and insufficient budget or resources (28.35%).

COMPAR E D TO PENET RAT IO N T ESTING Bug bounties are often compared to traditional application security assessment methods such as a penetration testing. The biggest differences between the two are volume of testers involved and the differing reward models. Bug bounties involve a large volume of researchers, as opposed to a select few penetration testers, and utilize a pay for results reward model rather than for effort. The volume and diversity of security researchers participating in bug bounty programs results in a diverse range of bug types, classes and criticality of vulnerabilities, and testing is usually performed without prior knowledge of the target.

©BUGCROWD INC.

RESEARCHERS


Private Programs

Program Growth Over Time In our first annual State of Bug Bounty Report, we outlined the growth of programs on Bugcrowd’s platform. Program growth continues to increase over 210% on average year over year. Additionally, in the 2015 report we highlighted the emerging trend of private programs, surpassing the launch of public programs. This trend continues in its trajectory; as of March 31 2016, 63% of all Bugcrowd program launches have been private programs. 200 Private Public

150

100

50

2013-Q1

2014-Q1

2015-Q1

2016-Q1

Figure 8: Private and public program launches on the Bugcrowd platform from January 2013 to March 2016 show that private launches surpassed public launches and continue to rise in popularity with more velocity than that of public programs.

P R I VATE PROG RAMS Private programs include both on-demand and ongoing programs in which researchers must be invited to participate. In general, we conclude that demand for private programs has continued for three main reasons:

• Organizations looking to start a public bug bounty program begin privately, incentivizing a smaller number of researchers while they build their response capabilities. Over time, the programs become public, allowing everyone to participate. • Organizations looking to access the benefits of crowdsourcing with specific business goals, complex technologies or environments benefit from a smaller testing pool. These organizations pay higher bounties to attract and maintain interest from the top researcher talent.

©BUGCROWD INC.

RESEARCHERS

PRIVATE RESEARCHERS In order to receive invitations to Bugcrowd’s private programs, researchers must score high in all of the following measures: trust, acceptance rate and overall submission quality, finding severity and activity. Private researchers not only have a good track record of adhering to community guidelines and program briefs, but also have a priority rate of better than 4.0, a minimum acceptance rate of 50% and have been active in the past 90 days. More on private program invitations >


Industry Diversification

Industry Diversification The expansion of bug bounties is proven by the growth in overall programs launched over time, as well as the diversity in industries. Out of the nearly 300 programs in the past three years, Bugcrowd has launched bug bounty programs for organizations from nearly every industry. Of all programs launched on the Bugcrowd platform (left), the top two industries represented are Computer Software, including companies like Heroku and Adobe, and Internet, including companies like Pinterest and Indeed.com. Following those industries, most notably, are Financial Services + Banking, Information Technology + Services, Computer + Network Security, and Retail + E-Commerce. These results are in line with public data of all known vulnerability disclosure programs (right). The most notable difference is a smaller portion of traditional industries such as Financial Services + Banking represented. This is likely because more of those organizations often choose to run private programs.

22.30% Computer Software

21.13% Computer Software

21.60% Internet

15.33% Internet

11.50% IT + Services

13.24% IT + Services

10.45% Financial Services + Banking

6.85%

Financial Services + Banking

6.27%

Computer + Network Security

4.76%

Business Services

4.88%

Non-profit

4.76%


3.83%

Retail + E-Commerce

3.27%

Computer Networking

3.14%

Consumer Electronics

3.27%

Entertainment

2.44%

Computer Networking

3.13%

Marketing and Advertising

2.44%

Marketing + Advertising

2.53%

Retail + E-Commerce

21.73% Other

11.15% Other

Figure 9: Breakdown of all programs launched on Bugcrowd’s platform by industry.

Figure 10: Company industries represented in public data of all known public bug bounty programs.

Over time, we’ve seen continued industry diversification on the Bugcrowd platform, with prominent traction from Retail + E-Commerce and Financial Services + Banking over the past twelve months. Overall, organizations from more “traditional” industries have seen year over year growth of over 217% on average, including Financial Services + Banking, Automotive, Healthcare, Education, Telecommunications, Hospitality, Real Estate, Utilities and Consumer Goods. 50 22.30% Computer Software 21.60% Internet 11.50% IT + Services

40

10.45% Financial Services + Banking

30

6.27%


4.88%

Non-profit

3.83%

Retail + E-Commerce

3.14%

Consumer Electronics

2.44%

Computer Networking

2.44%

Marketing + Advertising

11.15% Other

20

10

0 2013-Q1

2013-Q2

2013-Q3

2013-Q4

2014-Q1

2014-Q2

2014-Q3

2014-Q4

2015-Q1

2015-Q2

2015-Q3

2015-Q4

2016-Q1

Figure 11: Programs launched by quarter, broken down by industry shows increasing ‘traditional’ sectors starting more programs, quarter over quarter.

©BUGCROWD INC.

RESEARCHERS


Enterprise Entering the Market In addition to more varied industries adopting the bug bounty model, we’ve seen diversification in the sizes of companies adopting the bug bounty model as well. 9.79%

Most companies launched on the Bugcrowd platform have 500 to 5,000 employees (28.67%), followed by 51 to 200 employees (24.48%).

5,000 Employees+

24.48% 500 to 4,999 Employees 13.99% 200 to 499 Employees 23.08% 50 to 199 Employees 28.67% 1 to 49 Employees

As shown in the figure below, smaller companies were the first to adopt the model, but we are seeing more mid-market and enterprise interest. The enterprise, defined as organizations with 5,000+ employees, account for the fastest growth of program launches on the Bugcrowd platform over the past twelve months.

Figure 12: Company size by number of employees in all time programs launched.

50

40

5,000 Employees+ 500 to 4,999 Employees 200 to 499 Employees 50 to 199 Employees

30

1 to 49 Employees

20

10

0 2013-Q1

2014-Q1

2015-Q1

2016-Q1

Figure 13: Programs launched by quarter, by number of employees, showing that bigger sized companies are adopting bug bounties.

We attribute larger companies’ bug bounty adoption to the evolution of the traditional bug bounty model, and to the popularization of private programs. Private programs are more conducive to organzations with more compliance requirements such as the Payment Card Industry Data Security Standard (PCI DSS) and Sarbanes Oxley (SOX), while retaining the integrity of the bug bounty model and delivering the value of the crowd.

©BUGCROWD INC.

RESEARCHERS


SUBMISSIONS AND VULNERABILITIES We have received 54,114 submissions on the Bugcrowd platform between January 1, 2013 and March 31, 2016. Of those total submissions, 24,516 (45.38%) were marked invalid and 19,574 (36.23%) were marked duplicate. Valid, non-duplicate submissions account for the remaining 9,963 submissions, resulting in a signal-to-noise ratio of 18%. 60K Total Submissions (cumulative) Total Valid Submissions (cumulative)

50K

Non Duplicate Valid Submissions (cumulative)

40K

30K

20K

10K

0 2013-Q1

2014-Q1

2015-Q1

2016-Q1

Figure 14: Cumulative submission data broken out by all submissions, all valid submissions, and all valid non-duplicate submissions.

The signal-to-noise ratio significantly affects the total ownership cost of a program. The more time an organization spends on processing submissions that don’t produce a signal, the more overhead they experience in the program. Private programs overall have a much better signal to noise ratio of 29% compared to 13% in public programs. Market education, guidance and standardization will be key for both researchers and organizations as the bug bounty model matures, encouraging more high value and valid submissions, better communication with one another and aligned expectations.

Vulnerability Rating Taxonomy In 2016, Bugcrowd released its Vulnerability Rating Taxonomy (VRT) to help align expectations about the criticality of a bug bounty submission. Bugcrowd’s Technical Operations team follows the VRT, using it to rate the technical priority of submissions. Once Bugcrowd receives a submission, a VRT submission type and technical priority level are assigned, allowing the program owner to quickly understand the urgency of a submission. The criticality scale for a submission ranges from Priority 1 (P1) to Priority 5 (P5), 1 being the most critical, 5 being the least critical. This scale provides researchers and organizations a baseline for prioritization of a fix and potential reward amount.

©BUGCROWD INC.

RESEARCHERS

VRT EVOLUTION The VRT is a living document that changes regularly, so specific submission ratings and notes are frequently updated. To understand all considerations, implications and use cases for this document, download a copy of the VRT and read the accompanying report. Get the full guide >


Priority up

Vulnerabilities by Criticality To measure the health of individual bounty programs and the bug bounty economy as a whole, we have tracked the average priority of a bug across submissions. The current average priority of all time submitted bugs is 3.75, better than it was twelve months ago: 3.88. Lower impact issues (i.e. P5s) are easier to find and can often be discovered by automated vulnerability scanners, thus researchers are often discouraged from submitting them. On the flip side, more critical vulnerabilities are much harder to find. For that reason, P1 and P5 vulnerabilities account for a smaller portion of valid submissions.

V U LNE R ABI LIT IES BY PRIORIT Y:

2.81%

• P1 - Critical: Vulnerabilities that cause a privilege • • • •

escalation from unprivileged to admin or allow for remote execution, financial theft, etc. P2 - High: Vulnerabilities that affect the security of the platform including the processes it supports P3 - Medium: Vulnerabilities that affect multiple users and require little or no user interaction to trigger P4 - Low: Vulnerabilities that affect singular users and require interaction or significant prerequisites to trigger (MitM) to trigger P5 - Acceptable Risk: Non-exploitable vulnerabilities in functionality. Vulnerabilities that are by design or are deemed acceptable business risk to the customer

P1

14.62% P2 24.95% P3 45.23% P4 12.39% P5

Figure 15: All valid submissions, including duplicates, by priority.

Since we started to disincentivize P5 submissions, they have accounted for less of all submissions over time. Higher impact submissions are highly incentivized on Bugcrowd’s platform and reports are rising in frequency. As more P1s and P2s are submitted over time, coupled with less P5s over time, we’ve seen the average priority of a bug get significantly better. 5K P1 P2 P3

4K

P4 P5 3K

2K

1K

2013-Q1

2014-Q1

2015-Q1

2016-Q1

Figure 16: Submission volume by priority over time shows a steady decrease in lower priority submissions and an increase in higher priority submissions.

©BUGCROWD INC.

RESEARCHERS


XSS

Vulnerabilities by Type Cross site scripting (XSS) and Cross Site Request Forgery (CSRF) are still the top vulnerability submissions to all Bugcrowd programs, which is consistent with other publicly available bug bounty data. 38% of all valid and duplicate submissions fall into the category of XSS, CSRF, mobile, SQLi and clickjack. Uncategorized bugs omitted, XSS vulnerabilities account for 66% of valid submissions, followed by 20% categorized as CSRF. Read more about the prevalence of XSS, as well as other bugs not classified in this data in our October 2016 Addendum (pages 23-25 of this report). 25.35% XSS

66.24% XSS

7.54%

CSRF

19.71% CSRF

3.37%

Mobile

8.79%

Mobile

1.39%

SQLi

3.64%

SQLi

0.62%

Clickjack

1.62%

Clickjack

61.72% Other

Figure 17: Bug types across all valid submissions including uncategorized submissions.

Figure 18: Bug types across valid submissions excluding uncategorized submissions.

Consistent with the VRT, our ranking system discourages issues with low technical impact such as clickjacking, thus we’ve seen a decline in clickjacking and other low-impact reports over time. 3K

2K XSS SQLi Mobile CSRF Clickjack

1K

0 2014-Q1

2015-Q1

2016-Q1

Figure 19: Bug types across valid submissions shows a decline in low value bug types such as clickjacking, and steady submissions in XSS and mobile bugs.

©BUGCROWD INC.

RESEARCHERS


BOUNTY PAYOUTS Payouts Bugcrowd has paid out $2,054,721 across 6,803 paid vulnerability submissions and additional payments as of March 31, 2016. Total payout distribution is skewed heavily to a small portion of the crowd, signifying the strength and success of ‘super hunters’ which we will address in the following section. $120K

$100K

TOP BUG PAYOUTS

$80K

On the Bugcrowd platform, multiple top payouts of $15K have been made in the past year, up from last year’s top payout of $10K.

$60K

$40K

$20K

Unique Paid Submitters Figure 20: Distribution of cumulative payments cross individual paid submitters.

Beyond the distribution of payouts to individuals, we look at the average payout per bug as a metric of health for the entire bug bounty economy. The all time average payout of a bug is currently $294.70, up 47% from the all-time average reported in the 2015 State of Bug Bounty Report, $200.81. $600

Notice that as the average payout begins to stabilize at the beginning of 2015, we notice a significant upward trend quarter over quarter. In early 2016, we published our first ever Defensive Vulnerability Pricing Model to provide guidance to organizations considering a bug bounty on how to budget and reward submissions. This guidance is beginning to pay off; the average bug payout in just the first quarter of 2016 was at an all time high of $505.79.

$500

$400

$300

$200

$100

2013-Q1

2014-Q1

2015-Q1

2016-Q1

Figure 21: Average payout per bug shows steady increase quarter over quarter.

Defensive Vulnerability Pricing Model The Defensive Vulnerability Pricing Model (DVPM) first focuses on security maturity based on four variables: security philosophy, people, processes, and technologies. Based on the security maturity , an organization may be classified as ‘basic,’ ‘progressing’ or advanced.’ Using technical priority and an organization or application’s security maturity level, the DVPM helps organizations determine what a bug is worth. The DVPM set the first ever market rate for bug bounties, guiding and setting expectations for both researchers and organizations.

©BUGCROWD INC.

RESEARCHERS

DVPM EXAMPLE : According to the DVPM, a P4 submitted to an organization on the ‘basic’ end of the security maturity scale should be rewarded $100, while a P1 submitted to an organization on the ‘advanced’ end of the security maturity scale should be rewarded $15,000. Read the full guide >


RESEARCHERS As of March 31, 2016, 26,782 researchers had signed up on the Bugcrowd platform. Notably, 41% of those researchers signed up in the past twelve months alone, signifying steady community growth year over year. 30K

25K

20K

15K

10K

5K

0 2013-Q1

2014-Q1

2015-Q1

2016-Q1

Figure 21: Cumulative researcher sign-ups quarterly since January 2013.

Age and Education Of those who responded to our survey, 75% of researchers were between the ages of 18 and 29 followed by the second largest age group, aged 30 to 44, representing 19% of respondents. Additionally, 88% of them have at least one year of college under their belts. 55% of them have graduated with a bachelor’s or postgraduate degree. All respondents had at least a high school degree.

80%

60%

42.14% College Degree 28.70% Some College 12.76% Graduate Degree 12.30% High School Degree

40%

4.10%

Some Graduate School

20%

0%

< 18

18 - 29

30 - 44

45 - 59

Age in Years Figure 22: Researcher survey data showing the distribution of ages represented amongst researchers.

©BUGCROWD INC.

Figure 23: Researcher survey data showing level of education completed.

RESEARCHERS


Regional Researcher Activity Bugcrowd researchers hail from 112 different countries; activity and quality varies by region. The vast majority of researcher sign ups are from India (28.2%) and the United States (24.4%), followed by the United Kingdom (3.9%), Pakistan (3.5%) and Australia (2.4%). In terms of submission volume, however, the top ten submitting countries are, in order of highest submission volume to lowest, India, the United States, Pakistan, the United Kingdom, the Philippines, Germany, Malaysia, the Netherlands, Australia and Tunisia. Over time, we have seen steady submissions from India and the United States, with a steady decrease in submissions from Pakistan. 10K

8K

India

6K

United States Pakistan United Kingdom Philippines

4K

Other

2K

0 2013-Q1

2014-Q1

2015-Q1

2016-Q1

Figure 24: Submission volume by geographical location quarter over quarter.

We analyze activity and quality trends primarily in terms of total submissions, valid submissions and total amount paid. Total submission volume (left) versus total paid out (right) show differences in quality by region.

43.04% India

35.51% India


12.94% Portugal

11.50% Pakistan


3.87%

United Kingdom

12.13% United Kingdom

3.28%

Philippines

6.52%

Malaysia

1.63%

Germany

2.67%

Ukraine

1.52%

Malaysia

2.47%

Philippines

1.31%

Netherlands

2.18%

Pakistan

1.30%

Australia

1.68%

Netherlands

1.23%

Tunisia

1.40%

Germany

10.35% Other

18.53% Other

Figure 25: Breakdown of total submission volume by geography.

Figure 26: Breakdown of total payment volume by geography.

India ranked first for total submission volume as well as total amount of money paid out. Notably, Portugal is the second ranked country for payout volume without making the top ten submitting regions by total volume. Additionally, while countries like Australia and Tunisia are in the top ten for submission volume, they don’t make it in the top ten for total money paid out. This signifies that researchers in those regions contributed less valid submissions that resulted in payment.

©BUGCROWD INC.

RESEARCHERS


Regional Researcher Quality The above trends are also reflected in regional submission priority, as depicted by analyzing the breakdown of vulnerabilities by priority. The top five P1 submitters by region are the United States (33.81%), India (13.12%), Portugal (7.42%), the United Kingdom (6.42%) and Germany (3.99%). 37.44% India 19.60% United States 12.04% Pakistan 2.38%

United Kingdom


2.17%

Tunisia

13.12% India

2.14%

Hong Kong

7.42%

Portugal

1.96%

Philippines

6.42%

United Kingdom

1.25%

Germany

3.99%

Germany

1.22%

Australia

3.42%

Russia

1.16%

Netherlands

3.28%

Netherlands

3.00%

Canada

3.00%

France

2.28%

Australia

18.65% Other

20.26% Other

Figure 27: P5 submission volume broken down by geography.

Figure 28: P1 submission volume broken down by geography.

Looking at top submitting regions, one measure of quality is average vulnerability priority. Remember, the lower the average numerical priority, the better, meaning more critical vulnerabilities out of total submissions. Of the top fifteen submitting countries, Portugal has the top average priority of 2.91, compared to the overall average submission priority of 3.75 across all submissions.

3.97

India

3.65

United States

‘SUPER HUNTERS’ 4.58

Pakistan

As reported previously, a vast majority of payouts go to a select group of individuals. The top ten paid out researchers have made, collectively, 23% of total payouts.

3.28

United Kingdom

3.67

Philippines

3.58

Germany

These individuals from around the world have made names for themselves, garnering attention from the security researcher community.

3.42

Malaysia

3.34

Netherlands

3.60

Australia

Some, from less expected regions, have been so consistent and successful, they have put their entire countries on our radar. For example, Portugal’s success is from just a few researchers.

4.57

Tunisia

3.14

France

3.40

Russia

Super hunters, although not a entirely new phenomenon, are making more money than ever, as more complex and high profile bounty programs launch with higher stakes.

4.07

Egypt

2.91

Portugal

3.57

Italy

Meet Bugcrowd’s top bug hunters > P1

P3

P4

P5

Figure 29: Average priority by top submitting regions depicts above average priority in European nations such as Portugal, France, United Kingdom and Netherlands, and below average priority from India, Pakistan, Tunisia and Egypt.

©BUGCROWD INC.

RESEARCHERS


Bug Types and Specializations Another way we measure and segment researchers is by skillsets or specializations. Although there are less variances in bug type by geography, we notice a few key trends. While volume isn’t represented in the graph below, the breakdown of bug types of the top ten submitting region reveals key trends. SQLi, which is often categorized as a high value vulnerability, accounts for the greatest percentage of all vulnerabilities submitted by researchers in Tunisia, Portugal, and the United States. Similarly, Tunisia, Pakistan, India and the Philippines account for the most clickjack vulnerabilities, often categorized as a low value vulnerability. Additionally, mobile submissions account for the biggest portion of submissions from Australia and the United States. XSS

India

SQLi

United States

Mobile

Pakistan

CSRF

United Kingdom

Clickjack

Philippines Germany Malaysia France Netherlands Tunisia Russia Portugal Australia 0%

100%

Figure 30: Top submitting countries’ total valid submission volume by percentage of each bug type, excluding unclassified bugs.

When asked which technologies they had intermediate to advanced skill in, 95% of respondents of our aforementioned survey felt they had intermediate or advanced knowledge of web application testing, 48% in Android, 28% in iOS and 15% in IoT. While the Bugcrowd community is made up of security researchers with expertise across numerous technologies, accessibility, complexity and opportunity contribute largely to these responses. Web App APIs/Web Services Code Review Mobile App - Android Network Infrastructure Linux Desktop/Server software Mobile App - iOS Reverse Engineering Network Appliance Malware Analysis IoT/hardware Mobile App - other Mobile OS/Baseband SCADA 0%

25%

50%

75%

100%

Figure 31: Survey data showing researchers’ advanced and intermediate level skill sets.

As more organizations utilize bug bounties in different technologies and applications and education around complex technologies becomes more accessible and available, we foresee diversification of depth and breadth of skills to be manifested in the crowd.

©BUGCROWD INC.

RESEARCHERS


Researcher Engagement Based on survey responses, we discovered that just 15% of researchers participate in bug bounty programs full-time, with an additional 31% hoping to participate full-time in the future. Thus, 85% of Bugcrowd researchers participate in bug bounty programs as a hobby or view it as a part-time job currently, from which we infer their primary source of income is independent from bug hunting. Additionally, 70% spend fewer than 10 hours a week working on bounties. 50%

40% 15.00% Yes 54.09% No, part time only

30%

30.91% No, but hopefully someday 20%

10%

0

0-5

6 - 10

11 - 20

21 - 30

31 - 40

40+

Time Spent Per Week in Hours Figure 32: Survey responses to the question “Do you bug hunt full time?”

Figure 33: Average hours a week spent participating in bug bounty programs.

From our experience, while we’ve seen multiple super payouts of $10K and $15K, and ‘super hunters’ making anywhere from $9K to $20K monthly, most of the crowd is bug hunting as a secondary source of income. In other words, the majority of bug hunters are employed, making their living independent of bug hunting, oftentimes as developers, security engineers or penetration testers. Furthermore, when asked how much money they would need to make in order to do security research full time, respondents’ answers varied greatly, representative of the geographical diversity and differences in regional average incomes. 43% of respondents would need to make $0 to $50,000 while nearly 30% would need to make more than $100,000. $0 - $24,999 $25,000 - $49,999 $50,000 - $74,999 $75,000 - $99,999 $100,000 - $124,999 $125,000+ 0%

5%

10%

15%

20%

25%

Figure 34: Survey responses of how much they would need to make, annually to bug hunt full time.

Researchers are motivated by a range of incentives, extrinsic and intrinsic, from prestige or profit, to philanthropy or professional development. As the community grows and we learn more about it, we leverage these motivations to better assist this flourishing marketplace. While there is certainly more money becoming available in this marketplace, as proof of this report, Bugcrowd also has the unique opportunity to continue supporting the crowd. This community will be forever evolving and growing, and we will continue to analyze and report on the state of bug hunters and the security research economy.

©BUGCROWD INC.

RESEARCHERS


CONCLUSION In 2016, bug bounty programs are emerging as a key component of organizations’ security programs. The bug bounty path, paved by tech giants, is widening, enabling security teams of all sizes to create and manage robust application security assessment programs, get ahead of adversaries, and level the cybersecurity playing field. While we are clearly still in the early- to mid-adopter phase of this new market, this report proves that bug bounties are gaining momentum and evolving to meet those needs. Case-in-point, with over $2M paid to researchers and velocity increasing with a current average reward of over $500, bug bounty programs are becoming very lucrative for researchers around the world. Researchers are building their reputation and obtaining access to private programs, allowing them to earn even more. With more Financial Services + Banking organizations adopting the model and starting programs, we predict that rewards will continue to rise both in frequency and size. Bugcrowd is here to help you get started. If you’re interested in starting a program or learning more, we encourage you to reach out to us at https://bugcrowd.com.

©BUGCROWD INC.

ABOUT BUGCROWD The pioneer and innovator in crowdsourced security testing for the enterprise, Bugcrowd harnesses the power of more than 30,000 security researchers to surface critical software vulnerabilities and level the playing field in cybersecurity. Bugcrowd also provides a range of responsible disclosure and managed service options that allow companies to commission a customized security testing program that fits their specific requirements. Bugcrowd’s proprietary vulnerability disclosure platform is deployed by Drupal, Pinterest, Western Union and many others. Based in San Francisco, Bugcrowd is backed by Blackbird Ventures, Costanoa Venture Capital, Industry Ventures, Paladin Capital Group, Rally Ventures and Salesforce Ventures.


ADDENDUM

VOLUME AND IMPACT OF CROSS-SITE SCRIPTING IN BUG BOUNTIES SEPTEMBER 26, 2016 As of March 31, 2015, we reported that over 25% of all valid submissions were categorized as Cross-Site Scripting vulnerabilities, following the largest category of vulnerabilities, ‘Other’ (62%). Since releasing our second annual ‘State of Bug Bounty’ report in June, we’ve received inquiries about this statistic. This addendum will address those questions, as well as... • Provide context around the volume and velocity of XSS over the past 10+ years • Explore the potential impact XSS can have • Analyze our general state of our vulnerability data and other high impact vulnerabilities

Current State of Cross-Site Scripting Before we address the potential impact of XSS, it’s important to provide context around the current perception of XSS in the vulnerability disclosure space. The frequency and persistence of XSS in headlines, POCs and vulnerability databases over the past 10+ years have created ‘XSS-fatigue.’ The security community–consultants and researchers–have been submitting XSS in reports for a long time. But instead of proving the impact of an XSS vulnerability and exploiting a full attack by taking over accounts etc., the industry has standardized on a JavaScript pop-up or prompt box. In short, the industry has written off XSS as low-hanging fruit, partially because it has been around for so long, and also because bug reports are downplayed. In reality, however, this isn’t how it happens in the real world. There are many exploit frameworks that rely on XSS, including ransomware attacks, nation-state attacks, and more. While it is true that some XSS vulnerabilities have little notable impact, that is certainly not always the case. Bug bounties are here for when that isn’t the case.

Cross-Site Scripting Impact Our vulnerability data shows that 27% of all valid XSS were classified as P1 or P2, followed by 29% P3, which is to say not all XSS are ‘low hanging fruit.’ Our Vulnerability Rating Taxonomy (VRT) has multiple classifications for XSS, capturing priority variations for XSS within applications with multiple user privilege levels. Stored Cross-Site Scripting is listed as a Priority 2 (P2) vulnerability when privilege escalation can occur from non-admin to anyone, and a Priority 3 (P3) vulnerability for XSS when privilege escalation can occur from admin to anyone. When coupled with the appropriate business impact, XSS can, and has been, classified and rewarded as a Priority 1 (P1) vulnerability as explored in the following section. Generally, XSS can have critical business impact in two instances; when the attack vector is unusual or obscure, or when the users involved or business impact are notable.

©Bugcrowd inc.

RESEARCHERS


XSS in Bug Bounties This longevity and persistence is unique to XSS due to the technical cause of the vulnerability, as well as how difficult it is to avoid it. What is more–it shows no signs of being resolved anytime soon. It’s also important to note that most organizations don’t start a bug bounty program, public or private, without already having a mature application security program in place. The vast majority of our customers have already run various vulnerability scanners against their attack surface, as well as penetration tests. Most scanners historically fail to pick up many of these XSS findings–the power of human creativity cannot be replicated by scanners. Bug bounties are often useful to find ‘the unknown unknowns.’ Below are two specific examples XSS bugs submitted through a Bugcrowd bug bounty program with particularly high impact.

EXAMPLE 1: BLIND XSS The Company: Communications company The Bug: Blind XSS is a variant of XSS in which the attacker injects the script payload via a form that doesn’t immediately indicate to the attacker that the script was executed. In this specific example, the researcher injected the payload via the company’s interactive support chat. Eventually, this payload was executed by an employee of the company who viewed the other end of the chat conversation, allowing the researcher to thereby control a browser within the company’s internal network. Impact: In this scenario, the researcher was able to bypass several security mechanisms protecting the company’s internal network by executing a payload within the help desk technician’s internal browser. This kind of access to an internal network – especially with a privileged user account – is often the entry point for a much larger-scale network compromise. While some think that Blind XSS is just a form of advanced phishing, this is actually not the case, as the root cause is that the website itself allowed an attacker to send and ultimately rendered the attacker’s script payload. All websites – whether externally or internally facing – should always perform proper input validation and output encoding.

EXAMPLE 2 : PRIVILEGE ESCALATION AND REMOTE CODE EXECUTION VIA XSS The Company: E-commerce company using multi-tenant content management systems (CMS) The Bug: In this example, a researcher found an XSS that would allow an attacker to force administrative users to promote nonadmin users to an admin role. Privilege escalation is common in CMSs with multiple roles within an application, and in this instance, the admin had virtually unlimited access within in the CMS. After using XSS to allow for silent and automatic account upgrading to admin, the attacker was then able to to use his new account to inject more code, including adding a web shell to the server to take over the server. Impact: Some will not accept this bug because they believe it requires a phish on the part of the admin, which is unlikely to happen, but in most cases, admins are just as susceptible to phishing and XSS as anybody else. In this case, the attack in the JavaScript was so cleverly crafted with CSS modification and left all the hooks and codes in the background, so the admin never even knew what was going on. This wouldn’t have been possible without XSS.

©BUGCROWD INC.

RESEARCHERS


For additional examples of high-impact Cross-Site Scripting vulnerabilities, listen to our recent ‘Big Bugs’ Episode, XSS Fatigue, with Jason Haddix, Bugcrowd’s Head of Trust and Privacy.

‘Other’ Bugs As stated in this report, 62% of all bugs fall into the but type category of ‘Other.’ In writing this report, we included our vulnerability data as-is to provide an accurate, albeit simplified, snapshot of our vulnerability data while we work to classify all submissions into more granular categories. This will minimize vulnerabilities listed in ‘Other’ which actually holds a multitude of vulnerability types, providing richer, more meaningful data to the market as a whole. The goal in providing categories or classes of vulnerability types is to better understand the current state of web applications, mobile applications, and connected devices. Again, it’s important to keep in mind that the vulnerabilities we are finding are those that have been missed by scanners, or overlooked by penetration testers, which is why it is notable to call out that XSS and CSRF are such a big percentage of valid vulnerabilities, behind our unclassified portion. So what kinds of bugs are in this category? As one can imagine, it varies greatly, but notable bug types include IDOR, XXE, RCE and more. Below are just a few examples of ‘Other’ bug write-ups with significant impact.

A FEW EXAMPLES • Blackphone remote memory corruption vulnerability • Netgear remote code execution (RCE) • Aruba Reversing Firmware

Addendum Sources and Additional Resources • • • • • •

http://www.thesecuritypractice.com/the_security_practice/2010/11/how-cross-site-scripting-was-discovered.html https://www.owasp.org/index.php/Cross-site_Scripting_(XSS) http://techbeacon.com/sites/default/files/gated_asset/hpe-cyber-risk-report-2016.pdf https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Taking_still_photos https://www.owasp.org/index.php/XSS_Filter_Evasion_Cheat_Sheet https://www.owasp.org/index.php/XSS_%28Cross_Site_Scripting%29_Prevention_Cheat_Sheet

©BUGCROWD INC.

RESEARCHERS
