How It's Made
The Instacart Bug Bounty Program – How We Work with White Hat Hackers to Secure Instacart
Authors: James Cha, Vickie Li, Shashank Mirji, and Frank Filho
Let’s face it, there is no such thing as 100% bug-free code. And there is particularly no such thing as 100% security. That’s why a security program that catches security vulnerabilities is an important part of a well-rounded engineering organization.
Instacart’s Security Team works to mature scalable tooling to build guardrails for the Instacart engineering team. However, security blindspots inevitably exist — or at least, savvy security teams have to assume so. Our bug bounty program acts as the canary mechanism to let us know what we don’t know and what escaped our preventative/detective measures.
What does it take to run a standout Bug Bounty program?
Running a successful bug bounty program involves many moving parts, and organizations often struggle to maintain the balance between the stakeholders of a bug bounty program — researchers, bug bounty platform, and security engineers.
Over the past few years, Instacart’s bug bounty program has grown exponentially in terms of reports and payout amounts. As Instacart grows, we needed a way to grow our bug bounty program to keep pace with our engineering infrastructure. This article is about how Instacart matured our bug bounty program over the years. And our surveys have shown that researchers really enjoy hunting on our program! Based on our recent survey results, about 80% of our researchers would recommend the Instacart’s Bug bounty program to other researchers/friends.
We built a successful bug bounty program by:
- Introducing broad and well-defined scopes. So that the researchers can make the best use of their time and let their creativity out for chaining the found vulnerability across multiple Instacart domains.
- Helping researchers succeed (and save time!) by providing recon data like the API documents to the researchers and giving priority/impact rating on the in-scope domains.
- We payout competitively by benchmarking with the highest paying programs
- Rewarding fairly and transparently. We explain to the researcher why we decided on a payout amount
- Rewarding and retaining top performers by treating them well. We issue bounties fairly and honor their concerns
Building a Boutique VIP Invite-only Program
We run two bug bounty programs:
- A public program open to everyone
- VIP program for invited researchers only
Both programs are hosted on HackerOne. The scope for both programs is the same and includes most of Instacart’s core assets, including our web platforms, mobile apps, APIs, and any system or service within *.instacart.com or *.instacart.tools. Our public program is one of the top paying programs on HackerOne and is open to all researchers.
However, we invite the best performing researchers on Instacart’s public program to join the VIP program, through which they are provided with access to internal resources like API documentation and DNS zone file records to speed up their technical research.
These researchers have typically reported a high-impact finding before and have been working on our program over the years. These researchers help look deep into our technology stack and understand the business logic very well.
To acknowledge the extended effort and expertise in our platform, the maximum reward amount for the high and critical severity finding are increased significantly compared to the public program.
Bug Bounty Program Progress
Over the past two years, we have received 1400+ submissions and rewarded over $500,000 in bounties since 2020. We launched a VIP program in 2021 to attract top-tier researchers, resulting in the program becoming the 2nd highest ranked VIP program on HackerOne, with total bounty payouts above $200,000. We saw a 300% increase in the critical & high severity security bugs reported in the last two years thanks to increased participation in the program.
Here’s a summary of the number of submissions we received and the payout amount since 2018:
2021 was our biggest year for bounties; we experienced a 300% increase in the critical & high severity security bugs reported. This was due to the launch of our VIP program and the overall increase in the bounty amounts. The number of active researchers within our programs also increased by over 30% in 2021. We continue to see a similar trend in 2022, and based on our forecast, we anticipate rewarding around $250K+ bounties by the end of the year.
How we reward bounties
At Instacart, we aim to value our researchers by rewarding bounties fairly. When it comes to determining how much the bounty payout amount should be, the security team first looks at the overall severity of the bug found to determine the minimum and maximum payout amount.
The overall severity of the risk is calculated by examining the likelihood and impact of the vulnerability reported. Simply put, the easier the bug is to exploit, and the more severe its potential impact, the higher the reward.
We usually use the vulnerability’s bug class to guide bounty decisions. Each vulnerability type has a typical severity level; for example
These are all guidelines we use to make an objective decision, and sometimes submissions are more nuanced and require adjustments to our standard policy. As a rule of thumb, for non-standard submissions, the bigger the impact, the bigger the payout. For example, to determine payouts, we may use criteria like whether the finding results in a code or configurational change or the scale of the theoretical amount of data that might be exposed through the vulnerability.
To calibrate further, we also look at historical payout amounts for a vulnerability of the same category and impact statement. To be fair to all researchers working on our bounty program, we sometimes retroactively pay out additional boun ties when we find out about the bug’s impact later on or if we have re-calibrated the payout amount of a similar bug.
For example, we used to pay $1,000 for an unauthenticated Cross-Site-Scripting (XSS) bug and $3,000 for an authenticated XSS bug. But recently, we decided that those two bug types have a very similar, if not the same, impact. So we retroactively went back to the former bug reportings and matched them as $3,000.
However, sometimes it’s unclear how severe the finding could be. In that case, we will use the Common Vulnerability Scoring System (CVSS) Calculator to calculate a bug’s severity. This calculator is used to measure the severity of a vulnerability by carefully classifying its likelihood and impact.
During our weekly bug bounty meeting, we discuss all of the above as a group, try to sum up all the potential impact the finding can have, and agree upon the payout amount. This is when the security team members conduct a technical impact assessment of the bug found and try to explore further escalation or exploitation paths that the researcher may not have been aware of. In many cases, researchers may receive a much higher amount than expected.
Bug bounty guide
The security team at Instacart strives to create a rewarding, fair, and exciting bug bounty program for our researchers.
In our efforts to make our program more researcher friendly, we aim to make hunting and reconing easier for researchers in the future. In September 2019, we launched a bug bounty guide to our private VIP program. Instacart’s Bug Bounty Guide is a collection of hand-written resources with a high-level overview of how our systems work and automatically generated API documentation of our endpoints, expected inputs, and outputs.
Justin Gardner (@Rhynorater), based in Japan, leads the leaderboard and was awarded our highest bounty.
“I’ve really enjoyed working with the Instacart team — Not only do they offer great bounties, but they also triage quickly and issue bounties not long after. There are also great things waiting for you if you happen to make it into the VIP program!”
Contributors to the Bounty Program — Shoutouts!
Ankit A, David G, Frank F, James C, Kyriakos G, Shashank M, Tim L, Vickie L.
Most Recent in How It's Made
How It's Made
One Model to Serve Them All: How Instacart deployed a single Deep Learning pCTR model for multiple surfaces with improved operations and performance along the way
Authors: Cheng Jia, Peng Qi, Joseph Haraldson, Adway Dhillon, Qiao Jiang, Sharath Rao Introduction Instacart Ads and Ranking Models At Instacart Ads, our focus lies in delivering the utmost relevance in advertisements to our customers, facilitating novel product discovery and enhancing…...Dec 19, 2023
How It's Made
Monte Carlo, Puppetry and Laughter: The Unexpected Joys of Prompt Engineering
Author: Ben Bader The universe of the current Large Language Models (LLMs) engineering is electrifying, to say the least. The industry has been on fire with change since the launch of ChatGPT in November of…...Dec 19, 2023
How It's Made
Unveiling the Core of Instacart’s Griffin 2.0: A Deep Dive into the Machine Learning Training Platform
Authors: Han Li, Sahil Khanna, Jocelyn De La Rosa, Moping Dou, Sharad Gupta, Chenyang Yu and Rajpal Paryani Background About a year ago, we introduced the first version of Griffin, Instacart’s first ML Platform, detailing its development and support for end-to-end ML in…...Nov 22, 2023