Preview: Envisioning the Future of Grocery Shopping with GPT-4V

AI has long been at the heart of everything we do, and we’re always looking to partner with AI industry leaders and leverage the latest AI advancements to deepen the relationship people have with food and grocery shopping. Our early access and adoption of the new vision-enabled GPT-4 Turbo with Vision (GPT-4V) is the latest such example. Today, we’re sharing a preview of what we’re cooking up next: upgrading our existing AI-powered Ask Instacart search feature with this new vision capability. Soon, in addition to being able to ask food-related questions in natural language, Instacart users will be able to use Ask Instacart to convert handwritten recipes and shopping lists directly into digital, shoppable item lists in the Instacart app. The vision-enabled Ask Instacart will decipher ingredients and quantities in a single step so that our users don’t have to do the mental translation and manually look for one item at a time.

Here’s how it will work:

• 📸 Snap: Simply open the photo feature in your Instacart app and snap a photo of your list– perhaps it’s a family recipe for Thanksgiving apple pie lovingly handwritten by Grandma, or a scribbled shopping list of groceries you’ve jotted down throughout the week.  Don’t worry if the handwriting isn’t the cleanest; GPT-4V’s language capability takes into account context from other legible words to help clarify the meaning of any item that isn’t clear.

• ☑️ Select: Thanks to GPT-4V, the Instacart app will analyze the image, recognize the text, and convert it into a comprehensive, shoppable list of items.  From there, you can select specific ingredients you need, or de-select any ingredients you might already have at home– like cinnamon, in the case of Grandma’s apple pie.

• 🥧 Savor: With your new digital shopping list, you can instantly place an order and get all of the ingredients or grocery supplies you need delivered straight to your door in as fast as an hour– leaving you more time to savor the fruits of your efforts. 

We hope to launch this feature in the Instacart app to a small number of customers in the next few weeks and more widely in the coming months. This feature is the latest in a set of generative AI-enabled shopping experiences we’re building to help people more easily answer the eternal, “What’s for dinner?” question. From the Instacart plugin for ChatGPT to Ask Instacart, which lets people ask open-ended food questions in natural language– like, “What are dairy-free snacks for kids?” or even, “What should I make for dinner if I have kale and cheddar in my fridge?”–we believe that AI can help us create personalized, inspirational, value-driven shopping experiences that enrich peoples’ relationship with food and the retailers and brands they love. Now with our early adoption of GPT-4V, this new vision-enabled feature promises to be a big help to anyone following a recipe or looking to save some time tackling their grocery shopping list.

JJ Zhuang

JJ Zhuang is Chief Architect at Instacart, where he drives key technology and architecture decisions across all Instacart product pillars and ensures engineering investments are aligned with the company’s long term business strategy. Prior to Instacart, Zhuang was a Distinguished Engineer at Microsoft and led Office365 app technology. He was also a co-founder and CTO of Acompli where he co-created the app that Microsoft acquired and adopted as the modern mobile Outlook. He also served in Chief Architect roles at VMWare and Yahoo! Zhuang graduated from Shanghai Jiao Tong University with a Bachelors degree in Mechanical Engineering.