What Gemini and Google AI features can we expect?

Over the past year or so, Google has been previewing a number of Gemini-branded AI features across its consumer-facing apps. Here’s everything that’s been announced and when it might be available.

Pixel

At the end of Made by Google 2023, a Zoom Enhance feature that “intelligently fills in gaps between pixels and predicts fine details” was teased for the Pixel 8 Pro. Leveraging the “custom generative AI image model” on the device, Google pitched this as useful when you forget to zoom.

It’s an amazing application of generative AI, opening up a host of possibilities for framing and editing your photos. So the kind of zoom optimization you’re used to seeing in science fiction — right in the phone in your hand.

In October, Google said this feature was “coming later.” After launching three new features for the Pixel phone, the feature has not arrived yet. It is not clear whether the model Google is referring to is the Gemini Nano with Multimodality. At this point, this feature may debut with the Pixel 9 Pro as this phone’s main photography feature.

Google House

In the Google Home app, generative AI will be used to summarize events in a “simplified view of what’s happened recently.” This “quick and easy summary” will leverage dots, while you’ll also be able to conversationally “ask about your home” to find video history clips and trigger automations. The “experimental features” will be available to Nest Aware subscribers in 2024.

Fitbit

Fitbit Labs will allow Fitbit Premium users to test experimental AI capabilities and provide feedback on them.

One such feature is a chatbot that lets you ask questions about your Fitbit data in a natural, conversational way. This “personal coaching” that takes fitness goals into account aims to create “actionable messages and guidance,” with responses that can include personalized charts.

• “For example, you can dig deeper into how many active zone minutes (AZMs) you get and how they relate to how well you recover from sleep.”
• “This model may be able to analyze differences in your sleep patterns and sleep quality, and then suggest recommendations on how to change exercise intensity based on those insights.”

Behind the scenes, the app is powered by Fitbit and Google Research’s new Personal Health LLM program built on Gemini. Starting in March, it will launch “later this year” to “a limited number of Android users enrolled in the Fitbit Labs program in the Fitbit mobile app.”

Google images

Ask Photos lets you ask questions about the photos and videos in your library. Beyond searching for images, it can pull up information and give you a text-based answer. Gemini-powered queries include “Show me the best photo from every national park you’ve visited” and “What were the themes of our birthday parties?” and can be used to “suggest the best photos” and create annotations for them. Ask Photos is an “experimental feature” that will be rolling out soon, with Google already hinting at more possibilities in the future.

Gmail + Google Workspace

In the Gmail app for Android and iOS, you’ll find a Gemini button in the top right corner that lets you bring up the mobile equivalent of a side panel to enter full instructions. Gmail is also getting contextual smart replies that offer more personalized, detailed, and accurate suggestions. This feature will be released in Workspace Labs in July.

At its Cloud Next 2024 conference in April, Google also showed off a voice guidance feature for the Help me write in mobile Gmail app. Meanwhile, the Instant Refine feature will “turn raw notes into a full email with one click.”

On the desktop web, the side panel is available in Gmail, Google Drive, and Docs/Sheets/Slide. Gemini then comes to Google Chat to recap conversations and answer questions.

Google Maps

Last February, Google announced that its Maps would use LLMs to power its “Ask About” chatbot. You can use it to find places that match your claim with supporting follow-up questions. It is powered by details about 250 million places, photos, videos and reviews submitted by users.

Chrom

Gemini Nano comes to the Chrome desktop browser to power browser features like Help Me Typing. It should be available on most modern laptops and desktops.

Google search

Along with the launch of AI Overviews, Google previewed a number of upcoming features coming to Search Labs for the first time:

• You will be able to take an original AI overview and make it “simpler” (just a few sentences) or “break it down” (a longer response).
• Multi-step reasoning capabilities allow you to ask a complex question all at once rather than breaking it down into multiple queries.
• Meal and Trip Planning
• AI-organized search results page
• Video Searches: Record a video and ask a question about it.

Android

Gemini Nano with Multimodality will launch on Pixel “later this year” and power features like TalkBack on-device/offline descriptions, and real-time fraud alerts that listen in on a call for telltale patterns. Google will share more details later this year.

At I/O 2024, Google also previewed how Gemini on Android will soon become an overlay panel instead of opening up a full-screen UI to display results. Along with preserving context, this will let you drag and drop the generated image into the conversation. For Gemini Advanced subscribers, the “Ask this video” and “Ask this PDF” buttons will see Gemini videos and documents, respectively. This will roll out “over the next few months.” Additionally, Gemini Nano’s dynamic suggestions will use multimedia to understand what’s on your screen:

For example, if you activate Gemini in a conversation about baseball, suggestions might include “Search for baseball clubs near me” and “Rules of baseball for beginners.”

Another addition that will be particularly useful on mobile is the Gemini Extensions for Google Calendar, Tasks, and Keep. These will let you take a photo of a page with several upcoming dates that Gemini can turn into calendar events. In the coming months, the “Utilities” for Gemini mobile will allow you to access the watch app on Android.

We’re also looking forward to the Gemini mobile phone arriving on the Pixel Tablet this summer.

twin

Live streaming will allow you to have a two-way conversation with a Gemini. To make the experience more natural, Gemini will return brief responses that you can interrupt to add new information or ask for clarification. You can choose from 10 different voices, with Google imagining Gemini Live being useful for interview preparation or speech rehearsal. It will be available in the “coming months” to Gemini Advanced members.

Later this year, Gemini Live will let you turn on a live camera mode. Simply point to something in the real world and ask a question about it. This is done through Project Astra.

Gems are customized versions of Gemini that let you have a “gym buddy, sous chef, coding partner, or creative writing guide.” Gemini Advanced members will be able to create custom gems, while all users will have access to pre-made gems, such as the Learning Coach.

Simply describe what you want your Gem to do and how you want them to respond – such as “You are my running coach, give me a daily running plan and be positive, upbeat and motivated.” Gemini will take these instructions and, with one click, optimize them to create a gem that meets your specific needs.

Gemini Advanced users will also get a “comprehensive planner” that goes beyond just suggesting activities, but takes into account travel times and stops, as well as people’s interests, to create a detailed itinerary. Gemini will use Gmail trip/travel details, Google Maps recommendations for food and museums near your hotel, and search for other activities.