Bixby 2.0 brings improved speech recognition and third-party app support

There is no doubt that Samsung Bixby, the artificially intelligent assistant found on the company’s flagship Galaxy S8, is quite different from Apple’s Siri, Amazon’s Alexa, and the Google Assistant. Samsung’s take attempts to anticipate your needs and serve up contextualized reminders. Bixby edits photos, sends messages, and composes emails on command. Anything you do on your phone via touch, you should be able to do by voice with Bixby. Eventually, it’ll control smart home appliances and Internet of Things (IoT) devices, and work on smart fridges, TVs, and dozens of other Samsung devices.

Here’s how to set up Bixby, what’s new with Bixby, and everything you can do with it.

Bixby 2.0

Bixby isn’t perfect, but it’s slowly improving.

In April, an early version of Bixby without voice functionality shipped on the Galaxy S8. After months of delays and a limited preview program, the full version of Samsung’s assistant became available for all Galaxy S8 and S8 Plus devices. Most recently, in late August, Samsung expanded Bixby Voice’s English and Korean-language rollout to include a total of more than 200 countries globally, including Canada, the U.K., South Africa, and Australia. The company says more languages and territories are on the way.

Separately, at a developer conference in San Francisco, Samsung took the wraps off Bixby 2.0, the next generation of its AI-powered digital assistant. Among the improvements are enhanced speech recognition, which the company says will help Bixby recognize individual voices (à la Echo and Google Home) and “better anticipate their needs.” The deep linking feature will integrate the assistant more tightly with third-party apps. In the near future, for example, Bixby might understand commands like “Send Jack $20 for lunch” or “Book me a hotel room in Portland, Oregon, next weekend” without needing to ask follow-up questions.

Smartphones aren’t the only devices that stand to benefit. Samsung says that Bixby 2.0 is coming to the Samsung Smart TV, Family Hub refrigerator, and other products later this year, which will dovetail with the launch of a software development kit (SDK) in private beta. And Samsung briefly teased something called Project Ambience, a Bixby-compatible plug-in dongle that adds Bixby’s smart home controls to “a wide variety of objects.”

Getting started with Bixby and Bixby Voice

Pulling up Samsung Bixby and Bixby Voice couldn’t be easier. The Galaxy S8 and Galaxy S8 Plus feature a dedicated Bixby button on the right side, just below the volume rocker. Tapping on it brings up Bixby Home, the assistant’s hub and setup screen.

If you’re on the Galaxy S8’s home screen, you can access Bixby by swiping to the right. When you pull up Bixby for the first time, you’ll be asked to agree to the terms of service — and to grant it access your data. Hit “Agree to all” (after reading the terms), and then choose “Allow” when prompted to grant Bixby permissions.

To get to the Settings screen, tap the gear icon in the upper-right-hand corner of Bixby Home, and then hit Settings in the drop-down menu. You’ll see options for Bixby Home cards, a list of supported apps, and more.

There’s more to Bixby than a single app or screen. Rather, the Galaxy S8 and S8 Plus ship with three incarnations of Samsung’s Assistant: Bixby Home, Bixby Vision, and Bixby Voice.

Bixby Voice

Bixby Voice is like Siri on steroids — in fact, it can rap insults at Siri in Korean. Not only that, but it’s built to adapt to a person’s manner of speaking — rather than the other way around. According to Samsung, the assistant will be able to understand “Show me today’s weather,” “What’s the weather like today?,” or “What’s the forecast today?” — all variations of the same command.

Bixby Voice is now available for all Galaxy S8 and Galaxy S8 Plus owners. If you haven’t gotten a prompt to enable it, tap the Bixby button to start the update. To ensure that the update goes off without a hitch, make sure you’re running the latest version of Bixby by heading to the About page on the Bixby Home menu. You might have to clear the data and cache for Bixby apps by opening the Galaxy S8’s Settings screen and navigating to the management menu.

Once you’ve installed the software and completed Samsung’s Bixby Voice tutorial, you’re free to invoke the voice assistant at your leisure. You can activate it by saying, “Hey Bixby” or by holding down the Bixby button while you talk. If you don’t want to look like you’re talking to your phone in public, you can talk to Bixby by holding your phone up to your ear, like a phone call.

Bixby can perform tasks like sending text messages via the Galaxy S8’s Messages app and responding to basic questions about the weather, upcoming meetings, sports scores, and movie showtimes. But that’s just the tip of the iceberg — the voice assistant also works across apps like the Galaxy S8’s dialer, settings menus, camera app, contacts list, and gallery.

Bixby can handle complicated tasks like opening an app in split-screen view, rotating misaligned photos, reminding you where you parked, playing videos on a nearby TV, and composing emails. It can gather all the photos you took last week into a new album labeled “Vacation” and share it with your friends, or take a selfie and text it to someone. And it can perform two-step actions like, “Open Uber and rate my driver 5 stars,” or “Open Instagram and post my most recent photo.”

There are also some new capabilities with the official launch — namely, more hands-free functions. For example, you can now ask the assistant to “read out the latest messages,” and it’ll read you your texts or emails, assuming you use native Samsung apps.

“Instead of taking multiple steps to make a call — turning on and unblocking the phone, looking for the phone application, clicking on the contact bar to search for the person that you’re trying to call, and pressing the phone icon to start dialing — you will be able to do all of these steps with one push of the Bixby button,” Injong Rhee, Samsung’s executive vice president, told Digital Trends during the reveal of Bixby in March.

Bixby Voice is also customizable — you can change the gender of the assistant on the fly, and it supports shortcuts: You can shorten the lengthy commands you use most often to a single word or phrase.

Samsung says Bixby supports more than 3,000 commands, including app-specific commands in Facebook, Instagram, YouTube, Uber, Gmail, Google Maps, and more. The company says more will come later this year, and it has been working hard on improving Bixby’s response time — though it doesn’t give any figures on exactly how much it has improved.

Bixby Home

Samsung calls Bixby Home a “social stream for your device,” and that’s more or less accurate. It’s a unified card-like interface of reminders, contextually relevant information, and social media updates — sort of like a cross between Google Now and HTC’s BlinkFeed.

Bixby Home can be customized to a degree. You can dismiss, pin, or permanently hide cards by tapping the gear icon in the upper-right corner of an individual Bixby Home tab. You can now also set quick commands. For example, you could set the assistant to turn off all lights and set an alarm when you say the command “I’m going to bed.”

Samsung’s default apps supply most of Bixby Home’s content. You’ll see a local weather forecast, activity stats from Samsung’s Health app, and local files in the Galaxy S8’s Music app.

The list of third-party services that support Bixby Home is growing, albeit slowly. A Spotify card provides one-tap access to your playlists and music, and full support is on its way. CNN and Flipboard show trending news stories from around the web. Facebook, Foursquare, and Twitter show the latest updates from your social circles. And soon, Uber will show your recent rides.

Eventually, Samsung says Bixby Home will be able to learn from your routine and update to reflect how you use your phone throughout the day. A future iteration will serve up buttons that’ll control smart light bulbs and door locks, shortcuts to the apps you use most frequently, and more — all based on the time of day, your location, and other factors.

Bixby Vision

Bixby Vision taps machine learning (and Pinterest’s Lens Tool) to identify the objects around it. Much like Google Goggles or Amazon’s Flow, Bixby Vision tries to suss out whatever’s in front of the Galaxy S8’s camera, and serve up information depending on what it finds.

You launch it by tapping the Bixby button in the Galaxy S8’s camera app, Bixby Home, or the Gallery app.

Right now, Bixby Vision recognizes four categories of objects: Place, Text, Image, and Shopping.

Bixby Vision is surprisingly capable, if a little finicky. It’ll identify landmarks like the Empire State Building and suggest nearby places of interest, for example. If you point at a bag of Twizzlers, it’ll recommend similar candy from one of the supported online stores. If you point it at a bottle of wine, it’ll offer details on the label. It’ll automatically parse QR codes and barcodes, books, and logos, and you can use it to translate text.

It’ll only get better with time. Samsung says it’s working with partners including Amazon, Vivino, Pinterest, and others to improve the relevancy of Bixby Vision’s search results.

Reminders

Bixby’s Reminders is a little more robust than the reminders you can create with Siri or the Google Assistant. It’s built into the Galaxy S8’s native apps and lets you attach media like videos, websites, photos, and messages.

You can set a specific time you’d like to see a reminder or even a specific location. When the conditions are met, said reminder will appear in the form of a notification you can dismiss, snooze, or check off.

Update: Added news of Bixby 2.0’s launch.