Ep.31 — Helping the hearing impaired using Apple Watch’s ML capabilities
1,500,000,000 — The number of people who live with hearing loss in the world. That’s nearly 20% of global population! (source: who.int)
I recently experimented with Apple’s Sound Classification API and the results are pretty promising on the Apple Watch. So, I decided to release the BuzzWatch app and the corresponding source code so others can use it as a reference example; and build more assistive technology to help others.
The Watch App listens to user-selected sounds (e.g. baby cry, siren, car horn) in the background. It vibrates to notify the hearing-impaired when it happens. There are 300+ sounds that it can understand.
Here are the top 5 lessons learned in this journey:
- Sound classification is surprisingly mature with a low rate of false positives. In my use case, audio stream from Apple Watch is analyzed on the fly for sounds such as car horns and sirens. The detection time is pretty good too, less than 10ms.
- Haptic feedback is really useful, but it is often forgotten during user experience design. Engineers have fine-grained control over the type of vibrations on iOS using the UIFeedbackGenerator framework (it’s not available on watchOS yet.) BuzzWatch can only use the standard vibration for all types of important sounds. I couldn’t get it to work like Apple Maps navigation where you can customize the haptic experience. I hope Apple would improve watchOS in this area. Haptic feedback is very handy for wearables. (I wonder if Wear OS 3.5 on Android is better… time for a Pixel Watch?)
- One important reminder on using Haptics with AVAudioSession: When you are recording audio, system sounds / haptics (e.g. Notifications) are disabled. You can override it using
session.setAllowHapticsAndSystemSoundsDuringRecording(true). This one was tricky to discover because audio recording isn’t really related to haptics stuff and it’s hard to make that linkage when debugging. - SwiftUI is being forced upon us for more and more use cases. It started with Widget development last year and now with watchOS 9.0, you can’t even start a new watchOS project using Storyboard anymore on Xcode 14. If you have existing code written to WatchKit and the traditional storyboard-based UI, it’s time to start planning a migration. In this project, I started out with Xcode 13 with storyboard; but eventually had to give up and redo everything on the watch app using SwiftUI.
- Don’t assume the OS always behave correctly, especially with .0 releases. I made this mistake when trying to get custom UserNotification working on watchOS. After much head-banging-against-the-wall, it turned out that there was nothing wrong with the code. Custom notification view works consistently on watchOS 8.5, but watchOS 9.0 just doesn’t behave. I ended up filing a bug report to Apple, hoping they would address it at some point. ¯\_(ツ)_/¯
I hope this write-up helps. Cheers.
