Apple recently released a new promotional trailer for their upcoming F1 movie. They’re calling it the “Haptics Trailer”. As the name suggests, the trailer incorporates haptic feedback based on the scenes and sounds. It’s a novel experiment in enhancing user engagement during video consumption.
While haptic feedback isn’t new – gaming console controllers like Sony’s DualShock and Xbox’s haptic-enabled models have been doing this for decades – Apple’s Haptics Trailer introduces a refined, video-driven version of it.
There have been experiments related to video like this in the past, but Apple’s newest implementation is surely going to storm ahead. Apple simply outshines the competition with their “ecosystem”, and if anyone can make this concept popular, it’s going to be Apple!
DIAGNAL’s ENLIGHT iOS Template App solution already supports haptic videos out of the box through AHAP file side-loading.
What is the Apple Haptics Trailer?
If you have an iOS device, you can watch it here. This particular trailer uses Apple’s Taptic Engine (their fancy name for the vibration motor, but it’s not a motor with counter weights) to add a new dimension to how we experience video content. It definitely feels impressive, and as an F1 fan, I would say it really adds to the overall experience.
DIAGNAL’s ENHANCE CMS has built in support for providing multiple rich, immersive video experiences to end users. Haptic Video is one such capability. From a content or an editorial perspective, no additional steps or workflow changes are needed to provide a rich media experience to end users.
How does it work?
If you analyse the master manifest file of the trailer video, there is an additional EXT-X-SESSION-DATAwithDATA-IDset tocom.apple.hls.haptics.url andVALUE set to an “ahap” file url, which is the Apple Haptic Audio Pattern file which holds the haptics information tied to the video.
The tag EXT-X-SESSION-DATA allows arbitrary session data to be carried in a Master Playlist – think of it like additional data that’s readily available in the manifest. You can read the RFC doc here.
Apple magic
Since iOS 13.0+ and iPadOS 13.0+, Apple has included Core Haptics capability, which lets developers add custom haptic feedback to their apps. This provides a bit more physical engagement with on-screen interactions. You might already be familiar with some of the input controls having haptic feedback when you interact with them – the alarm time picker for example. You can read Apple’s documentation on Core Haptics here.
The Taptic Engine built into iPhones, iPads, and Apple wearables is remarkably precise – you can between a mild tap, a sharp knock, and a continuous rumble. No other mobile device offers this level of haptic feedback resolution.
What makes Core Haptics truly powerful is its ability to play a continuous sequence of haptic patterns. This is where Apple’s harmony between hardware and software really shines.
From an implementation standpoint, every haptic feedback event is represented by a CHHapticEvent from the Core Haptics library, which is essentially a simple way to represent one haptic event. There are both audio and haptic feedbacks – but let’s ignore the audio aspect for now.
Two main haptic event types
When it comes to haptics, there are 2 main event types based on the duration of the effect – transient and continuous.
- Transient events are short impulses
- Continuous events are vibrations or rumbles held over time
The image below shows how these events are represented against time:
On a high level, each haptic event can be represented with the following attributes:
- relativeTime: the start time of the event, relative to other events in the same pattern
- duration: the duration of the haptic event
- hapticIntensity: the strength of the haptic event
- HapticSharpness: the feel of the haptic event
Both hapticIntensity and hapticSharpness are float values between 0.0 and 1.0.
There are a few other parameters associated with Haptics. However, for the sake of brevity we can ignore them for now. But in case you’re interested, you can read more here.
Now the engineering masterpiece. Core Haptics allows developers to create a list of haptic events and play them back like an audio/video file.
Apple has a specification for these files – Apple Haptic Audio Pattern (AHAP) files. It’s a simple text file with an array of CHHapticEvent objects in a JSON format (Apple says JSON like). You can read Apple’s specifications here. Below is an example AHAP file with 2 haptic events:
Generating haptics (AHAP file generation)
Understanding how Apple delivers precise haptic sensations is one thing, but generating those sensations from a dynamic, real-world audio source like a film trailer is a different challenge entirely.
This is where the real magic happens. So how might Apple have created the haptic layer for its F1 trailer? And how can developers or curious engineers attempt something similar? Let’s break down how audio can be analysed, interpreted, and converted into an AHAP file to drive the experience behind the Apple Haptics Trailer.
An audio clip is essentially a continuous analog signal. The differences in frequency and amplitude of the signal is what we perceive as sound. Speakers are essentially motors that convert electrical energy to sound waves through electromagnetic effect, and the vibrations from a speaker cone are what we hear as sound.
If you connect a powerful enough amplifier to an electric motor, you can actually hear the music play because a motor is essentially a coil sitting in a magnetic field just like a speaker.
Keeping this in mind, on a high level – just feeding the audio signal to some kind of function that generates the corresponding haptic sequence is the key to haptic videos. This can can be done with the following steps:
- Extract the audio from the video
- Analyse the audio and extract audio features
- Convert these audio features and corresponding timestamps to AHAP format
Extracting audio from video is as trivial as running a simple one liner ffmpeg command, but analysing the audio and feature extraction is the challenging part. Creating the haptic sequence comes next, but it’s relatively easy because you can play with numbers.
The following video helps to visualise how an audio clip can be represented visually. If we just convert the spikes to transient haptic events, and the constant audio part to a continuous haptic event, it would still become a crude haptic sequence that will match the audio clip.
Audio feature extraction
We cannot convert all audio features to haptics. That would mean every cough, every scream and every cry would be converted to a haptic sequence, which would definitely not feel natural or pleasant. With limited research, along with some trial and error I’ve done so far, the audio feature extraction should look for the following for a natural and smooth experience:
- Tempo
- Beats
- Onsets
- Mood – Bright/dark (or technically Spectral Centroid)
Once this extraction part is done, separate out the components of the audio clip:
- Harmonics (For Sustained noises)
- Percussive Sounds (For Drums & transients)
- Bass
- Treble
Once we have these split out and available, it’s just a matter of converting these to corresponding continuous haptic events:
- Generate HapticTransient events from onsets & beats
- Generate HapticContinuous events from the sustained energy + harmonics
And then adjust the haptic feel (ParameterCurve) for intensity and sharpness based on the amplitude and other signal parameters.
Putting it all together
We have used the following video to do the analysis and Haptics generation, this video is also publicly available as an HLS stream, which allows for a quick POC using AVPlayer in a test app.
You can download the full AHAP file from here.
Making it play
There are a few ways to do this, what Apple has done with their original implementation is to include the AHAP file in the manifest file itself as a EXT-X-SESSION-DATA tag. However, what we have done is to load the AHAP file similar to a sidecar subtitle and time-sync the video & haptic playback timelines.
DIAGNAL ENHANCE + ENLIGHT = Out of the box haptic content support
DIAGNAL ENHANCE has out of the box capability to automatically generate AHAP files from input videos, combined with the ENLIGHT iOS Template app, content owners and publishers can offer an additional dimension of content consumption to their end users without having to change any operational workflows, thus enabling more engagement on exclusive titles.
Relevant code blocks
The following code blocks will give an idea about how this can be implemented, skipping boilerplate, player controls, and other essentials for brevity.
Setup of video player and the haptic player:
// Setup
func setup(videoURL: URL) throws {
// Setup video player
videoPlayer = AVPlayer(url: videoURL)
// Setup haptic engine
hapticEngine = try CHHapticEngine()
try hapticEngine.start()
// Load AHAP from URL
let ahapURL = URL(string: "https://blog.aravindvs.com/assets/others/haptic-video/tears-of-steel.ahap")!
let hapticPattern = try CHHapticPattern(contentsOf: ahapURL)
hapticPlayer = try hapticEngine.makeAdvancedPlayer(with: hapticPattern)
}
Start playback:
// Synchronized Playback, we want our haptics & video to be in sync
func play() throws {
// Setup synchronization
let interval = CMTime(seconds: 0.1, preferredTimescale: 1000)
//Setup a timeObserver so that we can keep track of the time to keep both players in sync
timeObserver = videoPlayer.addPeriodicTimeObserver(forInterval: interval, queue: .main) { [weak self] time in
self?.synchronize(videoTime: time)
}
// Start both players
videoPlayer.seek(to: .zero)
hapticPlayer.seek(toOffset: 0.0)
videoPlayer.play()
try hapticPlayer.start(atTime: CHHapticTimeImmediate)
}Sync:
// Do Synchronization
func synchronize(videoTime: CMTime) {
let videoSeconds = CMTimeGetSeconds(videoTime)
let hapticTime = hapticPlayer.currentTime
let timeDiff = abs(videoSeconds - hapticTime)
// Re-sync if drift > 100ms
if timeDiff > 0.1 {
try? hapticPlayer.seek(toOffset: videoSeconds)
}
// Pause haptics if video isn't playing
if videoPlayer.rate == 0 {
try? hapticPlayer.pause(atTime: CHHapticTimeImmediate)
} else {
try? hapticPlayer.resume(atTime: CHHapticTimeImmediate)
}
}Closing thoughts
Even though the technology is cool, it has a few issues related to user experience. Main one being battery drain. Phone battery went down by a percent or two while watching the F1 trailer – however, this is expected. Then there’s the fact, it’s only on Apple devices for now.
Plus, it’s only suitable for short videos like trailers and teasers, it won’t make much sense for longer videos in its current form.
DIAGNAL’s take on haptics at scale
Creating rich, immersive haptic experiences like the Apple Haptics Trailer doesn’t need to be a manual, developer-heavy process. At DIAGNAL, we’ve built native support for haptics into both our ENHANCE CMS and ENLIGHT Template App Solution.
With ENHANCE CMS, haptic files can be automatically generated – independent of the video transcoding pipeline – making the process scalable and consistent across your content library.
On the front end, the ENLIGHT iOS app delivers seamless playback of haptic effects, fully integrated with Apple’s Core Haptics and designed for smooth user experiences. Whether you’re creating trailers, teasers or interactive content, DIAGNAL enables you to bring haptics to life – without reinventing your delivery workflow.
If you’re interested in integrating haptics into your own OTT or mobile video experience, get in touch with DIAGNAL to explore how we can support your vision.
