AMD Working On Automatic In-Game Subtitles Using AI

Rundown:

• AMD was recently granted a patent for automatic in-game subtitle and closed caption technology for video games, aimed at improving accessibility for players with hearing impairments or challenging listening environments.
• The technology involves an overlay application that accesses a video game’s audio and video streams, processes the audio through a text conversion engine to generate subtitles, and determines where to display them on the screen.
• The subtitles are overlaid on a portion of the video stream, allowing players to read the dialogue and audio cues in the video game, potentially enhancing the gameplay experience for a broader range of players.
• The patent describes analysing the audio stream to identify the speaker associated with the subtitles and matching traits of the speaker with classifications stored in a voice profile database.
• The technology has the potential to be applied to online multiplayer video games, improving communication among teammates and potentially overcoming language barriers between players.

Earlier today, we encountered a recently published patent from AMD, the renowned multinational semiconductor company, titled, “AUTOMATIC IN-GAME SUBTITLES AND CLOSED CAPTIONS,” filed in December 2022 under the name of ADVANCED MICRO DEVICES, INC. The patent, published earlier this month, describes a technology that allows for automatic in-game subtitles and closed captions in video games.

“An approach is provided for a gaming overlay application to provide automatic in- game subtitles and/or closed captions for video game applications. The overlay application accesses an audio stream and a video stream generated by an executing game application. The overlay application processes the audio stream through a text conversion engine to generate at least one subtitle,” reads the abstract for the patent.

“The overlay application determines a display position to associate with the at least one subtitle. The overlay application generates a subtitle overlay comprising the at least one subtitle located at the associated display position. The overlay application causes a portion of the video stream to be displayed with the subtitle overlay.”

With this patent, AMD addresses the need for accessibility features for players with hearing impairments or challenging listening environments. The technology involves an overlay application that accesses a video game’s audio and video streams. It processes the audio through a text conversion engine to generate subtitles and determines where to display them on the screen.

The subtitles are then overlaid on a portion of the video stream, allowing players to read the dialogue and other audio cues in the video game. This technology aims to make video games more accessible and provide a better gameplay experience for a broader range of players.

According to the claims made by AMD, the system includes accessing the audio and video streams of a video game, converting the audio into subtitles using a text conversion engine, determining where to display the subtitles on the screen, generating a subtitle overlay with the subtitles in the determined position, and displaying a portion of the video stream with the subtitle overlay.

In addition, the patent specifies that the audio stream is processed in size-limited buffers for real-time or near-real-time processing. Furthermore, determining the display position involves analysing the audio stream to identify the speaker associated with the subtitles. The display position is set near an in-game object associated with that speaker in the video stream.

The patent also mentions analysing the audio stream to identify the speaker including matching traits of the speaker, such as age, gender, and dialect, with classifications stored in a voice profile database. Additionally, if the audio stream includes multichannel or positional audio, the in-game object associated with the speaker can be located by triangulation based on the audio information.

The generation of the subtitle overlay includes configuring the visual characteristics of the subtitles, such as font attributes, colour, size, and speech bubble type. Moreover, the visual characteristics can be adjusted for the subtitles by the player, which can also be determined based on player preferences, readability when displayed with the video stream, and the speaker’s sentiment analysed from the audio stream.

Interestingly, AMD also mentions that the technology can also be applied to online multiplayer video games, where the audio stream may include voice chat from participants in the gameplay session. The proposed technology can greatly enhance the gameplay experience in online multiplayer video games, especially in cooperative scenarios where clear communication among teammates is crucial.

It addresses the common challenge of understanding audio from teammates, particularly during intense moments. By implementing this technology, players can communicate more effectively, leading to improved gameplay and a more convenient video game experience overall. Furthermore, this technology has the potential to overcome language barriers that players may encounter.

“Even when spoken dialogue is clearly audible to the user, it may be spoken in a foreign language or in a dialect or accent that is not readily understood by the user. In these cases, subtitles or closed captions can assist the user in better understanding audio,” mentions the patent’s background, which implies that the technology can be used to bridge the gap between players who speak different languages, allowing them to understand and communicate with each other more effectively during gameplay.

AMD, widely recognised for its computer processors extensively utilised in PlayStation and Xbox consoles, appears to be entering the artificial intelligence realm. This move may indicate a potential competition with NVIDIA, its primary rival in the graphics processor market, which has already been at the forefront of the artificial intelligence revolution.

The proposed technology, if successfully implemented, has the potential to be a groundbreaking milestone in enhancing the accessibility and inclusivity of video games for a wider audience. It is, however, important to acknowledge that the current stage of development is limited to a patent, and its actual integration into existing and upcoming video game technologies is uncertain. Only time will reveal AMD’s plans and whether it will bring this innovative technology to fruition.

What do you think about this? Do tell us your opinions in the comments below!

Similar Reads: NVIDIA Targeting Toxic Players With In-Game Audio Censorship