Video format and codecs “behind the scenes“
November 13, 2023Working with video files always brings along new experience as video content changes, too. Especially with Extended Reality (XR) and Virtual Reality (VR) being on the rise, finding the matching video format becomes a crucial performance factor. Let’s sum up some basic facts about container formats and then move on to the magic of XR and VR content .
Videos are also a matter of format!
Video files come in different formats, depending on the recording device or later adjustments in an editing software. By the way, in an IT context you also call a video format a container format. This is because your recording device and computer automatically save all relevant data units in a virtual “container“ to keep them together. So basically like computers store all sorts of files. Video files are very complex, though, if you compare them to a two-dimensional image or a text file. So let’s take a closer look at some common formats now.
MP4 (MPEG-4 Part 14) – the “all-inclusive“ video format
Facts in a nutshell
MP4 is a very popular and widely used multimedia container format that can store video, audio, subtitles, and metadata.Most media players and end devices can play MP4 files, especially newer players and devices.
Performance issues
n case you use older devices, editing software or media players, MP4 files might not be compatible with them. The video format then has more recent standards than the output device. As a comparison, just think about the shift from VHS media to DVD and then to digital streaming portals. Video presentation always follows a flow.
Boosting your video performance
In case your end device or media player does not accept the MP4 video format, you can easily convert your files to other formats like AVI or MKV or select a different player.
AVI (Audio Video Interleave) – the “evergreen“ video format
Facts in a nutshell
AVI is a legacy video container format developed by Microsoft. Normally, you use it to store video and audio files. It is very common and playable on many output devices. Overall, it works best for “older” devices and players.
Performance issues
The AVI video format is only capable of a lower compression than MP4. Furthermore, it does not support all recent codecs depending on which ones are involved.
Boosting your video performance
In case a very recent media player or device cannot play this video format, it’s possible to easily convert AVI files to more modern formats like MP4 or MKV. Doing this, you can also improve compatibility and compression efficiency.
MKV (Matroska Video) – the flexible open-source buddy
Facts in a nutshell
With MKV, you rely on a popular open-source video format. It usually includes several relevant video, audio, and subtitle tracks at once.
Performance issues
In some cases, MKV shows compatibility issues with older media players and devices. This is due to the fact that this container format was originally designed for recent output devices and media players.
Boosting your video performance
You can naturally avoid compatibility problems by playing and editing MKV on a matching device. Alternatively, converting this video format e.g. to MP4 is a game changer, too.
H.265 (HEVC, MPEG-H Part 2) - the “brilliant diva“ with high demands
Facts in a nutshell
In general, you can consider H.265 a successor to H.264. Good to know: This video format is also called HEVC (High Efficiency Video Compression) or MPEG-H Part 2. The big advantage: HEVC provides better compression while maintaining a higher video quality compared to MP4.
Performance issues
As H.265 is one of the most recent container formats available, you might face compatibility problems with older output devices, editing software or media players. Furthermore, this brilliant format always demands a lot of storage space or bandwidth.
Boosting your video performance
For sure, you cannot prevent all compatibility issues. However, converting the H.265 video format only takes a few clicks. As an alternative, some universal players like the VLC media player support also the most recent and complex container formats and codecs.
Codecs determine your video format!
Every video format comes with codec settings. While codecs cannot be identified at first sight, they own the power to either run or stop the show from a “backstage position“. As a reminder: Video performance issues are often also codec compatibility issues. This means that codec combinations in your video files might not match and thus get into conflict. Read on to find out about common video codecs!
H.264 (MPEG-4 AVC) – the “powerful allrounder“
Facts in a nutshell
The H.264 video codec (also: MPEG-4 AVC) is well-known and popular for its high compression efficiency and quality. Practically, you often find it in MP4 video files designed for a small or “medium-sized” screen and in a streaming context.
Performance issues
Video files with a H.264 codec (also known as “High Definition“) might not play adequately on some older devices and and media players.
Boosting your video performance
In order to make your codecs compatible, you can either convert your files to a more time-tested video format or use a player that can decode HD contents.
H.265 (HEVC) – the challenging “state of the art“ solution
Facts in a nutshell
As mentioned above, H.265 (also: HEVC) is the “extreme effiency successor“ of the HD (H.264) video format including relevant codecs. HEVC offers a higher compression rate and thus especially can improve the performance of large and complex video files.
General “codec mismatches“
When you work with a HEVC video format and codec, not every output device and media player can read and “understand“ these very recent data structures. In fact, you will face compatibility issues with H.5265 codecs quite a lot as this high-tech container format still has a “exclusive touch“ and brings along many requirements. Just imagine you’re inviting an elegant lady or gentleman for dinner. Would they not expect more luxury than your best friends in town would, either?
Hardware-based performance issues
When it comes to output devices, some devices lack hardware support for HEVC decoding. This fact, in consequence, leads to playback issues or high CPU usage. Furthermore, videos in this container format definitely need more storage space and bandwidth than the “smaller” video formats described before.
Software hacks for your HEVC videos
Like all video files, you can also convert HEVC movies into MP4 or any other more common container format. You should further make sure that your media player and software are up to date, as new versions often add HEVC support. From an online marketer’s point of view, converting your HEVC movies is also helpful before broadcasting them on social media. Otherwise, if your viewers don’t have a matching output device, they cannot enjoy your content!
Hardware hacks for a brilliant performance
Sometimes it’s not the software causing issues, but hardware and firmware. In order to face less compatibility problems with H.265 codecs, consider upgrading to more recent output and editing devices with relevant hardware support. Also keep in mind your personal cost-benefit-equation, though. If you or our target groups don’t work with a H.265 video format on a regular basis and don’t need HEVC-capable hardware all the time, converting your videos back to H4.264 might do. Alternatively, you can naturally also rely on streaming eervices that support adaptive streaming.
Needless to say, every video format also comes with one or several audio codecs. However, audio fomats are a topic which is important and multifaceted enough to deserve its own editorial article. So you can already look forward to reading about the fascinating world of movie sound at a later occasion. For now, let’s move on to the techno-magical sphere of virtual realities!
Your video format in a multidimensional environment
Virtual Reality (VR), Extended Reality (XR), Augmented Reality (AR), and Mixed Reality (MR) require specialized video formats and codecs in order to create immersive and interactive experiences. These technologies often use specific video formats and codecs tailored to their unique requirements. Nonetheless, selecting “evergreen“ and established video format is often a safe choice for your source content to be used in a VR or XR context. As explained in a previous blog entry, the VR universe mainly includes four content categories – Virtual Reality (VR), Augmented Reality (AR), Extended Reality (XR) and Mixed Reality (MR). But for now, let’s focus on the choice of video format and codec combinations.
Virtual Reality (VR) – diving into virtual worlds
Selecting your VR video format
Basically, you can store your VR video content in various container formats, but common choices include MP4 or MKV. This is because most users don’t always go through VR worlds with the newest equipment and enjoy this sort of content on various devices anyway.
Video codecs for your VR experience
In order to ensure high-quality VR experiences, video codecs like H.264 or H.265 (HEVC) are commonly used. Some VR platforms also support Google’s VP9 codec, which is designed for streaming high-resolution movies.
Augmented Reality (AR) – adding virtual magic to the real world
Selecting your AR video format:
AR content often relies on streaming and real-time interaction. While standard video formats can be used, AR experiences are more about overlaying digital elements on the real world.
Video codecs for your AR experience
With AR, a dynamic interaction between real spaces and virtual elements is key. In consequence, AR always requires real-time video processing and streaming codecs. Formats like WebRTC (Real-Time Communication) and H.264 are suitable for streaming live AR content.
Extended Reality (XR) – exploring real and virtual horizons
Selecting your XR video format
XR content encompasses a broad range of experiences, from AR to VR and everything in between. In short: You can enrich both “dimensions“, the virtual and the real-life space, with virtual or real-life elements. As user scenarios (and also output devices) vary to a great extent, it is practical to store your content in standard video formats like MP4 or MKV.
Video codecs for your XR experience
When it comes to XR content, your codec choices depend on the specific user experience you want to create. In general, H.264 and H.265 are versatile options for all occasions. Nonetheless, it definitely raises the bar to apply real-time codecs and codecs optimized for streaming, like VP9 or AV1.
Mixed Reality (MR) – the “mix and match“ game
Selecting your MR video format
Similar to XR, MR combines aspects of both the physical and virtual worlds. However, it blende virtual and real-life elements blend in an even more realistic manner so you might face trouble to distinguish what was “real“ before putting on your MR glassen and what wasn’t. Due to its various user scenarios and output devices, MR content mainly requires standard video formats for content storage, such as MP4.
Video codecs for your MR experience
Like for XR, MR codec choices depend on the specific application. Overall, real-time codecs are essential for MR experiences. Therefore, you can apply H.264, H.265, or other real-time streaming codecs keeping in mind your future users and output devices.
So whatever sort of audiovisual and multidimensional content you’re about produce or edit – there is no “one fits all“ solution when it comes to the best-matching video format and codecs. So you always need to take into account some basiicfactors like the target platform, hardware, bandwidth, and interactivity requirements. Integrating all these aspects into your workflows, you can ensure optimal playback and user experiences with your VR, AR, XR and MR content. Just a side note: Don’t worry too much if your videos ever get damaged or – video repair apps like VRT and file recovery tools like MDG help you out with a few clicks.
German 
English