Hard Light Productions Forums

Modding, Mission Design, and Coding => FS2 Open Coding - The Source Code Project (SCP) => Test Builds => Topic started by: m!m on March 26, 2015, 07:58:16 am

Title: New cutscene player (more supported formats, better performance)
Post by: m!m on March 26, 2015, 07:58:16 am
Quote from: February 2015 Interview
Mjn.Mixael: H264 for cutscenes, music seek-to-time SEXPs, and model translation animation support
Hmm, H264 support, that doesn't sound like it's too difficult to implement.
After a few weeks of looking through documentation and cursing those who wrote it I arrived at an acceptable solution: http://www.mediafire.com/download/2dovsb3t83w5d2d/FSO-FFmpeg.7z
Source code: https://github.com/asarium/fs2open.github.com/tree/feature/ffmpeg

What does this do?
The changes in those builds add support for decoding a wide range of video and audio formats, notably H.264. It also improves the performance of cutscene playback by using a separate thread for video and audio decoding.
It uses the ffmpeg (https://www.ffmpeg.org/) libraries which do all the hard work. The currently version is capable of smoothly playing a 1080p Theora (Ogg) or H.264 encoded cutscene (I tried playing the amazing end cutscene of ST:R with RC5, it was horrible :shaking:).

Known issues
The build is based on the Antipodes branch which is a bit out of date so you should only use these build with retail data or mods that support older versions of FSO. The easiest way to test these build is to put the movie you want to see in data/movies and renaming it intro.ogg or intro.mp4 (depending on what format you are using).
Title: Re: New cutscene player (more supporte, better performance)
Post by: mjn.mixael on March 26, 2015, 08:20:49 am
Well well.... I know what I'm doing later.
Title: Re: New cutscene player (more supporte, better performance)
Post by: chief1983 on March 26, 2015, 09:29:32 am
We have to jump through quite a few hoops to legally use FFmpeg (https://www.ffmpeg.org/legal.html).  For this example, was #1 followed?

FWIW, OpenH264 (http://www.openh264.org/) is BSD-licensed, hasn't been out too long so it might have not been around the last time people were thinking about video codecs.
Title: Re: New cutscene player (more supporte, better performance)
Post by: m!m on March 26, 2015, 10:04:05 am
Well, looks like I have to compile ffmpeg myself (seriously, **** GPL. Who thought such a piece of crap would be a good software license? :hopping:).
I am currently using precompiled binaries that were compiled with --enable-gpl so I have to compile ffmpeg myself without that flag so I can link those libraries dynamically to FSO (I hope it will still have the necessary features...).
Title: Re: New cutscene player (more supporte, better performance)
Post by: mjn.mixael on March 26, 2015, 10:11:29 am
We have to jump through quite a few hoops to legally use FFmpeg (https://www.ffmpeg.org/legal.html).  For this example, was #1 followed?

FWIW, OpenH264 (http://www.openh264.org/) is BSD-licensed, hasn't been out too long so it might have not been around the last time people were thinking about video codecs.

I'm wary of trying/using yet another generally untested video thing...
Title: Re: New cutscene player (more supporte, better performance)
Post by: chief1983 on March 26, 2015, 10:20:05 am
There may be other options, it's just the first one I found that's not LGPL/GPL.  Not saying we can't do LGPL, but in the FFmpeg case we definitely have to disable the GPL and non-free parts, and we also have to meet all the other LGPL license requirements to the best of our ability.
Title: Re: New cutscene player (more supporte, better performance)
Post by: m!m on March 26, 2015, 03:24:33 pm
I compiled my own FFmpeg binaries without "--enable-gpl" and it looks like all the needed features are still there. Link in the first post has been updated.
For anyone curious, here are the resulting binaries: http://www.mediafire.com/download/irvgy2ry51r8z3l/FFmpeg-lgpl-shared.7z
Title: Re: New cutscene player (more supporte, better performance)
Post by: Cyborg17 on March 26, 2015, 10:38:54 pm
It also improves the performance of cutscene playback by using a separate thread for video and audio decoding.
 


Multi-threading in mai freespace???
Title: Re: New cutscene player (more supporte, better performance)
Post by: m!m on March 27, 2015, 04:40:54 pm
Multi-threading in mai freespace???
Don't worry, it's just one additional thread :P
Title: Re: New cutscene player (more supporte, better performance)
Post by: m!m on March 30, 2015, 03:46:34 pm
ngld kindly checked and fixed the code for linux so this should also compile and run on linux.
Title: Re: New cutscene player (more supporte, better performance)
Post by: Flaser on April 01, 2015, 02:43:26 am
First thing first: AWESOME job, this feature should make adding cutscenes to games/mods a lot easier! (Since devs can use more or less "standard" tools for creating video files instead having to deal with the vagaries of Theora).

IIRC, FFMpeg supports a bunch of codecs (https://www.ffmpeg.org/ffmpeg-codecs.html), are all these features available for developers or should they only stick with h264? If features are available, what codecs (at what specs, e.g. hi10 vs 8-bit encoding) are recommended for use?
Title: Re: New cutscene player (more supporte, better performance)
Post by: m!m on April 01, 2015, 03:29:39 am
The list of supported formats depends on how the libraries were compiled. I needed to disable a few features because we can't use the GPL version but here is a list of what the current libraries support:
Once FFmpeg is merged into trunk/master we probably will create some kind of list of formats that must be supported by "official" builds.
The current build still checks all available extensions but I have already changed that to only check ogg, mp4 and webm. Those formats will always be supported.
Title: Re: New cutscene player (more supported formats, better performance)
Post by: mjn.mixael on May 04, 2015, 11:53:09 am
As I said on IRC. I approve of this build.

It plays 1080p cutscenes (while scaling them for my current monitor) like they were nothing. I suspect I could stick to releasing just 1080p cutscenes and not multiple resolutions when this gets into the master branch.
Title: Re: New cutscene player (more supported formats, better performance)
Post by: m!m on May 04, 2015, 12:48:55 pm
I uploaded new builds that should be more in sync with the current master branch.
Title: Re: New cutscene player (more supported formats, better performance)
Post by: LaineyBugsDaddy on May 04, 2015, 01:37:40 pm
I'm looking forward to this. I remember that when I tried to play the 1080 version of the FS1 endgame cuscene when it first came out, I could only play it well in a dedicated media player. In game it froze and stuttered. This is gonna be awesome.
Title: Re: New cutscene player (more supported formats, better performance)
Post by: karajorma on May 04, 2015, 09:03:39 pm
Just out of interest, has anyone tried using this code to see what performance is like when the movie is in a VP file? Cause originally Diaspora VP'd all our movies but eventually put them loose in the data/movies folder due to performance issues.
Title: Re: New cutscene player (more supported formats, better performance)
Post by: jr2 on May 05, 2015, 08:19:20 pm
Hmm.  I thought .vp was uncompressed?  How would that cause a difference (besides for a split second to open the .vp)?  :confused:
Title: Re: New cutscene player (more supported formats, better performance)
Post by: m!m on May 06, 2015, 05:43:13 am
There is a bit of overhead from reading in a VP file as the read pointer has to be moved to the right position in the file. For optimal performance cutscenes should not be stored in a VP but I can't confirm or deny that VPs cause a performance hit (it would need to be very serious hit as the decoder thread is idle for quite a long time even when reading a 1080p video).
Title: Re: New cutscene player (more supported formats, better performance)
Post by: mjn.mixael on May 06, 2015, 07:38:29 am
I'll test it out this evening
Title: Re: New cutscene player (more supported formats, better performance)
Post by: mjn.mixael on May 06, 2015, 02:23:51 pm
Playing from a VP seems just as fast. I tested a 1080p .ogg video file. (STR's Endgame)
Title: Re: New cutscene player (more supported formats, better performance)
Post by: karajorma on May 06, 2015, 08:30:58 pm
Cool, thanks a lot for that mjn.
Title: Re: New cutscene player (more supported formats, better performance)
Post by: mjn.mixael on May 08, 2015, 08:03:33 am
You bet. Now let's get this merged! :)
Title: Re: New cutscene player (more supported formats, better performance)
Post by: m!m on May 08, 2015, 08:05:05 am
Not just yet, I found a few memory leaks :nervous:
Title: Re: New cutscene player (more supported formats, better performance)
Post by: m!m on May 15, 2015, 09:32:18 am
I think I fixed most of the remaining issues.
I also satisfied a few points of the "License Compliance Checklist" (now there is a notice about FFmpeg usage in the credits and a text file describes how to compile the libraries). Please be aware that the checklist is designed for proprietary applications and not for open-source projects that can't use the GPL version so some of the points on the checklist are not relevant for us.
Title: Re: New cutscene player (more supported formats, better performance)
Post by: chief1983 on May 17, 2015, 12:04:39 pm
To the FOSS community I doubt they would consider us legally "open source" due to the volition license so i'd be careful with that.
Title: Re: New cutscene player (more supported formats, better performance)
Post by: m!m on May 17, 2015, 12:14:14 pm
What should we be careful about? We may not be a legally open-source project but the source is still freely available and we satisfy as many points of the checklist as possible.
Title: Re: New cutscene player (more supported formats, better performance)
Post by: Echelon9 on May 17, 2015, 10:30:55 pm
Because of the copyleft (http://en.wikipedia.org/wiki/Copyleft) nature of the license to software we are considering using.

We need to demonstrate clearly that the combined work is not a derivative of FFmpeg, because the FS2Open project is not in a position to change the license on the Freeespace 2 code we use (only the rights holder can do that). And there are components to the existing Freespace 2 license which are incompatible with a free (as in libre) open source license.
Title: Re: New cutscene player (more supported formats, better performance)
Post by: logomancer on May 27, 2015, 12:55:18 am
As an alternative, you all could use VP8 (https://en.wikipedia.org/wiki/VP8) instead of H.264. The former has the benefit of being freer on a patent basis than H.264 with roughly the same (or slightly larger) file size for the same bitrate. In addition, it has its own self-contained decoder library in libvpx (https://github.com/webmproject/libvpx/), which is BSD licensed. Also, as it uses Vorbis as the audio stream, it wouldn't mean adding a new audio library. This also opens the door to use VP9 with the same library, which may offer even smaller file sizes still (though I wouldn't recommend it right now; it's still experimental).
Title: Re: New cutscene player (more supported formats, better performance)
Post by: mjn.mixael on May 27, 2015, 01:04:54 am
Ugh, vp8 is far inferior to h264 in many ways. vp8 is basically the same quality as theora, so we won't really gain any quality.. just another version of the same quality. The goal here is to get something better.
Title: Re: New cutscene player (more supported formats, better performance)
Post by: m!m on May 27, 2015, 02:35:00 am
The current version already checks the .webm extension which is VP8.
Title: Re: New cutscene player (more supported formats, better performance)
Post by: logomancer on May 27, 2015, 01:46:44 pm
vp8 is basically the same quality as theora, so we won't really gain any quality.. just another version of the same quality. The goal here is to get something better.

Not true. Theora was based on VP3, which was 5 iterations before VP8. And it's improved a lot since then. In your remastered cutscenes, Theora has blocks all over the place. VP8 (which you can see from YouTube, or I can post the file someplace) doesn't. And it's an eighth the size, even with more bandwidth allocated to the audio track. So even if it wasn't better than Theora -- which it is -- it is definitely smaller. When I looked at the H.264 YouTube rips, they were twice the size of the VP8 ones for no discernable difference in quality.

I admit, I don't whether you uploaded the original source files for processing or uploaded the Theora videos. So it may not have been an objective test. But if the idea is to look for something better, it behooves the project to look at all of the options (including even H.265 and VP9). Regardless, I will agree that Theora sucks and the sooner we retire it, the better, and it seems like the library is a step in that direction. So yay for that.
Title: Re: New cutscene player (more supported formats, better performance)
Post by: mjn.mixael on May 27, 2015, 01:58:58 pm
I realize this is a moot point since m!m's builds can apparently do both. But I have worked in the video production industry for about 7-8 years. Trust me when I tell you that h.264 is objectively better than VP8 in all ways except file size. Especially for our purposes. VP8 is great for web video, streaming things, and stuff you want to get the data of fast at the cost of some (minor) quality. (See: YouTube) However, we don't care about streaming... and the size of the cutscenes is not going to make or break the download size of mods big enough to have cutscenes. For sheer video quality, h.264 is the way to go.
Title: Re: New cutscene player (more supported formats, better performance)
Post by: m!m on September 09, 2015, 04:32:08 am
Just to keep everyone updated, I fixed a few issues of the cutscene player on linux and also added a way to use prebuilt LGPL binaries instead of relying on the system libraries which are GPL most of the times.

As far as I know GPL allows to use (and even modify) the libraries if they are not distributed so building a binary for private use may actually not be against the rules of the GPL. That would mean that only the release and nightly builds would need to include the LGPL libraries which would simplify development a lot. If someone knows more about this I would like to know if what I wrote is actually right as I don't know much about all that licencing stuff.
Title: Re: New cutscene player (more supported formats, better performance)
Post by: z64555 on September 09, 2015, 05:48:26 am
As far as I know GPL allows to use (and even modify) the libraries if they are not distributed so building a binary for private use may actually not be against the rules of the GPL.

Yep, section 2 of the GPL states this, that you can use and modify, and allow others to modify, the work under you sole discretion, provided that none of the involved party intends to distribute the "modified work" without appropriate licensing under the GPL. If you have the GPL'd library on your machine already, or otherwise can get it separately, there's nothing legal stating that you can't link the GPL'd library to the program.

The SCP can't distribute such versions of FSO, since that's expressively forbidden by the GPL, but conveying a version of FSO that's linked to an unmodified LGPL'd library is permitted, and of course the source code of FSO is freely available so folks savvy with build systems can use whatever GPL flavor they desire--so long as they don't distribute them.

The safe bet here for us would be to stick with LGPL libraries and leave the custom GPL linking to users that wish to do so, and probably warn them somewhere in bold friendly letters that they cannot distribute/convey copies of their GPL-linked version.
Title: Re: New cutscene player (more supported formats, better performance)
Post by: mjn.mixael on September 30, 2015, 10:10:57 am
So... I'm gonna poke this again. I don't understand all the license stuff, but I'm still fed up with Theora. :)
Title: Re: New cutscene player (more supported formats, better performance)
Post by: m!m on October 05, 2015, 04:27:17 am
As far as I can tell this project is ready to be merged but it still needs antipodes and CMake so it might take a while.
Mac support is also an issue because my CMake configuration currently doesn't work there and I don't have a mac so I can test it :sigh:
Title: Re: New cutscene player (more supported formats, better performance)
Post by: jr2 on October 06, 2015, 09:05:59 am
Someone needs to hackintosh (http://www.tonymacx86.com/).    Maybe your system isn't compatible, but maybe it is!  Then you could possibly test these things (actually, can't you already boot up mac in VirtualBox (http://lifehacker.com/5583650/run-mac-os-x-in-virtualbox-on-windows)?

Although all these things take Time (https://www.youtube.com/watch?v=_P8eFRCfno8).  :warp:
Title: Re: New cutscene player (more supported formats, better performance)
Post by: Echelon9 on October 08, 2015, 12:50:46 am
There's enough reasonable devs with Macs these days, that we can probably muddle our way through the necessary CMake changes.
Title: Re: New cutscene player (more supported formats, better performance)
Post by: m!m on October 08, 2015, 04:06:07 am
chief1983 and I had some success on his machine but without direct access to a Mac fixing bugs is very difficult and time consuming.
Title: Re: New cutscene player (more supported formats, better performance)
Post by: Bryan See on January 02, 2016, 12:25:24 pm
What about subtitles much like players? What about the .srt file?
Title: Re: New cutscene player (more supported formats, better performance)
Post by: The E on January 02, 2016, 01:01:35 pm
No plans for them at this time.
Title: Re: New cutscene player (more supported formats, better performance)
Post by: m!m on October 03, 2016, 09:38:09 am
These changes have been merged into the master branch and are available in the nightlies starting from this build (http://www.hard-light.net/forums/index.php?topic=92614.0).
Title: Re: New cutscene player (more supported formats, better performance)
Post by: mjn.mixael on October 03, 2016, 10:20:35 am
YeeaAAAHH
Title: Re: New cutscene player (more supported formats, better performance)
Post by: Bryan See on October 08, 2016, 12:13:54 pm
Are there any supported formats it can play? Why can't be in the Wiki?
Title: Re: New cutscene player (more supporte, better performance)
Post by: niffiwan on October 08, 2016, 05:42:05 pm
The list of supported formats depends on how the libraries were compiled. I needed to disable a few features because we can't use the GPL version but here is a list of what the current libraries support:
  • Formats (http://pastebin.com/YXKCiub6)
  • Codecs (http://pastebin.com/5ZHkbdM3)
Once FFmpeg is merged into trunk/master we probably will create some kind of list of formats that must be supported by "official" builds.
The current build still checks all available extensions but I have already changed that to only check ogg, mp4 and webm. Those formats will always be supported.