Hard Light Productions Forums
Modding, Mission Design, and Coding => FS2 Open Coding - The Source Code Project (SCP) => Cross-Platform Development => Topic started by: MasterGeek on June 11, 2015, 09:15:59 pm
-
Hi. I just joined the forum, btw. I've been looking for a project to do over my summer break, and I've been eyeing up the fs2open code. I've been looking for a place to jump in, and I feel like doing a CMake conversion of the build system. You see, I have a large amount of C/C++ experience, but little OpenGL experience. I've also programmed on a number of operating systems (linux is by far my favorite, though I still know my way around Windows and Mac). Also, I know from experience I'm pretty good at this sort of programming.
Anyway, if you're not familiar with http://cmake.org/ (http://cmake.org/), it's a meta build system that generates makefiles and project files for almost every build system and IDE under the sun. It makes solution files for every version of MSVC from version 6 (whenever that dinosaur was made) to version 14 (MSVC 2015 RC). It also makes project files for Eclipse and CodeBlocks, and makefiles for MinGW, NMake, and MSYS. Of course, it makes UNIX makefile, too, and when the UNIX makefiles are run, the output is color-coded and displays a percentage of completion (it's very shiny).
CMake can also be used to generate packaged installers like NSIS installers for windows and .deb files for debian, along with a few others.
I'm putting this under cross platform because the main advantage is it makes it easier to build code on multiple platforms, since when CMake runs it tracks down any build dependencies and generates a platform-specific makefile much like a configure file, except it doesn't care which OS it's on.
Enough of that. My guess is whoever reads this and intends to answer me is probably a programmer and has probably heard of CMake. How much need is there for something like this? Would you guys use it (assuming it worked)?
I'll probably start poking around the current build system by myself. I remember I tried to do something like this once before, and got the fs2_open executable to compile (can't remember if it ran), but fred2 was a completely different animal. When I last messed with the fs2open code was a couple years ago, but I remember that it was difficult to get a complete list of files that needed to be compiled to build the binaries. Any advice as to where I should start poking around the current build system would be appreciated.
-
https://github.com/asarium/fs2open.github.com/tree/cmake
-
Yeah, m!m has already implemented cmake, it just needs more testing on OSX before he's ready to unleash it.
Having said that, do you have any other particular areas of coding interest? ;7
-
Ah, well... so much for my speech. I search the forum for CMake, and got like 3 hits that seemed to be extremely dated and weren't even talking about porting fs2open to cmake, so I thought nobody was working on it. Anyway, I haven't done much in the way of graphics, but am willing to tackle anything else (and maybe graphics, too, but then I have to learn stuff before I can start hacking). I've done a lot of threading work in the past, and have a pretty good understanding of concurrency and related topics. I have a job as a web developer, and work mostly on the back end of things (think content management and system configuration). Consequently, I'm pretty good when it comes to networking and databases. I also have access to and experience with Linux, Windows, and Mac. I can also get a hold of absolutely any version of MSVC (even the 'Ultimate' editions) though my university. This was the main reason I wanted to take on the CMake project.
If you don't have any good ideas, how about that in-game config system that I've read people dreaming about occasionaly on this forum (the one that's supposed to handle most of the options that are currently command line options)? Is there much work done on that?
-
Unfortunately, the forum search is flat-out broken. You get better results if you use google with "site:hard-light.net".
I've done a lot of threading work in the past, and have a pretty good understanding of concurrency and related topics.
Making FSO multithreaded would basically require rewriting the whole thing. I mean, if you want to do that, I won't stop you, but that's pretty much the major reason it hasn't been done already (at some point it was attempted to just multithread the collision detection, but the added overhead from making only that be multithreaded basically removed any performance gains).
Consequently, I'm pretty good when it comes to networking
That could be useful; the netcode is in need of an overhaul.
how about that in-game config system that I've read people dreaming about occasionaly on this forum (the one that's supposed to handle most of the options that are currently command line options)? Is there much work done on that?
AFAIK no work has been done on that.
-
I would also recommend you join the #scp IRC channel on esper.net, it's where we coders hang out.
-
I am currently trying to get my CMake setup to work on Mac but I have no direct access to that OS. Do you have access to a Mac machine? If that's the case, could you try to generate project files from my branch? I am currently doing some iterative Mac development with chief1983, which is a bit slow so someone with CMake experience and a Mac would speed that process up immensely.
-
Unfortunately, the forum search is flat-out broken. You get better results if you use google with "site:hard-light.net".
I've done a lot of threading work in the past, and have a pretty good understanding of concurrency and related topics.
Making FSO multithreaded would basically require rewriting the whole thing. I mean, if you want to do that, I won't stop you, but that's pretty much the major reason it hasn't been done already (at some point it was attempted to just multithread the collision detection, but the added overhead from making only that be multithreaded basically removed any performance gains).
Consequently, I'm pretty good when it comes to networking
That could be useful; the netcode is in need of an overhaul.
how about that in-game config system that I've read people dreaming about occasionaly on this forum (the one that's supposed to handle most of the options that are currently command line options)? Is there much work done on that?
AFAIK no work has been done on that.
In regards to threading: have you checked out std::thread and std::async? C++11 added a lot of good stuff. Sounds like a big project, though. I think I'll pass for now.
In regards to networking: sounds interesting. What needs to be reworked?
I am currently trying to get my CMake setup to work on Mac but I have no direct access to that OS. Do you have access to a Mac machine? If that's the case, could you try to generate project files from my branch? I am currently doing some iterative Mac development with chief1983, which is a bit slow so someone with CMake experience and a Mac would speed that process up immensely.
I'll see if I can build it. I can't test it though, since the mac in question is at work, and it's the weekend. I'll be working over SSH.
-
Well, after a bit of hacking I got the thing building on mac. Most of the problems were related to how dependencies were found/handled/etc.
First of all, the 'code' target wasn't getting linked against OpenAL and OpenGL when it got built (and was missing the header includes as well). I fixed that in code/CMakeLists.txt.
Then, since OpenAL is a dynamic library on mac, it needs to be re-linked against the 'Freespace2' so code.a can get its symbols, so I fixed that in freespace2/CMakeLists.txt.
Then comes the oggvorbis crap. :banghead: I don't know where those binaries in lib/mac came from, but my linker said they were for an invalid format/architecture, and refused to use them. Since I don't believe in keeping binaries in a source tree, I deleted the entire lib/mac directory, and told lib/oggvorbis/CMakeLists.txt to pull the libraries and the headers from an external source (I just installed the libraries with Mac Brew). I should warn that lib/oggvorbis/CMakeLists.txt may not complain if it can't find the libs since I did it quick and dirty, so some code should probably be added there to check for that. I may split it off into its own find module, since that's how it's supposed to be done in CMake.
The way OpenAL was being imported before was causing weirdness. OpenAL is a framework, but after all the special processing lib/openal/CMakeLists.txt was doing to the import paths, by the time it got the the linker it wasn't specified as a framework but the framework's directory was passed as if it were a library file! Anyway, I made an exception that just bypasses lib/openal/CMakeLists.txt when on a mac and OpenAL gets imported the normal find_package way in code/CMakeLists.txt and freespace2/CMakeLists.txt... Honestly, simple find_package commands should replace all the complicated CMakeLists.txt in the lib directory for all operating systems. I don't think there's any need to keep dependencies in the source tree when CMake can handle them so well otherwise. If dependencies get managed manually in the CMakeLists.txt, then half the purpose of CMake is defeated.
Anyway, here are my changes: https://github.com/fjelliott/fs2open/tree/cmake-mac (https://github.com/fjelliott/fs2open/tree/cmake-mac).
If you're looking for a diff: https://github.com/fjelliott/fs2open/compare/cmake...fjelliott:cmake-mac (https://github.com/fjelliott/fs2open/compare/cmake...fjelliott:cmake-mac)
-
In regards to threading: have you checked out std::thread and std::async? C++11 added a lot of good stuff. Sounds like a big project, though. I think I'll pass for now.
Our specific problem is that most of the code uses global variables which means it's hard or impossible to multi-thread.
Well, after a bit of hacking I got the thing building on mac. Most of the problems were related to how dependencies were found/handled/etc.
First of all, the 'code' target wasn't getting linked against OpenAL and OpenGL when it got built (and was missing the header includes as well). I fixed that in code/CMakeLists.txt.
Then, since OpenAL is a dynamic library on mac, it needs to be re-linked against the 'Freespace2' so code.a can get its symbols, so I fixed that in freespace2/CMakeLists.txt.
Then comes the oggvorbis crap. :banghead: I don't know where those binaries in lib/mac came from, but my linker said they were for an invalid format/architecture, and refused to use them. Since I don't believe in keeping binaries in a source tree, I deleted the entire lib/mac directory, and told lib/oggvorbis/CMakeLists.txt to pull the libraries and the headers from an external source (I just installed the libraries with Mac Brew). I should warn that lib/oggvorbis/CMakeLists.txt may not complain if it can't find the libs since I did it quick and dirty, so some code should probably be added there to check for that. I may split it off into its own find module, since that's how it's supposed to be done in CMake.
I really don't like using the find_* functions anywhere but in the lib directory as that makes the code very hard to understand. Most of those commands only supply variables and no interface target which are easier to use IMO. I used the changes in your branch to do some fixes in my branch (although that broke the linux builds, I'll have to fix those). Given that we have different handling for the platforms a find module dedicated to mac doesn't make much sense to me.
The way OpenAL was being imported before was causing weirdness. OpenAL is a framework, but after all the special processing lib/openal/CMakeLists.txt was doing to the import paths, by the time it got the the linker it wasn't specified as a framework but the framework's directory was passed as if it were a library file! Anyway, I made an exception that just bypasses lib/openal/CMakeLists.txt when on a mac and OpenAL gets imported the normal find_package way in code/CMakeLists.txt and freespace2/CMakeLists.txt... Honestly, simple find_package commands should replace all the complicated CMakeLists.txt in the lib directory for all operating systems. I don't think there's any need to keep dependencies in the source tree when CMake can handle them so well otherwise. If dependencies get managed manually in the CMakeLists.txt, then half the purpose of CMake is defeated.
I changed the ADD_IMPORTED_LIB macro to use an interface target which should replicate what you have done in code/CMakeLists.txt (I did the same for OpenGL). The problem with find_package is that we supply prebuilt binaries for Windows (at some point I want to move those into a separate submodule) so that would need to be handled as a special case. I also like using pkg-config on linux as that handles the different platforms much better than a custom find module.
Did you have problems with SDL? We had issues getting that linked into the project as XCode was doing something weird.
-
Our specific problem is that most of the code uses global variables which means it's hard or impossible to multi-thread.
Apparently they hadn't heard of things like 'object orientated programming' and 'encapsulation' back in 1999... Yeah, that would make things hard if most of the code uses globals.
I really don't like using the find_* functions anywhere but in the lib directory as that makes the code very hard to understand. Most of those commands only supply variables and no interface target which are easier to use IMO. I used the changes in your branch to do some fixes in my branch (although that broke the linux builds, I'll have to fix those). Given that we have different handling for the platforms a find module dedicated to mac doesn't make much sense to me.
I completely agree that a find module dedicated to mac would be nonsensical. I was suggesting a find module that worked on all platforms. Then everything in lib/oggvorbis/CMakeLists.txt could be condensed to something like:
find_package(OggVorbis REQUIRED)
<convert vars to interface target if you so desire>
I guess you have a different opinion than me, but I consider the find_package command to be one of CMake's main advantages. Of course, the find module needs to be written correctly. CMake already has a bunch of find modules built into the base installation (but you probably already knew that). Whenever I use CMake for my own projects I abuse them as much as I can, and it greatly simplifies dependency management. Even if we had to write our own find modules, as long as they work for all platforms they would make everything in lib simpler.
I changed the ADD_IMPORTED_LIB macro to use an interface target which should replicate what you have done in code/CMakeLists.txt (I did the same for OpenGL). The problem with find_package is that we supply prebuilt binaries for Windows (at some point I want to move those into a separate submodule) so that would need to be handled as a special case. I also like using pkg-config on linux as that handles the different platforms much better than a custom find module.
Most find modules use pkg-config. I think it's possible to set a custom search path for find_package to use, though depending on the find module it may or may not be respected. Don't quote me on that, though. If it's possible, then all that needs to be done is set the custom search path for windows only and let the find module do its thing.
As I said previously, I don't believe in storing binaries in a source tree. What's the rationale for keeping the pre-built binaries for windows? I'd imagine that most people install the pre-built binaries on windows (myself included). I'm guessing that not many people beyond developers try to compile on windows just because of how difficult it can be to actually compile stuff on windows (and MSVC is expensive). If it's mostly developers who are trying to build this on windows, then it should be assumed they can track down the right dependencies.
If the problem is FSO needing the right version of a specific dependency, then the source code should probably be included in the source tree as opposed to the binaries. That would eliminate the need for any find_*, and would also deal with any linker problems like the ones I experienced.
Of course, if we _must_ have precompiled binaries for windows kept in the source tree, then there's not much to say. Windows will always be an exception then.
Did you have problems with SDL? We had issues getting that linked into the project as XCode was doing something weird.
...No... I don't see anything SDL-related in my changes... However, I do remember needing to install the SDL library in order to compile. However, CMake had no problem locating and using it once I did. Perhaps you had the wrong version? I installed SDL version 2 through mac brew (brew install sdl2). Then again, I was using regular makefiles and not XCode. I'm not sure if I mentioned, but the mac I have access to is my computer at my web dev job. The thing has a public static IP and a hostname so I can SSH into it, but I can't access any GUI apps. If you want me to try all this with XCode, then you need to wait until Monday.
I will pull your changes and try compiling again. Then I'll post back here. BTW: Do you lurk on the IRC channel that The E mentioned?
-
Apparently they hadn't heard of things like 'object orientated programming' and 'encapsulation' back in 1999... Yeah, that would make things hard if most of the code uses globals.
Actually, FS' core design is object oriented; It was just not implemented in a language that had support for it.
Do you lurk on the IRC channel that The E mentioned?
We all do, that's the thing :)
-
Do you lurk on the IRC channel that The E mentioned?
We all do, that's the thing :)
Do you know what m!m's name is on IRC, then?
-
I really don't like using the find_* functions anywhere but in the lib directory as that makes the code very hard to understand. Most of those commands only supply variables and no interface target which are easier to use IMO. I used the changes in your branch to do some fixes in my branch (although that broke the linux builds, I'll have to fix those). Given that we have different handling for the platforms a find module dedicated to mac doesn't make much sense to me.
I completely agree that a find module dedicated to mac would be nonsensical. I was suggesting a find module that worked on all platforms. Then everything in lib/oggvorbis/CMakeLists.txt could be condensed to something like:
find_package(OggVorbis REQUIRED)
<convert vars to interface target if you so desire>
I guess you have a different opinion than me, but I consider the find_package command to be one of CMake's main advantages. Of course, the find module needs to be written correctly. CMake already has a bunch of find modules built into the base installation (but you probably already knew that). Whenever I use CMake for my own projects I abuse them as much as I can, and it greatly simplifies dependency management. Even if we had to write our own find modules, as long as they work for all platforms they would make everything in lib simpler.
That should work, as long as the interface targets remain I have no objections but it might take a while until I get around to actually implementing that.
Most find modules use pkg-config. I think it's possible to set a custom search path for find_package to use, though depending on the find module it may or may not be respected. Don't quote me on that, though. If it's possible, then all that needs to be done is set the custom search path for windows only and let the find module do its thing.
As I said previously, I don't believe in storing binaries in a source tree. What's the rationale for keeping the pre-built binaries for windows? I'd imagine that most people install the pre-built binaries on windows (myself included). I'm guessing that not many people beyond developers try to compile on windows just because of how difficult it can be to actually compile stuff on windows (and MSVC is expensive). If it's mostly developers who are trying to build this on windows, then it should be assumed they can track down the right dependencies.
If the problem is FSO needing the right version of a specific dependency, then the source code should probably be included in the source tree as opposed to the binaries. That would eliminate the need for any find_*, and would also deal with any linker problems like the ones I experienced.
There are actually quite a lot of non-developers compiling their own builds on windows as the Community edition of VS2013 has everything we need and it is completely free.
The problem with finding libraries on windows is that there are no standard paths that can be searched by CMake which means that the developer has to specify most or all paths manually which is incredibly tedious. The other problem is that actually compiling or getting prebuilt binaries for our dependencies is pretty much impossible as most of those libraries are very hard to compile on windows (I tried to compile ogg/vorbis/theora and gave up after trying to solve some dependency issue).
...No... I don't see anything SDL-related in my changes... However, I do remember needing to install the SDL library in order to compile. However, CMake had no problem locating and using it once I did. Perhaps you had the wrong version? I installed SDL version 2 through mac brew (brew install sdl2). Then again, I was using regular makefiles and not XCode. I'm not sure if I mentioned, but the mac I have access to is my computer at my web dev job. The thing has a public static IP and a hostname so I can SSH into it, but I can't access any GUI apps. If you want me to try all this with XCode, then you need to wait until Monday.
That may be an issue with the prebuilt libraries then. I don't know how common homebrew is for Mac developers but if those binaries work then we should use them.
-
After working with m!m for a while, it seems that mac frameworks are just generally painful to use in cmake. Most of the dependency-related problems were fixed by installing the homebrew (brew.sh (http://brew.sh)) version of each library, which installs them to /usr/local/lib and /usr/local/include in a UNIX-like fashion. There will still be a couple cases where frameworks must be used, like with OpenAL and OpenGL, but it seems to just be easier to use homebrew to get the correct dependencies in most cases. For example, the Framework version of SDL2 caused problems with cmake, but the homebrew version of SDL2 worked immediately with no hacking required.
So as a question to the Mac developers on this forum, is homebrew an acceptable way to install compile dependencies? Most homebrew libraries include both the static and the dynamic version of the library, so this shouldn't affect pre-built binaries as they can be statically linked. But if homebrew makes it so easy to install the correct dependencies for development, why shouldn't we use that? Any thoughts?
-
Mostly because I've been avoiding installing Homebrew, MacPorts, etc on my new macbook. Before CMake, we've had a functioning setup for years with the pre-built libs in VCS, and I'm not saying we shouldn't keep doing that but it does add steps to the developer build process, and there may be other reasons to avoid actually installing the libs just to build it. Also some people might already have a setup that some libs installed via manual installation processes, recommending homebrew could lead to cross pollination of libs across the computer.
Also, we have a new OpenAL Soft framework that is about ready for testing. Using OpenAL Soft should really improve the audio performance on Mac.
-
That's my reason as well. I'm not saying it's not the best solution, just not sure. And I've managed to do quite disparate development on OS X without having to install one of the package managers like Homebrew or Macports so far.
I'll take a look at the cmake GitHub branch this weekend if there is some free time.
-
To clarify, by performance I don't mean make the game run faster, I just mean that it should improve the audio situation (more channels, better support for OpenAL's features where reference implementation support is lacking, etc).
-
Consequently, I'm pretty good when it comes to networking
That could be useful; the netcode is in need of an overhaul.
So since this sort of got lost as the discussion focused on cmake issues, I'd like to point you to this thread (http://www.hard-light.net/forums/index.php?topic=90397.0). We could REALLY use some help with the netcode. No joke.