Hard Light Productions Forums

Modding, Mission Design, and Coding => FS2 Open Coding - The Source Code Project (SCP) => Topic started by: Iss Mneur on April 09, 2010, 11:34:13 pm

Title: wxLauncher 2.0 Request for Comments
Post by: Iss Mneur on April 09, 2010, 11:34:13 pm
As you may already know kkmic and I, along with input from (alphabetical) chief1983, FUBAR, Fury, Karajorma, Portej05, The_E, Zacam, and others (sorry, I don't have records of all the names) have been developing a new launcher for FreeSpace and it's Total Conversions. It is now time for the wxLauncher team to start planning version 2.0, whose purpose is to complete the package and add the installer to wxLauncher. For those that remember the initial proposal that kkmic and I presented (http://www.hard-light.net/forums/index.php?topic=66190.0), we asked for suggestions on what features the community would like for this component of the launcher. I don't know about kkmic but I have received a great deal of feedback about what features are in demand.

This document is not about the how, but the what. That is, this discussion is to be about the short and long term features of wxLauncher 2.0, not specifics of how we implement (ie. algorithms and designs, not code libraries and languages) these features.  The discussion of the specific how will start at a later date.

Table of Contents
The following essay has the following major topic categories:

Terms
I know most of you will likely know the definitions of these terms, but I am defining them here so that we are clear.

Overview of wxLauncher 2.0
The general idea is to make wxLauncher into a platform like Steam or Impulse and allow for the complete management of MODs and TCs.

The specific ideas that we are pulling from Steam or Impulse are: a central location for all FSO engine content (TCs and MODs) to be catalogued, make installing the catalogued MODs and TCs easy for all users, and allow modders (TCs in particular) to offer off-line installers.

Contrary to some opinions, wxL2 would not invalidate the existing installs, except for non-mod.ini compliant installs.  Any MOD that uses the mod.ini will be a dropin for the wxLauncher. To be clear, in the eyes of the wxLauncher team, having wxL2 break existing installs and/or invalidate current install instructions is not negotiable.  The result of this policy is that even if wxLauncher was used to install everything, there is nothing stopping anyone from manually installing a MOD.  That being said, for installations that are not mod.ini compliant (user error (or age, but this will be harder because of need for meta-data)), wxLauncher will be capable of correcting a damaged and/or out-dated TC folder. That is, if the user has put VPs in the wrong spot or has truncated files wxLauncher will be able to detect and fix these issues on a per file basis.

The wxL2 platform's ability to integrate with off-line installers would also allow projects like the Bablyon Project who like to offer DVD iso installs of the content to be able to to still do this but use wxL2 as their actual installer and gain the benefits that wxL provides.  The wxL2 platform would also allow the modders to just provide an installer that is basically just a downloader.

The platform would also allow for a modder to provide a "net" installer that only downloads what is needed.  For example, Blue Planet, who are going to release Blue Planet 2 as a MOD that depends on having Blue Planet 1 installed (and updated).  wxL2 could for the release of BP2 could use wxL2 as a net installer that will check and update in needed the BP1 install then download the BP2 install.  As well, BP does require the MediaVPs so wxL2 could also check and update, if needed, the MediaVPs.

If this is starting to sound like a package manager, you are correct, package management is one of a few things that Steam is good at. In keeping with how wxLauncher is currently setup, anything that requires internet access will be optional and wxLauncher will prompt on first run about these types of things.

The general design of the central location to catalogue and distribute the MODs and TCs will likely be an integration of both the local wxLauncher front-end (install tab of the current wxLauncher builds) and a website/server to provide the back-end support required for the versioning that the installer will require. Potentially we would also allow for installing previous versions of MODs and TCs.

At this point we are not looking at supporting any FSO build older that 3.6.9, thought this could be bumped to 3.6.10 depending on the time lag, difficulty of implementing the support for .9 and the demand for .9 support. This also implys that retail executable support is outside of the scope of this project, unless of course there is sufficient demand. Though we do plan on implementing the ability for wxLauncher to read the retail installer files (gog.com and OEM/RTM disks) on all platforms.  This is so that we can expedite and ease the installation of FreeSpace on all platforms.  Yes, this means that wxLauncher is not going to be as simple to install/upgrade as the current (win32) launcher, but as the intention of this project is to make FSO easy(-ier) for new people, like the people that WCS and TBP already attract and the people that Diaspora and Fate of the Galaxy could attract to the community.

As for the installer capability, we are looking at being able to have wxLauncher able to edit .vp files on the spot and to have on-the-fly compression of the .vp files so that we can lower bandwidth requirements for updates. The purpose of this capability is to allow the wxL2 to apply diffs of the contents of .vps.

Looking for input
The following are items that we are looking for more information on.

Scripting
What language or syntax is meant by "scripting" for wxL2 is still up in the air.  At this point it is between Python, Lua, or a custom "script" based on .tbl.

Names
We are currently looking for a catchy name for wxL2 that is not as generic.  To be clear I am not opposed to it continuing to be called "wxLauncher 2.0".  This name will likely also be used in the domain for the wxL2 central catalogue website.  Current suggestions are (in no particular order):


Change log:
Title: Re: wxLauncher 2.0 Request for Comments
Post by: chief1983 on April 10, 2010, 05:27:19 am
I know you can't remember everyone, but me?  Even though we talk about this thing like every other day?  Is this punishment for not testing OS X soon enough?  :P
Title: Re: wxLauncher 2.0 Request for Comments
Post by: Tomo on April 10, 2010, 05:56:44 am
Not sure if you're already doing this, but there should be buttons for FS2-Open and FRED, and the option to run the Debug and Release of either, all using the same MOD.

The current launcher needs quite a bit of background knowledge and doesn't do much handholding, which isn't great for new people.

On rebuilding from source:

However, there does need to be a choice of Win32 Standard/SSE/SSE2 and Win64 builds under Windows - selection should be automatic based on the detected OS and CPU capabilities, but an 'advanced' user could override it (in case they want to try something or the detection is incorrect)

I don't use Macs so can't comment on that platform.

On old versions of MODs
Is there any particular reason for not leveraging an existing proper version-control system that already handles the full revision history?
- Each individual MOD folder would be its own repository and thus have a contiguous revision history.
(Eg Updates to BP2 do not affect BP1 revision numbers)

Each 'commit' to these version control repositories would only ever be a proper release (never beta), so all commits should be good. (At least as good as shrinkwrap software anyway)
Each MOD lists the revision numbers of all the dependencies that it has been tested with.
If the local version is older than the oldest 'tested' revision it warns and suggests updating to the newest or newest tested one.

Example:

It would then be up to a given modder to test with a sensible range of versions of the items it depends on. We don't have multiple inheritance so this isn't an onerous task - eg BP2 depends on BP1 and MediaVPs, but not anything that BP1 says it depends on.

If a MOD is old and therefore completely untested, then the latest version of all dependencies is assumed, but the end user can roll any dependency back to any older version.

When a modder wishes to test their MOD with other revisions, they activate the 'Expert mode' to get the full, dated list of dependency revisions to consider which to test with.
- Administrators of the new system can also add tags to any MOD to indicate recommended versions, to handle orphaned and untested works (eg People can post here saying that MOD x doesn't work with MVP release y but is fine with z)

New MODs are self-evidently tested by the modder with the versions of the dependencies that they had at the time, even if they didn't want to bother testing with other versions.
I expect that only very large MODs would be tested with more than one version of dependencies at release.

As the new system offers automatic mod.ini creation anyway, this same automatic system can be used by the modder to generate the 'release' mod.ini (while in Expert Mode)

This gives the information required to determine when to archive/discard very old versions of things:
- 'Untested' MODs are a problem, but they won't stay untested for long if anyone is playing them.

Otherwise we are likely to discard things that someone is relying on.

Selection of version control engine is left open. No holy wars please!
However, it should be something where you can switch between revisions without redownloading them, as otherwise testing across multiple revisions becomes a pain in the proverbial.

Scripting
Lua is the only choice.

It's the one used in FS2-Open, so many people already know it from both ends.
There is no point in asking modders to learn a new language, or coders to learn new ways of handling hooks.

Further notes
The entire system is going to need fairly powerful Administrators.
This is going to be quite a lot of work.

Turey's Installer fell over for a while because nobody was doing the important work needed to keep that up to date.

So whatever system is built, there will need to be a very simple and easy to use multiplatform tool to handle the updates that can be taught to your average modder in a few minutes over IRC or similar.

Otherwise, sooner or later the new system will lose a key person (even if only temporarily) and fall apart.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: portej05 on April 10, 2010, 06:52:33 am
One word: Versioning.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: Mongoose on April 10, 2010, 12:24:00 pm
I feel like this came up already in one of the other discussions on the subject, but the ability to create "profiles" of specific launcher flag combinations for particular mods (beyond the -mod flag itself) would be very useful.  It doesn't come up all that often, but there are certain mods that require specific launcher flags to be turned on or off; beyond the obvious TBP and WCS flags, I know Windmills needed certain lighting settings to achieve its effect.  I can't think of a specific application, but one could even go beyond just flags to general settings as a whole, like changing the video resolution.  I'm not sure if this would be best achieved by integration with the existing mod.ini system, or some external means, but I'll leave that to those who are actually technically-oriented.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: The E on April 10, 2010, 03:49:56 pm
I think I mentioned this on IRC before, but I'll mention it again here.

IMHO, the best model for version management would be something akin to Debians' apt. The launcher would record metadata about the mod (comprised of something like version + md5 checksum of any given asset), and use that to determine whether or not an update is needed.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: Iss Mneur on April 10, 2010, 10:57:38 pm
Is this punishment for not testing OS X soon enough?  :P
Yes :P (I have edited the first post to include you)

Not sure if you're already doing this, but there should be buttons for FS2-Open and FRED, and the option to run the Debug and Release of either, all using the same MOD.
Yes we are, something related to this is going to be in wxLauncher by 1.0.

The current launcher needs quite a bit of background knowledge and doesn't do much handholding, which isn't great for new people.
This is more related to wxLauncher 1.0, but yes, this is one of those "benefits that wxL provides".

On rebuilding from source:
  • On Linux, this feature is required due to the differences in APIs between distributions - a lot of apps don't work correctly (or at all) if not built from source.
  • Under Windows, this feature is not a good idea:
    • No build environments are included by default, so one would have to be provided. We can't provide MSVC which is the build environment currently used for Windows development as that's owned by MS.
    • It would take a very long time compared to what Windows users are used to. My PC runs FS2-Open pretty well, but takes around half an hour to do a full build of FS2-Open 3.6.12 in MSVC 2008. Windows users don't expect to do this.
Thank you, this is why it is on the list of questions, we are looking for more input on this item, namely on how integrated we want it to be with wxL.  That is, do we just provide some pre-built binaries for the major linux distros (.debs, .rpms, etc) and coerce the platform's package manger into doing our bidding; or do we make the linux users compile, then copy the binaries into the TC folder as we do now; or do we have wxL interface with the platform's build system to build the binaries for the user?

Also, VC++ Express editions are freely available to anyone that wants to download it. The VC++ Express edition allows the compiling of FSO on windows, even though FRED is currently not available (but FRED not being build-able is a problem on Linux as well).

However, there does need to be a choice of Win32 Standard/SSE/SSE2 and Win64 builds under Windows - selection should be automatic based on the detected OS and CPU capabilities, but an 'advanced' user could override it (in case they want to try something or the detection is incorrect)
While standard builds are supposed to vanish, it is an interesting thought.

On old versions of MODs
Is there any particular reason for not leveraging an existing proper version-control system that already handles the full revision history?
[snip]
This is essentially what I had in mind for the installer. Though it will likely be more ackin to a package manager than a version control system. Also, I am not convinced that we will be able to drop in an existing version control system for two reasons they don't handle large groups of binary files that well, and they are hard to tailor to the specifics of what we will need to do.  That being said, as I noted in the OP, I do not want to discuss the specifics of the implementation wxL2 at this point. Also see my response to The_E at the bottom of this post.

Scripting
Lua is the only choice.

It's the one used in FS2-Open, so many people already know it from both ends.
There is no point in asking modders to learn a new language, or coders to learn new ways of handling hooks.
Noted. Still looking for more input from others, of course.

Further notes
The entire system is going to need fairly powerful Administrators.
This is going to be quite a lot of work.

Turey's Installer fell over for a while because nobody was doing the important work needed to keep that up to date.

So whatever system is built, there will need to be a very simple and easy to use multiplatform tool to handle the updates that can be taught to your average modder in a few minutes over IRC or similar.

Otherwise, sooner or later the new system will lose a key person (even if only temporarily) and fall apart.
Agreed. The administrative interface of the wxL2 platform is something that is going to be a large part of this project.  Of course we will try and exploituse the input of the end users to help with the administration. 

Yes, this will be a lot of work, and to be clear this is not even in the prototyping phase. That is, actual technical work will not start on this until wxLauncher 1.0 is released.

I feel like this came up already in one of the other discussions on the subject, but the ability to create "profiles" of specific launcher flag combinations for particular mods (beyond the -mod flag itself) would be very useful.  It doesn't come up all that often, but there are certain mods that require specific launcher flags to be turned on or off; beyond the obvious TBP and WCS flags, I know Windmills needed certain lighting settings to achieve its effect.  I can't think of a specific application, but one could even go beyond just flags to general settings as a whole, like changing the video resolution.  I'm not sure if this would be best achieved by integration with the existing mod.ini system, or some external means, but I'll leave that to those who are actually technically-oriented.
Is this any different than the current planned implementation of profiles wxL? (See the Overview of wxLauncher linked in the first paragraph of the first post).

I think I mentioned this on IRC before, but I'll mention it again here.

IMHO, the best model for version management would be something akin to Debians' apt. The launcher would record metadata about the mod (comprised of something like version + md5 checksum of any given asset), and use that to determine whether or not an update is needed.

Yes, is why I mentioned package managers in the 6th paragraph of the "Overview of wxLauncher 2.0".  And a package manger model is the most likely solution, but Tomo's response is the reason that we have posted this Request for Comments to start with.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: Mongoose on April 11, 2010, 02:32:03 am
I feel like this came up already in one of the other discussions on the subject, but the ability to create "profiles" of specific launcher flag combinations for particular mods (beyond the -mod flag itself) would be very useful.  It doesn't come up all that often, but there are certain mods that require specific launcher flags to be turned on or off; beyond the obvious TBP and WCS flags, I know Windmills needed certain lighting settings to achieve its effect.  I can't think of a specific application, but one could even go beyond just flags to general settings as a whole, like changing the video resolution.  I'm not sure if this would be best achieved by integration with the existing mod.ini system, or some external means, but I'll leave that to those who are actually technically-oriented.
Is this any different than the current planned implementation of profiles wxL? (See the Overview of wxLauncher linked in the first paragraph of the first post).
No, that's pretty much exactly what I was hoping for. :) I thought I'd remembered it being discussed earlier, but I didn't happen to check the link.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: Tomo on April 11, 2010, 05:02:18 am
While standard builds are supposed to vanish, it is an interesting thought.
Oops - I didn't mean Standard/Inferno, rather 'no SSE', SSE and SSE2. (Although I suppose that eventually there won't be any machines that don't have SSE)

Thank you, this is why it is on the list of questions, we are looking for more input on this item, namely on how integrated we want it to be with wxL.  That is, do we just provide some pre-built binaries for the major linux distros (.debs, .rpms, etc) and coerce the platform's package manger into doing our bidding; or do we make the linux users compile, then copy the binaries into the TC folder as we do now; or do we have wxL interface with the platform's build system to build the binaries for the user?

Getting the core executables into the 'common' platform package managers would be a very good thing, but that would fall outside the scope of this project - if it's going into the genuine package manager then it needs to be managed by the distribution, not us.
I'm unsure of exactly how to ask Debian, Ubunbtu, Red Hat etc to add a given set of binaries to their package managers, and they may not anyway as the required game data isn't free, and can't be included in any free package.

So, getting wxL to interface with the platform's build system would be perfect, and a very good measure regardless of whether Debian et al add FS2-Open to their package managers.
The 'official' packages will always be a bit behind, and now that Linux is starting to go mainstream there will be an increasing number of users who aren't sure how to rebuild and copy themselves.

Also, VC++ Express editions are freely available to anyone that wants to download it. The VC++ Express edition allows the compiling of FSO on windows, even though FRED is currently not available (but FRED not being build-able is a problem on Linux as well).
While VC++ Express editions are available for zero money, it's not 'freely available'.
A Windows user has to download it from Microsoft (at a location that moves each year) and install then activate it providing quite a lot of info to Microsoft, and agreeing to a rather large licence agreement.

We can't ask Windows end users to do this simply to play the game - it will scare them!
- Anyone interested in the code for the game engine can be pointed to this forum, where the latest available of VC++ Express, suitable svn tools etc are easy to find.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: The E on April 11, 2010, 07:44:41 am
Well, what you can do is offer to build from source if VS Express or the full VS is installed.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: chief1983 on April 11, 2010, 02:25:24 pm
Tomo, you can probably fire up the command line compiler after install without ever needing to activate the project.  But there are non-MSVC compilers for Windows too, it's just that we don't use them currently for anything else in the SCP area.  But source releases for Windows end users are probably just a bad idea in general.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: The E on April 11, 2010, 02:39:12 pm
Yep. Using the precompiled exes is the best way to go on Windows.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: Iss Mneur on April 11, 2010, 02:57:52 pm
Tomo, you can probably fire up the command line compiler after install without ever needing to activate the project.  But there are non-MSVC compilers for Windows too, it's just that we don't use them currently for anything else in the SCP area.  But source releases for Windows end users are probably just a bad idea in general.

Yes you can, the activation is only for visual studio itself and not build tools.

To be clear, I was not planning on having wxLauncher build the binaries on Linux, OSX or especially Windows unless actually needed.  As the newbie linux users are likely to be on the major popular distros anyway, building on linux will not be needed in all cases. But as Tomo noted previously, the linux installs will likely have to build the releases anyway because we are not going to be able to maintain binary packages for all distros.  But because of this potential need to build the binaries anyway, there is no reason that we cannot expand this ability to all platforms.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: Spicious on April 11, 2010, 04:56:59 pm
Which APIs aren't compatible across distros?

What is scripting meant to achieve in this context?
Title: Re: wxLauncher 2.0 Request for Comments
Post by: Iss Mneur on April 11, 2010, 11:19:33 pm
Which APIs aren't compatible across distros?
Its not that the APIs necessarily change but the location of the .so (dynamic link library) is not consistent.  Some platforms have support for the LSB to help minimize this, but I don't know if the APIs that we require are actually specified in the LSB.

Also some of the APIs that we link to are only source compatible based on the version that is actually installed. Lua and the SDL would be good examples of APIs that we use that we are source compatible with, that is, we can use a range of versions of the APIs but only when compiled against those APIs.

What is scripting meant to achieve in this context?
Scripting mostly for configuring wxLauncher so that it is able to configure FSO.  That is, automatically create profiles for the user to choose.

Scripting may also be needed to deal with some of the non-standard installers that some mod teams use, though this may or may not be a problem depending on the specifics of how wxL2 is implemented and how it handles the MOD data.  That is, does wxL2 force all data to be distributed with the wxL2 platform as zipped .vps or do we allow the mod teams to provide an self extracting installer. If we do force .vp only systems we are also potentially going to need someway for the TCs (in particular) to create shortcuts to starting the TC.

I realize that everything that I describe above about scripting could be accomplished by a .tbl or .ini-like configuration file or even in the MOD/TC's meta-data that wxL2 has on file.  On the other hand allowing a MOD or TC to have some code executed on its behalf to implement features that wxLauncher will not implement could be handy (like extracting unsupported file archives).

The potential FSO binary building system could also exploit a proper scripting language to allow for different scripts to execute on different platforms so that we can move this data out of the wxL binary so that it can be updated without forcing an update to wxL itself to support a new platform.

I also realize that the above ideas about scripting could be needlessly complex, but that is why we are discussing this here, to help determine these things.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: Iss Mneur on April 30, 2010, 02:33:47 pm
What is scripting meant to achieve in this context?
Scripting mostly for configuring wxLauncher so that it is able to configure FSO.  That is, automatically create profiles for the user to choose.

Scripting may also be needed to deal with some of the non-standard installers that some mod teams use, though this may or may not be a problem depending on the specifics of how wxL2 is implemented and how it handles the MOD data.  That is, does wxL2 force all data to be distributed with the wxL2 platform as zipped .vps or do we allow the mod teams to provide an self extracting installer. If we do force .vp only systems we are also potentially going to need someway for the TCs (in particular) to create shortcuts to starting the TC.

I realize that everything that I describe above about scripting could be accomplished by a .tbl or .ini-like configuration file or even in the MOD/TC's meta-data that wxL2 has on file.  On the other hand allowing a MOD or TC to have some code executed on its behalf to implement features that wxLauncher will not implement could be handy (like extracting unsupported file archives).

The potential FSO binary building system could also exploit a proper scripting language to allow for different scripts to execute on different platforms so that we can move this data out of the wxL binary so that it can be updated without forcing an update to wxL itself to support a new platform.

I also realize that the above ideas about scripting could be needlessly complex, but that is why we are discussing this here, to help determine these things.

I remembered the other reason that I wanted scripting. I promised The_E (and the rest of the support ninjas) to have wxL help in diagnosing the users problems.  That is, have wxL run tests and diagnostics to help the users before the post even makes it to the forum.  Since starting wxL I have read the support forums very regularly, and it seems a large number of the issues can be dealt with by very simple automation.  Of course some of the automation can be done in C++, but other parts can and should be more easily changeable without requiring that everyone download a point release of wxL, when only a couple of lines change.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: chief1983 on April 30, 2010, 03:14:26 pm
That sounds like it could open up a bunch of opportunities, I like it.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: RedDwarf on July 08, 2010, 11:31:58 am
Thank you, this is why it is on the list of questions, we are looking for more input on this item, namely on how integrated we want it to be with wxL.  That is, do we just provide some pre-built binaries for the major linux distros (.debs, .rpms, etc) and coerce the platform's package manger into doing our bidding; or do we make the linux users compile, then copy the binaries into the TC folder as we do now; or do we have wxL interface with the platform's build system to build the binaries for the user?

Getting the core executables into the 'common' platform package managers would be a very good thing, but that would fall outside the scope of this project - if it's going into the genuine package manager then it needs to be managed by the distribution, not us.
I'm unsure of exactly how to ask Debian, Ubunbtu, Red Hat etc to add a given set of binaries to their package managers, and they may not anyway as the required game data isn't free, and can't be included in any free package.

So, getting wxL to interface with the platform's build system would be perfect, and a very good measure regardless of whether Debian et al add FS2-Open to their package managers.
The 'official' packages will always be a bit behind, and now that Linux is starting to go mainstream there will be an increasing number of users who aren't sure how to rebuild and copy themselves.

I have a package for openSUSE... but users need to build it. That's becasue the non-commercial part of the license. Such a thing is against OSI's Open Source definition and FSF's Free Software definition, so most distros will not package it. And it could be also a problem for openSUSE because of Novell (and RedHat/Fedora? Others?). Things would be easier without that part of the license, but I suppose it would be difficult to change.

Which APIs aren't compatible across distros?
Its not that the APIs necessarily change but the location of the .so (dynamic link library) is not consistent.  Some platforms have support for the LSB to help minimize this, but I don't know if the APIs that we require are actually specified in the LSB.

Also some of the APIs that we link to are only source compatible based on the version that is actually installed. Lua and the SDL would be good examples of APIs that we use that we are source compatible with, that is, we can use a range of versions of the APIs but only when compiled against those APIs.

The dynamic libraries can be anywhere specified at /etc/ld.so.conf (and you can add more paths with a file in /etc/ld.so.conf.d/ with a sane /etc/ld.so.conf). But in every normal distro they will be at /usr/lib or /usr/lib64. What's the standard place in Windows? If you prefer you can use rpath and put the libraries in the same dir than the binary or any other path (absolute or relative, with $ORIGIN). Or naturally static link them. But I don't know why this was raised, the binary has no the library paths hardcoded and you don't need to provide any library, only a binary.

About the libraries that the binary uses... If they break source or binary compatibility in Linux they will also break it in Windows, and:
- libSDL: stable if just because it is rarely updated. I don't understand what you said about it, all the 1.2.x series are binary compatible.
- libvorbis: stable
- libvorbisfile: stable
- libogg: stable
- libtheora: stable
- libpng: ok, it recently broke backward compatibility. But has been stable for a long time, and anyway distros will provide both 1.2 and 1.4 versions for a time.
- libGL: stable
- libGLU: stable
- libopenal: stable if just because it is rarely updated
- libjpeg: similar to png case
- liblua: no idea. It seems little stable, good candidate to static link.
- libstdc++: stable, broke for last time with... gcc 3.4? That was 2004, and because of an exceptional event. It even uses symbol versioning to make things easier.
- libc/m/pthread: rock stable. It even uses symbol versioning to make things easier.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: chief1983 on July 08, 2010, 11:46:13 am
libopenal is actually one of the worst recent offenders.  Some distros have switched to seemingly only offering openal-soft, which uses a differently names .so and does not provide a symlink even though the new one is backwards compatible.  Thus, binaries built on Debian don't seem to work on Ubuntu and vice versa.

Title: Re: wxLauncher 2.0 Request for Comments
Post by: RedDwarf on July 08, 2010, 11:58:06 am
libopenal is actually one of the worst recent offenders.  Some distros have switched to seemingly only offering openal-soft, which uses a differently names .so and does not provide a symlink even though the new one is backwards compatible.  Thus, binaries built on Debian don't seem to work on Ubuntu and vice versa.
Uhm, openSUSE provided the symliink, though everybody did.
But this is another very exceptional case. When an specific library breaks binary compatibility just static link (or rpath) it for some time*. Isn't that the same that you did in Windows?

It seems to me easier than try to automatically compile it (ask the native package manager for -devel packages with a different name in every distro).


* Or, in this specific case, dlopen it. If .so.0 fails try .so.1.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: chief1983 on July 08, 2010, 12:05:43 pm
No in Windows openal-soft is set up differently, we only use the reference implementation until recently and the openal-soft library is installed to the game folder on an as-needed basis currently.

I'd love if we could have a binary that works on most distros without hassles, but so far it hasn't always worked that way.  Help in that area would be appreciated.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: RedDwarf on July 08, 2010, 12:34:43 pm
No in Windows openal-soft is set up differently, we only use the reference implementation until recently and the openal-soft library is installed to the game folder on an as-needed basis currently.

I'd love if we could have a binary that works on most distros without hassles, but so far it hasn't always worked that way.  Help in that area would be appreciated.
Sorry, I never used the Windows version. "installed on an as-needed basis"? How, from the launcher? It was my understanding that the Windows version also needs of an OpenAL library. And since Windows will not provide it by default you would include it with the binary (either the reference version or openal-soft).

In Linux the libraries are searched (from man ld.so):
Quote
       The shared libraries needed by the program are searched for in the following order:

       o  (ELF only) Using the directories specified in the DT_RPATH dynamic section attribute of the binary if present and DT_RUNPATH attribute does not exist.  Use of DT_RPATH is dep-
          recated.

       o  Using the environment variable LD_LIBRARY_PATH.  Except if the executable is a set-user-ID/set-group-ID binary, in which case it is ignored.

       o  (ELF only) Using the directories specified in the DT_RUNPATH dynamic section attribute of the binary if present.

       o  From the cache file /etc/ld.so.cache which contains a compiled list of candidate libraries previously found in the augmented library path.  If, however, the binary was  linked
          with the -z nodeflib linker option, libraries in the default library paths are skipped.

       o  In the default path /lib, and then /usr/lib.  If the binary was linked with the -z nodeflib linker option, this step is skipped.
So you can build fs2_open against openal-soft using rpath to say the binary to search the library in the same directory the binary is installed (using "$ORIGIN" as rpath). The binary will search for libopenal.so.1 in the same dir the binary is installed (you distribute both). And if the user wants to use the system openAL library he can just delete the version distributed with fs2_open and the linker will correctly find the system's version.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: chief1983 on July 08, 2010, 12:40:14 pm
We have never used OpenAL-Soft until recently, it's apparently helping with some issues in the new audio code.  Currently that library is only being installed by hand for testing by copying it to the game directory.  If it helps enough we may find a way to have the new launcher/installer pull it down.

We're not distributing a version of libopenal.so.1 with fs2_open on Linux, but you're saying there's a way to have it find libopenal.so without needing a symlink?  That's essentially what we need in this particular case.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: Iss Mneur on July 08, 2010, 12:56:43 pm
I have a package for openSUSE... but users need to build it. That's becasue the non-commercial part of the license. Such a thing is against OSI's Open Source definition and FSF's Free Software definition, so most distros will not package it. And it could be also a problem for openSUSE because of Novell (and RedHat/Fedora? Others?). Things would be easier without that part of the license, but I suppose it would be difficult to change.
OpenSUSE doesn't allow third party binary packages and/or third party package repositories?

At least in the case of Ubuntu, IIRC FSO does qualify under the universe class though.  wxLauncher itself is GPL (because it is original code), but because wxL is useless without FSO it would probably also end up in universe (again IIRC).  Geting wxLauncher into the distro repos is something that one TC has expressed interest in to me about, so that they can simplify the release of there TC.

Removing the non-commercial clause of the source code would be great, but the license of FSO can only be changed by :V: and/or Interplay.  Aslo we would need to go through the code repo and get the permission of everyone that has contributed code to the project, some (most?) of whom are no longer active.

In any case, because the current linux build system that FSO uses handles the iterations and variations on the different distros rather well, wxL may just end up making sure that all of the required dependencies get installed and then run the build manually using Linux platform build system.

Which APIs aren't compatible across distros?
Its not that the APIs necessarily change but the location of the .so (dynamic link library) is not consistent.  Some platforms have support for the LSB to help minimize this, but I don't know if the APIs that we require are actually specified in the LSB.

Also some of the APIs that we link to are only source compatible based on the version that is actually installed. Lua and the SDL would be good examples of APIs that we use that we are source compatible with, that is, we can use a range of versions of the APIs but only when compiled against those APIs.

The dynamic libraries can be anywhere specified at /etc/ld.so.conf (and you can add more paths with a file in /etc/ld.so.conf.d/ with a sane /etc/ld.so.conf). But in every normal distro they will be at /usr/lib or /usr/lib64. What's the standard place in Windows? If you prefer you can use rpath and put the libraries in the same dir than the binary or any other path (absolute or relative, with $ORIGIN). Or naturally static link them. But I don't know why this was raised, the binary has no the library paths hardcoded and you don't need to provide any library, only a binary.

Interesting.  I did not know that.  That being said, the concern was not the directories that .so's are in, but the actual names of the libraries in those folders (yes, I know I said location in the quote above, but I meant names). That is, libraries that are called .so on one platform, .so.5 on anther, and .so.5.1 on yet another.  Yes symbolic links would solve this, but the distro does not provide them, asking the user to do this is not something that is "user friendly".

In windows case, the names of the .dlls are actually set down by the library maintainer so we avoid the entire name problem.  But currently FSO statically links every library it uses even the CRT except for the OpenAL redirector library (which is downloaded by users as a separate download).  This is actually something we want to move away from as wxL2 is implemented because wxL2 will then be able to install the the dependencies properly.  The currently the only reason that FSO doesn't dynamically link against the libs is because of the distribution method that we use.

To reiterate, we understand that libraries change, and we can mostly deal with that.  The issue is that the same binary library can have different names on different distros (even different versions of the same distro), this is our problem when distributing binaries for linux.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: RedDwarf on July 08, 2010, 01:04:23 pm
We have never used OpenAL-Soft until recently, it's apparently helping with some issues in the new audio code.  Currently that library is only being installed by hand for testing by copying it to the game directory.  If it helps enough we may find a way to have the new launcher/installer pull it down.
Ok, now I understood it.

We're not distributing a version of libopenal.so.1 with fs2_open on Linux, but you're saying there's a way to have it find libopenal.so without needing a symlink?  That's essentially what we need in this particular case.
Well, it will always find "libopenal.so", whatever it's the reference version of openal-soft. But if the user didn't installed the development package probably it will not exists. The binary will search for either "libopenal.so.0" or "libopenal.so.1". And no, you can't use "libopenal.so.1" if the binary searchs for "libopenal.so.0" if you don't use a symlink*. But what you can do is distribute the library that you used in the compilation (either "libopenal.so.0" or "libopenal.so.1", I would recommend openal-soft) and make the binary search for it in the same path the binary is installed (normally it doesn't). It would be equivalent to Windows:
- openAL is distributed with the binaries
- The binary uses that version, so we are sure it's 100% binary compatible
- If the user deletes the included openAL, and he has a compatible version installed in its system, the system version is used.

The thing is:
- Use system versions when you are sure they will be compatoble for all users
- If for whatever reason you are not sure that all users will have a compatible version, distribute an specific version with the binary. If the user knows his system has a compatible version he can delete it at his own risk.


* You can if you use dlopen(). But that means modifying the code.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: chief1983 on July 08, 2010, 01:22:56 pm
Well modifying the code is, obviously, within our abilities :)

We should probably look into something like that.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: RedDwarf on July 08, 2010, 01:27:27 pm
I have a package for openSUSE... but users need to build it. That's becasue the non-commercial part of the license. Such a thing is against OSI's Open Source definition and FSF's Free Software definition, so most distros will not package it. And it could be also a problem for openSUSE because of Novell (and RedHat/Fedora? Others?). Things would be easier without that part of the license, but I suppose it would be difficult to change.
OpenSUSE doesn't allow third party binary packages and/or third party package repositories?
It does. It could be added to Packman (the most important one)... but it's already a third party repository. And in this specific case I tried and got no answer (could try harder ;-) )

Interesting.  I did not know that.  That being said, the concern was not the directories that .so's are in, but the actual names of the libraries in those folders (yes, I know I said location in the quote above, but I meant names). That is, libraries that are called .so on one platform, .so.5 on anther, and .so.5.1 on yet another.  Yes symbolic links would solve this, but the distro does not provide them, asking the user to do this is not something that is "user friendly".
But every distro that uses LUA 5.0 will use .so.5; and every distro that uses LUA 5.1 will use .so.5.1. The names are different to explicitly say LUA 5.1 (.so.5.1) is not binary compatible with LUA 5.0 (.so.5). That's not something that you can fix with a symlink. If you create a symlink from .so.5 to .so.5.1 (binary compiled with LUA 5.0, user has LUA 5.1) the binary will find the library... but that doesn't means it will work. The libraries are binary incompatible, and the binary can break at any moment.

In windows case, the names of the .dlls are actually set down by the library maintainer so we avoid the entire name problem.
In Linux it is also the library maintainer who sets the names. Were LUA maintainer who decided to change the name to explicit say "these two libraries aren't binary compatible".


Normally library maintainer will maintain the binary compatibility longer. But LUA decides to break it more frequently. It's a problem with LUA, and it effects both Windows and Linux.
What you do is such cases is static compile... in Windows, in Linux and in any other OS.
At least in openSUSE case. What we also do is provide both versions. The default package, liblua5_1, will provide LUA 5.1. But if needed the user can install lua50-libs. They can be used in parallel.


Edit: I missread a patch. It seems at least version 5.1 of LUA can't be compiled as a shared library by default. So distributiosn had to patch it and each one could have added a different name. In the case of LUA 5.1 seems everybody used .so.5.1 but with LUA 5.0 some used ".so.5" and others ".so.5.0". This makes binary compatibility slightly worse in Linux than in Windows where everybody will use http://luabinaries.sourceforge.net/. But the binary incompatibility between LUA 5.0 and 5.1 argument still applies.

Edit2: Well, looking at the quantity of versions available at http://sourceforge.net/projects/luabinaries/files/5.1.4/Windows%20Libraries/ I'm not so sure about the "binary compatibility slightly worse in Linux than in Windows" thing. But I don't really know about the incompatibilities between dlls generated by different compilers in Windows. You really need different version for MSVCRT.DLL, MSVCR71.DLL, MSVCR80.DLL, MSVCR90.DLL, MSVCR100.DLL plus Cygwin and MingW??
Title: Re: wxLauncher 2.0 Request for Comments
Post by: RedDwarf on July 08, 2010, 02:05:14 pm
Well modifying the code is, obviously, within our abilities :)

We should probably look into something like that.
I don't think the work needed to use dlopen is worth. Right now isn't everybody already using openal-soft?
In openSUSE at least, 11.0 came only with reference version, 11.1 came with both, and 11.2 came only with openal-soft. And 11.0 is already unsupported.
Lenny (stable Debian) already came with openal-soft. "oldstable" is the only supported version that still used the reference version.
In Ubuntu it's included since Intrepid. The only supported versions that still uses the reference version are Dapper and Hardy. They are supported just because they are LTS. And if someone that wants to play games wants a LTS he probably is using Lucid.

Perhaps when openal-soft was released would have made sense. But now?
Anyway. You just need to use http://www.opengroup.org/onlinepubs/009695399/functions/dlopen.html and http://www.opengroup.org/onlinepubs/009695399/functions/dlsym.html. Try using dlopening with different names until it returns != NULL.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: Iss Mneur on July 08, 2010, 03:04:10 pm
Edit: I missread a patch. It seems at least version 5.1 of LUA can't be compiled as a shared library by default. So distributiosn had to patch it and each one could have added a different name. In the case of LUA 5.1 seems everybody used .so.5.1 but with LUA 5.0 some used ".so.5" and others ".so.5.0". This makes binary compatibility slightly worse in Linux than in Windows where everybody will use http://luabinaries.sourceforge.net/. But the binary incompatibility between LUA 5.0 and 5.1 argument still applies.
To be clear, I pulled the numbers out of the air, I wasn't specifically thinking of lua when I wrote it.  As such, I understand about the binary incompatibilities, and like I said before there is nothing that we can do about it.  Our problem is with binary compatible libraries like what you said about lua version 5.0.

Also, keep in mind that a large part of the general "no binaries for linux" stance is that the SCP started with the linux porting sometime in 2004, when the distros did not play along as well with each other as they do now.  Also my opinon of this is also coloured by my experience installing linux game servers for other games in the time period (I had not heard of FSO at that time) and running into this same problem.  One distro would provide all symlinks for everything (for example .so, .so.1, .so.1.0, and .so.1.0.0 all pointing to the same binary) and other would only provide one or two of them.

I image reality has changed some since then, but for better or worse, fs2open is trapped in its own reality (which is a very windows biased reality).  We would do well with more people that are experienced with packaging and distribution of applications across multiple distros and operating systems other than windows.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: chief1983 on July 08, 2010, 03:24:59 pm
I plan on getting linux nightlies up again soon too, and we can see what the current state of running a Lucid built binary on other distros is.  Maybe it'll play nice now after all.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: RedDwarf on July 08, 2010, 03:37:08 pm
To be clear, I pulled the numbers out of the air, I wasn't specifically thinking of lua when I wrote it.  As such, I understand about the binary incompatibilities, and like I said before there is nothing that we can do about it.  Our problem is with binary compatible libraries like what you said about lua version 5.0.
LUA is just a library. Static compile it and problem solved. Since upstream didn't provided a name each distro had to guess one. But normal libraries already come to be compiled as shared libraries, and distributions don't change its names.

Also, keep in mind that a large part of the general "no binaries for linux" stance is that the SCP started with the linux porting sometime in 2004, when the distros did not play along as well with each other as they do now.  Also my opinon of this is also coloured by my experience installing linux game servers for other games in the time period (I had not heard of FSO at that time) and running into this same problem.  One distro would provide all symlinks for everything (for example .so, .so.1, .so.1.0, and .so.1.0.0 all pointing to the same binary) and other would only provide one or two of them.

I image reality has changed some since then, but for better or worse, fs2open is trapped in its own reality (which is a very windows biased reality).  We would do well with more people that are experienced with packaging and distribution of applications across multiple distros and operating systems other than windows.
Well. I just downloaded http://swc.fs2downloads.com/builds/LINUX/fs2_open_3_6_10.tar.bz2... and it's plain perfect binary compatibility-wise.

Statically linked LUA, PNG and JPEG: the libraries that recently broke binary compatibility. And doesn't uses any feature from glibc newer than 2.3 (from 2002). The most common problem with that binary should be:
- "I use X86-64/PPC and the binary is only for x86-32. My distro doesn't has libvorbis for 32 bits in a x86-64 system". But neither I see x86-64 Windows binaries.
- The OpenAL-Soft thing. But as already said very few distros still have a supported version without OpenAL-Soft. And I'm pretty sure the guy that created that binary knows how to statically link OpenAL.

Is people really complaining??
As said trying to automatically compile from the installer seems to me a lot more difficult. The binary that you already have must work plain fine. At most I would add a x86-64 version.



Not important, but about the ".so, .so.1, .so.1.0, and .so.1.0.0" thing. Only a file is needed at run-time, the one that matches the DT_SONAME entry in the library (probably .so.1). And the ".so" is needed only to compile (and not really needed if you know what you are doing). ".so.1.0, and .so.1.0.0" are there just as information, but are not really needed... a distribution can or can't include them, isn't really important.


Edit: I also took a look at http://swc.fs2downloads.com/builds/LINUX/fs2_open_3_6_12_RC3_INF.tar.bz2 and isn't so good. It will only work in a system with glibc >= 2.11 (from November last year). And since January this year libpng 1.4 was released this wasn't the best moment to start dynamically linking to libpng 1.2.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: chief1983 on July 08, 2010, 04:27:11 pm
I made that binary.  I might be able to statically link OpenAL but we've erred in favor of not statically linking unless we need to.  In fact coming up Lua won't be statically linked anymore.  I think most of the linux libs are dynamically linked now (or at least compiled against system libs vs those in SVN, it may still link some of those in statically).
Title: Re: wxLauncher 2.0 Request for Comments
Post by: RedDwarf on July 08, 2010, 05:02:26 pm
I made that binary.  I might be able to statically link OpenAL but we've erred in favor of not statically linking unless we need to.  In fact coming up Lua won't be statically linked anymore.  I think most of the linux libs are dynamically linked now (or at least compiled against system libs vs those in SVN, it may still link some of those in statically).
Sure, if you have the option dynamic linking is always better.

From the number I though LUA 5.1 was a version released 6 months after LUA 5.0. But from its web it seems the first 5.0 version is from 2003-04-11 and the latest from 2004-03-17 (plus what seems an important bug fix release at 2006-06-19). The first LUA 5.1 is from 2006-02-20 and the latest from 2008-08-18. It all depends on how many distributions you want to support, but it seems any distro less than four years old will have LUA 5.1. So doesn't seems a big problem to dynamic link it.
It feels as is 5.2 will be released soon ("Lua 5.2 (work3)" released in May). When that happens new distros probably will install 5.2 by default but will maintain 5.1 packages for compatibility. But users will need to explicitly install those 5.1 versions. Again, it depends on which kind of support you want to provide.

About PNG: openSUSE 11.3 will be released next week. It will install libpng 1.4 by default. So if someone tries to run the 3.6.12 RC3 version he will need to explicitly install libpng12, the compatibility package. I suppose others will do the same.

About JPEG: If you start to dynamically link against it. You will find that an openSUSE 11.2 (and any other distro from the latest 10 years) user only has libjpeg.so.62. But an openSUSE 11.3 has libjpeg.so.8, he will need to explicitly install libjpeg6. It also existed a short lived libjpeg7, I expect nobody to use it.


So. It's a problem if users need to install compatibility packages? Every important distro provides these compatibility packages or only the latest versions? From the answer to these questions the decision has to be taken.


And then is the glibc thing. The 3.6.12 RC3 binary requires a very recent glibc. I think you need to compile with "-fno-stack-protector" to lower the requirement, but this way the binary will be less secure... what to do?
Title: Re: wxLauncher 2.0 Request for Comments
Post by: RedDwarf on July 09, 2010, 09:55:03 am
Looking for input
The following are items that we are looking for more information on.
  • Should wxLauncher be able to build FSO from source on Linux at least, maybe windows and osx?
  • Because we are keeping the old files so that we can provide specific diffs, do we allow the user to request older versions of resources? How long do we keep them the old files?
  • Who/Where are the mod files going to be stored?  In particular wxL2 would require MOD/TC data hosting with resume support that allows hot linking.
a) As already said. I think it's easier to provide binaries compatible with any distro than automatically compile (dependencies, code stops building with newer gcc versions, etc.)
b) I would provide a diff from X-1, X-2, X-N (decide N on a per-MOD basis) and the full version of the latest version. No need to provide full versions of old versions.
c) Each file could be in a different host, as now. In the central server you just need signed metadata that points to the data.
Title: Re: wxLauncher 2.0 Request for Comments
Post by: chief1983 on July 09, 2010, 10:24:20 am
Currently, the codebase seems to be compatible with libpng-1.2 and -1.4.

We could lower the glibc it's compiled against I suppose, until more distros catch up and lock stuff down.