[Hellzed]

Main goals of this project:

• Easy to use one-stop solution to installing, playing and installing/updating mods for FSO
• Make modding easier (this is a secondary goal but it is still important)

A few months ago, ngld and I worked on a somewhat similar project. The app itself worked pretty well for an alpha software (it had some bugs of course, but no big show stopper), but we ran into some big issues :

• Mods have currently no standard files structure. The only workaround is to handle many different types of archives, and use "descriptor" files to rebuild files structure from scratch, for each mod. It adds a maintenance burden on whoever runs the mode repository.
• fs2_open itself and VP files have no clue about versioning. Versions had to be handled by the "descriptor" files too. When you have mods from different sources (the main repository, and some updated or unsupported mod repos), you get conflicts. Solving these conflicts is a PITA.
• Some fs2_open command line flags are undocumented (especially anything related to sound). You can try to use OpenAL standards, and fs2_open behaves in a completely differend, or completely weird way. Development process becomes a game of trial and error, much more than it should.
• Same applies for saved settings : different defaults apply to Windows/*nix platforms. I believe it's fixed in trunk, with Windows builds switching to .cfg files, instead of using the registry.
  

You can try to fix these issues by yourself, which is ok, but it will quickly become a hassle.
It would be better to fix the root issue : fs2_open needs a mod package standard that every launcher and mod manager out there could implement.

ngld and I both prioritized RL since march, so our project is dead at the moment, but if you want to take a look, the dev logs are still on the forum, and the git repository is on github (everything is written in Python, with a Qt UI).
The end goal was to integrate a new version of the mod database and the launcher/installer like Steam does.

[m!m]

• Mods have currently no standard files structure. The only workaround is to handle many different types of archives, and use "descriptor" files to rebuild files structure from scratch, for each mod. It adds a maintenance burden on whoever runs the mode repository.
• fs2_open itself and VP files have no clue about versioning. Versions had to be handled by the "descriptor" files too. When you have mods from different sources (the main repository, and some updated or unsupported mod repos), you get conflicts. Solving these conflicts is a PITA.

My idea was to have multiple "repositories" of JSON files that contain information about what versions a mod has and where they are available, those would be what you call the descriptor files.
I don't understand what you mean by by "rebuild files structure from scratch". Mods have a very well defined file structure and any released mod will most be distributed as VP or later 7zip/Zip so the installer would only have to unpack the files somewhere and set the correct -mod paths. Could you explain what you mean by that?

Versioning is a problem but for our limited application we probably only need to handle "new version available? Download the whole thing!" although being able to have incremental patches would be a nice thing. Now that I think of, it shouldn't be that hard to implement...

• Some fs2_open command line flags are undocumented (especially anything related to sound). You can try to use OpenAL standards, and fs2_open behaves in a completely differend, or completely weird way. Development process becomes a game of trial and error, much more than it should.
• Same applies for saved settings : different defaults apply to Windows/*nix platforms. I believe it's fixed in trunk, with Windows builds switching to .cfg files, instead of using the registry.
  

Yeah, most of the launcher stuff is undocumented but I tend to just take a look at the FSO source code anyway so that is not a problem for me.

The configuration is more or less easy to handle, you just need to implement the different backends (registry on Windows and .ini files on other platforms). Currently every branch still uses the old way of saving/loading stuff from the config but I have a patch that moves everything to using .ini files and once that hits trunk I will gladly delete the registry stuff from this launcher

[Darklord42]

FYI as far as OSX is concerned, GTK# can make use of the quartz backend and does not need to use X11 (which is good as x11 doesn't even come with OSX anymore) However, I can tell you from experience that GTK# is a #$%@ to get bundled in an OSX app.  It is possible, but it took me many months to wrangle xboard into one.

[Hellzed]

@m!m: I know mods have a well defined file structure once installed, but many mods are broken into different archives for easier downloading, and these archives follow no rule regarding their root folder (some have all files directly put in the root of the archive, others use a folder named like the mod as their root). It can even change from one archive to the other for a single version of a mod. It gets even worse for mods not using VPs.
That's not a big issue, but you have to keep that in mind when you build your JSON descriptor files. The only reliable information you can feed your mod installer with will come from these files, because you can't trust what's in the archives.

How do you plan to handle conflicts ?
Let's say A mod needs an old version of fsport, and B mod needs a newer version of fsport. Both versions of fsport share the same name, and are supposed to install in the same folder.
ngld and I thought about mod.ini rewriting and alias creation but it would have been messy (imagine fsport_33 and fsport_34 aliases, and mod depending on these having their "primary" mod list rewritten to fsport_xx on install)

To me, the only "clean" way to solve this is to build a better mod selection logic inside fs2_open.
Each mod should know it dependencies (min and max version of said dependencies too), and pass them to fs2_open. fs2_open would then search the root folder for the right mod  version (whatever the mod folder name is), and output a clear error in some cases (dependency not found, dependency too recent or too old). Each mod should come with a "version file" containing a version string, so different versions of a same mod could be installed in parallel.

[m!m]

@m!m: I know mods have a well defined file structure once installed, but many mods are broken into different archives for easier downloading, and these archives follow no rule regarding their root folder (some have all files directly put in the root of the archive, others use a folder named like the mod as their root). It can even change from one archive to the other for a single version of a mod. It gets even worse for mods not using VPs.
That's not a big issue, but you have to keep that in mind when you build your JSON descriptor files. The only reliable information you can feed your mod installer with will come from these files, because you can't trust what's in the archives.

Now I understand, yes that is an issue but that can be handled by the JSON files.

How do you plan to handle conflicts ?
Let's say A mod needs an old version of fsport, and B mod needs a newer version of fsport. Both versions of fsport share the same name, and are supposed to install in the same folder.
ngld and I thought about mod.ini rewriting and alias creation but it would have been messy (imagine fsport_33 and fsport_34 aliases, and mod depending on these having their "primary" mod list rewritten to fsport_xx on install)

To me, the only "clean" way to solve this is to build a better mod selection logic inside fs2_open.
Each mod should know it dependencies (min and max version of said dependencies too), and pass them to fs2_open. fs2_open would then search the root folder for the right mod  version (whatever the mod folder name is), and output a clear error in some cases (dependency not found, dependency too recent or too old). Each mod should come with a "version file" containing a version string, so different versions of a same mod could be installed in parallel.

I was thinking of completely deprecating the usage of mod.ini files. The JSON mod descriptors have all the information needed to get a commandline to work. To handle conflicts I would install the mods into a folder structure like {modname}/{version}. The installer then could use that folder structure to construct the -mod commandline argument. I already build the mod handling with something like that in mind.

Integrating the mod selection logic inside FSO doesn't sound like a good idea as that is something the launcher should handle as implementing that logic inside FSO would take a lot of work and the launcher has all the information anyway.

[ngld]

How do you want to write / generate those JSON files and what are they going to contain?
I wrote a python script which takes Goober's installer text files, parses them, downloads the mod archives and generates JSON files which contain all mod dependencies, file checksums and so on.

If you want to make the launcher multi-platform you'll have to look out for path case issues: Most mod developers use Windows which is case-insensitive. If you install the MediaVPs, you'll end up with MediaVPs_2014 and mediavps_2014. The mod installer has to choose one of those folders and move all files inside that one. Goober's installer doesn't handle this correctly.

One problem left in fs2mod-py is dependency tracking. The mod.ini files list mod folders like "blueplanet" but you never know exactly which version is required and which FSO version is needed to run the mod.

[m!m]

Ideally the new format would be created based on a discussion in the community.
What I am currently thinking of is at least:

• The user facing name of the mod
• A description
• An unique ID that can be used for installation folders
• The current version in the Major.Minor.Release.Patch format to handle versioning consistently
• A list of mod IDs that this mod depends on along with a version specifier, either minimum and maximum or some special format (I like the format Composer uses but that may be a bit too complicated to implement)
• A list of URLs where the files of the mod can be downloaded along with mirrors and checksums
• I'm not really sure about this but maybe some sort of optional script to do special things on installation

While I don't think that case issues is something the launcher/installer should handle as that is a problem of the mod author I think extracting everything to lower case file names should not be hard to do.

Dependency tracking would be included in the JSON file along with information about what version of the dependency is required.

[Hellzed]

We need a common way to support for optional files in the mod (compatibility packages, hi res stuff, cutscenes...).
EDIT : we also have to add fs2_open compatibility information, with "branches" (standards fso builds, blue planet builds, diaspora builds...) and min/max versions for each branch. Installers should check if a compatible build is installed on mod install, and if it's not, download the latest compatible build and create launcher settings for the mod, with the right build.
At this point, it becomes too complicated for good old .fs2mod files, I want to drop them in favour of the new common JSON files.
EDIT 1.5 : some optional files will need the ability to override fs2_open build compatibility. Let's say a modder adds a new shader pack which only works with newer builds, this feature would be needed.
EDIT 2 : If we still allow mods to be packaged in weird ways (all kinds of archive formats, all kinds of file structure with no clear relation to fs2_open mod folders standards), the new JSON should store the final path for unpacked files.

Writing a php toolbox to manage these repositories shouldn't be a problem. With Symfony framework, I would only need the regex to make sure every bit of information we store is ok. Then generating JSON from a template would be easy.

[AdmiralRalwood]

with "branches" (standards fso builds, blue planet builds, diaspora builds...)

Except both Blue Planet and Diaspora run on standard builds...

[Fury]

with "branches" (standards fso builds, blue planet builds, diaspora builds...)

Except both Blue Planet and Diaspora run on standard builds...

They do now but they used to have custom builds because SCP couldn't keep up with official stable releases. It is not safe to assume that mods/tc's use official builds every time.

[Hellzed]

Ok, new try : Wing Commander Saga ?
Last march, before I took a break from working on fs2mod-py, BP was still using custom builds. Since then, I updated BP, but as I run on svn trunk builds I didn't check standard builds.
I don't have anything to add to what Fury said. Mods using different branches happened before, it may happen again. Better be on the safe side.

[mjn.mixael]

WCS doesn't count. They forked the code-base. For all real-world purposes, they are no longer using SCP at all.

[AdmiralRalwood]

And there's also an alpha release of a port of WCS to FSO: http://www.hard-light.net/forums/index.php?topic=85375.0

[X3N0-Life-Form]

And I'm working improving said release so that someday one will be able to run WCS on regular FSO builds without a hitch.

[karajorma]

To be honest, I'd rather see us get together and actually plan what we're going to do about a launcher properly before people run off and start coding them. Cause this sort of nonsense is exactly what happened last time.

[m!m]

I have been updating the launcher over the past few months and now I think I have implemented most features required for a standard launcher.
There is currently a discussion about what format to use for mod package management which I want to support when there is an agreement on what is needed.

To be honest, I'd rather see us get together and actually plan what we're going to do about a launcher properly before people run off and start coding them. Cause this sort of nonsense is exactly what happened last time.

It's fairly obvious what a simple launcher needs to do but I see your point. I would like to see a discussion about what the launcher should be able to do and how it should do so. If anyone has suggestions I would be glad to hear them and implement any changes that can improve the launcher.