Author Topic: [Cross-platform] FS2 GOG/Mod installer  (Read 72207 times)

0 Members and 1 Guest are viewing this topic.

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
@m!m: The link is https://fsnebula.org/repo/test.json. Your launcher has to support the "includes" field and TLS >= 1.0 to access that repo.
Thank you very much for setting up that domain and the server. I just tested my client an apparently .NET supports SSL/TLS out of the box :D

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
The amount of work put into this project in such a short time span astounds me.  I'm watching this project closely.

Thanks! However, most of these changes are UI changes which expose the functionality. I've been working on this new version for several months now.
The first commit was over a year ago but that back then I was still trying to use Goober's textfiles. m!m later convinced me to use a new schema. Hellzed wrote the website (fsnebula.org) which (once it's finished) will allow modders to add/publish new mods and updates.
I guess I just want to finally release a stable version.  :lol:

Thank you very much for setting up that domain and the server. I just tested my client an apparently .NET supports SSL/TLS out of the box :D
Nice!

I've just finished the last changes. Once the packages are done, I'll edit this post and add a link to the installation instructions.
EDIT: Done.

Here are the installation instructions. If you have problems installing Knossos or your OS is missing, feel free to post here or on the bugtracker.
It would be nice to get some feedback regarding the UI (Is it too complicated? Is it self-explanatory or should I add more text?).

I'm looking forward to your bug reports  :).
« Last Edit: January 29, 2015, 07:42:45 pm by ngld »

 

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: [Cross-platform] FS2 GOG/Mod installer
Oh I know.  I was referring to everything since page 1.  Has it been a year?  Still seems fast to me.
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline Goober5000

  • HLP Loremaster
  • 214
    • Goober5000 Productions
Re: [Cross-platform] FS2 GOG/Mod installer
back then I was still trying to use Goober's textfiles

They're not really mine; Turey invented the format and I continued to use it.  It would probably be a good idea to make the FSO Installer compatible with the Knossos spec.  Do you have a spec definition somewhere?
« Last Edit: January 29, 2015, 08:20:09 pm by Goober5000 »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
They're not really mine; Turey invented the format and I continued to use it.
I didn't know. I just saw that your installer used it and never actually wondered how Turey's installer worked.

It would probably be a good idea to make the FSO Installer compatible with the same spec.  Do you have a spec definition somewhere?
Here's the spec, it's split into two parts. The top part is the format my converter expects. The final JSON file contains all the fields listed in the first section but the "files" property is replaced with the format documented in the second section and the "filelist" property is added which contains the file layout of the installed mod.
As an example: This is the JSON file for FSO 3.7.2 RC5.
The filelist describes which files should be in the mod folder, which archive contains the file and the path in the archive. The "files" property lists all archives and their URLs.

 

Offline Goober5000

  • HLP Loremaster
  • 214
    • Goober5000 Productions
Re: [Cross-platform] FS2 GOG/Mod installer
Thanks.  I'll put this on my lengthy to-do list. :nervous:

 

Offline niffiwan

  • 211
  • Eluder Class
Re: [Cross-platform] FS2 GOG/Mod installer
Here are the installation instructions. If you have problems installing Knossos or your OS is missing, feel free to post here or on the bugtracker.
It would be nice to get some feedback regarding the UI (Is it too complicated? Is it self-explanatory or should I add more text?).

Nice work! The latest version showed up as an update in my package manager (on Mint 17) so super-simple to upgrade  :yes:

I also noticed a possible memory leak during the install; it's on the bugtracker.

Regarding the UI, it mostly seems fine to me. The dropdown + checkbox for the exec flags seems a little bit counter-intuitive.  That could be just because I'm used to seeing all the checkboxes in one big list (like wxLauncher).
Creating a fs2_open.log | Red Alert Bug = Hex Edit | MediaVPs 2014: Bigger HUD gauges | 32bit libs for 64bit Ubuntu
----
Debian Packages (testing/unstable): Freespace2 | wxLauncher
----
m|m: I think I'm suffering from Stockholm syndrome. Bmpman is starting to make sense and it's actually written reasonably well...

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
I've released 0.1.1. The new version should fix the memory leak and an issue which might cause a crash.
The installation instructions now also contain update instructions.

Nice work! The latest version showed up as an update in my package manager (on Mint 17) so super-simple to upgrade  :yes:
I've tried to make installation and updating as painless as possible on all supported platforms. Seems like the effort was worth it.  :)

Regarding the UI, it mostly seems fine to me. The dropdown + checkbox for the exec flags seems a little bit counter-intuitive.  That could be just because I'm used to seeing all the checkboxes in one big list (like wxLauncher).
Well, I copied that from YAL because that's what I've used until now. :nervous: I might change that in the next version.

 

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: [Cross-platform] FS2 GOG/Mod installer
Out of curiosity, how does the installer support .dmg files?  Is it shelling out to the native OSX command line tools or is it using some sort of native library for extracting them?  Or does it just leave the .dmg files in place, like the Java installer did for a while?  :P
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
I'm using 7zip (that's why I said that it shouldn't be a problem for the installer).
I just launch the 7z command in the background to extract the archives. It can even extract DMG files. You have to call it twice, though. The first time it decompresses the DMG file and extracts the contained sections. Then I call 7z a second time to extract the 3.hfs file which contains a HFS image to get the DMG's content.

If you're using Linux, it pretty much works like this:
Code: [Select]
td="$(mktemp -d)"
7z x -o"$td" somefile.dmg
7z x "$td/"*.hfs
rm -r "$td"
« Last Edit: February 03, 2015, 04:47:20 pm by ngld »

 

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: [Cross-platform] FS2 GOG/Mod installer
Ah, I think the problem with the java one is that the java bindings for 7zip don't seem to support dmg extraction or something like that.  So is 7zip made available?  Is it a prerequisite or a provided dependency?
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
The Windows and Mac builds both bundle 7z. The Linux packages have 7zip listed as dependency which means that the user never has to manually install 7zip.

EDIT: I've added FSPort to the package repo but I'm pretty sure the generated "-mod" flag is wrong so you probably won't be able to play it with my launcher.

@m!m: I use dependencies to generate the "-mod" flag but that gets tricky with FSPort since it uses "primarylist" and "secondarylist" in mod.ini. I'll try to think of a solution but I guess we'll have to add a new field to the mod object for that.

EDIT2: I've added nightlies to the repository. The list is updated once a day. If you want to install a specific nightly version, install the "FSO Nightly" mod, then click on the cog icon next to it, switch to the "Versions" tab and click on "Edit". A new window with all available versions/revisions will show up. Select the versions you want and click on "Apply".
After you installed a nightly, you should click on "Settings", select "Game settings" and select the nightly from the dropdown.

Inspired by this thread, I've thought about creating deltas/patches from one mod version to the next.
A patch would be a 7z archive which contains all added files, deltas (generated by xdelta) and a list of deleted files. I'll probably also add a list of checksums for all files from the old and new mod version.

I can implement the patch generation on the Nebula server which means that after a new mod version has been added to the Nebula, a patch would be generated automatically. However, I'm not sure if this is worth the effort...
« Last Edit: February 07, 2015, 10:07:26 am by ngld »

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
And.... I'm back ! Resuming work on Nebula right now.

 

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: [Cross-platform] FS2 GOG/Mod installer
One dream I had was that mods would include a list of supported binaries, and the launcher/installer would simply keep enough binaries available to cover any given mod.  There'd never have to be any manual binary downloading then.  And the binaries could theoretically live outside of the program data home folder, and just be told where to look for their game data...
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
@ngld: Sorry I missed your updates, apparently edits aren't shown as new in the thread list.

@m!m: I use dependencies to generate the "-mod" flag but that gets tricky with FSPort since it uses "primarylist" and "secondarylist" in mod.ini. I'll try to think of a solution but I guess we'll have to add a new field to the mod object for that.
Do you happen to know why fsports needs this special treatment? Maybe FSO can be changed so no special treatment is needed.

Inspired by this thread, I've thought about creating deltas/patches from one mod version to the next.
A patch would be a 7z archive which contains all added files, deltas (generated by xdelta) and a list of deleted files. I'll probably also add a list of checksums for all files from the old and new mod version.

I can implement the patch generation on the Nebula server which means that after a new mod version has been added to the Nebula, a patch would be generated automatically. However, I'm not sure if this is worth the effort...
Patching sounds like a great idea but FSO already has limited support for VP patching. If a mod needs to distribute an update it could publish a new version which depends on the old version. The launcher could then handle that just like any other dependency without needing to implement complicated patching mechanisms.

One dream I had was that mods would include a list of supported binaries, and the launcher/installer would simply keep enough binaries available to cover any given mod.  There'd never have to be any manual binary downloading then.  And the binaries could theoretically live outside of the program data home folder, and just be told where to look for their game data...
When the repositories contains all the relevant executable definitions it would be easy to determine a minimum executable version for a mod (a mod could even specify which executable versions to use).

 

Offline AdmiralRalwood

  • 211
  • The Cthulhu programmer himself!
    • Skype
    • Steam
    • Twitter
Re: [Cross-platform] FS2 GOG/Mod installer
@m!m: I use dependencies to generate the "-mod" flag but that gets tricky with FSPort since it uses "primarylist" and "secondarylist" in mod.ini. I'll try to think of a solution but I guess we'll have to add a new field to the mod object for that.
Do you happen to know why fsports needs this special treatment? Maybe FSO can be changed so no special treatment is needed.
It has nothing to do with FSO; primarylist and secondarylist are launcher-side features that just change how the -mod flags are ordered. Instead, it has everything to do with how FSPort works (or, to be more precise, Silent Threat: Reborn, since FSPort itself doesn't have a "[multimod]" section in its mod.ini). ST:R puts fsport-mediavps (or now fsport-mediavps_2014) in the primarylist and FSPort and the MediaVPs in the secondarylist; this is so that the model upgrades in the FSPort MediaVPs override the POF file definitions in ST:R's data files (because primarylist entries go before the selected mod).

The problem could probably also be fixed via a separate ST:R MediaVPs mod, but presumably the FSPort team decided that would be too much of a hassle when the primarylist feature exists.

EDIT: Corrected a silly brainfart.
« Last Edit: February 13, 2015, 11:46:38 pm by AdmiralRalwood »
Ph'nglui mglw'nafh Codethulhu GitHub wgah'nagl fhtagn.

schrödinbug (noun) - a bug that manifests itself in running software after a programmer notices that the code should never have worked in the first place.

When you gaze long into BMPMAN, BMPMAN also gazes into you.

"I am one of the best FREDders on Earth" -General Battuta

<Aesaar> literary criticism is vladimir putin

<MageKing17> "There's probably a reason the code is the way it is" is a very dangerous line of thought. :P
<MageKing17> Because the "reason" often turns out to be "nobody noticed it was wrong".
(the very next day)
<MageKing17> this ****ing code did it to me again
<MageKing17> "That doesn't really make sense to me, but I'll assume it was being done for a reason."
<MageKing17> **** ME
<MageKing17> THE REASON IS PEOPLE ARE STUPID
<MageKing17> ESPECIALLY ME

<MageKing17> God damn, I do not understand how this is breaking.
<MageKing17> Everything points to "this should work fine", and yet it's clearly not working.
<MjnMixael> 2 hours later... "God damn, how did this ever work at all?!"
(...)
<MageKing17> so
<MageKing17> more than two hours
<MageKing17> but once again we have reached the inevitable conclusion
<MageKing17> How did this code ever work in the first place!?

<@The_E> Welcome to OpenGL, where standards compliance is optional, and error reporting inconsistent

<MageKing17> It was all working perfectly until I actually tried it on an actual mission.

<IronWorks> I am useful for FSO stuff again. This is a red-letter day!
* z64555 erases "Thursday" and rewrites it in red ink

<MageKing17> TIL the entire homing code is held up by shoestrings and duct tape, basically.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
One dream I had was that mods would include a list of supported binaries, and the launcher/installer would simply keep enough binaries available to cover any given mod.  There'd never have to be any manual binary downloading then.  And the binaries could theoretically live outside of the program data home folder, and just be told where to look for their game data...
When the repositories contains all the relevant executable definitions it would be easy to determine a minimum executable version for a mod (a mod could even specify which executable versions to use).
Well, we're doing that already...  :p
If you run Knossos on a fresh retail install and tell it to install a mod, it'll automatically download the latest FSO build (which is 3.7.2 RC5 right now). It even uses the right version for the current OS and processor. The user still has to select a build in the settings window but I'm planning to automate that, too.

@ngld: Sorry I missed your updates, apparently edits aren't shown as new in the thread list.
I guess I'll just double-post next time.

@m!m: I use dependencies to generate the "-mod" flag but that gets tricky with FSPort since it uses "primarylist" and "secondarylist" in mod.ini. I'll try to think of a solution but I guess we'll have to add a new field to the mod object for that.
Do you happen to know why fsports needs this special treatment? Maybe FSO can be changed so no special treatment is needed.
Even if the problem with FSPort could be solved otherwise, it'd still be nice to have this field to make the modline generation easier. Resolving a mod's dependencies yields a tree which is troublesome (at least for me) because I'm not quite sure in which order the mods are supposed to be. If two mods in the dependency tree depend on the same mod, the exact order is unclear which makes it impossible to generate a proper mod flag. A simple, linear list could solve this.
IMHO, the best solution would be if the Nebula interface generated this list from the dependencies but allowed the modder to reorder it.

Inspired by this thread, I've thought about creating deltas/patches from one mod version to the next.
A patch would be a 7z archive which contains all added files, deltas (generated by xdelta) and a list of deleted files. I'll probably also add a list of checksums for all files from the old and new mod version.

I can implement the patch generation on the Nebula server which means that after a new mod version has been added to the Nebula, a patch would be generated automatically. However, I'm not sure if this is worth the effort...
Patching sounds like a great idea but FSO already has limited support for VP patching. If a mod needs to distribute an update it could publish a new version which depends on the old version. The launcher could then handle that just like any other dependency without needing to implement complicated patching mechanisms.
The idea was to conserve bandwidth and disk space. Although using VPs for patching would save bandwidth when a user updates, it'd use more disk space than a complete replacement. Another problem is that you can't remove any files that way and a fresh installation / download would use up more disk space and more bandwidth.
The patching mechanism isn't that complicated and the launcher wouldn't be required to support it since the normal downloads would still be available. For now, though, I'll concentrate on finding bugs and improving Knossos' interface.

 

Offline Goober5000

  • HLP Loremaster
  • 214
    • Goober5000 Productions
Re: [Cross-platform] FS2 GOG/Mod installer
@m!m: I use dependencies to generate the "-mod" flag but that gets tricky with FSPort since it uses "primarylist" and "secondarylist" in mod.ini. I'll try to think of a solution but I guess we'll have to add a new field to the mod object for that.
Do you happen to know why fsports needs this special treatment? Maybe FSO can be changed so no special treatment is needed.
Even if the problem with FSPort could be solved otherwise, it'd still be nice to have this field to make the modline generation easier.

This is not a "problem".  FSPort is using a launcher feature that has been around ever since RandomTiger's original launcher release.  If you want a mod to be loaded before the active mod, use primarylist.  If you want a mod to be loaded after the active mod, use secondarylist.  You need two lists in order to specify all possible mod orderings.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
I'm sorry if my wording was misleading. The fact that Knossos isn't generating a proper mod flag for FSPort is a problem. I didn't want to say that FSPort had a problem.

You need two lists in order to specify all possible mod orderings.
Not necessarily. The list I was thinking of includes the current mod. The list for ST:R would look something like "fsport-mediavps,fsport-str,fsport,mediavps_2014".

 

Offline Goober5000

  • HLP Loremaster
  • 214
    • Goober5000 Productions
Re: [Cross-platform] FS2 GOG/Mod installer
I'm sorry if my wording was misleading. The fact that Knossos isn't generating a proper mod flag for FSPort is a problem. I didn't want to say that FSPort had a problem.
Ah, fair enough then.  I suppose I should have read that more carefully.

Quote
You need two lists in order to specify all possible mod orderings.
Not necessarily. The list I was thinking of includes the current mod. The list for ST:R would look something like "fsport-mediavps,fsport-str,fsport,mediavps_2014".
Well, the Launcher always omitted the active mod (in this case fsport-str) because it was inferred from the selection.  If you include the current mod, then yes, only one list is needed, but this wouldn't be compatible with existing mod.inis.  Unless you're thinking of just the internal representation.