Author Topic: [Cross-platform] FS2 GOG/Mod installer  (Read 72024 times)

0 Members and 1 Guest are viewing this topic.

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
@Hellzed: Are you using Python 2? Knossos tries to use the function "escape" of the built-in module "glob" which is missing in your case.
I'm going to rewrite that part, anyway. I'll use the "filelist" instead to ensure consistency.

Knossos still parses the mod.ini but that might change in the future. Once I add support for multiple versions of the same mod, the dependencies listed in the mod.ini are bogus.

@m!m: That's a great idea.
@Hellzed: Maybe you could add a warning like "You specified an invalid dependency. Are you sure you want to submit?". I'm could re-write the dependency resolution algorithm in PHP if that helps (you'll need it if you want to check for version conflicts).
« Last Edit: October 11, 2014, 09:54:28 am by ngld »

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
I'm adding optional dependency enforcement to my TO DO list.
My next work items are :
- Allowing to copy a build across branches (if you want to copy the last release candidate from the unstable branch to the stable one, under a stable version number)
- Complete team logic (this is a big change, but a big chunk of it is already dormant the code, so ideally we wouldn't need to rebuild the database from scratch to activate it)
- Basic admin panel to pick which mods can publish directly to the global ("all.json") repository.
- Expand image uploads (add a Nebula specific high res logo/"cover art" and screenshots).
- Small things like list pagination on the all mods and modders pages.

These are the basic steps needed to build the public mod catalogue.

@ngld :
* Yes, I started ./launcher.py with default python binary in Ubuntu 14.10. It happens to be python2.
* Having a php version of the algorithm could be useful, as a custom validation constraint in the form. I'll tell you when I need it, but that could be in a little while, since it's not absolutely necessary yet.
I'll advise modders to test their builds in a separate branch before publishing to the default branch (once we have the build transfer feature).


EDIT : Could you add instructions to get the dependencies for the virtualenv too ? Once activated in the knossos-develop folder, python can't find basic dependencies installed from APT, like "six" and "PySide".
« Last Edit: October 11, 2014, 10:48:34 am by Hellzed »

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
Knossos still parses the mod.ini but that might change in the future. Once I add support for multiple versions of the same mod, the dependencies listed in the mod.ini are bogus.
I have put some though into that issue as well and here is what I implemented:
The mods that are downloaded are stored in folders that are uniquely identified by the mod ID and version (it looks like <FS root>/mods/packages/<id>/<version>.
I also decided that mod.ini files are just too cumbersome to handle so I switched to using JSON to store the information about the installed mods locally. When a new mod (or package) is installed the installer updates a file named mod.json that then includes all information that is needed later. I store the general meta data about a mod (title, version, ...) and the data about the installed packages in that file.
Here is an example of the file:
Code: (mod.json) [Select]
{
  "description": "The day is July 25, 2336. Sixteen months have passed since the Lucifer was destroyed, mysteriously locking the Sol system away. It's barely been thirteen months since the Hades superdestroyer was destroyed in orbit of Deneb, and the rogue GTI reduced to silence. There are many wounds to lick for the GTA, as the weakened fleet struggles to maintain order in its systems. Although the cease-fire treaty with their new Vasudan allies is still holding, the PVE is still busy dealing with the terrorist HoL faction.\r\n\r\nNow is our chance to get past them. We no longer need to so carefully hide. Now we can make our deal with the devil.",
  "title": "Between the Ashes: Mefistofele",
  "version": "1.1.2",
  "id": "BTA_Demo",
  "packages": [
    {
      "notes": "",
      "name": "BtA Demo",
      "status": "Required",
      "dependencies": [
        {
          "id": "BTA_Demo",
          "version": "*",
          "packages": []
        }
      ],
      "files": []
    },
    {
      "notes": "",
      "name": "Low End Compatibility",
      "status": "Optional",
      "dependencies": [
        {
          "id": "BTA_Demo",
          "version": "*",
          "packages": []
        }
      ],
      "files": []
    }
  ]
}

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
What about scripts (iirc some mods write data to their folders during missions) ?

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
As far as I know scripts can't write into the mod folder as FSO doesn't allow that but that isn't a problem anyway as FSO doesn't care where a mod is located as long as the path it is given is right.
Using this system we might even be able to install total conversion without special handling and if I add the right commandline option to FSO we could even handle retail FS2 as just another total conversion but that isn't relevant now, first we need to get the core features running :p

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
Looks like I ran into outdated docs about python3 in ubuntu. I had to use pip3 instead of pip, no need for a virtualenv...
I can now run Knossos with python3.

Your latest commit broke mod install : BtA fails to download but instantly appears as installed in the mods list.

Code: [Select]
INFO:Thread-7:util.get: Retrieving "http://dev.tproxy.de/knossos/version"...
DEBUG:Thread-5:progress._next_step: Entering step 1 of 2 in task InstallTask.
ERROR:Thread-5:progress.run: Exception in Thread!
Traceback (most recent call last):
  File "/home/loris/knossos-develop/lib/progress.py", line 120, in run
    task[0].work(*task[1])
  File "/home/loris/knossos-develop/lib/progress.py", line 380, in work
    self._steps[self._cur_step][1](arg)
  File "/home/loris/knossos-develop/lib/tasks.py", line 298, in work1
    for item, info in mfiles.items():
AttributeError: 'list' object has no attribute 'items'
DEBUG:Thread-5:progress._next_step: Entering step 2 of 2 in task InstallTask.
DEBUG:Thread-5:progress._next_step: Entering step 3 of 2 in task InstallTask.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
As I said I'm rewriting the InstallTask and I accidentally pushed broken code. :sigh: Once I'm finished it'll work again.

EDIT: It should work now.
« Last Edit: October 11, 2014, 12:58:32 pm by ngld »

 

Offline mjn.mixael

  • Cutscene Master
  • 212
  • Chopped liver
    • Steam
    • Twitter
Re: [Cross-platform] FS2 GOG/Mod installer
As far as I know scripts can't write into the mod folder as FSO doesn't allow that but that isn't a problem anyway as FSO doesn't care where a mod is located as long as the path it is given is right.
Using this system we might even be able to install total conversion without special handling and if I add the right commandline option to FSO we could even handle retail FS2 as just another total conversion but that isn't relevant now, first we need to get the core features running :p

Scripts can write to the mod folder and should. For example, the shipsave script from Admiral MS writes files to mod/data/config. Actually, I'm pretty sure Admiral MS created optional stuff to change what folder the script saves files to. I have also used a script to copy and rename files from within FSO in order to unlock missions.
Cutscene Upgrade Project - Mainhall Remakes - Between the Ashes
Youtube Channel - P3D Model Box
Between the Ashes is looking for committed testers, PM me for details.
Freespace Upgrade Project See what's happening.

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
I wasn't aware of that until Axem told me that it was possible but even with scripts it shouldn't be a problem to use the proposed mod path template.

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
On Ubuntu, it looks like scripts write to a separate folder, for example, BtA writes to :
Code: [Select]
.fs2_open/BtA Demo/data/scripts
Code: [Select]
FreeSpace 2/BtA Demo/datastays empty. I haven't fully replayed it yet, so i'm not sure.

I have no idea what fs2_open scripting actually does on the filesystem.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
It depends on the command line. If a -mod flag is passed, then FSO will pick the first path and write the files to "<FSO folder>/<mod path>/<whatever>" on Windows and to "~/.fs2_open/<mod path>/<whatever>" on Unix.
If the flag isn't passed, it writes the files to "<FSO folder>/<whatever>" or "~/.fs2_open/<whatever>".
Since the path depends on the "-mod" flag it won't cause any problems for us. (Just like m!m said.)

 

Offline mjn.mixael

  • Cutscene Master
  • 212
  • Chopped liver
    • Steam
    • Twitter
Re: [Cross-platform] FS2 GOG/Mod installer
On Ubuntu, it looks like scripts write to a separate folder, for example, BtA writes to :
Code: [Select]
.fs2_open/BtA Demo/data/scripts
Code: [Select]
FreeSpace 2/BtA Demo/datastays empty. I haven't fully replayed it yet, so i'm not sure.

I have no idea what fs2_open scripting actually does on the filesystem.

I think we had it writing to data/scripts/ in the demo release and changed it to config later. Can't remember.
Cutscene Upgrade Project - Mainhall Remakes - Between the Ashes
Youtube Channel - P3D Model Box
Between the Ashes is looking for committed testers, PM me for details.
Freespace Upgrade Project See what's happening.

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
@mjn.mixael : You might want to run a "copy & update" on BtA Demo. JSON generation has changed, and current BtA build won't download in the latest version of Knossos.

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
I cleaned up the repository, and added the "alpha" warnings.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
I've got a few bugs left to fix and add a few things to the UI but I've nearly finished support for multiple mod versions. This allows a user to install two versions of the same mod at the same time.
I have some question for everyone reading this:
  • How should updating be handled? As long as the user only has one version installed it's easy, just delete the old version and replace it with the new one. The best idea I can come up with is: If the mod's version was the latest one when it was installed, replace it with the newest version available in its branch.
  • How should the UI work for selecting versions? I'm talking about the "Mod Tree" tab here, the mod list will be updated once the Nebula's frontend is finished.
  • I'm saving the mod metadata in a file called "knossos.json" in the mod's directory. I didn't use the "mod.json" name since m!m's launcher is already using it and the schema's not compatible.

@Hellzed: I've updated the test server with your changes yesterday.
We should add a note to the "New build" form that tells the modder that the download links should always point to exactly this version. We should also allow to change the links later on (The server might have changed or maybe they want to add another mirror). If too many modders complain about this, we might be able to add a system which automatically creates a backup of all builds in the Nebula. I wonder how much disk space that would consume, though.  :blah:

This should make the FSCRP's job much easier (you would be able to download every version of a mod).
« Last Edit: October 14, 2014, 06:28:36 am by ngld »

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
  • How should updating be handled? As long as the user only has one version installed it's easy, just delete the old version and replace it with the new one. The best idea I can come up with is: If the mod's version was the latest one when it was installed, replace it with the newest version available in its branch.
Why not just keep all version installed (maybe add an optional "clean" action to delete old versions). For some mods you will need to keep some older versions anyway (e.g. the MediaVPs) so you can just apply that to all mods.

  • I'm saving the mod metadata in a file called "knossos.json" in the mod's directory. I didn't use the "mod.json" name since m!m's launcher is already using it and the schema's not compatible.
What makes it incompatible? It would be great if all launchers/installers could use the same data.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Why not just keep all version installed (maybe add an optional "clean" action to delete old versions). For some mods you will need to keep some older versions anyway (e.g. the MediaVPs) so you can just apply that to all mods.
I'm trying to save disk space, duh.  :P  For dependencies (like the MediaVPs) I'll check if they're still needed and keep them only in that case.

What makes it incompatible? It would be great if all launchers/installers could use the same data.
  • You're not storing the filelist, I'm saving it for integrity checks.
  • Why did you capitalize the package's status ("Required", "Optional", ...)? I've kept the same case from the schema.
  • How do you store which packages are installed? I'm only saving installed packages in the packages property but I suspect you're doing the same.

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
Why not just keep all version installed (maybe add an optional "clean" action to delete old versions). For some mods you will need to keep some older versions anyway (e.g. the MediaVPs) so you can just apply that to all mods.
I'm trying to save disk space, duh.  :P  For dependencies (like the MediaVPs) I'll check if they're still needed and keep them only in that case.
Hard disk space is cheap :p

What makes it incompatible? It would be great if all launchers/installers could use the same data.
  • You're not storing the filelist, I'm saving it for integrity checks.
  • Why did you capitalize the package's status ("Required", "Optional", ...)? I've kept the same case from the schema.
  • How do you store which packages are installed? I'm only saving installed packages in the packages property but I suspect you're doing the same.
  • I don't have support for file lists currently but when I implement that I will also add it to the installed mod information
  • That's because my enum values are CamelCase named and the serializer uses the actual names of the enum values, maybe you could use case-insensitivity to check the package status
  • Yep, when a new package is installed I add all relevant informations for a package and just add it to "packages"

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
We should add a note to the "New build" form that tells the modder that the download links should always point to exactly this version. We should also allow to change the links later on (The server might have changed or maybe they want to add another mirror). If too many modders complain about this, we might be able to add a system which automatically creates a backup of all builds in the Nebula. I wonder how much disk space that would consume, though.  :blah:

Yeah, we should end this habit of "sneaky updates". I'll add the note to the form.
Wouldn't allowing to change the links after a build has been validated go against strict semver enforcement ?
To me, the appropriate way to change links would be to run a "copy & update" on the build, replace the broken links, and validate the new build under the same version number with a changed "version metadata" field (for example 1.0.0 to 1.0.0+serverchange1).
I'm thinking about writing a Nebula "good practices" guide, now :p

This leads me to think we would still have a problem with outdated or broken links in older builds. We should use some automated testing. I'm thinking about a cron task, that would run Knossos-server from Nebula once in a while, and if a single file is unavailable from all urls, mark the build as "obsolete" and remove its cached JSON data from the database.
(that would make 4 states for a build : "success", "failed", "processing" and "obsolete")

Since many most files are big, we can't handle them directly from the build form, so I'm not sure how Nebula could handle file hosting.
However, as Knossos-server downloads the files before generating the checksums, maybe it could cache them and add the cache URLs to the lists before generating final JSON. (I'm not proposing this because that's less work for me, I actually think it's a good solution before we come up with a better file hosting system).

If my cloning code keeps behaving as expected, we should have a simple way to transfer a build to from a branch to another tonight. I'll also add a graphical clue to see which branch is the default one (in the mod/branches list view).

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Yeah, we should end this habit of "sneaky updates". I'll add the note to the form.
Wouldn't allowing to change the links after a build has been validated go against strict semver enforcement ?
To me, the appropriate way to change links would be to run a "copy & update" on the build, replace the broken links, and validate the new build under the same version number with a changed "version metadata" field (for example 1.0.0 to 1.0.0+serverchange1).
I'm thinking about writing a Nebula "good practices" guide, now :p
It wouldn't go against semver as long as the archives' checksums stay the same. If you only change the links and not the archives' contents, the actual mod data (content) stays the same and thus is still the same version. If you create a new build, that would mean there'd been an update. The client would then tell the user to update the mod since a new version means new/different content.

Since many most files are big, we can't handle them directly from the build form, so I'm not sure how Nebula could handle file hosting.
We can't? Why? Is there a browser limitation? :P

However, as Knossos-server downloads the files before generating the checksums, maybe it could cache them and add the cache URLs to the lists before generating final JSON. (I'm not proposing this because that's less work for me, I actually think it's a good solution before we come up with a better file hosting system).
I've already thought about this and it seems like a good solution. I'll add a mediafire downloader, so that modders can also paste normal mediafire links in addition to direct links.

If my cloning code keeps behaving as expected, we should have a simple way to transfer a build to from a branch to another tonight. I'll also add a graphical clue to see which branch is the default one (in the mod/branches list view).
Yay!

EDIT: @m!m: Here's a schema for you: mod_json_schema.txt What do you  think?

EDIT2: @m!m: Did you implement the "-mod" flag, yet? We can generate the -mod list by checking the dependencies but that only replaces the "secondarylist" in mod.ini. What about the "primarylist"? I guess we'll need to add another field to the repo schema.
@Hellzed: Can you allow the modder to reorder the dependencies? If we generate the -mod list using these, the order becomes important.
« Last Edit: October 15, 2014, 03:50:13 am by ngld »