Author Topic: [Cross-platform] FS2 GOG/Mod installer  (Read 72024 times)

0 Members and 2 Guests are viewing this topic.

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
You might want to clean tmp files on the converter server, I made some tests where Nebula would fail to retrieve the data.

2 Questions :
- What happens if knossos-server request to Nebula (webhook case) fails ? Does it keep the data until retrieved manually ?
- What happens if knossos-server "retrieve" url is called prematurely and the json is not ready yet ? (I want to know if I have to check the conversion status before trying to get the data)

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
  • Yes, it keeps the data.
  • Right now, it would delete the internal record and report a failure (success would be false). I'll add a safety check so that it only deletes tickets if they failed or succeded. I'll also add a new reponse property called "finished" which will tell you whether the conversion process is still running.

EDIT: Pushed the changes to GitHub and the test server.
« Last Edit: October 09, 2014, 01:37:48 pm by ngld »

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
Just added my local Nebula repository to Knossos, downloaded a mod... It works perfectly ! :D

I'm going to polish some rough edges (templates, better error handling, make sure the uploaded logo is an image and of the right size) and push the new version to github.

EDIT : if I send an empty build, Knossos server gets stuck ("WARNING: Added an empty task of type "ChecksumTask". Ignoring it!"). The build is not marked as ready or failed.

EDIT : Looks like I have crashed Knossos-server.
« Last Edit: October 10, 2014, 01:42:16 am by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
The converter didn't expect to get a mod but no files.  :)  It will now complain with "No files found!" and mark the build as failed.
I also fixed a weird bug which caused the log messages to be displayed multiple times.

You didn't crash the server, it was busy handling your running (stuck) requests.  :lol:

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
:D

Most of the small template and redirection changes (better errors, less duplicated HTML, build status reporting in the builds list and plenty of others...) are done. I have a list of tests to run to see if haven't broken anything.

Logo upload is working, but causing weird errors if I make them optional (a mod will need a logo to be published, but if a modder wants to run some test without having a logo, that should be allowed). I also have trouble generating urls for resources in the "web" directory (as logos), since they are not part of Nebula bundle file tree.
I may disable logo upload for tonight's update, and replace it later with a better solution.

I'm still not generating the "all.json" holding all mods, so that's currently one mod : one repo (but you can get the <mod>/<default branch>/<current build> url, so auto updates for a mod will work ! :) ). Modder teams and the global repo will be my work items for the next few days.

EDIT : It looks like i'm only getting the last line of log now (instead of adding new lines, they get replaced, so the log \<pre\> tag is never longer than one line). You may also check the "current task" field, my file download appeared 2 times.

Do you want me to add a specific response to the webhook if build processing has been cancelled by the user (there's an action in Nebula for that, called "forceFail"), so that it cleans data that no one needs ?

EDIT 2 : I've just made the change : If the webhook reaches a build that already has been marked as failed, Nebula answers with a {"cancelled": true} JSON response.
And I'm definitely seeing 2 times the same log entries.
« Last Edit: October 10, 2014, 10:38:31 am by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Does it work fine if you use http://s1.nebula.tproxy.de/converter ?
Thanks for the "cancelled" webhook response! I'll implement it on the server.

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
Disregard what I just wrote, it works, but i'm still getting the double output issue.

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
If anyone missed it on IRC, our test server has been updated.

http://nebula.tproxy.de/nebula

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
The converter issues should be resolved now.
A quick note to anyone who wants to test Knossos and uses Windows: You don't have to install Python or anything else. Knossos has its own installer.

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
@ngld :
Is the converter aware of the final mod file tree ?

If it is, I have an idea for a final check : if the mod is shipping a mod.ini, we have to make sure it's placed at the root of the mod folder.
If we have a mod.ini somewhere, but not in the right place, something probably went wrong with the "move" actions, and we should mark the build as failed (or give a clear warning) since it won't run anyway.

Some packages shipping only fs2_open builds don't need a mod.ini, so a build that doesn't have a mod.ini anywhere in the file tree should still pass.

That logic is not perfect, but should help avoiding most human errors. What do you think ?

You may also want to take a look at what went wrong with BTA_Demo_1.1.2 ( http://nebula.tproxy.de/nebula/mjnmixael/BTA_Demo/master/1.1.2/show_data ). I'm not sure about
Code: [Select]
"dest": "" behaviour... It looks like the files ended up not being moved at all instead of being moved to the BtA root folder, because Mjn's input looks ok to me.
Am I wrong ?

@MjnMixael :
So even after manually moving everything to the the right folder, fs2_open trunk complains about :
Code: [Select]
ERROR: "Could not load in 2_CommandBrief-mb!" at ui/window.cpp:85Not sure whats going on.
« Last Edit: October 11, 2014, 04:55:25 am by Hellzed »

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
I also tried testing the BtA data but something weird is going on.
  • The data shown by /show_data is different from the data the json URL returns. The JSON content somehow has an added dependency field.
  • That dependency results in a cyclic dependency (the package requires itself). I don't know if this error is also present in the mod configuration in your database.
A nice feature for the future would be more validation of the dependencies to make sure the server has valid dependency data.
« Last Edit: October 11, 2014, 07:02:35 am by m!m »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
@Hellzed: Yes, it is aware of the final mod file tree. Just look at the "filelist" property in the generated JSON. There you can see that everything was indeed moved successfully. If it's in the wrong folder on your hard drive, then it's a Knossos bug. I just tested it and everything went fine. (All files are placed in <FS2 dir>/BtA Demo/)
There's a bug in Nebula: The notes field contains "\\n" for line breaks but those should be "\n". Are you calling json_encode() for this string?

I could add a "WARNING: I found no mod.ini in your mod's install folder. Are you sure everything is where it should be?". Displaying the file tree is a good idea but I'm wondering whether we should display a simple list or an interactive tree (think of Windows' Explorer's sidebar).

@m!m: The data in /show_data is the converter input, your launcher should process the JSON file which should be the same as /show_data_finalised.
The dependency is added by Knossos' converter. I can probably remove it but I wanted to make sure that a required package depends on all other required packages of the same mod. Knossos handles cyclic dependencies without problems so it never caused issues.

The dependency validation sounds nice but I'm not sure where it should be handled. It would make the most sense to add this to the Nebula since the converter isn't aware of other mods. Should this also take care of conflicts and dependency cycles?

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
:banghead: I forgot to check that tab.
Adding the dependency seems fine but it shouldn't add a dependency that makes the package depend on itself. I would also suggest setting the version field in the dependency to the version of the mod that is currently being processed to make sure the dependency doesn't cause issues when there are multiple version of the same mod.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
I'll change the version field but I can't avoid creating the dependency cycle because every dependency on a mod implicitly includes all required packages.

Did you take a look at the "filelist" property? It basically tells the launcher how an installed mod should look. It makes checking installed mods much easier since it already takes the post-install actions into account.
I'm thinking about adding a "cmdline" property to the mod object. It would allow modders to specify a default command line for their mod. It's pretty much what wxLauncher already does although in that case it's a mod.ini flag.

There's another small issue I'd like to resolve: On every platform you can select only the FSO build which fits you. The only exception is Windows on a SSE2 processor: There you can install FSO's SSE and SSE2 builds. I'd like to hide the SSE builds in that case. The best solution would be a change to the "environment" conditions. Adding a simple "not" flag or something like that should solve this.
What do you think?

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
I'll change the version field but I can't avoid creating the dependency cycle because every dependency on a mod implicitly includes all required packages.
Can't you just skip adding the dependency when the current package name is the same as the package name of the dependency? A cyclic dependency is a something that compromises the integrity of the dependency system which is why I classified it as a fatal error if such a cyclic dependency is found.

Did you take a look at the "filelist" property? It basically tells the launcher how an installed mod should look. It makes checking installed mods much easier since it already takes the post-install actions into account.
I'm thinking about adding a "cmdline" property to the mod object. It would allow modders to specify a default command line for their mod. It's pretty much what wxLauncher already does although in that case it's a mod.ini flag.
Both are great ideas! A minor thing I noticed about the "filelist" is that is includes the files from all packages. How hard would it be to move that element to the "package" object and only include the files from that package in that list?
That would simplify the checking mechanism if the launcher/installer knows what packages should be installed. I guess I could implement the same logic in the installer but it might be easier to implement in the convert itself as it already has all the information that is needed.
"cmdline" would be nice but it would be better to have a generic "settings"-thing that would also handle things that are not configured via the commandline (e.g. TTS settings) but that is something for later, the "cmdline" element will be most useful for almost all other applications.

There's another small issue I'd like to resolve: On every platform you can select only the FSO build which fits you. The only exception is Windows on a SSE2 processor: There you can install FSO's SSE and SSE2 builds. I'd like to hide the SSE builds in that case. The best solution would be a change to the "environment" conditions. Adding a simple "not" flag or something like that should solve this.
What do you think?
I had the idea of somehow prioritizing a package when the "feature" value would be higher but the system you are proposing should work much better and will give the modder more control over how the installer processes the packages.

 

Offline mjn.mixael

  • Cutscene Master
  • 212
  • Chopped liver
    • Steam
    • Twitter
Re: [Cross-platform] FS2 GOG/Mod installer
Quote
@MjnMixael :
So even after manually moving everything to the the right folder, fs2_open trunk complains about :
Code: [Select]
ERROR: "Could not load in 2_CommandBrief-mb!" at ui/window.cpp:85Not sure whats going on.

Well, the BtA Demo requires MediaVPs 3.6.12 which isn't in the installer (yet). That might be it? I just tested the demo data (same files Nebula downloads) on a recent build and it works fine. So let's make sure it's not because it needs 3.6.12 and/or make sure Nebula installed everything properly.
Cutscene Upgrade Project - Mainhall Remakes - Between the Ashes
Youtube Channel - P3D Model Box
Between the Ashes is looking for committed testers, PM me for details.
Freespace Upgrade Project See what's happening.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
I'll change the version field but I can't avoid creating the dependency cycle because every dependency on a mod implicitly includes all required packages.
Can't you just skip adding the dependency when the current package name is the same as the package name of the dependency?
I don't quite understand what you're talking about. The JSON we are talking about is this:
Code: [Select]
{
    "packages": [],
    "version": "==1.1.2",
    "id": "BTA_Demo"
}
I guess I'll simply remove this dependency and leave it to the launcher to handle this case.

I'll move the "filelist" property to the package object, you're right that makes more sense. I guess I can remove the "contents" property, now, since it's redundant.
I'll probably rewrite Knossos' dependency resolution algorithm. How do you treat version conflicts? Let's say mod A depends on mods B and C. Mod B needs version 1.0.0 of mod D and mod C needs version 2.0.0 of mod D. This wouldn't work since FSO can't load two versions of the same mod.
However, installing mod B and mod C should work since they're not loaded together.

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
:banghead: I am such an idiot :banghead:
You are absolutely right, the error I am seeing is that I blindly add all required packages to a dependency if no package is specified which will result in one package depending on itself it is a required package.

A version conflict would result in an error in my implementation as FSO can't load that. A resolution would be to let the user decide what version to use but that would only be a troubleshooting solution. Ideally this would be handled by Nebula (another thing for the "mod dependency validation"-list :p) so a modder can't even create a mod that would cause this issue.

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
@ngld :
I checked again, there is a problem which causes the data to stay in BtA Demo/BtA Demo :
Code: [Select]
DEBUG:MainThread:qt.<module>: Using Qt API PySide.
DEBUG:MainThread:util.call: Running ['7z', '-h']
INFO:Thread-3:util.get: Retrieving "http://dev.tproxy.de/knossos/version"...
WARNING:Thread-2:tasks.work: File "BTA_DEMO_Patch_1_1.7z" for mod "Between the Ashes: Mefistofele" (BTA_Demo) is missing during uninstall!
WARNING:Thread-2:tasks.work: File "BtA Demo/mod.ini" for mod "Between the Ashes: Mefistofele" (BTA_Demo) is missing during uninstall!
WARNING:Thread-2:tasks.work: File "BtA Demo/BTA_DEMO_Root.vp" for mod "Between the Ashes: Mefistofele" (BTA_Demo) is missing during uninstall!
WARNING:Thread-2:tasks.work: File "BtA Demo/BTA_DEMO_Assets2.vp" for mod "Between the Ashes: Mefistofele" (BTA_Demo) is missing during uninstall!
WARNING:Thread-2:tasks.work: File "BtA Demo/BTA_DEMO_Assets1.vp" for mod "Between the Ashes: Mefistofele" (BTA_Demo) is missing during uninstall!
WARNING:Thread-2:tasks.work: File "BtA Demo/bta.bmp" for mod "Between the Ashes: Mefistofele" (BTA_Demo) is missing during uninstall!
DEBUG:Thread-2:progress._next_step: Entering step 1 of 3 in task InstallTask.
DEBUG:Thread-7:progress._next_step: Entering step 2 of 3 in task InstallTask.
INFO:Thread-7:util.download: Downloading "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_Assets2.7z"...
DEBUG:Thread-7:util.call: Running ['7z', 'x', '-o/tmp/tmpZontl3/content', u'/tmp/tmpZontl3/BTA_DEMO_Assets2.7z']

7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=fr_FR.UTF-8,Utf16=on,HugeFiles=on,8 CPUs)

Processing archive: /tmp/tmpZontl3/BTA_DEMO_Assets2.7z

Extracting  BtA Demo/BTA_DEMO_Assets2.vp
Extracting  BtA Demo

Everything is Ok

Folders: 1
Files: 1
Size:       361232708
Compressed: 203056300
DEBUG:Thread-7:tasks.work2: Creating path "/home/loris/FreeSpace 2 beta/BtA Demo/"...
DEBUG:Thread-7:tasks.work2: Creating path "/home/loris/FreeSpace 2 beta/BtA Demo/BtA Demo"...
DEBUG:Thread-7:tasks.work2: Moving "/tmp/tmpZontl3/content/BtA Demo/BTA_DEMO_Assets2.vp" to "/home/loris/FreeSpace 2 beta/BtA Demo/BtA Demo/BTA_DEMO_Assets2.vp"...
INFO:Thread-7:util.download: Downloading "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_Assets1.7z"...
DEBUG:Thread-7:util.call: Running ['7z', 'x', '-o/tmp/tmpPwJQqv/content', u'/tmp/tmpPwJQqv/BTA_DEMO_Assets1.7z']

7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=fr_FR.UTF-8,Utf16=on,HugeFiles=on,8 CPUs)

Processing archive: /tmp/tmpPwJQqv/BTA_DEMO_Assets1.7z

Extracting  BtA Demo/BTA_DEMO_Assets1.vp
Extracting  BtA Demo

Everything is Ok

Folders: 1
Files: 1
Size:       536321828
Compressed: 135848477
DEBUG:Thread-7:tasks.work2: Moving "/tmp/tmpPwJQqv/content/BtA Demo/BTA_DEMO_Assets1.vp" to "/home/loris/FreeSpace 2 beta/BtA Demo/BtA Demo/BTA_DEMO_Assets1.vp"...
INFO:Thread-7:util.download: Downloading "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_Root.7z"...
DEBUG:Thread-7:util.call: Running ['7z', 'x', '-o/tmp/tmpRuEXPa/content', u'/tmp/tmpRuEXPa/BTA_DEMO_Root.7z']

7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=fr_FR.UTF-8,Utf16=on,HugeFiles=on,8 CPUs)

Processing archive: /tmp/tmpRuEXPa/BTA_DEMO_Root.7z

Extracting  BtA Demo/bta.bmp
Extracting  BtA Demo/mod.ini
Extracting  BtA Demo/BTA_DEMO_Root.vp
Extracting  BtA Demo/data
Extracting  BtA Demo

Everything is Ok

Folders: 2
Files: 3
Size:       57822374
Compressed: 21822457
DEBUG:Thread-7:tasks.work2: Moving "/tmp/tmpRuEXPa/content/BtA Demo/BTA_DEMO_Root.vp" to "/home/loris/FreeSpace 2 beta/BtA Demo/BtA Demo/BTA_DEMO_Root.vp"...
DEBUG:Thread-7:tasks.work2: Moving "/tmp/tmpRuEXPa/content/BtA Demo/mod.ini" to "/home/loris/FreeSpace 2 beta/BtA Demo/BtA Demo/mod.ini"...
DEBUG:Thread-7:tasks.work2: Moving "/tmp/tmpRuEXPa/content/BtA Demo/bta.bmp" to "/home/loris/FreeSpace 2 beta/BtA Demo/BtA Demo/bta.bmp"...
DEBUG:Thread-7:tasks.work2: Creating path "/home/loris/FreeSpace 2 beta/BtA Demo/BtA Demo/data"...
DEBUG:Thread-7:progress._next_step: Entering step 3 of 3 in task InstallTask.
WARNING:Thread-4:progress.work: Either we still have some work to do (unlikely) or there are still some other threads running (2).
ERROR:Thread-7:progress.run: Exception in Thread!
Traceback (most recent call last):
  File "/home/loris/knossos-develop/lib/progress.py", line 120, in run
    task[0].work(*task[1])
  File "/home/loris/knossos-develop/lib/progress.py", line 380, in work
    self._steps[self._cur_step][1](arg)
  File "/home/loris/knossos-develop/lib/tasks.py", line 366, in work3
    path_prefix = glob.escape(path_prefix)
AttributeError: 'module' object has no attribute 'escape'
DEBUG:Thread-7:progress._next_step: Entering step 4 of 3 in task InstallTask.

@mjn.mixael & ngld :
I'm not sure if knossos uses mod.ini or dependencies from the JSON data in this case. I have already have "mediavps_3612".

@m!m :
I'm not strictly enforcing dependency validity in Nebula, but the good practice would be to rely on the dependency autocompletion to add these, just to make sure the ID and packages are right.
I think it would be a bad practice not to register dependencies in the right order (thus temporarily registering non existing deps), but to me that's modders decision, and should be possible as long as they know what they do (and add the missing deps quickly).
« Last Edit: October 11, 2014, 09:34:56 am by Hellzed »

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
@Hellzed: What do you think about adding the capability to check the dependencies of a mod and let the modder decide if they want to use it? This is obviously a feature that isn't required right now.
« Last Edit: October 11, 2014, 09:49:14 am by m!m »