Hard Light Productions Forums

Modding, Mission Design, and Coding => FS2 Open Tools => Topic started by: Hellzed on December 19, 2013, 02:59:14 am

Title: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on December 19, 2013, 02:59:14 am
A python cross platform port is now in developement, by ngld : http://www.hard-light.net/forums/index.php?topic=86364.msg1769614#msg1769614
Grab the code from git : https://github.com/ngld/fs2mod-py/

Hi !
I've worked a while on this little thing, and it's starting to look good enough for a first alpha ! Don't expect a nice GUI, it's only a bash script !

Sorry, the first upload v0.0.3 was broken (some Media VP stuff still hardcoded in the unpacking part). Version 0.0.4 is not OK either, it has download path issues.
Current version is 0.0.29. Supports GOG installer, a few mods.NO BINARIES INSTALL YET ! Add them yourself. And of course this is compatible with pre-existing FS2 installs. Just don't use "clean" option on a mod if you have modified tables and models.


innoextract must be installed !

So, this is my FreeSpace 2 bash installer :
Code: [Select]
[b][v0.0.4+][/b]
- it can set up a FreeSpace 2 install from a GOG installer
- it can manage mods (install/uninstall/put archives in a cache/remember mods for reinstalling later)
- it can retrieve a mod from a SINGLE file !
- it stores its cache and config files in a separate folder so it only works with mod files in the FS2 folder
- it may support gracefully file paths with spaces (i think it's covered, not sure)

[b][v0.0.6+][/b]
- when you change the FS2 root folder, the installer now compares your mod list and the folders in the new root folder and identifies what may be installed or not. This means you can now manage multiple FS2 installs on the same computer.

[b][v0.0.10+][/b]
- handle dependecies. BUT I'M HOSTING ON "UBUNTU ONE" SO LINKS KEEP BREAKING FOR ONLINE UPDATES.

[b][v0.0.12+][/b]
- deps install is now cleaner
- downloads fail with an error message (instead of hanging) when connection is lost

[b][v0.0.14+][/b]
- mod descriptor files can now auto-update when you install or check & update (they need to include an "update" file in "root")
- better offline behaviour (if we already have a mod descriptor and the online update fails, keep going anyway)
- [no difference in the code] Switched to dropbox ! Links do not break anymore when I update mod packages in the repository !

[b][v0.0.16+][/b]
- added a way to clean cache without uninstalling/removing/checking installed files
- added a "clean mod folder option" : remove everything that's not in the zip/7z files from a mod folder (tables, models... typically any user side modification)

[b][v0.0.21+][/b]
- cleaner mod updates, deleting outdated files is now supported. *Without touching user added tables models...* If they manually add stuff to a mod and this mod is broken after update, it is advised to manually run the cleaning tool included in the script : it will remove all unsupported content.
- a lot of small fixes, including texts and menu ordering

[b][v0.0.22+][/b]
- fixed cleaning after update throwing errors if running offline.

[b][v0.0.23+][/b]
- added innoextract detection. If it's not installed, and if we are on Ubuntu, we install it ! I'll add support for other distros soon.

[b][v0.0.24+][/b]
- better/moved innoextract detection (we don't need innoextract if the original game is already installed
- GOG installer search can now be interrupted.

[b][v0.0.25+][/b]
- 7z now updates corrupt files instead of hanging the script

[b][v0.0.27+][/b]
- added RAR archives compatibility
- added subfolders compatibility
- fixed : JADXA install process

[b][v0.0.28+][/b]
- if continued download are corrupt, retry from scratch.

[b][v0.0.29+][/b]
- tweak : delete empty folders on 3 levels inside the mod folder each time we install, update, uninstall, clean user files.

[b][v0.0.30+][/b]
- fixed : download verification error if the archive has spaces in its name (escaped some strings)
- tweak : silenced harmless cleaning errors


Not released yet :
- next version will manage binaries (launchers/engine/nightly builds...)
- an "advanced mode" will be added to force cache downloads or override old files even if their checksums are ok.



You will only need to get innoextract to get it running.

Have fun, and please report bugs ! (i think there is no more accidental wipes of FS2 mods, but you may not use it on a FreeSpace 2 that you want to keep safe)
About the code : it's globally uncommented, i have the comments in the few parts of code i used to build this thing. I'll re add them later.
Some parts are duplicated, and some code paths in the download logic are not used (because i found a better implementation mid-way). I'll remove them if it doesn't break anything.

Links :

The installer : https://www.dropbox.com/s/8p7t7yci099ohzd/fs2gogextract.sh?dl=1

Mod files (load them in the installer) :
(Oh, and don't worry about installing mediavps before derelict or blueplanet : the installer will do it for you !)

Media VP 3612 : https://www.dropbox.com/s/os2ihzqx8b1c4o9/mediavps_3612.fs2mod?dl=1

[NEW !] Media VP 2014 : https://www.dropbox.com/s/vbcxbblkeedmhdr/mediavps_2014.fs2mod?dl=1

Derelict : https://www.dropbox.com/s/whzbx5uav1s9gvf/Derelict.fs2mod?dl=1

Blue Planet: Age of Aquarius : https://www.dropbox.com/s/blch7bqqq11iywx/blueplanet.fs2mod?dl=1

Blue Planet: War in Heaven : https://www.dropbox.com/s/gjcl7r57x6q5jfj/blueplanet2.fs2mod?dl=1

FreeSpace Port : https://www.dropbox.com/s/qx7ww8a0dddyq52/fsport.fs2mod?dl=1

FS Port Media VPs : https://www.dropbox.com/s/yzcll3dicpwewcl/fsport-mediavps.fs2mod?dl=1

Silent Threat: Reborn : https://www.dropbox.com/s/ofg3ib5nz5iql48/fsport-str.fs2mod?dl=1

[NEW !] Just Another Day Xtreme Arcade (JADXA) : https://www.dropbox.com/s/f4pusp3hrcc05eu/JADXA.fs2mod?dl=1

[NEW !] Vassago's Dirge : https://www.dropbox.com/s/b23gkitwj66pqtb/vassago.fs2mod?dl=1

[NEW !] Transcend : https://www.dropbox.com/s/be2c6xf24gvoeie/Transcend.fs2mod?dl=1

[NEW !] Sync : https://www.dropbox.com/s/g6u5xqznfh0o2oo/sync.fs2mod?dl=1

(any contructive critcism on the usability is welcome, i'm going to make other mod installers... please report broken links too...)




Making your own mod files is an easy task ! (mini HOWTO)

You will need to have the .zip and/or .7z archives of your mod in a folder.

Code: [Select]
cd YOUR_MOD_ARCHIVES_FOLDER
md5sum *

The output should look like this example:
Code: [Select]
c019c57629f352ff68b00c0a5478dca5  MediaVPs_3612.zip
37d110c741e55141f8d71c64f989f850  MV_Advanced_3612.zip

Copy and paste the full output to you favourite text editor.
Then, for each line, replace the 2 spaces (between the MD5 sums and the name of the archives) by semicolons ";" :
Code: [Select]
c019c57629f352ff68b00c0a5478dca5;MediaVPs_3612.zip
37d110c741e55141f8d71c64f989f850;MV_Advanced_3612.zip

Now, for each line, add your archives download urls, using semicolon ";" separators :
Code: [Select]
c019c57629f352ff68b00c0a5478dca5;MediaVPs_3612.zip;http://mvp.fsmods.net/3612/MediaVPs_3612.zip
37d110c741e55141f8d71c64f989f850;MV_Advanced_3612.zip;http://mvp.fsmods.net/3612/MV_Advanced_3612.zip

Make sure there is no new line at the end of this file.

Save it under the name : download


Now you will need to have the .vp and other .ini, .bmp ... files of your mod in a new folder. Extract archives to this folder.

Code: [Select]
cd YOUR_MOD_FILES_FOLDER
md5sum *

The output should look like this example:
Code: [Select]
a22db283154c07e5cfb5951c410e3600  FSU-MVP.bmp
ef9c1864ee8d90bd661ebfdc624dc76a  mod.ini

Copy and paste the full output to you favourite text editor.
Then, for each line, replace the 2 spaces (between the MD5 sums and the name of the archives) by semicolons ";" :
Code: [Select]
a22db283154c07e5cfb5951c410e3600;FSU-MVP.bmp
ef9c1864ee8d90bd661ebfdc624dc76a;mod.ini

Now, for each line, add the name of the archive the file has been extracted from, using semicolon ";" separators :
Code: [Select]
a22db283154c07e5cfb5951c410e3600;FSU-MVP.bmp;MediaVPs_3612.zip
ef9c1864ee8d90bd661ebfdc624dc76a;mod.ini;MediaVPs_3612.zip

Make sure there is no new line at the end of this file.

Save it under the name : vp


Create a new text file.
It has only one line, with the full name of your mod as it will be read by human beings
Code: [Select]
Media VPs 3612
Save it under the name : title


Create a new text file.
This is where the script will get auto updates for the mod. If you leave it blank or write anything that's not a valid address, the script will only complain about failing to auto-update the mod. The file should only have one line.
Code: [Select]
https://www.dropbox.com/s/os2ihzqx8b1c4o9/mediavps_3612.fs2mod?dl=1
If you are using droppbox, you may need to skip this step, and add this to the archive later, after dropbox gives you the sharing link.
Obviously, many free hosting services won't let you update an already uploaded file, or will change the address each time you update this file. This makes these hosting services incompatible with mod auto-updates. Sorry.

Save it under the name : update


Now if your mod is depending on other mods (no need to create it if your mod has no dependencies):

Create a new text file.
On each line, write the file name of a mod it's depending on without its extension (fsport-mediavps.fs2mod -> fsport-mediavps). Then add the url of this mod file, separated by a semicolon :
Code: [Select]
fsport-mediavps;https://www.dropbox.com/s/yzcll3dicpwewcl/fsport-mediavps.fs2mod?dl=1
mediavps_3612;https://www.dropbox.com/s/os2ihzqx8b1c4o9/mediavps_3612.fs2mod?dl=1

Make sure there is no new line at the end of this file.

Save it under the name : dep


Create a .zip archive containing files : title update vp download (and maybe dep ), all in a folder called root
The first part of the name of the archive will be your mod folder name. The second part is the .fs2mod extension.
For example : mediavps_3612.fs2mod


This file should now be accepted by the script.





Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: headdie on December 19, 2013, 08:07:15 am
Hmm, interesting stuff.  Windows and fairly advanced user myself so probably wont use it (and cant test it atm) but what you are describing would certainly make for a useful filler until goober finishes his efforts.  so yes definitely a place for this in the community.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: Hellzed on December 19, 2013, 12:40:36 pm
How is goober's installer managing mods ? Does it come with a list of mods ? Some kind of repository ?
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: Hellzed on December 20, 2013, 03:33:16 pm
Up !
I changed a lot of things in the last few hours.
The script is still not able to manage binaries, but mods and GOG installer parts are in a usable state.

Anyone to try it ?
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: headdie on December 20, 2013, 05:31:09 pm
How is goober's installer managing mods ? Does it come with a list of mods ? Some kind of repository ?

a repository I believe is how goober is going
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: Hellzed on December 21, 2013, 02:36:41 am
For now, my script has 3 (-2 now !) big issues :

-it only handles files, not folders, inside mod archives. As long as there is no clear standard for zip and 7z mod files structure, i take all the files in the archive, drop the folder structure, identify what's usable by fs2_open, and put that in a mod folder with the right name.
That may be fixed if my script could identify specific folder names inside the zip, as /data/ /tables/ ... Not sure that the compatibility would be 100%...
Any thoughts ?
=>I could add a "path" field in the mod files.
FIXED (as of version 0.0.27) ! Added install path in mod descriptors, indeed.

- FIXED - updates : if files are added by an update, that's ok. If files are updated, that's ok too. If files are removed, it's getting tricky.
I could fix that by either performing a "clean" on the mod folder, but that would drop any user added stuff, like mjn's packs of ogg cutscenes for media VPs or fsport, OR comparing the list of old mod files with the new mod files, and drop what's not there anymore (more work for me, a few changes in the update logic). Some kind of "cleaner cleaning" that would spare user added stuff, but would not ensure that the mod is in a working state (what if user's old stuff breaks the new stuff in the mod ?)
Maybe default to soft clean process, and ask the users if they want to clean everything at the end of the install process...
What would be the best solution ?  => I went for saving the old files list, comparing with the new, and deleting what's in the old list but not in the new one. This way, user added tables and models are safe. The script output now recommends to do a complete cleanup if experiencing any trouble after update, though.
(i could get rid of some duplicated code too, or maybe not, depending on the probability to break something...)=> Didn't do that yet. But I think If I set an a better mod operations order like : ADD MOD > INSTALL > UNINSTALL > CLEAR CACHE > CLEAN (only auto files) > CLEAN (everything) > REMOVE MOD (including mod folder if empty) ; then if I make a better use of variables between steps ; and finally use generic wording (not dependent from previous step) at each step ; I could compltely remove duplicated code.

- My whole descriptor file structure IS NOT made to support optional files inside mod folders... For example : Blue Planet and BP Compat / Media VPs and MV Advanced. I cannot keep track of optional files, because that would mean I would would have store these options somewhere, and keep these options for the update or cleaning process... In terms of complexity, the only solutions I see are madness (change mod descriptors structure to show what's optional, and prompt the user during install process, store settings somewhere...)
Maybe i could find a workaround, by considering they are in fact "different mods" that go in the same folder. But that would lead us to both previous bullet point (how do i identify stuff that sould be here if the user decides to perform a "cleaning" ?) !
Not sur i'm willing to fix that. I mean, this was supposed to be a simple bash script...
=> I'm starting to see some kind of fix for that : when i'll add the "expert" mode, with the user will have the ability to skip download and install for some files. So people may skip the files they don't want. OR I could do "light" mod descriptor files, which would include only the bare minimum for running a mod. I'll have to find a way to make "light" and "complete" mod descriptors coexist (they would share the same mod folder but obviously the light mod install would break the "complete" mod install... FINALLY i'll go for the expert mode, i don't feel like supporting conditional dependencies.
Maybe it will even feel better for now users to have a choice between "Media VPs Complete" and "Media VPs light (old hardware)". Less confusing.


Among other things :
- Where shall I grab the builds (fs2_open and wxlauncher) ? Maybe I should make both 32 and 64 linux builds myself (including debug stuff), and host them ? How would I ensure best compatibility ? I know how to cross compile with LXC, so I may give it a try, but my goal was not to maintain a repository by myself.
Anyway : i may process the builds a bit like the mods (but this part is much more simple !), having a zip file with a list of available builds, and giving the user the possibility to install the ones he wants.

- deps issues : not sure of a good way to identify a mod that cannot be removed at some point because other mods are depending on it. I guess if a mod doesn't run, the user will do a "check and update". And the script will reinstall the missing mod ^^

- i'll add some destribution specific code (debian, ubuntu, fedora, arch, what else ?) to check if innoextract is installed. If it's not, use the distribution's package manager to get innoextract.

- maybe a CD data extraction module would be welcome. (I just found my old FS2 CDs in an old drawer)

Last thing : with BP_Compat, I ran into a small issue. BP_Compat archive has 2 files : one fore blueplanet, the other for blueplanet 2. Once my script has downloaded both mods, you have 2 copies of the same file, one in both mod's cache folders... That's ok because this is a really small file. But what if that happened with a bigger file ? That would not be ideal for disk space.
Actually, this issue would be solved if the cache folder was common to all mods... But that would break the cache checking/download logic when you have 2 versions of the same mod in the same FreeSpace 2 install, like Media VPs and Media VPs beta (IIRC some ZIP and VP file names are shared, that would make sense)...

That's all, folks ! If somebody in this community has a few answers and ideas i could use, I would be happy. This post will also be a useful personal reminder about the work left to do. Also, I'll need a better bash script editor, because I'm tired of scrolling in gedit. I'd like an editor that could match loops and conditional blocks, before starting to re-organise som parts of the code.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: niffiwan on January 03, 2014, 08:22:16 pm
I gave this a go to download the mediavps_2014 - in short, very nice :) :yes:  Download the mod spec file, give its location to the installer and just wait.

I've got a bit of feedback:

1) could you add an option to specify the location to the GOG installer, it takes quite a while to run that find command :)

2) can you check for the presence of innoextract prior to searching for the GOG installer and exit that screen gracefully if it can't be found in the PATH (I don't have innoextract available in the Mint 13 repo's, and I couldn't be bothered building it since I've got a working install already)

And on the question of providing binaries, I think the best way to deal with this would be to have statically linked binaries that will run on the majority of current distros. In theory native distro packages is better, but it takes a whole lot of effort to provide these for every distro around.  The OpenSuse build service helps somewhat, but it falls slightly short due to the... odd... support it has for DEB packages, and other minor quirks (like not being able to find libopenal1 in most ubuntu multiverse repos).

Anyway - this is a great effort, I'd thoroughly recommend other Linux users trying it out :)
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: Hellzed on January 04, 2014, 02:11:12 am
1) I'm moving this step, right now (users will now have the choice to do a search or not) -> Better solution : search can now be interrupted, but keeps partial results
2) Reading "/etc/lsb-release" looks good enough to detect most distros. Now I have to find which distros have innoextract in their repos (the innoextract official install guide has some useful information, but some parts are missing IIRC).

I tried to compile fs2_open with "-static" argument passed to g++. It fails at openal1 detection. I really don't understand... Without this argument, everything is ok !

(oh, and thanks for testing !)

EDIT : updated to v. 0.0.24
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: niffiwan on January 04, 2014, 05:10:07 am
yeah - we (i.e the SCP) need to do some work with autotools or cmake before we can properly build statically linked FSO execs.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: Hellzed on January 04, 2014, 06:19:40 am
About openal, I think the issue may be on the Ubuntu packaging side, since it doesn't ship libopenal.a (or any *.a file...).

EDIT : I'm taking requests too (next mod to make a spec file for ?)
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: ghostdancer on January 04, 2014, 10:50:12 am
Hi, i'm new around here so i expect i'm doing right , i found a difference between th md5sum of this file MV_Root.7z of MediaVP_2014  (96BF9C147C7E371ABDE42A10064D9714) and the sum you have inside the "mod file" for your script  mediavps_2014.fs2mod ( C6BF9C147C7E371ABDE42A10064D9714)  and is always marking it like corrupt download .  I'm using the script in Debian to install the mods and apart from that it looks like is working , if i find anything else i'll tell you. Nice work.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: Hellzed on January 04, 2014, 10:55:40 am
They must have silently updated MV_Root.7z. Wait 5 min from now, and retry the install process.

EDIT : they changed the readme... :D
It should work now !

EDIT 2 : there is also a bug in my download/checksum logic, because I had to wipe the cache for Media VP 2014, or even the right download will be identified as broken... I guess it has to do with how wget handles resuming downloads. I'm looking into this right now.

EDIT 3 : I think mjn's server has some trouble handling its current load, so resuming downloads fail once in a while.
I also discovered that the 7z command line utility does not update files by default. It just stops doing nothing (actually it asks a question, but the user of my script doesn't see directly 7z's output). This behaviour is now fixed.
New version 0.0.25 now online ! You may want to update the script.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: niffiwan on January 04, 2014, 08:04:45 pm
EDIT : I'm taking requests too (next mod to make a spec file for ?)

JAD:XA! :D
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: Hellzed on January 05, 2014, 03:44:25 am
That will take a bit longer than I thought, because JADXA is not entirely stored in the form of VP files (it has a "data" subfolder). I'll have to make a few changes in the script to handle that, hopefully without breaking all the installers I made previously.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: Hellzed on January 05, 2014, 05:46:16 am
@niffiwan :

Here's JADXA : https://www.dropbox.com/s/f4pusp3hrcc05eu/JADXA.fs2mod

This new version of the script should work with any previous mod file. JADXA mod file is not to be used with fs2gogextract prior to version 0.0.27 (it could only forget to install files from "data" subfolder, or it could wreck your fs2 install... haven't tried myself.)

Oh, and I hate rar archives. I have to know a file's path inside the archive to extract it, so the extraction logic will break much more easily with terribly packaged mods.

EDIT : Uninstalling is now fixed.
Cleaning user added files is still not fixed.

EDIT 2 : Cleaning is now fixed. Bumped version to 0.0.27.

@ghostdancer :
I finally found out why my script is identifying downloaded files as corrupt even after redownloading them.
Under some circumstances, "wget --continue" resumes the download process in the wrong place, hence corrupting the file. the download then ends normally, without an error message, so my script continues and runs the md5 checksum.
I could disable the "--continue" flag. But FreeSpace 2 mods are usually big, and this bug doesn't happen really often...
I think I'd rather odifiy my script to try to resume a failed download. If the checksum is wrong, then drop the corrupt file and retry to download from scratch.

EDIT : this case is supposed to be fixed in 0.0.28 (try to resume download, if it's still corrupt, we retry to download from scratch)
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: ghostdancer on January 05, 2014, 10:49:25 am
After the first fix began to work, it was a simple md5 difference , and yes i agree that the fs2 mod files are too big not to use continue, your idea of first try and resume and the if there's another error download from scratch is really good.  I always though that the best option for sharing those big files should be something like torrent where the files are verified in chunks and not all file so you just need to download the damaged part and not all the file.  I'll give it a try to the new version. Good work

P.D.: i also hate rar files,  ;)
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: Hellzed on January 05, 2014, 11:39:30 am
I would have preferred to use rsync as a backend, but it only takes advantage of a specific sync server. As most modders at HLP host their mods as HTTP downloads, it would be useless, and I never intended to create my own repository.

I just made a few tweaks in version 0.0.29 (mostly cleaning empty subdirectories). That's great to finally have testers ! I can see a lot of issues I would never have found out without some help ! :)

EDIT : JADXA has been updated, the mod file too ! It should auto update next time you run "check & update" on JADXA.

EDIT 2 : Still taking requests ! ^^
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: ghostdancer on January 05, 2014, 12:19:02 pm
Maybe you could have a repository for the fs2mod files, that when the user runs the program it updates the fs2mod files and looks what's new and if there's been an update in any of the installed ones .
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: LordMelvin on January 05, 2014, 01:41:33 pm
This will shortly be relevant to my interests, and I am posting here to say, "Good on you, keep it up, I'll try it soon."

Good on you, keep it up, I'll try it soon.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: Hellzed on January 05, 2014, 02:48:49 pm
@ghostdancer :
I may not add an "update everything" option to the installer, but I think I will soon add something to show if an update is available ("Update available" next to the mod in the list, and maybe "x updates available" next to the mod management option in the main menu)

I'm thinking about taking all the technical bits of this installer, make them into functions, and create a command line utility with an "apt-get" like syntax.
Something like :
Code: [Select]
$ fs2tool --add path/to/your_mod.fs2mod
$ fs2tool --install your_mod
$ fs2tool --update your_mod
...
Then, with a bit more code, I could make something like APT links. You click on a mod link with your usual browser, it downloads the mod descriptor, starts the script and installs the mod for you. Next step is a nicely presented mod database (nothing technical on pages, just nice screenshots, an introduction to the backstory, mod size... I was thinking about something like this : http://themefuse.com/demo/wp/photoartist/ )

I wonder if an HTML5/bash or python launcher for FS2 would even be possible. Hooks in bash or in python, and an Install/Update/Remove button on each mod page (think Firefox extensions, Gnome extensions...)...

Interesting reading here : http://ubuntuforums.org/archive/index.php/t-433548.html
It's easy to create a small protocol linked to bash scripts.
I will work on a "proof of concept" to adapt this to freespace 2 ASAP !

@LordMelvin :
Now that the mod management part seems to be working fine, I'm looking into fs2 binaries installation.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: Hellzed on January 05, 2014, 07:32:53 pm
Vassago's Dirge is now available ! https://www.dropbox.com/s/b23gkitwj66pqtb/vassago.fs2mod
Installer has been updated to 0.0.30, a bug of download verification if the archive has spaces in its name has been fixed. Some harmless cleaning errors have been silenced too.

Transcend is now available too ! https://www.dropbox.com/s/be2c6xf24gvoeie/Transcend.fs2mod

Sync : https://www.dropbox.com/s/g6u5xqznfh0o2oo/sync.fs2mod
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: ngld on January 26, 2014, 02:44:18 pm
Hmm.... Wouldn't it make sense to make this script compatible with Goober's installer configs (http://fsport.hard-light.net/installer/fsport.txt)?
That way you don't have to write your own mod installation scripts... (the .fs2mod) files.

I'd do it but I'm wondering if it would be more useful to implement this in python. It would certainly make the coding easier but it would add python as an additional dependency.
What do you think? I'd like to try...
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: Hellzed on January 26, 2014, 05:11:08 pm
I thought about it, and I like the idea. At first, I made fs2mod files this way because they are easy to generate with the md5sum command line tool and some very light editing, and they are easy to parse with tools like read/grep/cut/sed/awk. Goober's installer configs look a bit harder to parse in bash, and the logic behind them is really different (they are meant to be stored in a repository and somehow "invisible" to the end users, while fs2mod files are meant to be acquired and updated from various locations).

Porting this script to python would be nice, as it would be the first step to make it a cross-platform tool. I'm a beginner in python, so right now I have no idea how dependencies to innoextract, zip, 7zip, rar and some other command line tools are handled.

Did you try "apt" protocol links on Ubuntu ?
I have an idea, currently on hold (because Goober's installer is not stable yet, and we will have a talk with Goober and probably a few others about it once they have more spare time) :
a "fs2://" protocol to perform "one click" mod installs in FreeSpace 2, similar to "apt://" links in Ubuntu, and add these links to the mod database (and maybe make this database prettier). It's fairly easy to do, secure (because users only install what's available in their repository) but we would have to agree on the tools we want to use to do it (modify the new installer, or do something else...).
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: m!m on January 27, 2014, 03:38:31 am
Trying to parse the installer configs is probably not a good idea if we have the chance to switch it to a better format like JSON which would also make parsing the configs quite easy. Instead of having to write a parser yourself in your language you could just use an existing JSON parser.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: ngld on January 27, 2014, 04:12:58 pm
Thanks for the note m!m but I already finished the parser. The current format actually is pretty easy to parse since it uses linebreaks to separate values and uses only a few tokens.
JSON would surely make it easier to work with the files but I guess Goober has other priorities right now.

I've run into a small problem now. Goober's files don't contain all the information your script needs, Hellzed.
In particular the filelist (the vp file) and dependencies are missing.
I can write a script which automatically parses Goober's installer files and generates the needed information. I would be willing to host it, too.

Python scripts can run normal shell commands so it should be no problem to use innoextract, 7zip or other tools.

Your idea with the fs2:// protocol sounds nice! I think it would be nice if you could also add repositories using that protocol but with a question like "Do you want to trust this repository?".
That way every modder could use it without having to get their mod into a central repository. What do you think?
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: Hellzed on January 27, 2014, 09:57:39 pm
I really haven't studied how Goober's installer works... but how can it perform install/upgrades/checks without using a VP files list ? Isn't it supposed to handle dependencies ?
Correct me if I'm wrong, but I'm under the impression that Goober's installer is "action oriented" (download that file, add this folder, remove that stuff) while my script is "description oriented" (this is how a functioning mod looks like, then the script does whatever is needed to get to this state)

Now I'm worried I did something completely inefficient... I already knew it from a disk space point of view, as Goober's installer downloads and unzip files at the same time, while my script creates a cache of archives.

My primary concern with auto adding new repositories, even with a nice warning window, is that people say yes to nearly anything. I'd prefer a safer process, like people having to go to some settings screen -> add repository -> copy and paste the url there.

Anyway, to advance towards a "One-click" mod install/launch process, we would need 3 pieces of software client side :
- a fs2 protocol handler, knowing which mod is installed and which one is not (and start the appropriate action).
- a mod installer : either some modified version of Goober's installer (it would need to get a flag passed by the fs2 protocol handler, with the name of target mod), or some cross platform version of my script connected to a default repository of fs2mod files, and able to connect to user added repositories.
- a mod launcher : either a modified version of wxlauncher, skipping all the GUI part, and using the last options profile used with the mod called by fs2 protocol handler, or a new lightweight launcher (only managing fs2_open flags, per mod).

Server side :
- a new HTML5+CSS3 GUI viewable from any browser, connected to FreeSpace mod database, with the appropriate fs2:// links. I wonder if optional extras for mods (compatibility packages, advanced graphics) would have to be handled by the protocol on the web GUI side (checkboxes on the mod page, a bit like DLCs on Steam) or by the mod installer script GUI (checkboxes in the mod installation confirmation window opening after clicking on a fs2:// link)

And, as optional but interesting extras :
- a standalone, offline, mod browser. A simple Webkit/python window showing installed mods even offline, without using a full blown browser, in order not to be dependent from HLP being online to launch FreeSpace 2 and its mods.
- a dedicated installer, bundling all previous pieces of software together.

It's not that much work, for example, for example, a basic version of the standalone browser I dashed was 15 lines of python code, the protocol handler is only a few lines too, the online and offline HTML5+CSS3 GUIs would be based on the same template (I already have a working example, although incomplete and not ready for production), and if we can use Goober's installer and wxlauncher stripped from parts of their GUI, it would only be small patches.

The biggest parts would be the preprocessor (PHP, Java, Python, whatever else) to generate the mod catalogue from the mod DB using enhanced versions of my templates, porting my script to a cross platform python utility if we chose this solution, and creating a lightweight launcher/settings window.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: Goober5000 on January 28, 2014, 09:46:27 am
I really haven't studied how Goober's installer works... but how can it perform install/upgrades/checks without using a VP files list ? Isn't it supposed to handle dependencies ?
Correct me if I'm wrong, but I'm under the impression that Goober's installer is "action oriented" (download that file, add this folder, remove that stuff) while my script is "description oriented" (this is how a functioning mod looks like, then the script does whatever is needed to get to this state)
You're correct.  The installer lists files to download, and zips to extract.  The VPs are not explicitly specified, but they are obtained from unzipping the files.

Dependencies are not explicitly specified, but they are inferred from the installation hierarchy.  If a mod is listed as a child of another mod, it is assumed to depend on its parent.  (Of course, this can only be done if all mods are specified in the same config file.)
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: ngld on February 06, 2014, 09:45:37 am
Well... I'm mostly done with my Python port by now. I'll publish it once I've solved the last Windows bug.

Here are some quick screenshots:
(http://dev.tproxy.de/fs2/screen1.png)(http://dev.tproxy.de/fs2/screen2.png)(http://dev.tproxy.de/fs2/screen3.png)

BTW, is it fine to use the HLP logo as icon of the windows exe?
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: Goober5000 on February 06, 2014, 06:31:57 pm
BTW, is it fine to use the HLP logo as icon of the windows exe?
I would think so.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: ngld on February 08, 2014, 09:24:09 am
I've finished the first version of the python port. The code is on Github (https://github.com/ngld/fs2mod-py).

For Windows I've built a package which should work out-of-the box: fs2mod-py.exe (http://dev.tproxy.de/fs2/fs2mod-py.exe)

For everyone else: It runs on Python 2 and 3, needs either PySide or PyQt4 as well as six (another python module) and 7zip. It doesn't matter where 7zip is as long as it's in your PATH.
The mod list should be the same Goober's installer shows.

Click (or double-click if you're on Windows) to see details for each entry. The installer should automatically fix the problem with the MediaVPs folder name (all files will be placed in one mediavps_3612 folder and not in mediavps_3612 and MediaVPS_3612) on Linux. It should also automatically detect installed mods (if they are listed) and should be able to detect corrupted files.

Screenshots:
(http://dev.tproxy.de/fs2/win_screen1.png)(http://dev.tproxy.de/fs2/win_screen2.png)(http://dev.tproxy.de/fs2/win_screen3.png)
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: Hellzed on February 08, 2014, 11:25:34 am
I followed your instructions (installing dependencies and trying to run manager.py) but there is an error :
Code: [Select]
$ python manager.py
Traceback (most recent call last):
  File "manager.py", line 1011, in <module>
    main()
  File "manager.py", line 930, in main
    os.chdir(os.path.dirname(__file__))
OSError: [Errno 2] No such file or directory: ''
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: ngld on February 08, 2014, 11:30:58 am
Argh! Thanks, fixed it.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: Hellzed on February 08, 2014, 11:35:25 am
I'm not sure how exactly it mixes fs2mod files and Goober's installer files management, if it doesn't bother you, could you explain that part a bit ?
(is it possible to add our own fs2mod files yet ?)

EDIT : Anyway, this is very good work, I just installed Windmills, no issue.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: ngld on February 08, 2014, 12:07:40 pm
The code uses two classes to represent the two file types: ModInfo in fso_parser.py and ModInfo2 in fs2mod.py.
ModInfo2 extends ModInfo and adds a file list as well as a dependency list. Those are the only things present in your fs2mod files but not in Goober's installer files.

Right now I'm using converter.py to fetch and parse Goober's files and generate add the additional infos needed by inspecting the mods, the result is a JSON file (http://dev.tproxy.de/fs2/all.json) which is downloaded and parsed by manager.py .
The contained information is mostly the same except for the dep and update parts. I've left the update part out because the program (right now) uses a central repository to find the JSON files.
I'm generating the dependency list from the mod's mod.ini file so it's actually a list of folders the mod needs. The manager figures goes through its list of known mods to find matching mods (i.e. it would see mediavps_3612 in a dependency list and see that the MediaVPS contain a "mediavps_3612" folder).

You can actually use the converter to generate fs2mod files:

Right now the manager (nor the converter) can read fs2mod files. I've only implemented my special JSON version, yet, but I can add support for fs2mod files.
Should I add a button "Open fs2mod file" or something else to the manager? I'm thinking of letting the user add/remove repos as well.

EDIT: Fixed a small bug in converter.py, please update if you run into that.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: Hellzed on February 08, 2014, 12:35:30 pm
You could add a "Mod sources" tab, default source being the main Hard Light Project repository, other sources could be the different fs2mod files. I'm still thinking of a good way to  chose which source should have priority.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: ngld on February 08, 2014, 01:09:32 pm
The manager can load mods from multiple sources and display all of them at them same time. The priority will only matter if the installer encounters the same mod in multiple sources.
In that case it would make sense to pick the most recent version but what should happen, if the versions are the same? (i.e. The mod manager might find the Sync mod int the HLP sources and in an fs2mod file the user added.)

As long as the official versions are the most recent ones I'd say that they should have a higher priority but if the user adds a source with some newer version, that source sould be picked.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: Hellzed on February 09, 2014, 02:37:09 am
Maybe we could add explicit versioning in fs2mod files (a "version" text file). I would say, if users added a fs2mod file, we can safely assume that this is the version they want.

(Something is broken in the latest version of the installer : if I try to update the mod list, the progress bar window shows up and sits there, doing nothing)
Code: [Select]
INFO:Thread-4:util.get: Retrieving "http://dev.tproxy.de/fs2/all.json"...
ERROR:Thread-4:manager.work: Failed to decode "('json', 'http://dev.tproxy.de/fs2/all.json', "ngld's HLP Mirror")"!
Traceback (most recent call last):
  File "/usr/lib/python3.3/json/decoder.py", line 367, in raw_decode
    obj, end = self.scan_once(s, idx)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "manager.py", line 97, in work
    data = json.loads(data)
  File "/usr/lib/python3.3/json/__init__.py", line 316, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.3/json/decoder.py", line 351, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.3/json/decoder.py", line 369, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: ngld on February 09, 2014, 06:54:16 pm
Sorry, my repository was broken. It should work again.

I've finished my implementation of fs2mod files. Now that it's working I see another problem: There is no way to detect if two mod entries actually refer to the same mod.
For example:

To load a fs2mod file (in the current dev version) a user can now go to "Settings", click "Add", select the file and press "OK". The next time the user clicks on "Update List" te selected file and all dependencies will appear in the mod list.
This means that that you will see both "FSU MediaVPs, version 3.6.12" and "Media VPs 3612" in your list. The script should mark both either as installed or not as they refer to the same files.
If you don't have the MVPs installed it should show something like "Not installed (10 shared files)" which means that the manager found 10 files this package shares with others.

Maybe comparing the filelists will help finding duplications...
Right now this version is on the develop branch on Github. I'll merge it into the master later.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: niffiwan on February 09, 2014, 09:37:18 pm
This is looking good.. unfortunately I haven't been able to get it running on Linux so far :)

I've tried on Mint 13, but the version of six seems to be old and it doesn't have six.PY2 available. Do you know what the the minimum version requirement for six is?

I've also tried on Centos 6 (w/ EPEL).  It runs with python2, but the button "Select installed FS2 (Open)" doesn't seem to do anything.  I get the same results from both the master and develop branches...
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: Hellzed on February 09, 2014, 09:51:35 pm
I have updated the fs2mod file for Derelict : it now has a "version" file, with a version string. It should make version management easier.
Other fs2mod files will be updated soon.

Vassago's Dirge fs2mod file has been updated with a version string too, and now compatible with media VPs 2014.

EDIT : I think there's an issue with the update process, though, since it doesn't get rid of the outdated VPs.

EDIT 2 : it's around line 1294 in the original bash script, a saved version of the old fs2mod file is compared with the updated fs2mod file, and files that don't belong there any more are removed.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: ngld on February 10, 2014, 03:46:05 am
Thanks, Hellzed! I'll add support for the version files.
You're right about the update process. Right now it simply compares the filelist in the fs2mod file or from Goober's files with the files the user has and replaces any files with wrong checksums and adds missing files. I'll add detection for outdated and user-added files.

@niffiwan: Thanks for testing! I'm using the latest version of six which is 1.5.2. If you have pip installed you can run "pip install --upgrade six" to update it. If you don't want to update the package system-wide (or don't have pip) you can simply download the latest version (https://pypi.python.org/packages/source/s/six/six-1.5.2.tar.gz#md5=322b86d0c50a7d165c05600154cecc0a) and put the six.py file in the same directory as manager.py. That should fix the issue on Linux Mint.
"Select installed FS2 (Open)" should open a dialog which asks you to select a directory. What desktop environment are you using? Did the script show something on the console?
Just to check: If you click on "Install FS2 with the GOG installer", a window should open asking you for the path to the installer and a destination. If you click on the three dots next to "GOG installer", does anything happen?
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: niffiwan on February 10, 2014, 04:50:06 am
Thanks, I installed six 1.5.2 on Mint (home) PC and it's working there as well now. With python2 at least, python3 complains that it can't find 7z, but it's installed! And I get the same issue with the "Select installed FS2 (Open)" bit :)

I'm using the MATE desktop.

There was nothing output to the console when clicking on the "Select installed FS2 (Open)" button (although clicking on other buttons output "FIXME: handle dialog start.")

When selecting "Install FS2 with the GOG installer" and either source or destination "..." button I get the normal directory browser that you'd expect.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: ngld on February 10, 2014, 05:26:06 am
I'm not sure what the exact problem is but I've added a workaround to the develop branch which avoids the native dialog.
Can you check if this solves the problem?
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: Hellzed on February 10, 2014, 06:09:01 am
I was wondering : do you think it would be convenient to register fs2mod-py as a protocol handler for "fs2://" links ?
Imagine you click on "fs2://run/Derelict". Instead of showing the main window, it updates the list of available mods from the current sources, then checks if an item named "Derelict" is in the list. If it is not installed, we start installing Derelict.

I'm still trying to familiarise with python and github, so I forked the project and patched it with a few lines to read the version string from fs2mod files. I'm also looking into libunity API to check if we can do something for a better Unity/Ubuntu integration (using hard light icon, and maybe adding a progress bar on the icon when something is installing), the problem being that it's distro specific and I have no idea how not to break everything else trying to implement this.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: ngld on February 10, 2014, 06:23:11 am
Did you take a look at the documentation in the Ubuntu Wiki (https://wiki.ubuntu.com/Unity/LauncherAPI#Python_Example)?

To avoid breaking other platforms you can use a try ... except block like this:
Code: [Select]
try:
  from gi.repository import Unity
except ImportError:
  # Can't find Unity.
  Unity = None

If you want do anything Unity-specify in the code just wrap it in an if:
Code: [Select]
if Unity is not None:
  # Update the icon....

If the user has Unity, the import succeeds and you can use the module in your code. If the user doesn't have it, the import fails and Unity is set to None. Any Unity-specific part would be skipped in that case.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: niffiwan on February 10, 2014, 06:29:26 am
I'm not sure what the exact problem is but I've added a workaround to the develop branch which avoids the native dialog.
Can you check if this solves the problem?

I tried commit 0b6c47fd5ad0b42f23c9623b1b350a5a487c580b on the develop branch and I get the same result.

I should also clarify; I get the output to the console from the working-as-expected "Install FS2 with the GOG installer" directory picker. 
I don't get any console output from clicking on the "Select installed FS2 (Open)" directory picker.

i.e. here's the output I get from running manager.py, clicking on "Select..." multiple times, then clicking on "Install..." once only, then pressing cancel.

Code: [Select]
$ python manager.py
Got bus address:  "unix:abstract=/tmp/dbus-NgcGCjGTPB,guid=b3a6fba5e5ce8542b3576e6e0000076b"
Connected to accessibility bus at:  "unix:abstract=/tmp/dbus-NgcGCjGTPB,guid=b3a6fba5e5ce8542b3576e6e0000076b"
Registered DEC:  true
Registered event listener change listener:  true
FIXME: handle dialog start.
FIXME: handle dialog end.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: ngld on February 10, 2014, 07:10:39 am
Do you have the qt-at-spi package installed? That might cause this problem. (https://bugs.launchpad.net/ubuntu/+source/qtiplot/+bug/997462)
It seems that the package can cause crashes (https://bugs.launchpad.net/ubuntu/+source/qt-at-spi/+bug/857790) when a qt application opens the directory picker but I'm not sure since the program doesn't crash.
If this doesn't help, try this build (http://dev.tproxy.de/fs2/fs2mod-py.bin)(33MB!) it should use my version of Qt.

EDIT: @Hellzed: Your idea with fs2:// links sounds nice. I've tried to implement it in the current dev version. It works on Windows and in Firefox on Linux. If you click on this link (http://dev.tproxy.de/fs2/launch.php?run/Derelict) Firefox should ask you how it should open it. If you select the manager.py file (and that file is executable) and press OK it should start fs2mod-py.
On Windows you can simply start the script then go to "Settings" and click on "Install handler for fs2:// links". Windows will ask you if you want to modify the registry. If you accept all fs2:// links on windows should be opened with fs2mod-py.
I'm not shure about other browsers on Linux... Chromium uses xdg-open which in turn always complains "xdg-open: file 'fs2://run/Derelict' does not exist".
Well, at least it works in some cases.

EDIT2: HLP doesn't allow non-http links... Fixed it using redirection.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: AdmiralRalwood on February 10, 2014, 11:33:39 am
Did you take a look at the documentation in the Ubuntu Wiki (https://wiki.ubuntu.com/Unity/LauncherAPI#Python_Example)?

To avoid breaking other platforms you can use a try ... except block like this:
Code: [Select]
try:
  from gi.repository import Unity
except ImportError:
  # Can't find Unity.
  Unity = None

If you want do anything Unity-specify in the code just wrap it in an if:
Code: [Select]
if Unity is not None:
  # Update the icon....

If the user has Unity, the import succeeds and you can use the module in your code. If the user doesn't have it, the import fails and Unity is set to None. Any Unity-specific part would be skipped in that case.
Python programmer here, just chipping in that the latter check can be written more concisely thusly:
Code: [Select]
if Unity:
    # do stuff

(It also avoids silly bugs, like if you accidentally typed "Unity = False", it won't say, "Hey, 'False' isn't 'None'! Let's use it!")
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: Hellzed on February 10, 2014, 11:53:03 am
@ngld : I just tried to update both Blue Planet mods. It says it's only missing bp-compat.vp and bp2-compat.vp, but it started to download all archives for both mods when I confirmed. Is it a bug, or a check not implemented yet ?

From command line (python3 manager.py fs2://run/Derelict), directly installing Derelict worked, but I still cannot register the fs2:// protocol in firefox.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: AdmiralRalwood on February 10, 2014, 12:12:56 pm
@ngld : I just tried to update both Blue Planet mods. It says it's only missing bp-compat.vp and bp2-compat.vp, but it started to download all archives for both mods when I confirmed. Is it a bug, or a check not implemented yet ?

From command line (python3 manager.py fs2://run/Derelict), directly installing Derelict worked, but I still cannot register the fs2:// protocol in firefox.
Does this help (http://kb.mozillazine.org/Register_protocol)?
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: Hellzed on February 10, 2014, 12:19:20 pm
I tried this first, didn't help. Now the protocol is registered but Firefox complains about no app being able to open it.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: ngld on February 10, 2014, 03:18:18 pm
Thanks, Hellzed, that was actually a bug. The script generated wrong paths while checking during installation. (https://github.com/ngld/fs2mod-py/commit/f65aa9fc126133071277717ff321737c120abeec)

Does ./manager.py work? Maybe you need to change the first line of manager.py to "#!/usr/bin/python3" to make it work.
Firefox will complain about having no app for that protocol if it can't run the program it knows. Did you try entering the link or did you use my link (http://dev.tproxy.de/fs2/launch.php?run/Derelict)?

EDIT: Try to put the following in ~/.local/share/applications/fs2mod-py.desktop.
Code: [Select]
[Desktop Entry]
Name=fs2mod-py
Exec=/path/to/manager.py %U
Type=Application
Terminal=false
MimeType=x-scheme-handler/fso;

I had to change the protocol to fso:// because xdg-open doesn't allow numbers in protocol names.
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: Hellzed on February 10, 2014, 04:01:57 pm
You may also check the repository/fs2mod file adding logic : I can add empty stuff in there, and that leaves blank spaces in the sources list.

A lot of these bugs look awfully similar to the bugs I encountered with the bash version (wrong path forcing full re-download, adding empty stuff to the cache...) :D

Also, we're lucky that Dropbox keeps copies of old revisions, because (Ireally don't know how) Derelict.fs2mod was replaced by an empty text file using the same name, 2 times. Any idea what could be causing that ? (maybe it had something to do with adding and deleting empty repositories and it's fixed already ?)
(I'm not sure why, but when a fs2mod file is added, it's also always rewritten. Looks unsafe to me)

Each time I switch between launching the installer with python2 and python3, it loses all its cache and settings, and ends up doing all checksums again on next launch. Is it possible to avoid the issue ?
"#!/usr/bin/python3" may be necessary in Ubuntu, since it defaults to python2 if we directly launch the script.

The desktop entry alone doesn't work, it also needs to be referenced in "~/.local/share/applications/mimeapps.list"
Code: [Select]
[Desktop Entry]
Name=fs2mod-py
Exec=python3 /home/loris/fs2mod-py-develop/manager.py %U
Icon=/home/loris/fs2mod-py-develop/hlp.png
Type=Application
Terminal=false
MimeType=x-scheme-handler/fso;

Append this to "~/.local/share/applications/mimeapps.list"
Code: [Select]
x-scheme-handler/fso=fs2mod-py.desktop;
Something else : secondary progress bars appear in python3, not in python2. The main progress bar is ok in both.

I am also experiencing priority issues between the HLP repository and fs2mod files. Not sure which priority is intended behaviour.
(how complicated would it be to allow reordering of sources in the settings tab ?)

If I try to launch a mod from manager.py, fso currently crashes. I heard parsing flags and tabbed settings for fs2_open is not something easy. You could be interested in wxLauncher source code. I'm not sure what is your intention about launching mods directly from the mod installer (something like a "light" launcher ?)

Currently, wxLauncher is "profile centric" : it can store different settings profiles, with basic options, advanced options, and mod settings, for each profile.
A one-click mod install/launch process would need a "mod centric" launcher : base settings (screen resolution, AA, AF, controller, sound device) should be shared between all mods, while advanced settings (especially lighting, hud features...) should be set separately for each mod. When you chose to launch a mod, right settings are selected. wxLauncher has no way to that currently, and implementing it would end up with a bloated code base.

Something like fso://settings (for default profile, the one which is copied for default settings when we start a new mod for the first time) and fso://settings/Derelict (for Derelict profile) link to bring up a really simple settings screen would be great, with only the base settings, and a button to access a text zone to enter advanced flags (a bit like what steam does).

Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: ngld on February 10, 2014, 04:46:37 pm
Well, that's the update mechanism... Right now I'm fetching the file from the given URL in root/update whenever the user clicks on "Update List". If this fails, it leaves an empty file.
I guess I should save the downloaded file somewhere in /tmp and only replace the actual file if the download succeeds.
BTW Dropbox is kinda annoying when you want to download the fs2mod files. Right now I'm sending a "curl/7.x.x" (x are actual numbers in the code) User-Agent to convince Dropbox to give me the file and not a download page. The program also has to follow a redirect on every fetch.

When the fs2mod file is replaced with an empty file, did the script display some kind of error or exception?

EDIT: The settings are saved with pickle. Python 3 supports the new protocol version 3 (http://docs.python.org/3.3/library/pickle.html#pickle.dump) while python 2 doesn't (and won't) support it. I'll set it to 2 for now, that should prevent the settings loss.
I'm not sure about the progress bars... might be a PySide bug.
Title: Re:%
Post by: Hellzed on February 10, 2014, 05:50:23 pm
I didn't save the console logs, sorry.
My bash script made a copy of fs2mod files inside its config directory, before running the update on the copy. The original, wherever it is on the user's disk then becomes useless.

I guess we'll have to move fs2mod files somewhere on HLP. Since they are small files, they can be hosted as attachments on the wiki. I'll have to ask about it.

I think fso:// links should use fs2mod files names instead of the actual mod title, since space and special characters in urls are generally a bad idea (I just experienced some issues with "Media VPs 3612"...)
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: ngld on February 10, 2014, 06:18:48 pm
@niffiwan: I've finally found and fixed the bug you found. PyQt4 behaved a little bit different than PySide and thus skipped my directory picker. The develop branch should work now.

The python 2/3 settings issue is partly resolved. It works but all images will be lost when switching from 2 to 3. The manager should now ignore all mod entries which have no name.
I hope that I've fixed the fs2mod file issue now. The issue with the secondary progress bars is also resolved now the script had a rounding issue.

Actually allowing the user to sort the mod sources isn't too hard. Applying that order in tasks.py (in FetchTask.finish) is more work because I'll have to reorder the results there but it shouldn't be too hard. I'll see how long it takes.
FSO crashes on your installation? I'd like to see the error message...
The logic to run fs2 is actually pretty simple (run_mod() and run_fs2() in manager.py): It reads ~/.fs2_open/data/cmdline_fso.cfg on Linux and <FS2>/data/cmdline_fso.cfg on all other systems. It then tries to find the -mod flag and replaces whatever value follows that flag with the selected mod. Once that's done it simply runs the given FS2 binary. Could you post your cmdline_fso.cfg, please?

Right now it works for me but I'll take a look at the wxLauncher. Thanks for the pointer!
I could host the fs2mod files, if you want. I'm hosting the JSON files already anyway or you could use GitHub Pages (http://pages.github.com/) to host them.

EDIT: What happend to the thread title? :wtf:
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on February 10, 2014, 06:48:36 pm
Some connection issues with the forum, the page reinitialised while I was typing a quick reply.

I'm not sure about the current syntax for fso:// links : which string should I use now ?

This is my cmdline_fso.cfg :
Code: [Select]
-mod mediavps_3612 -missile_lighting -3dshockwave -post_process -fxaa -dualscanlines -orbradar -rearm_timer -ship_choice_3d -weapon_choice_3d -3dwarp -warp_flash -snd_preload -fb_explosions -keyboard_layout "azerty" -ambient_factor 75 -spec_exp 11 -spec_point 0.6 -spec_static 0.8 -spec_tube 0.4 -ogl_spec 60
I just tried to install mediavps_3612.fs2mod and some (if not all) files were actually downloaded 2 times. They are downloaded a first time, then their size drops to zero, and they are downloaded again. Any clue ?
Title: Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Post by: niffiwan on February 11, 2014, 05:36:43 am
Do you have the qt-at-spi package installed? That might cause this problem. (https://bugs.launchpad.net/ubuntu/+source/qtiplot/+bug/997462)
It seems that the package can cause crashes (https://bugs.launchpad.net/ubuntu/+source/qt-at-spi/+bug/857790) when a qt application opens the directory picker but I'm not sure since the program doesn't crash.

@niffiwan: I've finally found and fixed the bug you found. PyQt4 behaved a little bit different than PySide and thus skipped my directory picker. The develop branch should work now.

Hurrah! It works now on both my PCs, Mint 13 (which I needed to remove qt-at-spi from) & Centos 6.  And it looks really good  :yes:  Installing mods works very well, as does using innoextract to pull the FS2 data out of the GOG installer.  I like the download progress bars and how they remove themselves from the window when they finish.

My only comment is something that I think I previously mentioned to Hellzed (:D); I believe that the FS2 VPs from the GOG installer should be extracted into the root directory just like the retail installer did, and not into the data dir.  It's also what all our manual install guides recommend, and I believe it's what all the people on HLP who do support expect.

Another comment is that selecting a nested option doesn't automatically select the parent.  i.e. if you select "Derelict Voice Pack", "Derelict" itself is not selected or installed. It'd be nice to have the parent automatically selected when the child is selected.  It would also be good to apply this to dependencies.

Aside from that, this is a great effort  :yes:
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on February 11, 2014, 06:34:44 am
Found a bug, latest develop branch :
Code: [Select]
INFO:Thread-9:util.download: Downloading "https://www.dropbox.com/s/os2ihzqx8b1c4o9/mediavps_3612.fs2mod?dl=1"...
ERROR:Thread-9:fs2mod.update_info: Failed to update Media VPs 3612!
Traceback (most recent call last):
  File "/home/loris/Bureau/fs2mod-py-develop/fs2mod.py", line 286, in update_info
    shutil.move(path, self.update[2])
NameError: global name 'shutil' is not defined
WARNING:Thread-9:progress.add_task: Added an empty task of type "FetchTask". Ignoring it!

Maybe the "Working" window should be a bit more robust : currently if you close the window while something is downloading, you can restart the same download at the same time. I think it should send a warning if you try to close it, then kill all current tasks so we are sure we don't restart the resume the same tasks next time it's open.

By the way, my personal branch now has full Unity launcher integration (it has a quicklist to access tabs from right click, not really interesting except for right click -> Add Source which is a direct access to the "Add mod" window. And the icon in the Unity launcher shows a copy of the global progress bar.). I'll push it to my repository once it's rebased.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on February 11, 2014, 07:43:24 am
Thanks, niffiwan. The develop version automatically selects all dependencies of a selected mod (including its parent) but the default mod source (http://dev.tproxy.de/fs2/all.json) is missing some dependencies. If you change it to http://dev.tproxy.de/fs2/hlp.json it should work. The newer repo isn't compatible with the released version (master branch) so I changed the link.

@Hellzed: Thanks, fixed again... I should actually test the code before I commit.
Your cmdline looks alright so that shouldn't cause the crash... hm.
I've been able to reproduce the double download bug but I'm still trying to find out why it happens.

You don't need to change your fso:// links, I've just added support for links like "fso://run/FreeSpace Port". Special characters and whitespace are automatically encoded by the browser which results in "fso://run/FreeSpace%20Port". Fs2mod-py can now decode that and will look for "FreeSpace Port" and not "FreeSpace%20Port".

I'll prevent the user from closing the "Working..." window and instead add a "Cancel" button but I'm not sure how to implement it. After all, I can't simply abort a task but I'll have to revert all changes made by it.
Thanks for your work on the Unity integration!

Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on February 11, 2014, 07:48:32 am
You published some new code right after I have rebased my repository. If I send a pull request, what happens ? Can you solve the conflict on your side ? (my files are missing the reset code)
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on February 11, 2014, 08:11:57 am
It should work alright... You actually pulled in the reset code according to the commit graph (https://github.com/ngld/fs2mod-py/network).
When you create the pull request, Github should show you a list of changed lines. If it only shows your changes it's all right.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on February 11, 2014, 10:07:45 am
I posted the pull request, but I didn't see any conflict warning about the reset code. It's only shown in red, as deleted code, so I guess you will have to be careful about this one.

EDIT : Now the installer is almost in a working state (except for the double download bug), I will resume work on the HTML5 + CSS3 template for a mod browser.. Hopefully, most users won't have to directly use the mod browsing/sources interface (it has to be there for advanced users), but will use their web browser or a dedicated webkit/Qt browser (with a local mod list cache if people are offline) as the main hub for playing FreeSpace 2.

EDIT : oh, I see the changes you have made in my commits (indentation, spacing). I will try to respect PEP 8 next time. I'm still reading tutorials about python.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on February 11, 2014, 04:26:11 pm
I merged your changes and pushed my fix for that bug. I've made some changes (https://github.com/ngld/fs2mod-py/commit/f14a0be2ccd5053461fbccee6fb00f554a784df6). Is that alright?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on February 11, 2014, 04:52:01 pm
Thanks ! That's a good way to learn !

On a side note, we may need a higher resolution icon, since it renders upscaled (and blurred) in Unity dash, probably in Gnome-shell and Windows 8 too, but i'm not sure.

Edit : the error I get if I try to launch fs2_open : "Starting FS2 Open (/home/loris/FreeSpace 2 beta/fs2_open_3.7.1_llvm_BP) failed! (return code: -11)"

Edit 2 : I think fs2mod format needs a new evolution to support optional VPs.

- A keyword (like "opt") may be added at the very end of the line for some VPs (packagers will have to be careful about the number of ";" separators, because they have to take into account the *subdirectory* optional argument).
- An "opt" file listing the optional VPs : easier to do, cleaner. *All* VPs are still in the "vp" file, and this list only have optional VP names. This way, some of these VPs can be excluded selectively from the install process.

These not ideal solutions, and I recognize fs2mod is not that a great format (I just wanted something easy to parse in bash and to generate from command line md5sum tool when I wrote the first one) but it has the advantage not to break the legacy bash script (I may modify it... someday).

What fs2mod format does not need is support for mod description and picture, since description, pictures, download size, maybe user comments are to be included on the "launcher" web page.

What do you think ?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on February 11, 2014, 07:43:15 pm
Well here's the HLP icon (http://i37.tinypic.com/2076xzn.png), I just scaled it down because it's too big.

I think it's easier to list the optional VPs in a separate file like you suggested. However, the packager should be able to provide a description of the optional VP files similiar to what the Media VPs have. Without a description the user wouldn't be able to tell if he wants that file or not.

The description and picture would be nice to have but I might implement a way to automatically read those once a mod is installed.
BTW, what do you think about a wizard to create a fs2mod file? It would ask you to enter all URLs, then download the archives and generate the hashes (for the download file). Afterwards it can look inside the archives to generate the vp file. That would leave the title, update and dep files. (And, of course, the opt file). Title and update URL can be simple input fields, I guess. For the dep file I'd list all folders referenced by the mod.ini and allow the packager to enter a fs2mod link for each dependency or leave it empty (in which case it would be ignored). For the opt file, the wizard could simply show a checklist.

The fs2mod format is a simple format which makes it easy to parse and extend (just look how short read_zip() is...) IMHO. If you wanted to implement a parser for VP files, it would be much more work.

I guess I'll create issues over on GitHub for some of the current TODOs. For example, fs2mod-py still doesn't have a proper mod update routine, the user can't specify the mod source order (aka priority), I haven't fixed your FSO crash (might check wxLauncher's source on that) and it can't check for a new version (which can be important if the mod source / fs2mod format changes again).
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on February 12, 2014, 05:03:52 am
256x256 hlp.png icon will be in my next pull request. It already looks better in Unity.

A wizard is a good idea, of course.

I'm searching for a way to trigger an event if the main window is raised, what's the best way to it ? (it has to do with a last bit of Unity integration : highlight the icon and make it wriggle once all tasks are finished (for example, firefox and my torrent software do that), don't do it if the window is already raised, stop doing it if the user raises it). It will use libnotify too.

I'll study the install/update task code, at first for better debugging purpose, and maybe contribute later.

[attachment deleted by an evil time traveler]
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on February 12, 2014, 05:56:48 am
Here's the documentation for the Qt classes (http://qt-project.org/doc/qt-4.8/classes.html) and this a list of members of main_win (http://qt-project.org/doc/qt-4.8/qmainwindow-members.html).

You can check if the main window is active (raised) by calling main_win.isActiveWindow() (http://qt-project.org/doc/qt-4.8/qwidget.html#isActiveWindow-prop). There's no signal like "clicked" or "itemActivated" that is triggered but the focusInEvent (http://qt-project.org/doc/qt-4.8/qwidget.html#focusInEvent) is raised once the window gets focus. If you want to use it just write
Code: [Select]
main_win.focusInEvent = your_handler_function

The install/update code is run by the InstallTaskin tasks.py. Most of the actual code is in fso_parser.ModInfo and fs2mod.ModInfo2.

EDIT: That looks nice!
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on February 12, 2014, 07:54:30 am
Actually,
Code: [Select]
main_win.focusInEvent = your_handler_function wherever I put it in manager.py (I mostly tried to put it in the block of unity specific code in main() ), the handler function never gets called. There is no error in the console. Did I misunderstand something ?

(Apparently, it has to be reimplemented to be used)
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on February 12, 2014, 09:44:37 am
You're right, my bad. Apparently you have to use changeEvent. So the complete solution would be:
Code: [Select]
def focus_handler(event):
  if event.type() == QtCore.QEvent.Type.ActivationChange:
    if main_win.isActiveWindow():
      print('I\'m active!')
    else:
      print('I\'m inactive!')
 
  return super(QtGui.QMainWindow, main_win).changeEvent(event)

[...]
  main_win.changeEvent = focus_handler

Since we are overriding a method of the main window a cleaner solution would be to actually subclass QMainWindow, i.e.:
Code: [Select]
class MainWindow(QtGui.QMainWindow):
  def changeEvent(self, event):
    if event.type() == QtCore.QEvent.Type.ActivationChange:
      if main_win.isActiveWindow():
        print('I\'m active!')
      else:
        print('I\'m inactive!')
   
    return super(MainWindow, self).changeEvent(event)
Then you would have to replace
Code: [Select]
main_win = util.init_ui(Ui_MainWindow(), QtGui.QMainWindow()) with
Code: [Select]
main_win = util.init_ui(Ui_MainWindow(), MainWindow()).
The end result is the same but the latter solution is cleaner.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on February 12, 2014, 11:57:17 am
It works perfectly, now. Though, I had to access the "unity_launcher" global var from inside the new class. Is it considered ok, or is there any cleaner way to do it ? (it should not cause any issue we would never use unity_launcher without initialising main_win first)

I prefer to wait before adding notifications, since better notifications (2 cases : mod install successful or failed) will depend on the changes you plan to make to the tasks (cancelling task from progress_win, timeout on downloads, a good way to know if a mod install ha succeeded or failed...).

EDIT : Found a bug in the sources reordering code : when you reorder things, sources duplicate.

EDIT 2 : Also, I found a problem with dependencies handling (at least with fs2mod) : if you mark if you mark a (not installed)  mod dependecy for install and its parent mod (installed) for uninstall, the installer won't complain.

EDIT 3 : I'm not sure what exactly is happening, it could also be an issue caused by broken re-ordering, under some conditions, mods dependent on others are marked as broken because there is a source with higher priority than the one of their parent.

A mockup of a new mod browsing site relying on fso:// links (the cart is for redirecting to Good Old Games):
(http://pix.toile-libre.org/upload/original/1392259690.png)

Bonus video : https://www.youtube.com/watch?feature=player_detailpage&v=LpzkGHFkEpo
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on February 13, 2014, 06:24:00 pm
Most likely I'll have to re-implement the re-ordering mechanism because I haven't found a way to reliably read the new order of the sources after the user finishes the drag & drop.
Right now I'm not forcing any dependencies on the user because most of those are automatically detected. In that case that this auto-detection generates wrong results I want to allow the user to correct the program.
Maybe I'll add a new option on the Settings tab called "Enforce dependencies" which will be on by default.

The broken re-ordering shouldn't affect the dependency tracking. A mod is only marked as broken if one of the files has a different checksum than listed in the mod source. Can you post the the check messages in the notes area of the broken mod, please?

I just watched your video... it looks nice! :D
What do you think about adding some kind of splash screen to the mod manager while it's starting FS2? At first you see the "Working..." window but after that you don't know if the manager crashed or if it just disappeared because it's starting FS2... Some kind of window that says "Starting FS2..." (or whatever else fits) would be nice.
I also want to change the automatic list update: If the manager finds the mod and it's installed it can simply launch FS2 otherwise it could update the list and ask the user if he wants to install the mod. What do you think?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on February 14, 2014, 04:34:49 am
So, I really don't know what's happening. I'm missing something in this bug pattern.
Somehow, one time it worked : the repository had a low priority, Silent Threat Reborn (fs2mod) had a higher priority. As fsport and fsport-mediavps are dependencies for Silent Threat Reborn, fsport and fsport mediavps were tagged as higher priority than the repo, and marked both as installed (no warnings, fs2mod version number).
Then I removed both the repo and STR fs2mod, re-added them (first, the repo, then STR fs2mod). Now only fsport and fsport-mediavps say they are  installed (!) and Silent threat reborn says it's missing files. Also, STR only shows in the FreeSpace Port tree, and doesn't show a second time as an independent mod in the global list. fsport-mediavps shows as the fs2mod version "3.4" in the global list, but as in the repo "version 3.4" in the fsport tree.

And of course here's the (predictable: files from the fsmod are checked against the repo reference) error message
Code: [Select]
Check messages:
* File "ToggleMediaVPs.bat" is missing.
* File "mod.ini.without-mediavps" is missing.

Dependencies:
* FSPort Media VPs (installed)
* FSU MediaVPs, version 3.6.12 (installed)
* FreeSpace Port (installed)

Contents:
* fsport-str/ToggleMediaVPs.bat
* fsport-str/fsport-str.bmp
* fsport-str/fsport-str.vp
* fsport-str/mod.ini
* fsport-str/mod.ini.without-mediavps
* fsport-str/readme.txt

A splash screen is a good idea, and not auto-updating a mod in a working state if driectly launching it was requested is in my opinion a good behaviour ! (if it's not broken, don't fix it !).

Bug : I can add the same fs2mod file more than one time.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on February 14, 2014, 02:39:51 pm
I've fixed the mod source order now, could you check if this resolves your bug?
I also added a splash screen. It's simple but I think it's enough. I also added an error message in case the user tries to add the same fs2mod file twice. It only works if you try to add the same file twice. If you copy a fs2mod file and add the copy the script won't complain (yet).
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on February 14, 2014, 03:10:20 pm
Currently it doesn't run :
Code: [Select]
Traceback (most recent call last):
  File "./manager.py", line 42, in <module>
    from ui.splash import Ui_MainWindow as Ui_Splash
ImportError: No module named splash
(looks like a typo)
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on February 14, 2014, 03:13:03 pm
Forgot to add a file to git (ui/splash.py). Should be uploaded now.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on February 14, 2014, 03:36:32 pm
Bug fixed ! One strange thing though : is it intended behaviour that optional VPs from lower priority sources are appended to higher priority sources in the mod list ? (for example Silent Threat Reborn Voice pack (from the main repo) is appended to Silent Threat Reborn (from the fs2mod file) in the main list. And STR doesn't appear any more in the FreeSpace Port Tree) This is something rather interesting. I'm not sure, but it may even be useful in some rare cases. Or break things.
If optional VPs are implemented in fs2mod files, maybe it would be better not to mix optional files from 2 different sources.

You should add a hint that mods can be reordered, and where is the higher priority (bottom). Actually, it's the contrary of most pieces of UI I know (top priority on top, for example languages in Gnome language properties, keyboard maps in Gnome control center...).

The "Cancel" button doesn't seem to do anything at the moment. I guess it's only added to the UI and not connected to any callback yet ?
EDIT : I just checked the code, it is connected to a function which is supposed to abort tasks. So, this is a bug.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on February 14, 2014, 03:59:53 pm
Actually it does something: It calls the "abort" method on all currently running tasks but as noted in progress.by (where this method is implemented):
Code: [Select]
# Empty the work queue, this won't stop running workers but it will
# stop calls to the work() method.
It won't stop the current work but it won't attempt to process any queued work (i.e. if it's still downloading files, it won't abort running downloads but it won't attempt to start a new download).
Also "self.aborted" is set to True which can be handled in the "finish()" method.
I'll be able to add support for aborting running downloads and innoextract but I'm not sure if I'll be able to interrupt 7z when it's extracting files.

You're right about the priority order, I'll change it.
The problem with the mod tree currently is that every mod has a given parent and that parent is identified by its name. Now the STR Voice Pack has STR listed as its parent. The STR listed isn't the STR from the STR Voice Pack source but the manager doesn't detect this edge case currently. I guess I'll have to remove all optional parts of a mod (aka all children of that option) from the mod list when a package is overriden by another package from another source.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on February 14, 2014, 05:14:12 pm
In the current code, where is the best place to know the final state of the tasks ? I didn't see a clear test to check if something has failed. (from a UI point of view, when the "Working" window closes -> this question has to do with freedesktop.org linux notifications/Unity notifications)

[About the overridden packages : overriding all optional parts of a mod (said mod should still show optional parts coming from the same source as proposed version) should be done only if the mod is from an explicit source (a source that appears in the sources list).
For mods coming from implicit sources (sources not appearing in the sources list because they have been set by the dependencies system) they should still show up. ]*

Imagine you install STR from a fs2mod file, which will have higher priority than STR from the main repo :
-> FreeSpace Port will be updated from the fsport fs2mod file which url is specified in STR fs2mod. That's ok.
-> If all fsport children (main repo version) are replaced by fsport children (fs2mod version), and if Awakenings fs2mod file hasn't been installed... well, we can't install Awakenings now : it's not any more in the list.

*We can reasonably assume if some users only installed STR fs2mod, and *not* fsport fs2mod, they may still actually want to install a fsport child from the main repo. Which, by chance, could happen to be compatible with his fs2mod version new parent.

*If they installed both STR and fsport fs2mod files (or at least fsport fs2mod has higher priority), there are chances they don't want to use anything from the main repo. So we don't have to worry about not showing, for example, Awakenings.

Maybe(or maybe not)  all this cross-sources dependencies and optional files thing should be part of the "enforce dependencies" setting :
- if it's "on" we disallow cross-sources dependencies (aka a mod can never "adopt" optional parts from another source, and by the way, we make sure that no broken mod will be installed)
- if it's "off" we allow cross-sources dependencies (and broken mods could be installed, if their parent is from a version which is to different to allow them to work properly)
As it's on by default, it's safer. We could name this setting "Safe mode" (with a tooltip explaining that without this option, mods that may miss dependencies, or broken by other (updated or older) mod parents may be installed).

Any thoughts on the issue ?
*EDIT : I just noticed I actually proposed 2 different default behaviours in what i just wrote. You can ignore the paragraph between brackets.

EDIT :
New screenshot (no slider yet in this one, but i'm working on it !)
New video, refined html and css, with an HLP nav bar.

EDIT : second screenshot, daily update on the web slideshow.

Another edit : I have attached a ui file for a simple settings window ( https://www.dropbox.com/s/kk15ermb0j8g4bo/settings.ui ).  I'm working on adding support for fso://settings/mod links in protocol.
The right behaviour would be (assuming we will have a unique settings profile per mod):
-> if a fso://run/mod link is accessed AND there is no existing settings profile for this mod, the launcher duplicates the default profile and open a settings window.
-> if a fso://run/mod link is accessed AND there is an existing settings profile for this mod, we launch the game directly.
-> if a fso://settings/mod link is accessed : we launch the settings window for the mod (this is what the "settings" icon next to each "Play" button on the web page is for)
-> if the fso://settings link is accessed : we launch the settings window for the default profile
-> if a fso://settings/<invalid string> : we display a warning about the mod not being found, BUT we launch the settings window for the default profile.

And of course, this window (either default settings or mod profile settings) should be available from somewhere in the main window.

I'm still not sure if tabled settings should be common to all mods or set per profile. The command line should be set per profile, since different mods may require different features or lighting settings. We should save the string stored in cmdline.cfg in the installer settings, for each mod, and the last command line, to restore once the game is launched.

I can start implementing it myself, but I will need some help understanding how to call the same settings window both as a main window (if accessed from a link)
and a modal window (if the ful mod manager is launched).

[attachment deleted by an evil time traveler]
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on February 17, 2014, 07:59:17 am
Sorry for the late reply!  :nervous:

Most tasks have a "finish" method which is called once the task completes. I guess that would be the best place to check if it suceeded or failed. You can also check if the user aborted the task by looking at self.aborted (read the FetchTask's finish method for an example).

On the issue of cross-source dependencies: Your ideas are nice but I'm seeing a problem here. If the user installs STR using the fs2mod pacakge and we disallow cross-source dependencies, it will try to install the fsport fs2mod pacakge to fulfill its dependency. That, in turn, might break already installed fsport mods because the fsport fs2mod package might install different vp files. If we allow cross-source dependencies the STR fs2mod package might break because it isn't compatible with the installed fsport mod.

I propose a different solution:

Let's assume the most recent FSPort version is 1 and both the HLP package and the fs2mod package contain this version. The user installs Awakenings and the mod manager will install HLP's fsport to fulfill its dependency. Afterwards the user installs STR using your fs2mod file. Now the mod manager will see the fsport dependency but since the checksums for all vp files match up it will assume that the fsport dependency is fulfilled and simply install STR.

Now, the FSPort project releases a new version (2) and you update your fs2mod file. If the user installs Awakenings, the mod manager will of course use the HLP package (version 1) to fulfill the dependency. If the user installs STR afterwards, the mod manager will detect the conflict between the HLP package and the fs2mod package. To resolve the conflict, it installs the new FSPort (version 2) into fsport_2_fs2mod (or whatever makes more sense) and change STR's mod.ini to load fsport from fsport_2_fs2mod.

What do you think? This should avoid any possible conflicts and still allow the user to install mods from different sources even if their dependencies aren't compatible.

I think it only makes sense to make the command line flags mod-specific. You could also make the fs2 build mod-specific, if someone does the same thing BP did but most of the time mods should work with the most recent fs2_open release.
The wxLauncher even lets mod authors specify cmdline flags in their mod.ini files (https://code.google.com/p/wxlauncher/wiki/ModIniOptions#secondarylist).

To use your settings window in the mod manager you need to place the .ui file in the ui folder and run "compile_ui.sh". It generates a .py file which you can then import using
Code: [Select]
from ui.settings import Ui_Dialog as Ui_Settings
To open the window in scheme_handler do
Code: [Select]
settings_win = util.init_ui(Ui_Settings(), QtGui.QDialog(splash))
settings_win.setModal(True)
settings_win.destroyed.connect(app.quit)
settings_win.show()
otherwise use
Code: [Select]
settings_win = util.init_ui(Ui_Settings(), QtGui.QDialog(main_win))
settings_win.setModal(True)
settings_win.show()

I added the extra line which connects the destroyed signal to app.quit() so that fs2mod actually quits once the user closes the settings window.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Fury on February 17, 2014, 12:51:05 pm
What do you think?
I think you're awesome!
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on February 17, 2014, 04:24:59 pm
@ngld :
This is a very interesting way to deal with dependencies. I'm seeing some issues, though : let's imagine fsmod-py (including its config files) is completely uninstalled from the user's computer. We have some modified mods (duplicated parent mods because children were depending on different versions of these). Then the user re-installs fs2mod-py : when it searches for installed mods, what happens ?

If two or more versions of a mod with the same name are installed (but modified mod folder name and mod.ini), what happens with fso:// links ?

This solution also assumes different checksums for parent mod VPs mean children will not be compatible, but some children mods are actually compatible with different versions of their parent mod. How do we deal with that case ? (this is only a disk space issue, I guess we don't care that much, and if we want something as reliable as possible, we should only do what the source tells us to do...)

How does it work with updates ? The previous (planned) behaviour was that adding a new source or updating a source (as long as the mods share the the same name) would enable a mod update. I guess that would be no longer the case. Here, different versions of a mod would end up installed in parallel, nothing would be updated.
Maybe real updates (instead of parallel install) should be trigger only if the mod has no current children that cannot be update from any of the current sources.

Also, it seems to make the priority mechanism useless.

Am I getting this right ?
This is getting pretty complex. (I guess that's why big package managers out there prefer to mark packages as broken instead of rewriting paths and installing deps in parallel)


I'm currently trying to understand how wxlauncher autodetects some settings like the available screen resolutions, controllers and sound devices, and how to fill our settings menu with the presets. The settings profiles part should be straightforward.
EDIT : I think this is above my level in python for now. It needs SDL and openAl dependencies. I'm reading stuff about it, but don't expect useful code from me right now. It's around lines 1421 in /code/tabs/BasicSettingsPage.cpp in wxLauncher.

EDIT 2: what do you think about pygame ? It looks like it makes using SDL much simpler :
Code: [Select]
import pygame.display
pygame.display.init()
pygame.display.list_modes(32)
returns
Code: [Select]
[(1366, 768), (1360, 768), (1024, 768), (800, 600), (640, 480)]
It seems that wxlauncher code for getting the same piece of information is much more complex.
Do you think we can add a dependency on pygame ?

EDIT3: apparently pygame is outdated, and using pySDL2 would be better. It can detect joysticks, screen modes and sound cards. Pretty much all we need for the settings screen.

EDIT: pySDL2 doesn't seem to be in ubuntu repositories yet, so I'm experimenting with pygame.
Also, neither pygame or pySDL are able to list sound devices. But Qt does : http://qt-project.org/doc/qt-4.8/qaudiodeviceinfo.html
After trying this, i still can't list sound devices. Phonon example code returns me an empty list of sound cards (https://github.com/PySide/Examples/blob/master/examples/phonon/capabilities.py). I can't find a way to install QtMultimedia for PySide and Qt4. It's nowhere to be found in ubuntu repositories.

Anyway, I have attached my daily screenshot of the html template (not a lot of changes, except it's not broken any more in webkit based browsers, and that it should look better on wide screens).

EDIT (again) : I finally got fs2mod-py to parse fs2_open.ini, currently only the graphics part. :D
The rest will be easier. I'll need your help to integrate it with the rest of the code !

[attachment deleted by an evil time traveler]
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on February 20, 2014, 11:43:29 pm
@ngld:
You can take a look at my last commit here : https://github.com/Nepenthes/fs2mod-py/commit/6d58d7fff2e7591efdbc4b07895eefe328054935
(it's not yet windows compatible... Someone will have to deal with the registry, and I have no windows pc at home)
(also, i forgot to include it in the commit message, but there's a new Unity quicklist item opening HLP in the default browser. Later, it will be the mod catalogue)

EDIT : found a bug : if 2 or more instances of fs2mod-py are launched at the same time, the settings are wiped...
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on March 02, 2014, 07:18:28 am
I think you're awesome!
Thanks!  :D

@Hellzed: I've commented on the pull request now, I made several changes to your new code (which should also fix the bug you found).
Pygame isn't really outdated. They haven't done an official release since 2009 but if you look at their page over on BitBucket (https://bitbucket.org/pygame/pygame/downloads) you'll see that it's still being developed. It doesn't support SDL2 (yet?) but that shouldn't matter in our case because FS2 still uses SDL-1.2.
Since we only need a few SDL functions a complete binding package like pygame, pyglet or PySDL would be overkill. Instead, I added some wrappers (in clibs.py) which use ctypes (http://docs.python.org/3/library/ctypes.html) to call SDL and OpenAL.

I'll implement the package management once I've got enough time, I already solved most issues but I'm not sure if it'll work.

P.S.: Sorry for the long delay, RL interfered.   :blah:
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on March 02, 2014, 07:25:24 am
It doesn't support SDL2 (yet?) but that shouldn't matter in our case because FS2 still uses SDL-1.2.
Usage of SDL2 is currently being tested and should arrive in trunk soonTM so you should consider using something like PySDL2 (https://pypi.python.org/pypi/PySDL2/0.2.0) if you want to support the newest versions of FSO.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on March 02, 2014, 07:19:50 pm
m!m, thanks for the notice! I've added support for SDL2 to the ctypes wrapper.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on March 02, 2014, 08:11:27 pm
@ngld :
Everything looks good to me. The settings window works. How do you think it should integrate with the fs2mod-py main window ?
I'll look into installing python on Wine, and start working on the registry part.
Next goal is to save a different cmdline_fso.cfg for each mod.
I was also thinking about setting the LastPlayer (in fs2_open.ini) value for each mod.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on March 02, 2014, 08:56:06 pm
I guess it would make the most sense to add a "settings" button to the mod info window which opens the settings window.
The QSettings class can modify the registry. You should only need to change the line where the QSettings object is created (http://qt-project.org/doc/qt-4.8/qsettings.html#accessing-the-windows-registry-directly).
Setting the LastPlayer is a great idea but how do you determine the last player? You'll have to wait until FS2 quits to read that value, don't you?

As for python on wine... You might want to look at win_build/build.sh. It can set up a wineprefix with python and all dependencies.
Code: [Select]
cd win_build
./build.sh
WINEPREFIX="$PWD/_w" wine python ../manager.py

build.sh will generate a windows build (the exe file) but will also create the _w directory which should contain everything you need to run fs2mod-py on wine.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on March 03, 2014, 04:02:44 am
I'm getting an error with build.sh :
Code: [Select]
Building...
wine: cannot find L"C:\\windows\\system32\\python.exe"

And yes, I think we have to wait for FS2 to quit to read the value. (signals ?)
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on March 03, 2014, 01:39:25 pm
Did you change the installation destination of Python? The script expects it to be in C:\Python27.
If you want to try again, you have to remove the _w directory and re-run the build.sh script.

We'll have to create a new thread if we want to check when FS2 quits. I'll add a new signal (fs2_quit ?) and implement it.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on March 03, 2014, 04:25:40 pm
I didn't change any default path. I'm using wine 1.7.13.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on March 03, 2014, 05:23:31 pm
Weird, I'm using the same version.
Could you please run "ll _w/drive_c/windows/python.exe" in win_build?
If the symlink is missing (you get a "File or directory not found" error), you have an old _w directory. Deleting it and running build.sh again should fix that.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on March 03, 2014, 05:35:26 pm
That's the weird part : I just re-created _w, and I'm still getting the error.
However, I have a python.exe symlink in win_build.
If I manually move python.exe to _w/drive_c, it starts building, but I get an error with 7zip : "IOError: [Errno 2] No such file or directory: '7z.exe'"
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on March 03, 2014, 06:54:07 pm
You're using Ubuntu, right? You probably need to run "bash ./build.sh".
Dash apparently doesn't support the pushd and pulld commands...

Because of the wine error build.sh didn't complete and thus didn't extract 7z. If you manually place 7z.exe (and 7z.dll) as well as SDL.dll and openal.dll in the win_build directory that error should go away.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on March 03, 2014, 07:13:52 pm
It worked with bash.

EDIT : I'm trying to run the settings window in wine, with a change to write to the registry, but I'm getting this error :
Code: [Select]
fixme:msvcrt:type_info_name_internal_method type_info_node parameter ignored
p11-kit: couldn't load module: /usr/lib/i386-linux-gnu/pkcs11/p11-kit-trust.so: /usr/lib/i386-linux-gnu/pkcs11/p11-kit-trust.so: Ne peut ouvrir le fichier d'objet partagé: Aucun fichier ou dossier de ce type
p11-kit: couldn't load module: /usr/lib/i386-linux-gnu/pkcs11/gnome-keyring-pkcs11.so: /usr/lib/i386-linux-gnu/pkcs11/gnome-keyring-pkcs11.so: Ne peut ouvrir le fichier d'objet partagé: Aucun fichier ou dossier de ce type
Traceback (most recent call last):
  File "<string>", line 45, in <module>
  File "Z:\home\loris\fs2mod-py\win_build\PyInstaller-2.1\PyInstaller\loader\pyi_importers.py", line 270, in load_module
    exec(bytecode, module.__dict__)
  File "Z:\home\loris\fs2mod-py\win_build\build\fs2mod-py\out00-PYZ.pyz\windows", line 20, in <module>
  File "Z:\home\loris\fs2mod-py\win_build\PyInstaller-2.1\PyInstaller\loader\pyi_importers.py", line 270, in load_module
    exec(bytecode, module.__dict__)
  File "Z:\home\loris\fs2mod-py\win_build\build\fs2mod-py\out00-PYZ.pyz\clibs", line 155, in <module>
  File "Z:\home\loris\fs2mod-py\win_build\build\fs2mod-py\out00-PYZ.pyz\clibs", line 52, in load_lib
Exception: libopenal.so.1.15.1 could not be found!
fixme:msvcrt:__clean_type_info_names_internal (0x130f4d0) stub
fixme:msvcrt:__clean_type_info_names_internal (0x6578cca4) stub
fixme:msvcrt:__clean_type_info_names_internal (0xd24460) stub
fixme:msvcrt:__clean_type_info_names_internal (0x3bba30) stub
fixme:msvcrt:__clean_type_info_names_internal (0xc354f0) stub
fixme:msvcrt:__clean_type_info_names_internal (0x10198e30) stub
fixme:msvcrt:__clean_type_info_names_internal (0x36a948) stub
fixme:msvcrt:__clean_type_info_names_internal (0x39ab68) stub
fixme:msvcrt:__clean_type_info_names_internal (0x6726023c) stub
fixme:msvcrt:__clean_type_info_names_internal (0x1d1b5210) stub
fixme:msvcrt:__clean_type_info_names_internal (0x1e24d188) stub

I'm not sure why a windows build of fs2mod py would be searching for "libopenal.so.1.15.1" ?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on March 04, 2014, 09:47:50 am
It's trying various names (libopenal.so.1.15.1, openal.dll and whatever gcc -lopenal would use). As long as you have a SDL.dll and openal.dll in _w/drive_c/windows or in your current directory (build.sh should have put them in win_build already).
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on March 04, 2014, 10:31:42 am
So, I tried to set the different registry keys. It's not as simple as we thought, because there is no equivalent for "Default" section of the config file in the registry. These values instead go at the root of "FreeSpace2" key. And an other issue is that some values in the registry are typed : some values (including some numbers) are typed as strings, and some others as binary data.

EDIT : I have fixed most of the issues.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on March 05, 2014, 06:06:30 pm
Merged your changes.
Please use "git rebase" (http://git-scm.com/docs/git-rebase) to rebase and not GitHub's pull request. This avoids all the "Merge ... rebase" commits.

I've put a "FS2 Settings" button on the FS2 tab since none of the settings are actually mod-specific.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on March 05, 2014, 06:55:01 pm
I'll do it for next commits !

That's true, no setting is mod specific at the moment. The only thing that will be mod specific is the command line stored in cmdline_fso.cfg. I'll make a few changes to store it in fs2mod-py data, an restore it to the dedicated file on game launch (and if no command line exists yet for a mod, it will be created from a "default" command line).
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on March 06, 2014, 04:24:41 pm
Will you provide checkboxes for the most common options (like wxLauncher, YAL, ...) or will you simply show a text input?
You should probably run FS2 with "-get_flags" to generate the "flags.lch" file and read that to get a list of all available options.
I can implement that if you want...
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on March 06, 2014, 05:35:35 pm
I haven't started to work on it on my side, so feel free to implement it. At first, I was aiming for a simple text input, but if we can parse "flags.lch", that's great. An improvement on what wxLauncher currently does would be sliders for lighting settings.

I'm currently checking which settings fs2_open supports on each platform, so we can avoid showing settings that will not be used by the engine (I'm thinking about antialiasing on Win*). It will only be a set of minor changes.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on March 07, 2014, 08:42:46 pm
I've added a settings window to edit the flags. (You can reach it by clicking "Settings" in a Mod Info window or by clicking "FS2 Settings" > "Default command line flags".) Since FS2 generates the flag list itself, it should only show relevant options.
I also added you to the copyright statement (https://github.com/ngld/fs2mod-py/commit/a6eab41a2eded53d0a4d27f46ab82fad9ff44fb7), is that alright? (I should have thought of that earlier...)
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on March 08, 2014, 04:56:04 am
Yes, that's fine !
(I checked the current branches, and develop is missing some of the ui files.)
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on March 08, 2014, 06:27:19 am
Which ones? I'm seeing none missing...  :confused:
(master doesn't have all of them, of course, but that's because master is still the old version I first released. I'll update master as soon as develop is stable and the new package management is merged.)

BTW, would be nice if this becomes a standard (option in mod.ini ?) (http://www.hard-light.net/forums/index.php?topic=86723.0). I guess that's what your comment (https://github.com/ngld/fs2mod-py/blob/7501cf6401f1a3df7a7aea48572d90066846f2f5/windows.py#L105) in the settings window code was about, wasn't it?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on March 08, 2014, 06:39:30 am
Sorry, I was wrong, I inadvertently put the content of the master branch into the develop directory.

And yes, my comment was about this specific issue. (the settings window currently only reads available builds and lists them, but it doesn't use the user input)
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on August 08, 2014, 03:02:37 pm
Sorry for the long slience...  :nervous:

AFAIK the current TODO list would be:

Did I miss anything?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on August 09, 2014, 08:24:23 am
No worries !

That's pretty much everything for the core features.
I'll also need to write a real working fs2_open version selector (the one I made is only a stub) since a unified build working with all mods isn't out yet. And I'll have to get ready for fs2_open Windows builds switching to .cfg files (but i'll keep the registry settings for retail compatibility). Ubuntu/Unity integration may have to be refreshed for newer distros.

Now, I have thoughts about the UI, assuming the existing UI is considered the "advanced UI". All I need will be a Qt/WebKit view, a way to parse very simple HTML templates, and use callbacks similar to the fso handler actions (play/install/uninstall mod).

I guess some files in my dropbox mod repository are outdated, i'll updated them next week.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on August 25, 2014, 07:14:49 am
@ngld : I see you have made some changes to fs2mod-py-packager in the last few days, to support Goober's installer new text files... Does it mean you're not using a json mirror any more (last time I checked, http://dev.tproxy.de/fs2/all.json was down) ?

By the way, now I see 4 big shortcomings with the text files :

- some mods are considered "parts" of other mods. Depending on the UI, for example, end users would have to know Silent Threat Reborn or Awakenings are parts of fsport to find them. To me, it's only relevant to the installer, not the user. To the user, all mods should be equal and sorted by relevant criteria (universe, before FS2 or after FS2 main story, author...)
Currently it's hard to tell if something is an optional vp or a mod of its own.

- the update mechanism is not that efficient. Look at the last version of fsport checking for ALL the old versions to get rid of their files : http://fsport.hard-light.net/installer/fsport.txt
I prefer the idea of having an "image" of a working mod directory with all checksums, and a script to keep/add/update/delete what should be in order for the mod to reach a playable state.

- overrides (an optional part of the mod changing one of the files, like fsport-mediavps changing fsport mod.ini to enable hd graphics) are currently possible with the RENAME token, but not automatically reverted when you disable the optional part of said mod.

- a mod cannot be associated with a specific build of fs2_open by default. Even if many mods use the default builds in a near future, I think it's still a useful feature.

Finally, it also misses a lot of "semantic" tokens like "AUTHORS", "MIN SPECS" etc... that are not used by the installer, but would help create better mod pages.
(adding this here : for optional files, we should also have a "recommended" flag)

Any thoughts ?

With that in mind, I'm writing a Symfony bundle that would handle storage of mod informations into a database, with user friendly forms.
My first goal is to replicate Goober's installer text files as a demo. Once it's working, I'll add parsing options (maybe text files with a few more/different tokens, or json).
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on August 25, 2014, 04:25:15 pm
I'm rewriting the repo backend (which was the point of the packager branch). I deleted the all.json file along with some others because it was outdated.
I've written a schema (https://github.com/ngld/fs2mod-py/blob/packager/converter/schema.txt) for my new json format. The "requirements" will be a list of tokens (i.e. "os=windows" for the FSO Windows builds or "fso>=3.6.*" for a mod which requires FSO to be 3.6.x or newer.) Let me know if you have any ideas to improve it.

Right now you can run "python converter.py import repo.json" to convert Goober's installer text files to my json format (this example is updated every hour (https://dev.tproxy.de/fs2/mirrors/fs2mod-py/files/fsoi.json)).
"python converter.py checksums repo.json repo.gen.json" creates checksums for all mod archives and their contents (pretty much like the "image" you described).
A big improvement over the old converter is that it only downloads changed archives which saves a lot of bandwith.

fs2mod-py is supposed to be the script you mentioned. :) The biggest problem are user changes. fs2mod-py will effectively undo all changes made by the user whenever he/she updates a mod. I guess I'll create a "user-changes" subfolder which will be ignored by fs2mod-py and passed to FSO using the -mod flag. If you want to add a new file or change an exisiting file, you'll just put the changed / new file in the "user-changes" folder.

A mod isn't supposed to change another mod's files. "Overrides" (the way you described them) aren't possible right now. After installing fsport-mediavps, you have to select it as the active mod instead of fsport to get HD graphics. FSPort's mod.ini isn't changed in any way.

Now that I've rewritten the server-side backend, I just need to finish implementing the new format in fs2mod-py. I'll make a new release once it actually works again.
If it helps your fsmoddb project, I can implement a "python converter.py export repo.json repo.txt" command which generates Goober's text files from my json schema...
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on August 25, 2014, 07:15:15 pm
Quote
fs2mod-py is supposed to be the script you mentioned. :) The biggest problem are user changes. fs2mod-py will effectively undo all changes made by the user whenever he/she updates a mod. I guess I'll create a "user-changes" subfolder which will be ignored by fs2mod-py and passed to FSO using the -mod flag. If you want to add a new file or change an exisiting file, you'll just put the changed / new file in the "user-changes" folder.
The good old bash script used to do this exactly, with a clear warning that all user made changes would be deleted during mod "cleaning". Steam does the same, without the warning !
The logic was in the bash script was:
- download the files list and checksums,
- if the file doesn't match or is missing, check the cached archive,
- if the archive doesn't match or is missing, download the new one from repository.
- Then warn the user that the mod will be cleaned, and if the user confirms, every file in the mod folder that's not on the new list is deleted.
It saves a lot of bandwidth indeed...

Quote
A mod isn't supposed to change another mod's files. "Overrides" (the way you described them) aren't possible right now. After installing fsport-mediavps, you have to select it as the active mod instead of fsport to get HD graphics. FSPort's mod.ini isn't changed in any way.
Sorry I explained poorly, and current fsport was a bad example, but i'm absolutely sure i have seen these kind of hacks in the past, when a mod ships with 2 versions of the same file, and a script to rename/activate the right one.
I thought about overrides because an optional file for a mod should be able to specify it own "requirements" tokens. Imagine a mod that runs on default builds, with optional extras that need an experimental build (and of course one of the requirements token should be "branch").

And I think the new json format is a very good start :)
(I may send you a proposal tomorrow about a small change, but I'm still thinking about.)
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on August 26, 2014, 05:58:24 am
@ngld: The JSON format you propose looks very promising but I noticed some things that might be worth changing:

Code: (files example) [Select]
{
  "files": [
    {
      "isArchive": true,
      "destination": "/",
      "urls": [
        "http://swc.fs2downloads.com/builds/WIN/fs2_open_3.7.2_RC3.zip",
        "http://scp.fsmods.net/builds/WIN/fs2_open_3.7.2_RC3.zip",
        "http://scp.indiegames.us/builds/WIN/fs2_open_3.7.2_RC3.zip"
      ]
    }
  ]
}

Code: (actions example) [Select]
{
  "actions": [
    {
      "type": "delete",
      "files": [
        "test.vp",
        "foo.vp"
      ]
    }
  ]
}

Code: (dependencies example) [Select]
{
  "dependencies": [
    {
      "id": "<ID>",
      "version": "1.2.*"
    },
    {
      "id": "<ID>",
      "version": ">=1.0"
    }
  ]
}
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on August 26, 2014, 06:53:42 am
@ngld :
I think I found something, and I don't know how to understand it :
 
Code: [Select]
"FSOI#FreeSpace Port.Command Briefing Animations.1024x768 Command Briefing Animations": {
     "files": [
            [
                [
                    "http://fsport.freespacemods.net/releases/"
                ],
                {
                    "tango_hi4_fs1.zip": {
                        "dest": "fsport",
                        "is_archive": true
                    },
                    "tango_hi5_fs1.zip": {
                        "dest": "fsport",
                        "is_archive": true
                    },
                    "tango_hi2_fs1.zip": {
                        "dest": "fsport",
                        "is_archive": true
                    },
                    "tango_hi3_fs1.zip": {
                        "dest": "fsport",
                        "is_archive": true
                    },
                    "tango_hi1_fs1.zip": {
                        "dest": "fsport",
                        "is_archive": true
                    }
                }
            ]
        ],
        "description": "",
        "dependencies": [],
        "logo": null,
        "submods": [],
        "requirements": [],
        "title": "1024x768 Command Briefing Animations",
        "notes": "",
        "source": "#FSOI",
        "version": "Version 1.2",
        "delete": []
    }

This submod of "FSOI#FreeSpace Port.Command Briefing Animations.1024x768 Command Briefing Animations" doesn't list any dependencies, so how does fs2mod-py know that it has to install parents mods first ? Does it read the mod <ID> and use the points "." as separators to build the dependency tree ?

In the end (if the answer to the previous question is yes), what is the difference between setting the "dependencies" and "submods" values in parent/children mod objects, and mapping dependencies from mod IDs ? (I must be missing something because it looks a bit redundant to me)

Also, how do you plan to differentiate "user facing" submods (the ones that are presented directly to moddb users) like Silent Threat: Reborn and Cardinal Spear from "technical" submods like MV_Advanced ? I don't think Goober's installer text files provide this piece of data, but the new json format should provide it.


Here's something I dashed, for native mods, but I don't think it would play well enough with Goober's text files logic to be a solution :
(and with this syntax, a "package" cannot have dependencies like other "packages" in the same mod or a "package" inside another mod, this could be an issue.
This syntax would also help me to create a user friendly "mod builder" in Symfony.)
Code: [Select]
/* Repo configuration */
// Most of this is similar to Goober's text files (http://www.hard-light.net/wiki/index.php/FSO_Installer_Text_Files)

{
    "<ID>": {
        "title": "...",
        "version": "...",
        "description": "...",
        "features": ["<FEATURE1>", "..."], //Technical details (added functionnality, models, ships) that don't fit in the description
        "author": "...",
        "tags": ["<TAG1>", "..."], //Tags allow to quickly search for a mod in the moddb
        "logo": "<will be extracted from the archives if null>",
        "notes": "...",
        "submods": ["<ID1>", "..."],
        "dependencies": ["<ID1>", "<ID2>", "..."],
        "source": "<only when imported otherwise null>",
        /* Packages would replace "technical" submods that should not appear on their own in the moddb.*/
        "packages": [
            {
                "name": "...",
                "notes": "...",
                /*A package can be :
                - "required" (always installed with the mod, in fact these are the base files of the mod),
                - "recommended" (automatically selected for installation, but the user can skip them),
                - "optional" (not automatically selected, but user can add them during the install process) */
                "status": "<required|recommended|optional>",
                "requirements": ["<req>"] //A package should be able to set different requirements that override the mod requirements (example : a high res/shader package that requires an experimental build
                "files": [
                    [
                        [
                            "<link1>",
                            "..."
                        ],
                        {
                            "<filename>": {
                                "is_archive": true,
                                "dest": "<destination path>"
                            }
                        }
                    ]
                ]
            }
        ],
        "delete": ["<file1>", "..."]
        "requirements": ["<req>"] // TODO
    },
    "#include": [
        "<repo file 1>",
        "..."
    ]
}

There is a more conservative version of this, and it may play better with the text files conversion :
(this version is more limited in terms of things we can recommend on install)
Code: [Select]
/* Repo configuration */
// Most of this is similar to Goober's text files (http://www.hard-light.net/wiki/index.php/FSO_Installer_Text_Files)

{
    "<ID>": {
        "type": "..." //Something like "MOD" for user facing mods, "ADVANCED" for technical packages, "BUILD" for fs2_open builds.
        "title": "...",
        "version": "...",
        "description": "...",
        "features": ["<FEATURE1>", "..."], //Technical details (added functionnality, models, ships) that don't fit in the description
        "author": "...",
        "tags": ["<TAG1>", "..."], //Tags allow to quickly search for a mod in the moddb
        "logo": "<will be extracted from the archives if null>",
        "notes": "...",
        "submods": ["<ID1>", "..."],
        "dependencies": ["<ID1>", "<ID2>", "..."],
        "source": "<only when imported otherwise null>",
        "files": [
            [
                [
                    "<link1>",
                    "..."
                ],
                {
                    "<filename>": {
                        "is_archive": true,
                        "dest": "<destination path>"
                    }
                }
            ]
        ],
        "delete": ["<file1>", "..."]
        "requirements": ["<req>"] // TODO
    },
    "#include": [
        "<repo file 1>",
        "..."
    ]
}
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on August 26, 2014, 08:50:44 am
@m!m:
About using an array as an external wrapper for mods :  I don't know what ngld's original reason was, but in my php framework documentation they recommend using objects instead of arrays as external wrapper for a json response, in order to avoid some possible XSSI cases.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on August 26, 2014, 08:57:45 am
I didn't mean the external wrapper but the definition of the mods. I would propose something like the following:
Code: (mod definition) [Select]
{
  "mods": [
      {
        "id": "foo",
        ...
      }
    }
  ],
  "#include": [
    "<repo file 1>",
    "..."
  ]
}
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on August 26, 2014, 09:31:22 am
@m!m
Code: [Select]
{
  "files": [
    {
      "isArchive": true,
      "destination": "/",
      "urls": [
        "http://swc.fs2downloads.com/builds/WIN/a.zip",
        "http://scp.fsmods.net/builds/WIN/a.zip",
        "http://scp.indiegames.us/builds/WIN/a.zip"
      ]
    },
    {
      "isArchive": true,
      "destination": "/",
      "urls": [
        "http://swc.fs2downloads.com/builds/WIN/b.zip",
        "http://scp.fsmods.net/builds/WIN/b.zip",
        "http://scp.indiegames.us/builds/WIN/b.zip"
      ]
    },
    {
      "isArchive": true,
      "destination": "/",
      "urls": [
        "http://swc.fs2downloads.com/builds/WIN/c.zip",
        "http://scp.fsmods.net/builds/WIN/c.zip",
        "http://scp.indiegames.us/builds/WIN/c.zip"
      ]
    },
    {
      "isArchive": true,
      "destination": "/",
      "urls": [
        "http://swc.fs2downloads.com/builds/WIN/d.zip",
        "http://scp.fsmods.net/builds/WIN/d.zip",
        "http://scp.indiegames.us/builds/WIN/d.zip"
      ]
    },
    {
      "isArchive": true,
      "destination": "/",
      "urls": [
        "http://swc.fs2downloads.com/builds/WIN/e.zip",
        "http://scp.fsmods.net/builds/WIN/e.zip",
        "http://scp.indiegames.us/builds/WIN/e.zip"
      ]
    }
  ]
}
Using my syntax I'd have to write:
Code: [Select]
{
  "files": [
    [
      [
        "http://swc.fs2downloads.com/builds/WIN/",
        "http://scp.fsmods.net/builds/WIN/",
        "http://scp.indiegames.us/builds/WIN/"
      ],
      {
        "a.zip": {
          "is_archive": true,
          "destination": "/"
        },
        "b.zip": {
          "is_archive": true,
          "destination": "/"
        },
        "c.zip": {
          "is_archive": true,
          "destination": "/"
        },
        "d.zip": {
          "is_archive": true,
          "destination": "/"
        },
        "e.zip": {
          "is_archive": true,
          "destination": "/"
        }
      }
    ]
  ]
}

It's also easier to add mirrors since you only have to add 1 line.

I'd like to shorten your "dependencies" syntax to this:
Code: [Select]
{
  "dependencies": ["<ID>=1.2.*", "<ID>>=1.0"]
}

I don't see the point of the version requirement. Can't we just assume that a mod needs the most recent version of its dependencies?


@Hellzed
Yeah, you found a bug in the converter. I didn't test it, yet. Apparently I have to add the mod's parents to its dependencies.
The "." seperators aren't parsed in any way. It's how Goober's installer tracks dependencies. (http://www.hard-light.net/wiki/index.php/FSO_Installer_Text_Files#DEPENDENCIES) It was the easiest way to generate unique IDs for the converted mods and keep the dependency references intact. You can use pretty much anything as the mod's ID as long as it's unique.

I'm not differentiating between real submods and features. It's a nice idea but I'm not sure if it's worth the trouble. The converter would have to guess if a submod is an actual mod or a feature. How would you display the difference to the user? However, if we're going to add this, I'd pick the first example since the hierachy of mods / features is much clearer.

The XSSI concerns don't really apply here since we aren't processing user input (which could contain malicious code).
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on August 26, 2014, 10:55:44 am
Most of the JSON files will be generated automatically anyway so limiting the capabilities (every file has to have the same name on every mirror) in favor of saving some typing doesn't seem like a good idea. Instead the format should aim to reduce programming overhead.

I'm not sure if "environment" is a fitting name for this field. It's supposed to specify all requirements (OS, FSO version, maybe CPU features, ...) that don't fit in dependencies.
Well, the things you have listed describe the environment the mod will run in so the naming makes sense to me. I would also advise to have the FSO requirements specified in the dependency field. That would allow to handle executables just like a mod without much special handling.

The "actions" array sounds like a great idea. Any suggestions on additional actions?
I can't think of any actions right now but with the actions array adding a new action type should be easy to do when needed.

I'd like to shorten your "dependencies" syntax to this:
Code: [Select]
{
  "dependencies": ["<ID>=1.2.*", "<ID>>=1.0"]
}

I don't see the point of the version requirement. Can't we just assume that a mod needs the most recent version of its dependencies?
That version syntax is much harder to process and can be ambiguous which means specifying the version requirement and the id separately solves not only that problem but will also allow future expansions without breaking older mod definitions.

Specifying version requirements is needed to make sure a mod runs with the correct version of the MediaVPs. Assuming that a mod will work with the latest version of a dependency will only cause problems.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on August 26, 2014, 11:29:18 am
Most of the JSON files will be generated automatically anyway so limiting the capabilities (every file has to have the same name on every mirror) in favor of saving some typing doesn't seem like a good idea. Instead the format should aim to reduce programming overhead.
I could transform my files syntax into your syntax in the same step where the checksums are added. What do you think about that?

Specifying version requirements is needed to make sure a mod runs with the correct version of the MediaVPs. Assuming that a mod will work with the latest version of a dependency will only cause problems.

Well, here we have a problem. As long as we generate these files from Goober's text files we can only install the most recent version of each mod. If a mod depends on the 2014 MediaVPs, it has to put "FSOI#2014 FSU MediaVPs" in its dependencies. If it depends on the 3.6.12 MediaVPs, it has to put "FSOI#FSU MediaVPs, version 3.6.12" in its dependencies. Those two versions are two seperate mods in Goober's files.

I would also advise to have the FSO requirements specified in the dependency field. That would allow to handle executables just like a mod without much special handling.

Uh, what special handling are you saving here? Actually, this would require more special handling since we don't have one "FSO" package a mod could depend on. We have one for every OS and several for Windows.

The cleanest solution would be to write the JSON files from scratch but then you have to maintain them and update them whenever Goober changes his files. That's the only way we could get proper version handling.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on August 26, 2014, 02:59:34 pm
@ngld : I currently have a group of php/symfony related entities mapping exacly your json format. Now, I'm playing with the "form builder" to create a dynamic form with jquery, based on the entities. (click "create mod", then fill in basic fields, then click "add  file" for each file. Auto completion for dependencies would be a cool feature too)
Once it's done, rendering a json file or a txt file, depending on the request, shouldn't be too hard.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on August 27, 2014, 07:20:32 am
I could transform my files syntax into your syntax in the same step where the checksums are added. What do you think about that?
Sounds good. I would also advise to move the checksums to the file objects as those things belong together.

Well, here we have a problem. As long as we generate these files from Goober's text files we can only install the most recent version of each mod. If a mod depends on the 2014 MediaVPs, it has to put "FSOI#2014 FSU MediaVPs" in its dependencies. If it depends on the 3.6.12 MediaVPs, it has to put "FSOI#FSU MediaVPs, version 3.6.12" in its dependencies. Those two versions are two seperate mods in Goober's files.

[...]

The cleanest solution would be to write the JSON files from scratch but then you have to maintain them and update them whenever Goober changes his files. That's the only way we could get proper version handling.
This new format should replace the old format used by the installer so the old format should only be used to get an initial version of the JSON files which are then adapted to the new format which includes merging different versions of the same mod.

Uh, what special handling are you saving here? Actually, this would require more special handling since we don't have one "FSO" package a mod could depend on. We have one for every OS and several for Windows.
The special handling would be that the launcher has to know that the "FSO" package contains the actual executables for the game. Everything other than that can be handled by using the "environment"/"requirements" option for the executable to determine which executable should be used. That would also resolve the "What is SSE/AVX?" questions as the launcher can determine if the CPU supports that instruction and use the correct executable.
Having the executables as standard packages would need support by the nightly/release build script to publish every new build to the repository but that is nice to have and no requirement.

@ngld : I currently have a group of php/symfony related entities mapping exacly your json format. Now, I'm playing with the "form builder" to create a dynamic form with jquery, based on the entities. (click "create mod", then fill in basic fields, then click "add  file" for each file. Auto completion for dependencies would be a cool feature too)
Once it's done, rendering a json file or a txt file, depending on the request, shouldn't be too hard.
That would be great! A mod author could just create the mod and the information would automatically be propagated to the users.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on August 27, 2014, 03:16:03 pm
I've updated the schema (https://github.com/ngld/fs2mod-py/blob/packager/converter/schema.txt).
@Hellzed: I didn't add your "required" packages and kept the "files" list instead. I also renamed your "packages" into "features". What do you think?

Since we aren't restricted by Goober's files anymore, I made semantic versioning (http://semver.org/) a requirement for the "version" field. I suggest we use composer's syntax for versioned dependencies.

I've added an example for the "environment" option. It's a bit long but should be more flexible this way.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on August 27, 2014, 07:31:08 pm
Quote
Since we aren't restricted by Goober's files anymore, I made semantic versioning a requirement for the "version" field. I suggest we use composer's syntax for versioned dependencies.
So you want to drop txt files conversion to json in fs2mod-py ? (the php server is far from ready, but It will get there eventually)

I've just read the new schema :

- I really like the new "dependencies", "environment" and "actions" handling. (single level lists of objects in json help me a lot on the database and form building side)

- I'd like to keep my object map as simple, and as close as possible to the json data structure, for obvious parsing reasons.

    *With your previous json schema, I was using file groups. File groups are custom objects that contain all files sharing the same mirrors, and these mirrors urls. For both database management and dynamic form building, I'm mapping a relation from multiple file groups to a single mod object.

    *With the new json schema, I'll have to map a relation from multiple file groups to both the mod object (because of the "files" string) and to a new type object (because of the "features" string). It would make my objects map a bit less readable, and data operations cascades (like removing orphan file groups on mod removal) a bit more complicated.

    *The goal of "required" "packages" was for me to completely drop the file group relation to the mod object, and replace it with a relation to the "package" instead.
Short version : having "files" at 2 different depth in the data structure is not ideal on my side. If the new json schema makes things easier on the python side, I'm ok with it, I will find a solution (either at the database level or parsing level).

    *Anyway, including additional/overridden "dependencies" in the "features" is a good idea. Will the fs2_open builds to install on the users system be specified through dependency (if the build is considered as a standard "mod", I guess) or environment  variables? I'm asking this because if it's set with environment, it could be useful to get "environment" inside "features" too.

- Now we nearly have a good dependency handling, isn't "submods" list going to be obsolete ?

EDIT : At first I wanted modders to manually enter md5 hashes into the mod building form. I know it can easily be automated in php, but it also looks heavily server dependent (reaching all kinds of limits : execution time, file size...) for big files like VPs. Thoughts ?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on August 28, 2014, 08:10:21 am
EDIT : At first I wanted modders to manually enter md5 hashes into the mod building form. I know it can easily be automated in php, but it also looks heavily server dependent (reaching all kinds of limits : execution time, file size...) for big files like VPs. Thoughts ?
You could do it locally by using the JavaScript FileAPI and a JavaScript implementation of MD5 (for example this: https://github.com/satazor/SparkMD5).

    *The goal of "required" "packages" was for me to completely drop the file group relation to the mod object, and replace it with a relation to the "package" instead.
Short version : having "files" at 2 different depth in the data structure is not ideal on my side. If the new json schema makes things easier on the python side, I'm ok with it, I will find a solution (either at the database level or parsing level).
Dropping the file group and adding a "required" feature status seems like a good idea.

    *Anyway, including additional/overridden "dependencies" in the "features" is a good idea. Will the fs2_open builds to install on the users system be specified through dependency (if the build is considered as a standard "mod", I guess) or environment  variables? I'm asking this because if it's set with environment, it could be useful to get "environment" inside "features" too.
I would like to have the executable dependency handled by the "dependencies" array but moving the "environment" specification into the feature object sounds like a great idea. That way one version of the FSO executable "mod" could specify multiple features that would be selected based on the users environment.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on August 28, 2014, 08:35:04 am
If you're generating the first version ("repo configuration"), you don't need to include the checksums. The second version ("generated configuration", I still need a better name for this. :blah:) can be generated with "python converter.py checksums".
You could simply host a "repo configuration" and I'll be running the converter every hour to update the central repo with your mod information and the appropriate checksums.

If you want to do it yourself, you can use some kind of queue and background processes.

Quote from: m!m
You could do it locally by using the JavaScript FileAPI and a JavaScript implementation of MD5
That only works if the user is uploading his files, not if he inputs links. You would also need unzip.js and similiar scripts to extract the archives.

Alright, I'll be removing the "files" property and include "required" packages with m!m's changes.

Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on August 28, 2014, 08:38:01 am
That only works if the user is uploading his files, not if he inputs links. You would also need unzip.js and similiar scripts to extract the archives.
The FileAPI gives you access to the raw byte contents of the file without uploading it anywhere. I also don't understand why you would need to extract the archives to get the checksum of the file. All modern archive formats have some sort of archive integrity system in place to make sure archive corruption gets detected.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on August 28, 2014, 09:01:51 am
I know about the File API. My point was that if the user doesn't have the file locally and pastes a link like http://mvp.fsmods.net/..., you won't be able to read the file using JavaScript.
The checksums of the archive's contents are necessary so that fs2mod-py can check the local files without downloading the archive. It's also a lot easier to use MD5 checksums for everything.

EDIT: I've updated the schema again.
@Hellzed: We still need the "submods" property to build a tree structure.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on August 28, 2014, 11:26:38 am
The latest schema looks good to me. I guess everyone is ok with this version ? (if that's the case, I'll rebuild my database accordingly)

If you're generating the first version ("repo configuration"), you don't need to include the checksums. The second version ("generated configuration", I still need a better name for this. :blah:) can be generated with "python converter.py checksums".
You could simply host a "repo configuration" and I'll be running the converter every hour to update the central repo with your mod information and the appropriate checksums.

Regarding the checksums, I think I can leave the choice to the modders : manually enter all checksums, or leave it to our hosted version of fs2mod-py and wait for the next hourly refresh. From what I read about php and big files, that's the most reliable solution.

I'll add the option not to publish a mod after configuring the repository, as a way to save current work without propagating an unfinished mod/update to the users, and a few other mod management features.
With the new packager branch logic, will fs2mod-py still be able manually add a json source like it used to? (I'm asking because I can also add a "private repository" feature for modders who would want to test their mod deployment)
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on August 28, 2014, 12:01:08 pm
The latest schema looks good to me. I guess everyone is ok with this version ?
I'm not sure. You should probably wait for m!m's opinion.
@m!m: With the new schema it's still not possible to make the MediaVPs 3.6.12 and MediaVPs 2014 a single mod. I'm not sure if it's worth the extra effort. What do you think? FSO however, can be unified across platforms and CPU architectures. (This means that there will only be one FSO package the user can select.)

Regarding the checksums, I think I can leave the choice to the modders : manually enter all checksums, or leave it to our hosted version of fs2mod-py and wait for the next hourly refresh. From what I read about php and big files, that's the most reliable solution.
So... are you generating two json files? I'm asking because I want to add your database to the central repo.

I'll add the option not to publish a mod after configuring the repository, as a way to save current work without propagating an unfinished mod/update to the users, and a few other mod management features.
With the new packager branch logic, will fs2mod-py still be able manually add a json source like it used to? (I'm asking because I can also add a "private repository" feature for modders who would want to test their mod deployment)
The users will still be able to add additional repos but I'll drop the support for fs2mod archives (the format you invented for the original bash script).

Will you be releasing the source code to your project? I'd be interested.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on August 28, 2014, 12:40:42 pm
I think the schema is pretty good now. Do you intend to write a parser for this format in python? If that is the case can you bundle it in a library so I can use that in the launcher I am writing?

I'm not sure. You should probably wait for m!m's opinion.
@m!m: With the new schema it's still not possible to make the MediaVPs 3.6.12 and MediaVPs 2014 a single mod. I'm not sure if it's worth the extra effort. What do you think? FSO however, can be unified across platforms and CPU architectures. (This means that there will only be one FSO package the user can select.)
Yeah, one "mod" definition for multiple versions is not possible but having the MediaVPs share an ID (for example "mediavps") is possible so depending on the newest version of the MediaVPs is as easy as specifying the dependency "mediavps". Basically you would have multiple mod definitions with different versions.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on August 28, 2014, 01:05:42 pm
I think the schema is pretty good now. Do you intend to write a parser for this format in python? If that is the case can you bundle it in a library so I can use that in the launcher I am writing?
Yes, I'll write a parser in python for this. What kind of library do you mean? A .dll file?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on August 28, 2014, 01:36:23 pm
Yes, I'll write a parser in python for this. What kind of library do you mean? A .dll file?
No just the python source code that I could then execute using IronPython.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on August 28, 2014, 02:21:24 pm
Regarding the checksums, I think I can leave the choice to the modders : manually enter all checksums, or leave it to our hosted version of fs2mod-py and wait for the next hourly refresh. From what I read about php and big files, that's the most reliable solution.
So... are you generating two json files? I'm asking because I want to add your database to the central repo.
In the process I was describing, my script would generate 2 json files. If the modder enters all checksums manually, a full "generated configuration" can be generated. If the modder doesn't, only a "repository configuration" file is generated. I thought your server could grab the list of repo configs and finish the job.
(I'll think of better server interactions and hosting policy when I'm done with dynamic mod forms. I guess you'll have better ideas in the meanwhile)


I'm waiting for the project to be a bit more advanced, but yes, everything will be released. :)
I've known php for years, but I'm still learning how to use symfony framework. The only thing I've done with it yet is a blog with the help of the official walkthrough, and this project uses a slightly different set of features, so that's a lot of documentation reading while I'm coding.

I think the schema is pretty good now. Do you intend to write a parser for this format in python? If that is the case can you bundle it in a library so I can use that in the launcher I am writing?

@m!m: With the new schema it's still not possible to make the MediaVPs 3.6.12 and MediaVPs 2014 a single mod. I'm not sure if it's worth the extra effort. What do you think? FSO however, can be unified across platforms and CPU architectures. (This means that there will only be one FSO package the user can select.)
Yeah, one "mod" definition for multiple versions is not possible but having the MediaVPs share an ID (for example "mediavps") is possible so depending on the newest version of the MediaVPs is as easy as specifying the dependency "mediavps". Basically you would have multiple mod definitions with different versions.
This is exactly what I plan to do inside the database. By default, you'll always get the latest version of a mod, but nothing stops you from requesting an older version if it's still in the db.
www.example.com/nebula/all.json // you get the full public repository
www.example.com/nebula/mod/<ModID>.json // you get <ModID>'s latest version repository
www.example.com/nebula/mod/<ModID>.<ModVersion>.json //  you get <ModID>'s <ModVersion> repository

Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on August 28, 2014, 03:10:28 pm
I think we can still improve the dependency handling: Right now a mod's packages can depend on other mods but not on specific packages (or "features"). I suggest we add a "package" option to the dependency object so that it looks like this:
Code: [Select]
{
    "id": "<ID>",
    "packages": [],
    "version": "1.2.*" // https://getcomposer.org/doc/01-basic-usage.md#package-versions
}
By default, a mod depends on another mod's required packages but if it requires specific packages, those can be specified in the packages option.

@m!m: I'll either put the parser code in a single file or an external module that you can import in IronPython.

Quote from: Hellzed
I thought your server could grab the list of repo configs and finish the job.
That was actually my intention.  ;)

Quote from: Hellzed
I'm waiting for the project to be a bit more advanced, but yes, everything will be released. :)
Thanks!
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on August 28, 2014, 03:40:44 pm
If I get it right, a "package" could depend on another one in the same mod ? Does it use the full current mod id, or a special keyword in that case (like "CURRENT_MOD") ?
("optional" and "recommended" packages usually only depend on the "required" packages, but are independent from each others, I know. But I've seen this type of dependency used for fsport hi-res command briefings. fsport hi-res command briefings are optional (and should be recommended), and "higher"-res  CBs are depending on hi-res CBs, not on mod core files...)

EDIT : @ngld : do you want me to store mod logos inside the db, as remote a remote ressources for fs2mod-py (just in case you need it for mods that are not installed) ?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on August 28, 2014, 03:52:54 pm
I was actually thinking of MV_Advanced which requires MV_Assets and MV_Effects. Both are otherwise optional (but should be recommended).

This feature should also allow the same for different mods: Blue Planet's bp2-adv should depend on bp-adv but BP:AoA and BP:WiH are different mods.

EDIT: @Hellzed: You can make that optional. Just as the schema says, the logos are extracted and hosted by my server whenever they aren't linked (i.e. the "logo" property is null).
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on August 29, 2014, 06:18:05 am
Should I get ready to support more than one hashing types ? (SHA ?)
I'm not asking for it as a feature, but I was thinking maybe someone would need it, even us if someday we change default hashing type. I'm ok if we stick with MD5 as only hashing type.

@ngld :
I've decided not to implement the possibility to manually enter checksums, to avoid human error, and always use fs2mod-py. Maybe fs2mod-py could output some very simple log file during the hashing process ? Somethinglike " [timestamp] action done/failed", as hashing progresses ?
After the mod form is submitted, php would call fs2mod-py for current mod, and redirect the modder to a page showing the fs2mod-py log file as it updates (with ajax requests). When it's over, "generated" json response is stored in the database, mod is flagged as "ready", and final json download links are generated.
This would also avoid using a json parser in php at each end user request, as final json is cached in the db for a single revision of a mod.
db gets a lot bigger (we create a new entry for every new revision of a mod, and the cached final json nearly doubles the size of any entry). I could refine the logic a bit, so that patch entries would replace a previous patch, and we only create a new entry on minor/major revisions, but I'm not sure it's worth the extra work. (EDIT : this was bad idea : acocrding to http://semver.org : " 3. Once a versioned package has been released, the contents of that version MUST NOT be modified. Any modifications MUST be released as a new version.")
Is that close to how you wanted to integrate db and central repo ?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on August 29, 2014, 07:34:37 am
Should I get ready to support more than one hashing types ? (SHA ?)
I'm not asking for it as a feature, but I was thinking maybe someone would need it, even us if someday we change default hashing type. I'm ok if we stick with MD5 as only hashing type.
MD5 should work well enough because we aren't hashing passwords but checking file integrity.

@ngld :
I've decided not to implement the possibility to manually enter checksums, to avoid human error, and always use fs2mod-py. Maybe fs2mod-py could output some very simple log file during the hashing process ? Somethinglike " [timestamp] action done/failed", as hashing progresses ?
The converter.py script already does that.

After the mod form is submitted, php would call fs2mod-py for current mod, and redirect the modder to a page showing the fs2mod-py log file as it updates (with ajax requests).
So you're going to run fs2mod-py on your server?

Is that close to how you wanted to integrate db and central repo ?
I wanted to put your all.json in my "includes" list but your solution is better. This way modders can get immediate feedback whether the download links work.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on August 29, 2014, 08:46:25 am
So you're going to run fs2mod-py on your server?
Yes. I'll add a way to track "failed" mod versions that didn't pass the file availability test.
Basically, modders will have a view listing all versions for the mod, including published versions, private versions, failed versions and working copies. Only working copies can be edited, according to semver recommendation.
Default new mod version will be created from the lastest available data stored in the db (public, private, failed or working copy).
Ideally, a new mod version could also be created (as a working copy) from any other version, and maybe other mods (so it would be some sort of forking mechanism).

I'll start to implement all these database operations (they're actually pretty easy, that's only multi-field sorting and entries copy) once the base template for mod dynamic form is working as intended. I'm struggling to get JS to do exactly what I want with symfony generated html (handling multiple levels of subsections, while being able to add new subsections), but I'll get there eventually. When it's done, live testing of database operations will get much easier.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on August 30, 2014, 05:03:43 pm
Just a quick update: I've finished the new implementation in fs2mod-py. I haven't had a chance to extensively test it, yet. It might destroy your FS2 installation. </disclaimer>
For now FSO and the MediaVPs 2014 are in the repo. Here's the repo configuration (http://dev.tproxy.de/fs2/all.raw.json) and here's the file fs2mod-py uses (http://dev.tproxy.de/fs2/all.json).

I'll finish the package for m!m next.
Updated TODO list:
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on August 31, 2014, 04:12:19 pm
I'll give it a try soon. I always have multiple FS2 installations on my hard drive ;)

I've also made good progress today (finished working on javascript for the mod form, now I can feed it any Symfony form, and it will be able to recursively add/delete nested form elements, mirroring what Sf database expects.)
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on September 01, 2014, 04:33:10 am
For now FSO and the MediaVPs 2014 are in the repo. Here's the repo configuration (http://dev.tproxy.de/fs2/all.raw.json) and here's the file fs2mod-py uses (http://dev.tproxy.de/fs2/all.json).
I see you are still using the old installer like "files" syntax. Is this going to change at a later point?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on September 02, 2014, 08:42:12 pm
I went for the current syntax when I rebuilt the database, but it can be changed (it's just a matter of automated code generation).
Good news : tonight, for the first time, a dummy mod (with its full object structure and metadata) was successfully registered from the dynamic form. :)

(http://pix.toile-libre.org/upload/thumb/1409705518.png) (http://pix.toile-libre.org/?img=1409705518.png)

My current to-do list is now :
- Double check object relations to make sure the database stays clean (and is not broken) after mod removal,
- Validate user input,
- Add a JSON parser and a way to get raw JSON output and feed it to fs2mod-py,
- Add a Bootstrap theme to the dynamic form (it's currently hardly readable with 3 to 4 embedded form levels),
- Install a "user bundle" (probably FoSUserBundle) to manage multiple users and access control lists,
- Optimise DB queries,
- Fix the url rewriting.

When it's done, I'll upload the code to github.

After that, I'll keep working, not necessarily in that order :
- Integrate with the standalone repo parser,
- Refine mod administration tools,
- Add the public pages (with generated fso:// links) and an actually good looking theme (simple Bootstrap for the modders side, something more elaborate for the public side),
- Add cool features like auto-completion (I've already looked into jquery-ui) and images/screenshots preview in the dynamic form
- maybe import/export txt files.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on September 06, 2014, 04:33:29 pm
I could transform my files syntax into your syntax in the same step where the checksums are added. What do you think about that?
Sounds good. I would also advise to move the checksums to the file objects as those things belong together.

For now FSO and the MediaVPs 2014 are in the repo. Here's the repo configuration (http://dev.tproxy.de/fs2/all.raw.json) and here's the file fs2mod-py uses (http://dev.tproxy.de/fs2/all.json).
I see you are still using the old installer like "files" syntax. Is this going to change at a later point?

The second file actually uses your syntax. I've added the filenames as keys and changed some key names. Here's a prettier version of the second file:
Code: [Select]
{
   "generated" : 1409428735.93665,
   "mods" : [
      {
         "title" : "MediaVPs",
         "version" : "2.0.0-2014",
         "id" : "MediaVPs_2014",
         "logo" : "logofzicrk73.jpg",
         "packages" : [
            {
               "notes" : "Required: The bare minimum required for running FSO, also contains the mod.ini, FSU-MVP.bmp and Readme.txt and sets up the mod folder",
               "files" : {
                  "MV_Root.7z" : {
                     "contents" : {
                        "MediaVPs_2014/FSU-MVP.bmp" : "1c5972372b4d2a38bc1e5a7d8653f580",
                        "MediaVPs_2014/mod.ini" : "976bc289676bef464a37f70683baba92",
                        "MediaVPs_2014/readme.txt" : "4c9a122950141e828ae2f2a3aa73580b",
                        "MediaVPs_2014/MV_Root.vp" : "c7729dea15e7b23de1936cd79e5a036c",
                        "MediaVPs_2014/FSU-MVP_small.bmp" : "2fe5111a95f37f0008ad200896175035"
                     },
                     "md5sum" : "96bf9c147c7e371abde42a10064d9714",
                     "is_archive" : true,
                     "dest" : "",
                     "urls" : [
                        "http://mvp.fsmods.net/2014/MV_Root.7z",
                        "http://www.lunardigitalproductions.com/downloads/MediaVPs2014/MV_Root.7z"
                     ]
                  }
               },
               "name" : "MV_Root",
               "dependencies" : [
                  {
                     "id" : "FSO",
                     "version" : ">=3.7.1"
                  }
               ],
               "environment" : [],
               "status" : "required"
            },
            {
               "status" : "recommended",
               "notes" : "Enhanced OGG Music Package",
               "name" : "MV_Music",
               "files" : {
                  "MV_Music.7z" : {
                     "urls" : [
                        "http://mvp.fsmods.net/2014/MV_Music.7z",
                        "http://www.lunardigitalproductions.com/downloads/MediaVPs2014/MV_Music.7z"
                     ],
                     "dest" : "",
                     "is_archive" : true,
                     "md5sum" : "ec577fef3c101553ecb1bff86e7ad8ae",
                     "contents" : {
                        "MediaVPs_2014/MV_Music.vp" : "121300bbe229d0373cd731a37b205b3a"
                     }
                  }
               },
               "environment" : [],
               "dependencies" : []
            },
            {
               "notes" : "All the High Poly Models and their Maps",
               "files" : {
                  "MV_Assets.7z" : {
                     "contents" : {
                        "MediaVPs_2014/MV_Assets.vp" : "bb65f1e147c3417a86dd5464f268feb5"
                     },
                     "md5sum" : "17bc0c46a08de9753e342c445c6f9850",
                     "is_archive" : true,
                     "dest" : "",
                     "urls" : [
                        "http://mvp.fsmods.net/2014/MV_Assets.7z",
                        "http://www.lunardigitalproductions.com/downloads/MediaVPs2014/MV_Assets.7z"
                     ]
                  }
               },
               "name" : "MV_Assets",
               "dependencies" : [],
               "environment" : [],
               "status" : "recommended"
            },
            {
               "status" : "recommended",
               "dependencies" : [],
               "environment" : [],
               "files" : {
                  "MV_Effects.7z" : {
                     "is_archive" : true,
                     "dest" : "",
                     "urls" : [
                        "http://mvp.fsmods.net/2014/MV_Effects.7z",
                        "http://www.lunardigitalproductions.com/downloads/MediaVPs2014/MV_Effects.7z"
                     ],
                     "contents" : {
                        "MediaVPs_2014/MV_Effects.vp" : "cd3fef0af8308087f619d0e19cf4b620"
                     },
                     "md5sum" : "0d71c9ea7e4d620fd7902d1941161a35"
                  }
               },
               "name" : "MV_Effects",
               "notes" : "All the touched up or redone Effects (Weapons, Starfields, etc)"
            },
            {
               "notes" : "For users with higher end machines, this package contains many significantly more demanding textures and effects that may cause poor performance on low end machines",
               "name" : "MV_Advanced",
               "files" : {
                  "MV_Advanced.7z" : {
                     "md5sum" : "6f3275e66746e7997d060ee71bbd0b47",
                     "contents" : {
                        "MediaVPs_2014/MV_Advanced.vp" : "2646219d1d67250adc4bcf4ceaa0176f"
                     },
                     "urls" : [
                        "http://mvp.fsmods.net/2014/MV_Advanced.7z",
                        "http://www.lunardigitalproductions.com/downloads/MediaVPs2014/MV_Advanced.7z"
                     ],
                     "dest" : "",
                     "is_archive" : true
                  }
               },
               "environment" : [],
               "dependencies" : [
                  {
                     "packages" : [
                        "MV_Assets",
                        "MV_Effects"
                     ],
                     "version" : "*",
                     "id" : "MediaVPs_2014"
                  }
               ],
               "status" : "recommended"
            },
            {
               "name" : "MV_A-Glows",
               "files" : {
                  "MV_A-Glows.7z" : {
                     "md5sum" : "bed0e25ac2af663dd41574b458f37fd5",
                     "contents" : {
                        "MediaVPs_2014/MV_A-Glows.vp" : "f38badfdc5e3af3bacb356809a8ad8a6"
                     },
                     "dest" : "",
                     "urls" : [
                        "http://mvp.fsmods.net/2014/MV_A-Glows.7z",
                        "http://www.lunardigitalproductions.com/downloads/MediaVPs2014/MV_A-Glows.7z"
                     ],
                     "is_archive" : true
                  }
               },
               "dependencies" : [
                  {
                     "id" : "MediaVPs_2014",
                     "version" : "*",
                     "packages" : [
                        "MV_Assets"
                     ]
                  }
               ],
               "environment" : [],
               "notes" : "For users with extra texture space, contains all the Animated Glow Maps",
               "status" : "optional"
            },
            {
               "name" : "MV_RadarIcons",
               "files" : {
                  "MV_RadarIcons.7z" : {
                     "contents" : {
                        "MediaVPs_2014/MV_RadarIcons.vp" : "45b22fb976ada8c8eceb96592b940d33"
                     },
                     "md5sum" : "60a99d17d4be13b478dabdb8da23550d",
                     "is_archive" : true,
                     "urls" : [
                        "http://mvp.fsmods.net/2014/MV_RadarIcons.7z",
                        "http://www.lunardigitalproductions.com/downloads/MediaVPs2014/MV_RadarIcons.7z"
                     ],
                     "dest" : ""
                  }
               },
               "dependencies" : [],
               "environment" : [],
               "notes" : "Provides ship Icons for the 2D Radar View",
               "status" : "recommended"
            },
            {
               "status" : "recommended",
               "notes" : "Part of a two part pack that updates all of the cbanims to hi-res 660x300.",
               "environment" : [],
               "dependencies" : [],
               "name" : "MV_CB_ANI_1",
               "files" : {
                  "MV_CB_ANI_1.7z" : {
                     "md5sum" : "a521e079e0343de8a2873757b1c7e901",
                     "contents" : {
                        "MediaVPs_2014/MV_CB_ANI_1.vp" : "67ef346aba6a38e60e3517306d0dedca"
                     },
                     "dest" : "",
                     "urls" : [
                        "http://mvp.fsmods.net/2014/MV_CB_ANI_1.7z",
                        "http://www.lunardigitalproductions.com/downloads/MediaVPs2014/MV_CB_ANI_1.7z"
                     ],
                     "is_archive" : true
                  }
               }
            },
            {
               "notes" : "Part of a two part pack that updates all of the cbanims to hi-res 660x300.",
               "dependencies" : [],
               "environment" : [],
               "files" : {
                  "MV_CB_ANI_2.7z" : {
                     "urls" : [
                        "http://mvp.fsmods.net/2014/MV_CB_ANI_2.7z",
                        "http://www.lunardigitalproductions.com/downloads/MediaVPs2014/MV_CB_ANI_2.7z"
                     ],
                     "dest" : "",
                     "is_archive" : true,
                     "md5sum" : "d39066b360194429e634d01366568778",
                     "contents" : {
                        "MediaVPs_2014/MV_CB_ANI_2.vp" : "b25264ca441b654bc147c1e7da574eea"
                     }
                  }
               },
               "name" : "MV_CB_ANI_2",
               "status" : "recommended"
            }
         ],
         "notes" : "",
         "folder" : "Mediavps_2014",
         "submods" : [],
         "description" : ""
      },
      {
         "id" : "FSO",
         "version" : "3.7.2-rc4",
         "logo" : "http://www.fsoinstaller.com/files/installer/java/FS2.bmp",
         "packages" : [
            {
               "status" : "required",
               "notes" : "Windows builds for FreeSpace 2 Open, compiled with SSE optimizations.",
               "environment" : [
                  {
                     "os" : "windows",
                     "type" : "os"
                  },
                  {
                     "feature" : "sse",
                     "type" : "cpu_feature"
                  }
               ],
               "dependencies" : [],
               "files" : {
                  "fs2_open_3.7.2_RC4_SSE.7z" : {
                     "is_archive" : true,
                     "urls" : [
                        "http://swc.fs2downloads.com/builds/WIN/fs2_open_3.7.2_RC4_SSE.7z",
                        "http://scp.fsmods.net/builds/WIN/fs2_open_3.7.2_RC4_SSE.7z",
                        "http://scp.indiegames.us/builds/WIN/fs2_open_3.7.2_RC4_SSE.7z"
                     ],
                     "dest" : "",
                     "contents" : {
                        "fred2_open_3_7_2_RC4_SSE.exe" : "7de9615d5b212e2489dd59833bdbf8f5",
                        "fred2_open_3_7_2_RC4_SSE-DEBUG.exe" : "bc0d622342b6ecda22ffa17187097b26",
                        "fs2_open_3_7_2_RC4_SSE-DEBUG.exe" : "24057ff0d1ba3360e6ba164641b45727",
                        "fs2_open_3_7_2_RC4_SSE-DEBUG.map" : "d687df1a222ca0d8951819444a059a18",
                        "fs2_open_3_7_2_RC4_SSE-DEBUG.pdb" : "ccdd1b98d18580f6c0170954af92e9dd",
                        "fred2_open_3_7_2_RC4_SSE-DEBUG.pdb" : "1a764e4eac166a792bfe026a4d0db5ae",
                        "fred2_open_3_7_2_RC4_SSE-DEBUG.map" : "5a0797ebe00c8144716105e3e6be9d6a",
                        "fs2_open_3_7_2_RC4_SSE.exe" : "da4a73536061ede4e5e659b1e6bbb704"
                     },
                     "md5sum" : "70ecb06d9bf8a447d3a93d212628de37"
                  },
                  "Launcher55g.zip" : {
                     "md5sum" : "20bdf9dc2f225c30961df672be438e37",
                     "contents" : {
                        "Launcher.exe" : "798f7da18e3ce3a7caaa9c4aec2d3072"
                     },
                     "dest" : "",
                     "urls" : [
                        "http://swc.fs2downloads.com/files/Launcher55g.zip",
                        "http://scp.fsmods.net/builds/Launcher55g.zip",
                        "http://scp.indiegames.us/builds/Launcher55g.zip"
                     ],
                     "is_archive" : true
                  }
               },
               "name" : "Windows SSE"
            },
            {
               "status" : "required",
               "notes" : "Windows builds for FreeSpace 2 Open, compiled with SSE2 optimizations.",
               "dependencies" : [],
               "environment" : [
                  {
                     "os" : "windows",
                     "type" : "os"
                  },
                  {
                     "type" : "cpu_feature",
                     "feature" : "sse2"
                  }
               ],
               "name" : "Windows SSE2",
               "files" : {
                  "fs2_open_3.7.2_RC4.7z" : {
                     "md5sum" : "a865d781aad15e8503a8bf4e0d9546e0",
                     "contents" : {
                        "fred2_open_3_7_2_RC4-DEBUG.map" : "10e17002485c8b4e074cd7a833668f95",
                        "fs2_open_3_7_2_RC4.exe" : "2166833b7cf3f795316ed0fc28fe7efd",
                        "fs2_open_3_7_2_RC4-DEBUG.pdb" : "be9e90a78812527df9a18e00f8710aea",
                        "fred2_open_3_7_2_RC4.exe" : "f2968bb9dcc30bc7893637f08e54de16",
                        "fred2_open_3_7_2_RC4-DEBUG.pdb" : "89cdc982273b3ecba95f4134bff0c477",
                        "fred2_open_3_7_2_RC4-DEBUG.exe" : "0ce97a2b8fda1b9b99a0fcbff57cd5d9",
                        "fs2_open_3_7_2_RC4-DEBUG.map" : "df19268d1680729c327b87e006221424",
                        "fs2_open_3_7_2_RC4-DEBUG.exe" : "ac71a1eb4815fe514829b3b6de1c5f8c"
                     },
                     "dest" : "",
                     "urls" : [
                        "http://swc.fs2downloads.com/builds/WIN/fs2_open_3.7.2_RC4.7z",
                        "http://scp.fsmods.net/builds/WIN/fs2_open_3.7.2_RC4.7z",
                        "http://scp.indiegames.us/builds/WIN/fs2_open_3.7.2_RC4.7z"
                     ],
                     "is_archive" : true
                  },
                  "Launcher55g.zip" : {
                     "urls" : [
                        "http://swc.fs2downloads.com/files/Launcher55g.zip",
                        "http://scp.fsmods.net/builds/Launcher55g.zip",
                        "http://scp.indiegames.us/builds/Launcher55g.zip"
                     ],
                     "dest" : "",
                     "is_archive" : true,
                     "md5sum" : "20bdf9dc2f225c30961df672be438e37",
                     "contents" : {
                        "Launcher.exe" : "798f7da18e3ce3a7caaa9c4aec2d3072"
                     }
                  }
               }
            },
            {
               "status" : "required",
               "notes" : "Linux builds for FreeSpace 2 Open.",
               "files" : {
                  "fs2_open_3.7.2_RC4.tar.bz2" : {
                     "dest" : "",
                     "urls" : [
                        "http://swc.fs2downloads.com/builds/LINUX/fs2_open_3.7.2_RC4.tar.bz2",
                        "http://scp.fsmods.net/builds/LINUX/fs2_open_3.7.2_RC4.tar.bz2",
                        "http://scp.indiegames.us/builds/LINUX/fs2_open_3.7.2_RC4.tar.bz2"
                     ],
                     "is_archive" : true,
                     "md5sum" : "5fd897dd72ec0c76eb7eea8377aeffcf",
                     "contents" : {
                        "fs2_open_3.7.2_RC4_DEBUG" : "3f3db65b0f4fadd54a5fa1b269894abf",
                        "fs2_open_3.7.2_RC4" : "027cf0ffa7a96bbfb5239293feada1f3"
                     }
                  }
               },
               "name" : "Linux",
               "dependencies" : [],
               "environment" : [
                  {
                     "type" : "os",
                     "os" : "linux"
                  }
               ]
            },
            {
               "status" : "required",
               "notes" : "",
               "dependencies" : [],
               "environment" : [],
               "files" : {
                  "mod.ini" : {
                     "contents" : {},
                     "md5sum" : "d0576b36ef22b599186f67876e9b3aec",
                     "is_archive" : false,
                     "dest" : "",
                     "urls" : [
                        "http://www.fsoinstaller.com/files/installer/java/mod.ini"
                     ]
                  },
                  "scptrackir.zip" : {
                     "md5sum" : "b611edc87ef30ce57133b9721db01b59",
                     "contents" : {
                        "scptrackir.dll" : "6617fd0f9421ca6594cf2c1eebb20381"
                     },
                     "urls" : [
                        "http://scp.indiegames.us/builds/scptrackir.zip"
                     ],
                     "dest" : "",
                     "is_archive" : true
                  },
                  "FS2.bmp" : {
                     "is_archive" : false,
                     "dest" : "",
                     "urls" : [
                        "http://www.fsoinstaller.com/files/installer/java/FS2.bmp"
                     ],
                     "contents" : {},
                     "md5sum" : "faa4c82b1aa0d3cf5427207e10150711"
                  }
               },
               "name" : "Common files"
            }
         ],
         "title" : "FreeSpace Open",
         "description" : "Recent builds of FreeSpace Open (the engine)",
         "submods" : [],
         "notes" : "",
         "actions" : [],
         "folder" : ""
      }
   ]
}

Anyway, I'm adding this syntax to the converter. It'll accept both variants.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on September 06, 2014, 05:47:42 pm
I think I'm going to use this syntax in the forms :
I was using a "file group" syntax, where a "file group" is defined as "one or more files sharing the same set of mirror servers".
As I expect modders will use a lot of copy-paste to fill in this part of the form, using full URLs is better (even if adds some duplicated text).

Is the "isArchive" boolean really necessary ? Can't we guess that from the mime type ? (i'm betting no mod will ever ship 7z, zip or rar files that don't need to be extracted)
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on September 06, 2014, 06:14:48 pm
What you're describing is pretty much what m|m wanted. It's already in the repository syntax but I've added it as an alternative to the repo configuration (converter.py's input). Now you can simply write
Code: [Select]
{
  /* ... */
  "files": {
    "filename1.tar.gz": {
      "dest": "my_path",
      "urls": [
        "http://host.tld/path/to/file.tar.gz",
        "http://other.host.tld/path/to/other/file.tar.gz"
      ]
    },
    "filename2.7z": {
      "dest": "my_path",
      "urls": [
        "http://host.tld/path/to/file.7z",
        "http://other.host.tld/path/to/other/file.7z"
      ]
    }
  }
}

I've included "is_archive" in the schema because some archives aren't meant to be extracted (i.e. jar files).
It's actually optional. The converter and fs2mod-py assume that the given file is an archive if you don't set the "is_archive" property.
Here's the list of file extensions Goober's installer uses to decide whether a file is an archive or not: zip, tar, split, rar, lzma, iso, hfs, gzip, gz, cpio, bzip2, bz2, 7z, z, arj, cab, lzh, chm, nsis, deb, rpm, udf, wim, xar

I'll add a note to all optional properties in the schema...
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on September 06, 2014, 11:57:26 pm
I really don't like using the file names as keys for some data structures. The problem I see here is that you can't have a direct class <-> JSON relationship in object-oriented languages. I would prefer to use a standard array of objects that would contain a "filename" key with the file name.

Is the "isArchive" boolean really necessary ? Can't we guess that from the mime type ? (i'm betting no mod will ever ship 7z, zip or rar files that don't need to be extracted)
I have FSO builds that support 7zip and zip archives but those wouldn't be called 7z or zip :P
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on September 08, 2014, 01:32:42 pm
I've been working on the templates. No real php behind it yet (except including the right bits), but here's how it looks. Comments are welcome.
(http://pix.toile-libre.org/upload/thumb/1410196642.png) (http://pix.toile-libre.org/?img=1410196642.png)
(http://pix.toile-libre.org/upload/thumb/1410196657.png) (http://pix.toile-libre.org/?img=1410196657.png)
(http://pix.toile-libre.org/upload/thumb/1410196675.png) (http://pix.toile-libre.org/?img=1410196675.png)

EDIT 11/09/2014 : Progress ! Most of the mod and branch storage logic is done. I still have to refactor my first draft of the build storage logic and integrate it with the rest, polish the forms a bit, and install a third party account management bundle. After these steps we'll be ready for a first test. Expect this to be in something like 10 days.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on September 12, 2014, 03:58:34 am
EDIT 11/09/2014 : Progress ! Most of the mod and branch storage logic is done. I still have to refactor my first draft of the build storage logic and integrate it with the rest, polish the forms a bit, and install a third party account management bundle. After these steps we'll be ready for a first test. Expect this to be in something like 10 days.
:yes: That's great news!

I have been looking at the installation process a bit and I noticed that there is no way of specifying the folder of the package file that should be extracted. New mods should have all files in the root folder of the zip file but to support mods that were created before that we need some way of specifying the folder name (for example "MediaVPs_2014" would be the value used for the MediaVPs 2014 package).
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on September 12, 2014, 12:27:13 pm
Make it one more day than what i thought. I've spent my daily development time trying to figure out some badly documented feature of my framework.

If you're interested in the details, it's called "ParamConverter". It safely converts a variable from the page url into an object from your database, or throws an error 404 if no corresponding object exists, just by using a single keyword (php "type hinting"). It avoids a lot of duplicated checks in the php code (or extending the class just to perform said checks).
I've started to rely a lot on this to achieve a github-like hierarchy (owner_object/branch_object/build_object), but it's terribly documented (basically, it says : go check the source...).
It works well on a single object, but fails badly to deal with a group of related objects...
I know exactly what the workaround should be, but I keep having version specific issues (many forum threads are too old for my version of the framework...).

Hopefully I'll find a solution later tonight. Or I'll switch to good old variable checks.

EDIT : Finally found how to write a custom paramconverter from scratch (and that works with Sf 2.5).
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on September 15, 2014, 06:06:39 am
I am also making good progress with the implementation of the client library.
It can currently handle parsing the JSON data into an object representation of the repository. I also just finished implementing dependency resolution of a package. Now I can begin implementing the actual installation backend but that shouldn't be a big problem :nervous:
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on September 18, 2014, 11:25:16 am
That's great ! Hopefully the server is up when you need it.

The basic build storage is in, and we now fully support semver. That means you can use urls like : "/nebula/owner/my_mod/testing/1.1.0-beta2+cockpitspatch/show_data" or "/nebula/owner/my_mod/testing/current/show_data" (for the latest build in the branch).

I've finished reading a big tutorial on FoSUB (the user management bundle I'll be using on the server), I'll start installing it in the project tonight. There are many steps to follow carefully, but it's well documented and nothing is really tricky.
Last week, I stripped my main objects (mod, branch, build) from their secondary objects (logo, package, screenshots...) to make database testing easier. I'll probably re-add them this week-end. From there, I'll only need to pass it to the default php JSON parser, so I'm still aiming to do a first code release on github in a matter of days.

EDIT : Adding FoSUB went smoother than anything I expected :)
Now the app has login/logout/profile/password_reset pages and a "User" table in the database. I'm already writing a Sf Voter (a php class with a method to do a simple check, like verifying if the mod/branch/builds belongs to the current user). Voters are great because they can also be called directly from the template engine (to show the "edit"/"remove" buttons or not, for example).
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on September 18, 2014, 06:47:07 pm
Damn, I'm too slow.
@m!m: I guess you don't need the repo parser anymore, do you?

I've updated the schema (https://github.com/ngld/fs2mod-py/blob/packager/converter/schema.txt) to include the second files syntax (@m!m: Is this more to your liking?  :)) and document which fields are optional and which ones are required.

I should be able to finish a stable fs2mod-py release this weekend.
@Hellzed: I'm looking forward to test your Nebula. I'll implement a browser (Qt Webkit) similiar to Steam's shopping tab. I'll add a JS API (if possible) so that your web page can check which mods (and features) are installed. Any thoughts on that?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on September 19, 2014, 03:19:20 am
Damn, I'm too slow.
@m!m: I guess you don't need the repo parser anymore, do you?
Apparently not :p Writing the parser was much easier than I imagined. Actually writing the logic to get downloading and installation to work is the hard part.

I've updated the schema (https://github.com/ngld/fs2mod-py/blob/packager/converter/schema.txt) to include the second files syntax (@m!m: Is this more to your liking?  :)) and document which fields are optional and which ones are required.
Much better ;)
I have noticed some things while implementing my installer:
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on September 19, 2014, 05:42:08 am
I have noticed some things while implementing my installer:
  • The "mod" object has both "description" and "notes". What's the difference? It would be helpful to know what that should be used for.
  • What do we need "submod" for? I am not sure what I should do with that.
  • Can the "environment" specification be moved into the file specification? That would simplify dependency resolution for me and also seems like the better place.

Code: [Select]
- Mod 1
+ Mod 2
L Submod 1
L + Submod 2
| L Subsubmod 1
| L Subsubmod 2
L Submod 3
- Mod 3

I'm just wondering... how did you implement the dependency tracking? Can your resolver handle conflicts and cycles?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on September 19, 2014, 07:05:27 am
  • The description is supposed to match the mod.ini's description. The notes should contain important things a modder wants to tell the user before installing (pretty much what release posts on the forum contain).
  • Submods are mods that are related to a given mod (i.e. ST:R is a submod of FSPort 1). I'll display the tree structure in fs2mod-py like this:
Code: [Select]
- Mod 1
+ Mod 2
L Submod 1
L + Submod 2
| L Subsubmod 1
| L Subsubmod 2
L Submod 3
- Mod 3
Thanks for the clarification.

  • Hm... I'm not sure. In the repo example I did, the fs2_open mod has four packages: Common files, Windows, Linux and Mac OS builds. If we would move the environment specification to the files, I'd only make one package but I'd have to add the OS requirement to every file.
    How do you handle the environment checks, anyway? My repo parser checks the environment requirements whenever a repo is loaded and skips all packages which don't meed the requirements. This way you won't see the Linux or Mac OS builds of fs2_open on Windows.
Currently I am checking the environment specifications while resolving the dependencies of a package. That solution will not work correctly but I didn't want to spend much time implementing it.
Simply skipping the packages might work but then you will run into issues if a mod depends on the package "Windows SSE" which will not be present on other platforms. By making packages available everywhere but restricting the files that are being installed you solve that issue.

I'm just wondering... how did you implement the dependency tracking? Can your resolver handle conflicts and cycles?
For discovering the dependencies of a package I simply search through all known modifications and select the packages that the given package depends on. I then use that information to build a dependency graph which can be used to get a sorted list of all dependencies of a package.
If the algorithm detects a cycle it throws an exception as that is a case that can not be handled by FSO. For now version conflicts are not handled at all but I'll implement that at a later point.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on September 19, 2014, 08:48:09 am
@m!m :
Simply skipping the packages might work but then you will run into issues if a mod depends on the package "Windows SSE" which will not be present on other platforms.
To me, that's only a case of broken dependency. It happens in every package management system. It's not the responsibility of the package management system developers to avoid this, it's the responsibility of packagers not to do silly things (in FreeSpace 2 modding world : don't do mods that depend exclusively on platform specific packages, inside or outside your own mod.)
I agree with ngld : environment specification is in the right place at the package level.
Anyway, just in case it happens, we should warn the user at some point, and tell them to report the issue to the package manager for the broken mod.

Regarding the description/notes issue on server side : I chose not to add the possibility to override the description and other fields of mod metadata at the build level. Instead, I've added a "notes" fields to the branch and build. They will be appended to mod notes.

@ngld :
JS API on the web pages sounds cool.
Showing mod hierarchy is useful for the advanced UI (that we currently have), but I think the simple UI mod browser should present all mods on the same level. It's irrelevant to a newcomer who's just seen a youtube video of ST:R and who wants play it to know it's a submod of fsport (of course, this information will be shown before starting the install process).

To me, the simple UI should look like :
Code: [Select]
-------------------------------------
                      Quicksearch
-------------------------------------
Local mods
-------------------------------------
Mod            |play|settings|remove
-------------------------------------
Mod 2          |play|settings|remove
-------------------------------------
Available mods
-------------------------------------
Mod 3                       |install
-------------------------------------
Mod 4                       |install

-Every mod has its own background image (specific to the collapsed view, meaning modders should provide it as well).
-When you click on a mod title, you get a mod page adapted for a narrow view.
-Actions like "play"/"settings"/"remove"/"install" should only appear on mod mouseover.
-When internet connection is not available, instead of "Available mods", it shows "Go online to browse available mods !"
-When you're on a mod page, the quicksearch block should be replaced by a "Back to mods" button aligned on the left.
-If the "quicksearch" block is native Qt and not a part of the webview, to the right of the search box, we should have a button to switch to the advanced UI.

I'm not sure what the mods list cache management policy should be, but installed mods list and pages should be available while offline.
I'll do the templates if you want.

EDIT : Sf Voters are now working, meaning modders can only work in mods and branches they own.
I'm looking into adding modder teams that could also own mods (effectively allowing multiple modders to work on a single mod). The best way to do it is to add an "owner" abstraction layer, with the owner being either a "modder" or a "team of modders" object. This is a major change (since a lot of my functions revolve around the modder object), but I have to do it now, or it will turn into a massive refactoring later.

EDIT 2 : Implementing the "owner" logic (with modders and teams) will delay the initial git commit (it won't be on sunday, but early next week). I'll keep you posted.

EDIT 3 : @ngld : is JS API what https://extensions.gnome.org/ uses to know if you are running gnome-shell or not ?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on September 20, 2014, 03:11:05 pm
Um... this post wasn't supposed to be this big.  :nervous:

Replies

EDIT 3 : @ngld : is JS API what https://extensions.gnome.org/ uses to know if you are running gnome-shell or not ?
Yes. gnome-shell adds an object called "SweetTooth" which the site uses to interact with it. I'll add a "fs2mod" object.

@m!m :
Simply skipping the packages might work but then you will run into issues if a mod depends on the package "Windows SSE" which will not be present on other platforms.
To me, that's only a case of broken dependency. It happens in every package management system. It's not the responsibility of the package management system developers to avoid this, it's the responsibility of packagers not to do silly things (in FreeSpace 2 modding world : don't do mods that depend exclusively on platform specific packages, inside or outside your own mod.)
I agree with ngld : environment specification is in the right place at the package level.
Anyway, just in case it happens, we should warn the user at some point, and tell them to report the issue to the package manager for the broken mod.
fs2mod-py would complain with "Mod Windows SSE wasn't found!" if you tried to install a mod that depends on it on any OS other than Windows (or on a processor that doesn't support SSE instructions). The message is only logged, right now, but I'll add a message box for the user.
If a mod depends on a package that isn't available on your platform, you won't be able to install it since you can't expect it to run without that dependency.

I'm not sure what the mods list cache management policy should be, but installed mods list and pages should be available while offline.
I'll do the templates if you want.
I'm currently writing and testing the JS API by implementing the simple mod list you mentioned. It's using the JS API to retrieve fs2mod-py's mod list so that shouldn't be a problem.
I'm wondering whether this interface should be stored locally (static files in the script's folder) or served by a server and saved in the application cache (https://developer.mozilla.org/en-US/docs/Web/HTML/Using_the_application_cache)...
If we'd serve the interface from a server we could change it without having to update the fs2mod-py client.


JS API

Here's an early draft of the JS API:

Here's an example of what you could do with it:
Code: [Select]
<script>
$(function () {
  var list = $('<ul>').appendTo('body');
  $.each(fs2mod.getInstalledMods(), function (i, mod) {
    var row = $('<li>').appendTo(list);
    row.append(mod.title);
    row.append($('<a href="#">Run</a>').click(function (e) {
      e.preventDefault();
      fs2mod.runMod(mod.mid);
    }));
    row.append(' | ').append($('<a href="#">Settings</a>').click(function (e) {
      e.preventDefault();
      fs2mod.showSettings(mod.mid);
    }));
  });
});
</script>


Recent changes

Recent changes I've done:

@Hellzed: Whenever I think of emebedding your nebula web interface in fs2mod-py, I think of Steam. I'd like to do something similiar. (Not a simplified UI but the full web interface with Install / Play / Settings buttons.)
So... any thoughts?

Once I'm done with the JS bridge / local html interface, I'll make a release.
Current TODO list:

The auto-update feature is going to be a PITA. There will be ~5 ways a user can install fs2mod-py: Use git, download the Windows EXE, use pip, download an archive from github, use a Linux bundle.
The update method probably will be different for every single case.

P.S.: @m!m: Sorry, missed this post earlier.
EDIT 11/09/2014 : Progress ! Most of the mod and branch storage logic is done. I still have to refactor my first draft of the build storage logic and integrate it with the rest, polish the forms a bit, and install a third party account management bundle. After these steps we'll be ready for a first test. Expect this to be in something like 10 days.
:yes: That's great news!

I have been looking at the installation process a bit and I noticed that there is no way of specifying the folder of the package file that should be extracted. New mods should have all files in the root folder of the zip file but to support mods that were created before that we need some way of specifying the folder name (for example "MediaVPs_2014" would be the value used for the MediaVPs 2014 package).
Isn't that the purpose of the "dest" property? If an archive contains the MediaVPs_2014 folder, then you simply set dest to "". If it contains the VP files, you set it to "MediaVPs_2014". Or did you mean something else?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on September 21, 2014, 01:12:45 am
fs2mod-py would complain with "Mod Windows SSE wasn't found!" if you tried to install a mod that depends on it on any OS other than Windows (or on a processor that doesn't support SSE instructions). The message is only logged, right now, but I'll add a message box for the user.
If a mod depends on a package that isn't available on your platform, you won't be able to install it since you can't expect it to run without that dependency.
Ok, I'll just remove the packages that don't match when I initially parse the repo information.

P.S.: @m!m: Sorry, missed this post earlier.
EDIT 11/09/2014 : Progress ! Most of the mod and branch storage logic is done. I still have to refactor my first draft of the build storage logic and integrate it with the rest, polish the forms a bit, and install a third party account management bundle. After these steps we'll be ready for a first test. Expect this to be in something like 10 days.
:yes: That's great news!

I have been looking at the installation process a bit and I noticed that there is no way of specifying the folder of the package file that should be extracted. New mods should have all files in the root folder of the zip file but to support mods that were created before that we need some way of specifying the folder name (for example "MediaVPs_2014" would be the value used for the MediaVPs 2014 package).
Isn't that the purpose of the "dest" property? If an archive contains the MediaVPs_2014 folder, then you simply set dest to "". If it contains the VP files, you set it to "MediaVPs_2014". Or did you mean something else?
I meant that there should be some way of specifying which folder inside the archive contains the files. Currently the mediavp archived all have their files in the MediaVPs_2014 folder. My implementation of the installation doesn't install the files into your FreeSpace directory but instead into a directory "mod/<id>/<version>". This makes sure that there are now conflicts between two versions of the same mod. That is currently not possible as some archives don't have the files in the root directory of the archive. You could upload a new version of the archive but uploading multiple GB just for moving files in an archive doesn't seem like a good idea.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on September 21, 2014, 10:08:12 am
I see two solutions to this problem:

Add a file map which allows you to specify paths inside the archive and where their contents should be placed.
Code: [Select]
{
    "filename": "<filename>", // required
    "is_archive": true, // optional, default: true
    "dest": "<destination path>", // optional, default: ""
    "urls": ["<url1>", "<url2>"], // required
    "file_map": { // optional
        "Media_VPs": ""
    }
}

Add a "move" action:
Code: [Select]
"actions": [
    {
        "type": "move",
        "files": ["Media_VPs/*"],
        "dest": ".",
        "glob": true
    },
    {
        "type": "delete",
        "files": ["Media_VPs"]
    }
]
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on September 21, 2014, 04:43:37 pm
I like adding the move action. That feature can also be used for other things so it's not something that can only be used in specific cases.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on September 21, 2014, 05:09:01 pm
Updated the schema (https://github.com/ngld/fs2mod-py/blob/packager/converter/schema.txt) once again. I've added the move action and documented the delete action.

The move action is also useful if you want to replace a single file. You could just add the patched file to the downloads and add something like this:
Code: [Select]
"actions": {
    "type": "move",
    "paths": ["example.vp.patched"],
    "dest": "example.vp"
}
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on September 21, 2014, 07:07:13 pm
Quote from: ngld
@Hellzed: Whenever I think of emebedding your nebula web interface in fs2mod-py, I think of Steam. I'd like to do something similiar. (Not a simplified UI but the full web interface with Install / Play / Settings buttons.)
Providing the same level of functionality in the client and on the website would be good indeed. I'm just worried the client would feel a bit cluttered if it features the whole website UI without any adaptations. I kinda liked the "lighter" feeling of a smaller window retaining the "launcher" spirit, where the essential information (mods that are installed and ready to play) is immediately available.
Compared to the full website, the only things missing from the simple UI would be the advanced mod search and the current highlights (this one can be added easily).
I'll think again about it, my mind is not set yet.

Anyway the more I think about, the more I like the idea of using JS API.

Today has been a good day :
- The base logic for modder teams is working, with single user and teams both being considered valid owners for a mod. (This feature will be disabled on first release because I'd have to write a simple team control panel, and this is not the current priority).
- I have made mod/branch/build removal safer, with better confirmation pages.
- Some small fixes went into the mod/branch forms.
- Most secondary objects have been re-added to the database structure. I still have to do the "package" objects, though.

What's left to do :
- Re-add packages, and fix their subforms.
- Render fs2mod-py usable JSON.
- Make registration invite-only.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on September 22, 2014, 02:09:09 pm
I'm trying to get rid of the FS2 tab which looks kind of awkward IMO. That leaves us with three tabs: The mod tree, FSO settings and fs2mod-py settings.
I'll add a "nebula / steam" mode which opens the web view by default. That mode will have three buttons above the web view, one for each tab. If you click on one of those buttons, the current window is opened and the appropriate tab is selected.
If you use the "traditional" mode, it will open the current window and offer a button to open the nebula window. The mode can be changed in fs2mod-py's settings at any time.
I'll keep the "simple mod list" in both modes and make it adapt to the different screen sizes.

What's left to do :
- Re-add packages, and fix their subforms.
- Render fs2mod-py usable JSON.
- Make registration invite-only.

Yay! Any idea on when you're going to release it?  :D
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on September 22, 2014, 05:38:43 pm
Really soon ! :p

Packages (and package sub-objects) are now correctly attached to their parent build. But forms are a mess with no validation (except if you consider "qsfdcqdqzddsc" being a valid server mirror). I'm currently writing some validation rules. Next, JS fixes !
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on September 23, 2014, 03:53:44 am
I am also making good progress with my implementation. Downloading and installing a mod works fine but now I need to implement the handling of installed mods and I would also like to add some JSON caching.

Just to make sure we don't do too much duplicated work, what does fs2mod-py actually do? Is it a full FSO launcher or is it just the package installer?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on September 23, 2014, 07:17:52 am
@m!m : Both a package installer and a launcher !

@ngld : here's a small change request, could we change this :
Code: [Select]
"environment": [ // optional
{
"type": "cpu_feature", // required
"feature": "<SSE|SSE2|AVX|...>" // required if type == "cpu_feature"
},
{
"type": "os", // required
"os": "<windows|linux|macos>" // required if type == "os"
}
into this :
Code: [Select]
"environment": [ // optional
{
"type": "cpu_feature", // required
"value": "<SSE|SSE2|AVX|...>" // required
{
"type": "os", // required
"value": "<windows|linux|macos>" // required
}

To me, these environment vars are always type:value pairs, so it makes sense to always store the value as the same property (value) instead of having 2 properties (os and feature).

Also, I think the "dependencies" field is missing the "packages" list. (your post http://www.hard-light.net/forums/index.php?topic=86364.msg1761165#msg1761165 )

(And yes, If I'm asking these questions, that means I'm overhauling the last bits of the old storage logic)
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on September 23, 2014, 07:19:56 am
@m!m: It's a full FSO launcher and package installer. It can even install FS2 from GOG's installer (it's the same method Goober's installer uses because Hellzed developed that feature).
You can change FSO's settings and cmdline flags. fs2mod-py saves a global ("default") cmdline which every mod uses by default and can be changed by the user. You can also set cmdline flags for a specific mod, then that cmdline will be used instead of the global default.
It can reliably detect installed mods using the repo information and by checking the files in a user's FS2 folder.
Are you planning to implement somehting like that, too? It's really useful if you already have a big FS2 installation and you're using fs2mod-py for the first time.

@Hellzed: Sure. Any objections, m!m?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on September 23, 2014, 07:38:53 am
@m!m: It's a full FSO launcher and package installer. It can even install FS2 from GOG's installer (it's the same method Goober's installer uses because Hellzed developed that feature).
You can change FSO's settings and cmdline flags. fs2mod-py saves a global ("default") cmdline which every mod uses by default and can be changed by the user. You can also set cmdline flags for a specific mod, then that cmdline will be used instead of the global default.
It can reliably detect installed mods using the repo information and by checking the files in a user's FS2 folder.
Are you planning to implement somehting like that, too? It's really useful if you already have a big FS2 installation and you're using fs2mod-py for the first time.
I am doing the smae thing. Currently is is a standard launcher with multiple profiles and support for multiple game roots. I want to implement a new type of mod management using this package management.
My plan is to not install mods like they are installed now. Instead my installer will install them into subfolders according to their Ids and version. By doing that mod.ini files are not needed anymore and having two mods that depend on different versions of the same mod doesn't cause major problems anymore.

@Hellzed: Sure. Any objections, m!m?
No, I will have to change my enum handling but that should simplify my JSON classes.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on September 23, 2014, 08:58:29 am
Also, I think the "dependencies" field is missing the "packages" list. (your post http://www.hard-light.net/forums/index.php?topic=86364.msg1761165#msg1761165 )
Nice catch! Fixed it.

I added some comments to the paths in  the schema (https://github.com/ngld/fs2mod-py/blob/packager/converter/schema.txt) ("folder" for mods, "dest" for files, "paths" and "dest" for actions).

@m!m: Is this fine or does it break your launcher?
As far as I understood it, you just have to force the mod's "folder" property to "mod/<id>/<version>" and get exactly the paths you wanted.

@Hellzed: I hope I didn't break anything in the Nebula with this change.  :nervous:
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on September 23, 2014, 09:15:47 am
Actually, your comments match exactly the behaviour I expected from fs2mod-py, except for the mod "folder" defaulting to mod title. Shouldn't it be the mod id instead ?
As I understand it, the mod id is a url friendly canonical string (no spaces, no special chars...), while mod title is a full utf-8 compatible string. For example, for a mod with title "Blue Planet : Age of Aquarius", id is "blueplanet".

EDIT : A few months ago, someone said fs2mod-py was not a "catchy" name. We're all used to it now, but we must admit it's true :p
The "Nebula" is a big reference to FreeSpace 2, as this is where the campaign really takes off, and has a "cloud" feeling, as this is a server side software... What do you think about "Nereid", a tribute to the GTD Nereid (the GTD Orion that should have ended the war in Capella), or "Nephthys" (Vasudan support ship) ?
I think Nebula/Nereid would convey a sense of integration between both tools...
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on September 23, 2014, 09:43:04 am
Well, the mod ID can be pretty much anything and isn't shown to the user. It's main purpose is to uniquely identify a mod (in my first version it was supposed to be a UUID).
I'll add your explanation to the schema and change the comment for the mod folder.

EDIT: @Hellzed: "Knossos" would also fit since it's the portal that leads into the nebula. Nereid sounds nicer, though, and ties better with Nebula.

@m!m: You know that your path change will most likely break every single mod that uses cf.something functions in its LUA scripts, don't you?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on September 23, 2014, 08:48:31 pm
Small things left to do :
- last bits of form validation (mostly matching relative paths)
- recycle the old form theme, add it to the new forms (biggest chunk left actually).
- clean the JSON builder, make sure Nebula doesn't send NULLs to fs2mod-py (Nereid ? Knossos sounded cool too)
- copy-paste the "invite only" script into the security bundle.
- add licensing, write a quick start configuration guide (there's a fair bit of configuration so that NebulaBundle and FOSUB are registered by Symfony).
- run the converter from php to generate final json, and cache the result

Then, it goes straight to github ! Probably on tuesday. We should have a server running this week-end.

I'll have plenty of room for improvement :
- public activity tracking, for modders, teams, mods, branches...
- show more info about the objects (fill the "details" tabs)
- fix the object counts (always showing how many mods, branches, builds a modder has)
- write Nebula first level pages, like "Getting started", and add the modders, mods and activity common public lists.
- optimise DB queries
- activate modding teams
- start working on the "public" part of the website (mod public pages, default branch and build selection).

[attachment kidnapped by pirates]
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on September 24, 2014, 01:03:52 pm
@ngld : how strict are we regarding the relative paths ("folder", "dest"...) ? what about trailing slashes ? (I know an initial slash would be totally wrong for a relative path, but what happens if fs2mod-py gets one ? Should Nebula quietly remove these slashes while saving data, or fail to validate and give a warning ?)

I also figured that changing a mod/branch unique IDs would break dependencies in cached generated files, so I'll lock this field on mod/branch edit and add a warning on mod/branch creation.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on September 24, 2014, 05:00:32 pm
Trailing slashes actually don't matter. You can strip them if you want. You should strip the leading slash (if there is one) because it makes no sense (just as you said). A warning shouldn't be necessary since it's pretty obvious that this is a relative path and that the software is going to interpret it like one.
I never did a security check on my repo parser. Right now, fs2mod-py installs all mods with a leading slash in their "dest" path in the root of your harddrive (i.e. C:\MediaVPs_2014).  :banghead: I'll fix this right away and see if I can find similar problems.

BTW, I merged the packager branch back into the develop branch (https://github.com/ngld/fs2mod-py/tree/develop) and will push all new commits on that branch. The name Nereid is already used by a webframework for python (http://nereid.readthedocs.org/en/develop/) so I guess Knossos would be a better choice...
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on September 25, 2014, 11:22:18 pm
To anyone wondering what happened to the "first release on tuesday", I noticed in the last tests that if the form was resent after failing to validate, the small piece of javascript powering the dynamic form would go completely mad (overwriting form fields "name" and "id" properties, compromising final mod integrity)...
Button placement has always been ugly with this script anyway.

So I had to rewrite it from scratch to handle more things on php side, and play better with form themes...
Good news : now, it works !

Last things to do : change the registration setting to "invitation only", licensing, uploading to github.

Edit : I followed Symfony guide instructions to upload the code to github, but i noticed some of the most used bundles follow different guidelines (they ship less bits of Symfony kernel).
https://github.com/Nepenthes/hlp-nebula
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on September 26, 2014, 11:28:14 pm
@ngld (there's no hurry to answer, I have a lot of other things to add to Nebula) :
What is the current state of converter.py ? (I read it has been updated 2 days ago)
When I try to run it,  I get :
Code: [Select]
INFO:Thread-4:util.download: Downloading "http://www.mediafire.com/?zmh14h7o2122
 444"...
Unhandled exception in thread started by
                                         Unhandled exception in thread started by Unhandled exception in thread started by
                                           Unhandled exception in thread started by

    Unhandled exception in thread started by


The file was :
Code: [Select]
{
    "mods":  [
        {
            "id": "bta-demo-copy",
            "title": "Between the Ashes : Mefistofele (copy)",
            "version": "1.1.0-demo+nebula_test",
            "description": "The BtA Team is proud to release this small 2 mission demo. Consider it more like a prequel to BtA: Slaves of Chaos. In Mefistofele, you'll fly with local criminals and smugglers as they seek to make a simple profit, but they might have gotten in over their heads this time.",
            "notes": "MOD NOTES :\\nThis is a copy of Between the Ashes : Mefistofele, a mod by Mjn.Mixael.\\n\\nBRANCH NOTES :\\nThis is a stable branch.\\n\\nBUILD NOTES :\\nLet's register a build to test the database !",
            "folder": "BtA Demo",
            "packages": [
                {
                    "name": "BaseInstall",
                    "notes": "Packages are the base elements of your mod. They can be \"required\", \"recommended\" or \"optional\".\r\nThis is one \"BaseInstall\" the bare minimum to run this mod.\r\n\r\n- \"Client requirements\" are designed to distribute builds of fs2_open : our client knows about the OS, architecture and instruction sets !\r\n- \"Dependencies\" can be other mods, other mod packages, and other packages inside the same mod.\r\n- \"Files\" are the archives the client will download to install your mod. They can have multiple mirrors.\r\n\r\nAll checksums will be calculated by a server side script. (but we don't do full hosting (yet?))",
                    "status": "required",
                    "dependencies": [
                        {
                            "id": "mediavps_3612",
                            "version": ">=3.6.12",
                            "packages": [
                                "core"
                            ]
                        }
                    ],
                    "files": [
                        {
                            "filename": "BTA_DEMO_Root.7z",
                            "is_archive": true,
                            "urls": [
                                "http://www.mediafire.com/?sxayb4bhxkd22vf",
                                "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_Root.7z"
                            ]
                        },
                        {
                            "filename": "BTA_DEMO_Assets1.7z",
                            "is_archive": true,
                            "urls": [
                                "http://www.mediafire.com/?zmh14h7o2122444",
                                "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_Assets1.7z"
                            ]
                        },
                        {
                            "filename": "BTA_DEMO_Assets2.7z",
                            "is_archive": true,
                            "urls": [
                                "http://www.mediafire.com/?sfxkg4zqccdod6i",
                                "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_Assets2.7z"
                            ]
                        },
                        {
                            "filename": "BTA_DEMO_Patch_1_1.vp",
                            "is_archive": false,
                            "urls": [
                                "http://www.mediafire.com/download/cm0uu5qt9fww3n4/BTA_DEMO_Patch_1_1.vp",
                                "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_Patch_1_1.vp"
                            ]
                        }
                    ]
                },
                {
                    "name": "LowEnd",
                    "notes": "Use this package if you have a low end computer.",
                    "status": "optional",
                    "files": [
                        {
                            "filename": "BTA_DEMO_00LowEnd.7z",
                            "is_archive": true,
                            "urls": [
                                "http://www.mediafire.com/?7rsur95dvv6n07e",
                                "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_00LowEnd.7z"
                            ]
                        }
                    ]
                }
            ]
        }
    ]
}

Any idea what's going on ?


EDIT 27/09 : I've just finished implementing build cloning (php has a useful "clone" keyword to trigger specific code on object copy, but it doesn't apply to sub-objects).
So this was mostly a matter adding a __clone() method to the build object and its sub-objects, to make sure the whole tree gets cloned and we don't accidentally update sub-objects that belong to the original object (that should stay unchanged).

It will be useful in two cases :
- when modders push a small update and they want to keep content from a previous build, they can now click "Copy and update" on any existing build.
- there will be a possibility to copy a build from one branch to another. Lets say the last release candidate in the unstable branch was good enough, it can be copied to the stable branch under a stable version number.

It also helps enforcing semver since a published build can never be edited. Instead, clones are edited, and then published under a new version number.

EDIT 28/09 : I will definitely clear the current github repository, and upload everything again in a better way (easier to distribute/install with composer).
Now, mods/branches/builds counters work fine. I also have an early version of dependency autocompletion for mods.

EDIT 29/09 : Small changes in forms :
- added missing server side validation constraints and slash trimming (mostly on path lists)
- added more client side validation, for HTML5 compatible browsers.
- mod ID and branch ID are now locked after mod or branch creation (no more risk of breaking deps.)
- mod unique ID has now to be unique *across the whole mod table* instead of *for a single owner*, for easier administration of public mod access
- more help text in the build form.
- autocompletion on dependency mod id.

@ngld : what is the exact syntax of the "environment" item ? I don't see the architecture flag, only "os" and "cpu_feature". If the syntax is strict and small enough, i can handle that with dynamic lists in the form.
(I've just seen the converter fix on github ! thanks !)
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on September 29, 2014, 05:42:17 pm
I'm not sure how you ran the converter but somehow your PHP script didn't capture the whole output (the stacktrace is missing).
What do you think about this thing? (http://s1.nebula.tproxy.de/converter)  :D
During my tests I found two errors:

Your progress sounds great! I had a look at the PHP code in your GitHub repo but I didn't have a chance to test it (yet).
However there are two things I want to add: Real-time display of the converter's progress (see the link above) and uploading. Since direct-link hosting (for free) is rare, I want to allow modders to upload the files directly to the Nebula.

Quote from: Hellzed
@ngld : what is the exact syntax of the "environment" item ? I don't see the architecture flag, only "os" and "cpu_feature". If the syntax is strict and small enough, i can handle that with dynamic lists in the form.

Here's the relavant section from the schema.txt (https://github.com/ngld/knossos/blob/develop/converter/schema.txt):
Code: [Select]
"environment": [ // optional
    {
        "type": "cpu_feature", // required
        "value": "<SSE|SSE2|AVX|...>" // required
    },
    {
        "type": "os", // required
        "value": "<windows|linux|macos>" // required
    },
    ...
],

There's no architecture flag, yet, but I can add it (I'm assuming you mean x86_64 / x86_32). I'm also thinking about extending this. In the current version you only see the builds which fit your platform. However, if you're using Windows and have a processor which supports SSE2, you'll see both the SSE and SSE2 builds. If we extend the logic for the environment section (i.e. add a "invert": true property), we can fix that.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on September 29, 2014, 06:42:29 pm
I'm not sure how you ran the converter but somehow your PHP script didn't capture the whole output (the stacktrace is missing).
I actually ran it from the command line, and it became unresponsive. No way to get the output past that point. I haven't tried the new version yet.

What do you think about this thing? (http://s1.nebula.tproxy.de/converter)  :D
During my tests I found two errors:
  • You used mediafire.com links. Knossos needs direct links (i.e. if you access them with your browser, it will prompt you to save the file). I'll add proper warnings for this case.
  • There was a bug in my JSON generation code, it's fixed now.

I currently can't do a full test of the online converter, but I did a quick test :
- first one failed because my json was corrupted by an annoying bug in Nebula (when there's only one url for a file, the "urls" field becomes an object with a single key named "1" instead of a list...)
- second one went better, with actual md5 generation.

Code: [Select]
INFO:MainThread:__init__.main: Parsing repo...
INFO:MainThread:__init__.main: Updating checksums...
INFO:Thread-4:util.download: Downloading "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_Assets2.7z"...
INFO:Thread-2:util.download: Downloading "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_Assets1.7z"...
INFO:Thread-3:util.download: Downloading "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_Patch_1_1.vp"...
INFO:Thread-1:util.download: Downloading "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_Root.7z"...
INFO:Thread-5:util.download: Downloading "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_00LowEnd.7z"...
ERROR:Thread-3:util.download: Failed to load "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_Patch_1_1.vp"!
Traceback (most recent call last):
  File "./knossos/lib/util.py", line 272, in download
    result = urlopen(Request(link, headers=headers))
  File "/usr/lib/python3.4/urllib/request.py", line 153, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.4/urllib/request.py", line 461, in open
    response = meth(req, response)
  File "/usr/lib/python3.4/urllib/request.py", line 571, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.4/urllib/request.py", line 499, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 433, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 579, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

BTA_DEMO_Patch_1_1.vp actually gives a 404.

In both case, it didn't really differentiate success from failure. Is it possible to show less python ouptput, and better error messages in case of failure, at the end of the script ?
(and make the application that called the script aware of what happened ? Build objects in Nebula have two boolean properties : isReady and isFailed. Both are false when a new build is created, and the build is marked as "validating" in the UI. I'm planning to use converter return value/output to change these vars to the right state.)
Anyway, awesome work ! Can't wait to see this page integrated into Nebula !

Your progress sounds great! I had a look at the PHP code in your GitHub repo but I didn't have a chance to test it (yet).
However there are two things I want to add: Real-time display of the converter's progress (see the link above) and uploading. Since direct-link hosting (for free) is rare, I want to allow modders to upload the files directly to the Nebula.
I was thinking about uploading too, but it will add some complexity (I can take a "snapshot" of a build to enforce semver and duplicate a few text entries, but doing so with files would be terribly inefficient. I have some ideas to optimise it, though.)
We should focus on getting a first version online without the file upload system, then it will become my #1 priority.
I was planning to use screenshots and background images uploads as a benchmark, to know exactly what to expect with file uploads.


Quote from: Hellzed
@ngld : what is the exact syntax of the "environment" item ? I don't see the architecture flag, only "os" and "cpu_feature". If the syntax is strict and small enough, i can handle that with dynamic lists in the form.

Here's the relavant section from the schema.txt (https://github.com/ngld/knossos/blob/develop/converter/schema.txt):
Code: [Select]
"environment": [ // optional
    {
        "type": "cpu_feature", // required
        "value": "<SSE|SSE2|AVX|...>" // required
    },
    {
        "type": "os", // required
        "value": "<windows|linux|macos>" // required
    },
    ...
],

There's no architecture flag, yet, but I can add it (I'm assuming you mean x86_64 / x86_32). I'm also thinking about extending this. In the current version you only see the builds which fit your platform. However, if you're using Windows and have a processor which supports SSE2, you'll see both the SSE and SSE2 builds. If we extend the logic for the environment section (i.e. add a "invert": true property), we can fix that.
Good ideas indeed. I'll adapt the forms when you make the changes in Knossos.


EDIT : THAT FEELS GOOD !!!
(JSON corruption bug fixed. Not an easy one :
- finding the pattern : corruption only happened if a value was deleted from a list in the form (files urls, actions paths, dependency packages...) AND this value was *not the last* in the list AND the build was a clone from a previous build.

- what actually happened : cloning was causing items stored in lists (files urls, action paths, dependency packages) to keep their "keys" in the php arrays.
Then, when lists were stored as MySQL arrays in the database. When all keys were numbers following each other, beginning at 0, MySQL would detect a simple ("value", "value"...) array. When keys were not following each others or not starting at 0, MySQL would detect a ("key"=>"value" , "key => "value"...) arrray.
All of this would go undetected unless you are watching every entry of the database.

When it all arrives at the JSON parser, ("value", "value"...) arrays become JSON arrays.  ("key"=>"value" , "key => "value"...) become objects.
Hence the {"1": "url"} issue, instead of ["url"].

Fix was actually simple : use array_values() on lists I get from the form, to ensure arrays stay simple ("value") arrays to MySQL.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on September 30, 2014, 04:26:28 am
I cloned your repository and installed the composer dependencies but I have been unable to properly test it.
What are the steps to get it to run? I have no experience with Symfony so that might be an issue.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on September 30, 2014, 07:23:25 am
Yes, there's a bunch of things to do to get it running. You don't really want to run this version, though, because I'll do a full upload again. Instead of shipping a full Symfony project, I'd like to ship the HLP/NebulaBundle/* part only, and add a small install script. I've also changed A LOT of things in the last 3 days.

On your local install, you should make sure that :
- you rename "app/config/parameters.yml.dist" to "app/config/parameters.yml"
- inside "app/config/parameters.yml" change these options to match your local mysql install:
Code: [Select]
database_host: 127.0.0.1
    database_port: null
    database_name: symfony
    database_user: root
    database_password: yourpassword
- from Symfony folder, run :
Code: [Select]
sudo chmod 777 -R app/cache
sudo chmod 777 -R app/logs
php app/console doctrine:database:create
php app/console doctrine:schema:update --force

If Symfony folder is at the root of your local web server, you can access pages from http://localhost/Symfony/web/app_dev.php/nebula/ .
Register as a new user from http://localhost/Symfony/web/app_dev.php/nebula/register/
Once you are connected, the main pages (get started, mods, modders...) are empty, but you can access your own mod repository with a link on the right side of the nav bar.
(no theme on FoSUserBundle pages yet, sorry !)

I can't remember of anything else. If you used composer to get the deps, it should run.
If it doesn't, that means some yml config file didn't make it to github :p

If you want to start reading the code, you should know a few things :
- src/HLP/NebulaBundle holds most Nebula source code
- Resources/config/routing.yml lists all Nebula urls
- Resources/views contains the html.twig templates
- Controller contains the php controllers : they get variables from the router ({variable} in routing.yml), and return a response (generated html view, json response, redirection...)
- Request contains the ParamConverters. these classes handle variables from the route just before Controllers can use them. They make sure routing variables have a corresponding database entry.
- JSONBuilder is self explanatory :p
- Entity contains entities and repositories : every object  specific to Nebula is in there. Repositories hold custom database queries.
- Forms are not actual forms, but php form builders : Symfony call these classes to build forms with my form templates.
- Security/Authorization/ModVoter is a class that makes sure you own the page. I call it directly from the templates to show "new/edit/buttons" buttons, and from the controllers before generating forms.
- Good to know : the method to create a new branch from scratch is actually in the *mod* controller. It's logical if you think about it :) (same goes for every sub-object)

There are a few things I don't like in the current code :
- pagination, because it loads all entities from the database, and only display the right slice. I had to chose between efficient object preloading in the ParamConverter and efficient pagination. This is a limitation of Doctrine2 (the database abstraction layer above MySQL), but can be fixed by writing some complex native MySQL query. I'll do it at some point.
- ParamConverters sometimes preload too much stuff to avoid making more db queries (loading subobject lists when not needed). That's a small optimization, and not my current priority.
- The ModVoter only checks if you own the repository, and gives you the authorization to add objects. Templates use this "add" authorization even to show "delete" and "edit" buttons. I'll add a bit more granularity to this logic, and make the authorization name clearer.

@ngld  : I'd like to take a look at the server side of the online converter. Could you send me the files ?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 02, 2014, 10:17:57 am
@ngld : Any idea on what could cause this : (i'm trying to run converter.py from the command line to prepare the generated json storage. I must definitely do something wrong, because if I copy and paste Nebula's output to the online converter, it works.)
Code: [Select]
python converter.py checksums repo.json generated.json
Code: [Select]
{
    "mods": [
        {
            "id": "fs2_open",
            "title": "FreeSpace Open engine",
            "version": "3.7.2-rc4",
            "packages": [
                {
                    "name": "linux32",
                    "status": "required",
                    "files": [
                        {
                            "filename": "fs2_open_3.7.2_RC4.tar.bz2",
                            "is_archive": true,
                            "urls": [
                                "http://swc.fs2downloads.com/builds/LINUX/fs2_open_3.7.2_RC4.tar.bz2"
                            ]
                        }
                    ]
                }
            ]
        }
    ]
}
Code: [Select]
INFO:MainThread:__init__.main: Parsing repo...
INFO:MainThread:__init__.main: Updating checksums...
Traceback (most recent call last):
  File "./converter.py", line 26, in <module>
    converter.main(sys.argv[1:])
  File "/home/loris/knossos-develop/converter/__init__.py", line 238, in main
    prg_wrap(core)
TypeError: 'bool' object is not callable

By the way, it would be nice if the converter could do without wrapping the data inside a "mods" tag. Server side, it will only run on one build at a time.
This way, i could do the whole repository wrapping in Nebula, on demand, depending on the url accessed by the client (that's the easiest way to implement urls that reference the "default" branch and "current" build, like /nebula/modder/mod/default/current.json).
Also I'd avoid manipulating json inside Nebula, only storing it.

I'll also have to check what's the best way to grab the json output and store it into the database.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on October 02, 2014, 02:55:21 pm
Uh, that was my fault. I accidentally added a bug to the converter when I implemented the web frontend. It's fixed now.

I've added a bunch of new checks to the converter (which should find most errors now) and simplified the output. The online converter now tells you if the conversion was a success ("Done!" message and a green "Continue" button) or a failure ("Failed!" and a red "Continue" button).

The online converter works like this:

If you look at the source code of the online converter (http://s1.nebula.tproxy.de/converter), you'll see that the JavaScript is doing exactly that. However, the two POST request are supposed to be handled by the server to make sure the API key is kept secret and because the JSON response has to be stored on the server anyway.

The source code is on GitHub (https://github.com/ngld/knossos-server/tree/develop). It's written in Python because it's easier to integrate the converter and handle WebSockets that way.
To setup the server, you'll have to run the following commands (assuming you have python3 and virtualenv installed):
Code: [Select]
git clone -b develop https://github.com/ngld/knossos-server
cd knossos-server
git submodule update --init
python3 /usr/bin/virtualenv py-env
source py-env/bin/activate
pip install -r requirements.txt
python setup_db.py

To run the server, you should run:
Code: [Select]
source py-env/bin/activate
uwsgi --master -p 4 --http localhost:8080 --http-websockets --wsgi server:app --enable-threads

I'll think about a way to solve the mods issue but I don't understand why you don't want to just change the JSON in PHP.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 02, 2014, 03:43:07 pm
Ok, I'll change the JSON in the PHP script ! No need to change anything on your side.
(to me it's only a matter of removing the "mods" container, concatenating the right builds of mods, and adding a new "mods" container. I only wanted to know if things could be made one step simpler on both our sides, but that's really a tiny detail.)

I'll start writing the new build validation page in Nebula tomorrow. Thanks to your work on Knossos-server, it should go pretty fast.

Have you tried running Nebula ?

EDIT :
I'm still pretty new to all these things ^^
I never needed before to send a request to an external web service from php. I've read i'll need to use cURL.

From the 3 steps you described, I get the impression that someone has to try and access the "build in progress" page in Nebula so that the second POST request is sent, and the generated data registered in the database.
What happens if the modder who started the validation process quits the page and doesn't come back ?
I can circumvent this issue in Nebula by accessing http://s1.nebula.tproxy.de/api/converter/retrieve whenever a mod known to be configured is requested but has no cached final JSON. (to do this, I only need to save tickets and tokens in the database and use them when needed, right ? I mean Knossos server doesn't loose track of mods if there's more than one at the same time ?)

I'm thinking of another solution : maybe Knossos server could call the right URL in Nebula once it's done, to trigger the database registration ?
To avoid the security risk of getting an arbitrary JSON entry in the database, we could either :
- use a simple public/private key mechanism.
- only accept requests coming from Knossos-server's domain.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on October 02, 2014, 05:53:26 pm
Sorry, I didn't get to test the Nebula but I'll probably have time tomorrow.

I've already thought of that problem but I didn't get to implement a solution (yet). My idea is that I'll add an optional field to the /api/converter/request endpoint called webhook. The Nebula server fills a URL in this field and once the conversion is finished, the Knossos Server will send a POST request to that URL containing the ticket ID. The Nebula can then access the retrieve endpoint to get the actual JSON. Since the JSON will be fetched from the Knossos Server, no one can insert malicious JSON. What do you think?
The Knossos server forgets a mod once the generated JSON is retrieved to avoid wasting space. I haven't found a better solution. There's also a problem with the mod logos: I haven't figured out a good way to include them in the result so I simply embedded them using data: URLs but there's something wrong with the base64 encoding which breaks the images...

Have you had a look at Guzzle (http://docs.guzzlephp.org/en/latest/)? It should make accessing the API much easier.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 02, 2014, 06:45:17 pm
Guzzle looks nice but i'll probably won't need it. I already have all the JSON and incoming request classes I need in Symfony.
Sending the requests is actually only a few lines of php with cURL, and looks like this (I've just tried it for the first request, it works) :
Code: [Select]
<?php
$url 
'http://s1.nebula.tproxy.de/api/converter/request';
$fields = array(
'passwd' => urlencode($passwd),  //$passwd is the API key
'data' => urlencode($data), //$data is the original JSON
);

//url-ify the data for the POST
foreach($fields as $key=>$value)
{
  
$fields_string .= $key.'='.$value.'&';
}
rtrim($fields_string'&');

//open Knossos-server connection
$ksconn curl_init();

//set the url, number of POST vars, POST data
curl_setopt($ksconn,CURLOPT_URL$url);
curl_setopt($ksconn,CURLOPT_POSTcount($fields));
curl_setopt($ksconn,CURLOPT_POSTFIELDS$fields_string);

//execute post
$result curl_exec($ksconn);

//close connection
curl_close($ksconn);

//show the raw JSON token+ticket response
echo $result;

?>

As it will happen only twice each time a mod is validated, I'll put that code in a Symfony "service" that can be called from anywhere in the controllers, and the urls settings in a Symfony YML config file...
I already did that with the custom JSONBuilder.

Webhook sounds good to me.

Mod logos will be handled by Nebula, probably as a special item in the mod image gallery (3 optional special images : the logo, the large background image for the mod page, the "collapsed" background image for Knossos mods list ; and of course the screenshots).
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on October 03, 2014, 10:20:31 am
I'd move the urlencode() (http://php.net/urlencode) call in the foreach loop. You can use parse_str() (http://php.net/parse_str) to parse the response into an array.
I've installed the Nebula on the server. You can reach it at http://nebula.tproxy.de/nebula and register at http://nebula.tproxy.de/nebula/register .

I installed it with Composer which was nice because the install script already created the app/config/parameters.yml for me. Everything else worked without problems.
I've found a small bug: The "Modders & Teams" tab (/nebula/owners) doesn't work because Symfony tries to call BaseController::ownersAction which doesn't exist. I guess the OwnerController's index method should be called instead?
I've got a couple of suggestions:

To make the integration of the online converter easier for you I'm writing a simple client. To display the progress page you could do something like this:
Code: [Select]
<div id="progress-container"></div>
<script type="text/javascript" src="http://s1.nebula.tproxy.de/static/converter.js"></script>
<script type="text/javascript">
var cv = new Converter('http://s1.nebula.tproxy.de/ws/converter', <?php echo json_encode($ticket); ?>);
cv.bootstrap_ui($('#progress-container'));
cv.on('done', function (success) {
  // Do something to tell the user that the conversion is finished.
});
cv.connect();
</script>

The JS client isn't finished, yet. I'll release it together with the new webhook feature once I'm done.

EDIT: Oh, I didn't know Hard-Light had syntax highlighting. That PHP code is reaaally legible.  :lol:
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 03, 2014, 12:50:16 pm
Nice job with the js client ! I was reaching the point where I would have needed to implement this !

Could you render the widget  (content of #progress-container) as :
Code: [Select]
      <h4><span class="glyphicon glyphicon-cog"></span> Progress</h4>
      <hr/>
      <div class="progress">
          <div id="master-pg" class="progress-bar progress-bar-success" style="width: 0%;"></div>
      </div>
      <pre id="pg-well"></pre>
      <br/>
      <h4><span class="glyphicon glyphicon-list"></span> Log</h4>
      <hr/>
      <pre id="logging"></pre>

I already have the template for this page, and with this it will blend perfectly with the theme.

Quote
Could you add a validation email to the registration process to make sure everyone registers with a real email address?
FoSUB has a simple switch to activate this, IIRC. But at least for the early tests, I'll make the registration "invite only" (and maybe keep it this way since FS2 modders are only a small number.). I have to copy a few templates from FoSUB wiki to do this, because this feature is not included in their code base.

Quote
Could you create a default branch whenever a modder creates a new mod and redirect to the "New build" page to make the process of adding a new mod easier and faster?
That's smart. I'll add this too, at the same time as the "default branch" logic.

Quote
The post-install actions are meant to give the package creator more control over where the extracted files are put.
This is necessary for some mods where the archive contains a folder (i.e. MediaVPs). The actions are a great way to change the file structure without having to change any of the archives.
Oh, ok ! I didn't knew that (I thought Knossos tried to guess the actual archive root, like my old sh script did, and this was only intended for post-install/update cleaning. I'll change the form help-text ).

Quote
Why is the unique branch URL empty?
As "default branch" and "default build" logics are not in yet, I can only generate URLs for a single build at a time.


I'll add a temporary fix for the owners list error. (the definitive fix will come when I'll finish implementing modders teams, the objects are in but there's no control panel yet).

I've also figured I'd have to refactor the count logic to make it a bit more efficient. According to Symfony and Doctrine devs, SQL COUNT()  should be avoided, and instead, a count field should be created in the parent object and updated whenever a sub-object is created or deleted.


EDIT : I thought about the player UI. I originally planned to do a dynamic "tags" system like the new Steam tags. Freespace has a lot of mods, but that's still a few dozens, not thousands like Steam games. The risk is a catalogue "fragmentation", where mods end up being alone in their too specific categories, making navigation harder and wasting UI space.
Instead, I'll do 2 completely separate things : a static list of categories (think netflix-like), and a list of custom search keywords for each mod. A mod could be a part of multiple categories (some of these will be taken from this page http://www.hard-light.net/wiki/index.php/Campaign_List, but I'll add a few others, a single dynamic "New !" category and a "virtual" category "Related mods", depending on the mod you are viewing).
Keywords will be used by the search engine to bring better results, and not directly shown to the end user.

@ngld : I noticed your test server is offline. did something wrong happen (critical bug, security issue, spam bot) ? I only gave the link to MjnMixael and Axem so they could see the forms and comment on the usability.
(I'll read how to fine tune Symfony to switch it to production mode)

EDIT 2 : I noticed you've made changes to the api, can I start sending requests ?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: pecenipicek on October 06, 2014, 01:09:02 am
All i have to say is good job guys :)





also, ew, symfony :p
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 06, 2014, 09:18:20 am
@pecenipicek : Thanks for your support !
By the way, I chose Symfony/PHP because I'm not even a real programmer, but there was a good MOOC about it on my usual e-learning website :P (IRL, I'm a political science/European history student, but I've been doing small stuff with php since I was in high school)
If you are worried about Symfony's performance, yes, it's slow in debug mode and on first generation of a page, but it will get much better in production mode, once the cache is warm.


Here's my current list of mod categories :
- Essentials : Media VPs, FSPort, Silent Threat : Reborn
- New : the latest 4 or 5 mods
- Highlights : mods matching some criteria like being voice-acted, latest Media VPs compatible...
- Classics : FSCRP mods
(next ones are from the wiki)
- Pre-Great War Era
- Great War Era
- Reconstruction Era
- Capella Era
- Post-Capella Era
- Alternate timelines/parallel universes
- Parodies
- Total conversions

Imagine a Netflix like UI : each category is a line, with a short (less than one line) presentation text. You can either scroll to the right or left to see more mods in the same category, or click the category title to get a full page of these mods.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: AdmiralRalwood on October 06, 2014, 03:42:23 pm
I'm not sure "Capella Era" and "Post-Capella Era" are the most spoiler-free category names... suppose a brand-new user, never having played FS2, buys it from GOG or Steam and decides to check out FSO with mods. Sure, they may only be grabbing the Essentials... but from one look at the category list, they might now spend the entire campaign waiting for something to happen to Capella.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: niffiwan on October 06, 2014, 04:22:28 pm
How about "2nd Incursion" and "Post 2nd Incursion"? Hardly any spoiler in that as you already expect the Shivans to be back :)
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: AdmiralRalwood on October 06, 2014, 04:38:25 pm
That sounds better than "FS2 Era" and "Post-FS2 Era", which was all my brain was suggesting.  :yes:
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 07, 2014, 10:12:36 am
I'm liking "Second Incursion" and "Post-Second Incursion" ! :)

Here are some early screenshots of the highlights and mod browsing pages. Sorry ngld, I haven't decided yet where to put Knossos download link in the navbar...

- The navbar needs a better nebula logo
- I used the default mod logos, so they are a bit stretched... I'll ask modders to upload better ones.
- I'll add a mouseover effect on the mod items, to show a few lines of their presentation text and the play button, and probably place the mod title on top of the mod image, with some transparency.
- The items are not huge, my screen is really small. Expect more items per line on bigger screens.


[attachment kidnapped by pirates]
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: pecenipicek on October 07, 2014, 05:29:14 pm
@pecenipicek : Thanks for your support !
By the way, I chose Symfony/PHP because I'm not even a real programmer, but there was a good MOOC about it on my usual e-learning website :P (IRL, I'm a political science/European history student, but I've been doing small stuff with php since I was in high school)
If you are worried about Symfony's performance, yes, it's slow in debug mode and on first generation of a page, but it will get much better in production mode, once the cache is warm.
i'm a php dev by trade these days apparently :p

my beef with symfony is that its this big massive beast which pretends to do everything right in the php world :p

and dont worry about "real programmer" stuff. i think that those days are gone.

i'm just curious, on what will you guys host this?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Goober5000 on October 07, 2014, 07:37:17 pm
and dont worry about "real programmer" stuff. i think that those days are gone.

None of us are Real Programmers anymore...
http://www.catb.org/jargon/html/story-of-mel.html
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 08, 2014, 10:31:20 am
@pecenipicek : I don't know yet. ngld is hosting the current test server, so we'll discuss this when we're ready.

@Goober5000 : Awesome story !

@ngld :
(http://pix.toile-libre.org/upload/thumb/1412782205.png) (http://pix.toile-libre.org/?img=1412782205.png)
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on October 08, 2014, 01:45:38 pm
The new API (with the webhook feature) and the JS client are now on GitHub and the test server. I'll later add some documentation for it to the GitHub wiki.

@pecenipicek: Are you asking about the hardware or software?
The server is a dedicated box. I'm running Apache, PHP (in FastCGI mode) and MySQL on it. The converter is written in Python and runs inside a uwsgi server.
HAProxy forwards all requests depending on the domain name (nebula.tproxy.de is forwarded to Apache and s1.nebula.tproxy.de is forwarded to uwsgi). It also handles idle connections to ease the load on the Apache server (similiar to what lingerd does).

@Hellzed: Looks nice! :) I'd suggest using "mod manager" instead of "modding client", though.
I'm working on the auto updater function for Knossos next.
I already have a working installer for windows, creating an updater should be easy for this case.
I'm not so sure about Linux and Mac OS.

EDIT: The whole API / schema documentation is now linked here: https://github.com/ngld/knossos/wiki
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 08, 2014, 08:10:24 pm
I found a small mistake in the documentation, the websocket address is wrong :
Code: [Select]
http://s1.nebula.tproxy.de/ws/converterit should be replaced by (like in the online converter, that replaces http and https by ws):
Code: [Select]
ws://s1.nebula.tproxy.de/ws/converter
I will try the webhook tomorrow.

EDIT: i have no idea how to distribute a software package for OS X.
Have you ever tried building .deb and .rpm packages ?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on October 09, 2014, 08:49:41 am
Yeah, I missed the ws protocol. Should be fixed now.

I'm not sure about OS X but you essentially have to distribute a "Knossos.app" folder which the user would copy to their Applications folder.
I've built deb and rpm packages before but it's been a while. I can write PKGBUILD files and will definitively add Knossos to Arch's AUR.

The windows installer and updater work now. I just have to add an update check to Knossos and the auto update feature will be done.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: pecenipicek on October 09, 2014, 12:10:45 pm
@pecenipicek: Are you asking about the hardware or software?
The server is a dedicated box. I'm running Apache, PHP (in FastCGI mode) and MySQL on it. The converter is written in Python and runs inside a uwsgi server.
HAProxy forwards all requests depending on the domain name (nebula.tproxy.de is forwarded to Apache and s1.nebula.tproxy.de is forwarded to uwsgi). It also handles idle connections to ease the load on the Apache server (similiar to what lingerd does).
Wouldnt nginx on its own be perfectly enough for all the "balancing"? also, if by FastCGI you dont mean PHP-FPM, you are in big trouble :p

(reading up on HAProxy, it looks like another one of those "ITS NEW SO IT MUST BE BETTER THAN ANYTHING THAT CAME BEFORE!" solution -.- i'm sure it does the job well, i'm just personally on the front of "keep the software stack as small as possible so you dont have to remember 14 different config conventions and at least 6 different 'languages'")


@Goob point well made :p
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on October 09, 2014, 12:22:22 pm
Wouldnt nginx on its own be perfectly enough for all the "balancing"? also, if by FastCGI you dont mean PHP-FPM, you are in big trouble :p

(reading up on HAProxy, it looks like another one of those "ITS NEW SO IT MUST BE BETTER THAN ANYTHING THAT CAME BEFORE!" solution -.-
Uh, HAProxy is pretty old actually... the first release was in 2001. I had to install it a while back because nginx didn't support WebSockets, yet. It does now but there's no point in replacing HAProxy with nginx.

And what's wrong with mod_fcgid ?  :wtf:
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 09, 2014, 12:26:52 pm
You might want to clean tmp files on the converter server, I made some tests where Nebula would fail to retrieve the data.

2 Questions :
- What happens if knossos-server request to Nebula (webhook case) fails ? Does it keep the data until retrieved manually ?
- What happens if knossos-server "retrieve" url is called prematurely and the json is not ready yet ? (I want to know if I have to check the conversion status before trying to get the data)
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on October 09, 2014, 12:31:59 pm

EDIT: Pushed the changes to GitHub and the test server.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 09, 2014, 01:41:59 pm
Just added my local Nebula repository to Knossos, downloaded a mod... It works perfectly ! :D

I'm going to polish some rough edges (templates, better error handling, make sure the uploaded logo is an image and of the right size) and push the new version to github.

EDIT : if I send an empty build, Knossos server gets stuck ("WARNING: Added an empty task of type "ChecksumTask". Ignoring it!"). The build is not marked as ready or failed.

EDIT : Looks like I have crashed Knossos-server.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on October 10, 2014, 06:06:38 am
The converter didn't expect to get a mod but no files.  :)  It will now complain with "No files found!" and mark the build as failed.
I also fixed a weird bug which caused the log messages to be displayed multiple times.

You didn't crash the server, it was busy handling your running (stuck) requests.  :lol:
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 10, 2014, 09:40:14 am
:D

Most of the small template and redirection changes (better errors, less duplicated HTML, build status reporting in the builds list and plenty of others...) are done. I have a list of tests to run to see if haven't broken anything.

Logo upload is working, but causing weird errors if I make them optional (a mod will need a logo to be published, but if a modder wants to run some test without having a logo, that should be allowed). I also have trouble generating urls for resources in the "web" directory (as logos), since they are not part of Nebula bundle file tree.
I may disable logo upload for tonight's update, and replace it later with a better solution.

I'm still not generating the "all.json" holding all mods, so that's currently one mod : one repo (but you can get the <mod>/<default branch>/<current build> url, so auto updates for a mod will work ! :) ). Modder teams and the global repo will be my work items for the next few days.

EDIT : It looks like i'm only getting the last line of log now (instead of adding new lines, they get replaced, so the log \<pre\> tag is never longer than one line). You may also check the "current task" field, my file download appeared 2 times.

Do you want me to add a specific response to the webhook if build processing has been cancelled by the user (there's an action in Nebula for that, called "forceFail"), so that it cleans data that no one needs ?

EDIT 2 : I've just made the change : If the webhook reaches a build that already has been marked as failed, Nebula answers with a {"cancelled": true} JSON response.
And I'm definitely seeing 2 times the same log entries.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on October 10, 2014, 10:43:28 am
Does it work fine if you use http://s1.nebula.tproxy.de/converter ?
Thanks for the "cancelled" webhook response! I'll implement it on the server.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 10, 2014, 11:06:05 am
Disregard what I just wrote, it works, but i'm still getting the double output issue.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 10, 2014, 04:52:57 pm
If anyone missed it on IRC, our test server has been updated.

http://nebula.tproxy.de/nebula
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on October 10, 2014, 07:24:55 pm
The converter issues should be resolved now.
A quick note to anyone who wants to test Knossos and uses Windows: You don't have to install Python or anything else. Knossos has its own installer (https://dev.tproxy.de/knossos/installer.exe).
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 11, 2014, 04:32:03 am
@ngld :
Is the converter aware of the final mod file tree ?

If it is, I have an idea for a final check : if the mod is shipping a mod.ini, we have to make sure it's placed at the root of the mod folder.
If we have a mod.ini somewhere, but not in the right place, something probably went wrong with the "move" actions, and we should mark the build as failed (or give a clear warning) since it won't run anyway.

Some packages shipping only fs2_open builds don't need a mod.ini, so a build that doesn't have a mod.ini anywhere in the file tree should still pass.

That logic is not perfect, but should help avoiding most human errors. What do you think ?

You may also want to take a look at what went wrong with BTA_Demo_1.1.2 ( http://nebula.tproxy.de/nebula/mjnmixael/BTA_Demo/master/1.1.2/show_data ). I'm not sure about
Code: [Select]
"dest": "" behaviour... It looks like the files ended up not being moved at all instead of being moved to the BtA root folder, because Mjn's input looks ok to me.
Am I wrong ?

@MjnMixael :
So even after manually moving everything to the the right folder, fs2_open trunk complains about :
Code: [Select]
ERROR: "Could not load in 2_CommandBrief-mb!" at ui/window.cpp:85Not sure whats going on.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on October 11, 2014, 05:59:39 am
I also tried testing the BtA data but something weird is going on.
A nice feature for the future would be more validation of the dependencies to make sure the server has valid dependency data.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on October 11, 2014, 07:09:15 am
@Hellzed: Yes, it is aware of the final mod file tree. Just look at the "filelist" property in the generated JSON (http://nebula.tproxy.de/nebula/mjnmixael/BTA_Demo/master/1.1.2/show_data_finalised). There you can see that everything was indeed moved successfully. If it's in the wrong folder on your hard drive, then it's a Knossos bug. I just tested it and everything went fine. (All files are placed in <FS2 dir>/BtA Demo/)
There's a bug in Nebula: The notes field contains "\\n" for line breaks but those should be "\n". Are you calling json_encode() for this string?

I could add a "WARNING: I found no mod.ini in your mod's install folder. Are you sure everything is where it should be?". Displaying the file tree is a good idea but I'm wondering whether we should display a simple list or an interactive tree (think of Windows' Explorer's sidebar).

@m!m: The data in /show_data is the converter input, your launcher should process the JSON file which should be the same as /show_data_finalised.
The dependency is added by Knossos' converter. I can probably remove it but I wanted to make sure that a required package depends on all other required packages of the same mod. Knossos handles cyclic dependencies without problems so it never caused issues.

The dependency validation sounds nice but I'm not sure where it should be handled. It would make the most sense to add this to the Nebula since the converter isn't aware of other mods. Should this also take care of conflicts and dependency cycles?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on October 11, 2014, 07:16:04 am
:banghead: I forgot to check that tab.
Adding the dependency seems fine but it shouldn't add a dependency that makes the package depend on itself. I would also suggest setting the version field in the dependency to the version of the mod that is currently being processed to make sure the dependency doesn't cause issues when there are multiple version of the same mod.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on October 11, 2014, 07:37:13 am
I'll change the version field but I can't avoid creating the dependency cycle because every dependency on a mod implicitly includes all required packages.

Did you take a look at the "filelist" property? It basically tells the launcher how an installed mod should look. It makes checking installed mods much easier since it already takes the post-install actions into account.
I'm thinking about adding a "cmdline" property to the mod object. It would allow modders to specify a default command line for their mod. It's pretty much what wxLauncher already does (https://code.google.com/p/wxlauncher/wiki/ModIniOptions#%5Brecommendedlighting%5D) although in that case it's a mod.ini flag.

There's another small issue I'd like to resolve: On every platform you can select only the FSO build which fits you. The only exception is Windows on a SSE2 processor: There you can install FSO's SSE and SSE2 builds. I'd like to hide the SSE builds in that case. The best solution would be a change to the "environment" conditions. Adding a simple "not" flag or something like that should solve this.
What do you think?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on October 11, 2014, 08:30:07 am
I'll change the version field but I can't avoid creating the dependency cycle because every dependency on a mod implicitly includes all required packages.
Can't you just skip adding the dependency when the current package name is the same as the package name of the dependency? A cyclic dependency is a something that compromises the integrity of the dependency system which is why I classified it as a fatal error if such a cyclic dependency is found.

Did you take a look at the "filelist" property? It basically tells the launcher how an installed mod should look. It makes checking installed mods much easier since it already takes the post-install actions into account.
I'm thinking about adding a "cmdline" property to the mod object. It would allow modders to specify a default command line for their mod. It's pretty much what wxLauncher already does (https://code.google.com/p/wxlauncher/wiki/ModIniOptions#%5Brecommendedlighting%5D) although in that case it's a mod.ini flag.
Both are great ideas! A minor thing I noticed about the "filelist" is that is includes the files from all packages. How hard would it be to move that element to the "package" object and only include the files from that package in that list?
That would simplify the checking mechanism if the launcher/installer knows what packages should be installed. I guess I could implement the same logic in the installer but it might be easier to implement in the convert itself as it already has all the information that is needed.
"cmdline" would be nice but it would be better to have a generic "settings"-thing that would also handle things that are not configured via the commandline (e.g. TTS settings) but that is something for later, the "cmdline" element will be most useful for almost all other applications.

There's another small issue I'd like to resolve: On every platform you can select only the FSO build which fits you. The only exception is Windows on a SSE2 processor: There you can install FSO's SSE and SSE2 builds. I'd like to hide the SSE builds in that case. The best solution would be a change to the "environment" conditions. Adding a simple "not" flag or something like that should solve this.
What do you think?
I had the idea of somehow prioritizing a package when the "feature" value would be higher but the system you are proposing should work much better and will give the modder more control over how the installer processes the packages.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: mjn.mixael on October 11, 2014, 08:44:04 am
Quote
@MjnMixael :
So even after manually moving everything to the the right folder, fs2_open trunk complains about :
Code: [Select]
ERROR: "Could not load in 2_CommandBrief-mb!" at ui/window.cpp:85Not sure whats going on.

Well, the BtA Demo requires MediaVPs 3.6.12 which isn't in the installer (yet). That might be it? I just tested the demo data (same files Nebula downloads) on a recent build and it works fine. So let's make sure it's not because it needs 3.6.12 and/or make sure Nebula installed everything properly.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on October 11, 2014, 08:58:11 am
I'll change the version field but I can't avoid creating the dependency cycle because every dependency on a mod implicitly includes all required packages.
Can't you just skip adding the dependency when the current package name is the same as the package name of the dependency?
I don't quite understand what you're talking about. The JSON we are talking about is this:
Code: [Select]
{
    "packages": [],
    "version": "==1.1.2",
    "id": "BTA_Demo"
}
I guess I'll simply remove this dependency and leave it to the launcher to handle this case.

I'll move the "filelist" property to the package object, you're right that makes more sense. I guess I can remove the "contents" property, now, since it's redundant.
I'll probably rewrite Knossos' dependency resolution algorithm. How do you treat version conflicts? Let's say mod A depends on mods B and C. Mod B needs version 1.0.0 of mod D and mod C needs version 2.0.0 of mod D. This wouldn't work since FSO can't load two versions of the same mod.
However, installing mod B and mod C should work since they're not loaded together.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on October 11, 2014, 09:05:33 am
:banghead: I am such an idiot :banghead:
You are absolutely right, the error I am seeing is that I blindly add all required packages to a dependency if no package is specified which will result in one package depending on itself it is a required package.

A version conflict would result in an error in my implementation as FSO can't load that. A resolution would be to let the user decide what version to use but that would only be a troubleshooting solution. Ideally this would be handled by Nebula (another thing for the "mod dependency validation"-list :p) so a modder can't even create a mod that would cause this issue.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 11, 2014, 09:31:01 am
@ngld :
I checked again, there is a problem which causes the data to stay in BtA Demo/BtA Demo :
Code: [Select]
DEBUG:MainThread:qt.<module>: Using Qt API PySide.
DEBUG:MainThread:util.call: Running ['7z', '-h']
INFO:Thread-3:util.get: Retrieving "http://dev.tproxy.de/knossos/version"...
WARNING:Thread-2:tasks.work: File "BTA_DEMO_Patch_1_1.7z" for mod "Between the Ashes: Mefistofele" (BTA_Demo) is missing during uninstall!
WARNING:Thread-2:tasks.work: File "BtA Demo/mod.ini" for mod "Between the Ashes: Mefistofele" (BTA_Demo) is missing during uninstall!
WARNING:Thread-2:tasks.work: File "BtA Demo/BTA_DEMO_Root.vp" for mod "Between the Ashes: Mefistofele" (BTA_Demo) is missing during uninstall!
WARNING:Thread-2:tasks.work: File "BtA Demo/BTA_DEMO_Assets2.vp" for mod "Between the Ashes: Mefistofele" (BTA_Demo) is missing during uninstall!
WARNING:Thread-2:tasks.work: File "BtA Demo/BTA_DEMO_Assets1.vp" for mod "Between the Ashes: Mefistofele" (BTA_Demo) is missing during uninstall!
WARNING:Thread-2:tasks.work: File "BtA Demo/bta.bmp" for mod "Between the Ashes: Mefistofele" (BTA_Demo) is missing during uninstall!
DEBUG:Thread-2:progress._next_step: Entering step 1 of 3 in task InstallTask.
DEBUG:Thread-7:progress._next_step: Entering step 2 of 3 in task InstallTask.
INFO:Thread-7:util.download: Downloading "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_Assets2.7z"...
DEBUG:Thread-7:util.call: Running ['7z', 'x', '-o/tmp/tmpZontl3/content', u'/tmp/tmpZontl3/BTA_DEMO_Assets2.7z']

7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=fr_FR.UTF-8,Utf16=on,HugeFiles=on,8 CPUs)

Processing archive: /tmp/tmpZontl3/BTA_DEMO_Assets2.7z

Extracting  BtA Demo/BTA_DEMO_Assets2.vp
Extracting  BtA Demo

Everything is Ok

Folders: 1
Files: 1
Size:       361232708
Compressed: 203056300
DEBUG:Thread-7:tasks.work2: Creating path "/home/loris/FreeSpace 2 beta/BtA Demo/"...
DEBUG:Thread-7:tasks.work2: Creating path "/home/loris/FreeSpace 2 beta/BtA Demo/BtA Demo"...
DEBUG:Thread-7:tasks.work2: Moving "/tmp/tmpZontl3/content/BtA Demo/BTA_DEMO_Assets2.vp" to "/home/loris/FreeSpace 2 beta/BtA Demo/BtA Demo/BTA_DEMO_Assets2.vp"...
INFO:Thread-7:util.download: Downloading "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_Assets1.7z"...
DEBUG:Thread-7:util.call: Running ['7z', 'x', '-o/tmp/tmpPwJQqv/content', u'/tmp/tmpPwJQqv/BTA_DEMO_Assets1.7z']

7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=fr_FR.UTF-8,Utf16=on,HugeFiles=on,8 CPUs)

Processing archive: /tmp/tmpPwJQqv/BTA_DEMO_Assets1.7z

Extracting  BtA Demo/BTA_DEMO_Assets1.vp
Extracting  BtA Demo

Everything is Ok

Folders: 1
Files: 1
Size:       536321828
Compressed: 135848477
DEBUG:Thread-7:tasks.work2: Moving "/tmp/tmpPwJQqv/content/BtA Demo/BTA_DEMO_Assets1.vp" to "/home/loris/FreeSpace 2 beta/BtA Demo/BtA Demo/BTA_DEMO_Assets1.vp"...
INFO:Thread-7:util.download: Downloading "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_Root.7z"...
DEBUG:Thread-7:util.call: Running ['7z', 'x', '-o/tmp/tmpRuEXPa/content', u'/tmp/tmpRuEXPa/BTA_DEMO_Root.7z']

7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=fr_FR.UTF-8,Utf16=on,HugeFiles=on,8 CPUs)

Processing archive: /tmp/tmpRuEXPa/BTA_DEMO_Root.7z

Extracting  BtA Demo/bta.bmp
Extracting  BtA Demo/mod.ini
Extracting  BtA Demo/BTA_DEMO_Root.vp
Extracting  BtA Demo/data
Extracting  BtA Demo

Everything is Ok

Folders: 2
Files: 3
Size:       57822374
Compressed: 21822457
DEBUG:Thread-7:tasks.work2: Moving "/tmp/tmpRuEXPa/content/BtA Demo/BTA_DEMO_Root.vp" to "/home/loris/FreeSpace 2 beta/BtA Demo/BtA Demo/BTA_DEMO_Root.vp"...
DEBUG:Thread-7:tasks.work2: Moving "/tmp/tmpRuEXPa/content/BtA Demo/mod.ini" to "/home/loris/FreeSpace 2 beta/BtA Demo/BtA Demo/mod.ini"...
DEBUG:Thread-7:tasks.work2: Moving "/tmp/tmpRuEXPa/content/BtA Demo/bta.bmp" to "/home/loris/FreeSpace 2 beta/BtA Demo/BtA Demo/bta.bmp"...
DEBUG:Thread-7:tasks.work2: Creating path "/home/loris/FreeSpace 2 beta/BtA Demo/BtA Demo/data"...
DEBUG:Thread-7:progress._next_step: Entering step 3 of 3 in task InstallTask.
WARNING:Thread-4:progress.work: Either we still have some work to do (unlikely) or there are still some other threads running (2).
ERROR:Thread-7:progress.run: Exception in Thread!
Traceback (most recent call last):
  File "/home/loris/knossos-develop/lib/progress.py", line 120, in run
    task[0].work(*task[1])
  File "/home/loris/knossos-develop/lib/progress.py", line 380, in work
    self._steps[self._cur_step][1](arg)
  File "/home/loris/knossos-develop/lib/tasks.py", line 366, in work3
    path_prefix = glob.escape(path_prefix)
AttributeError: 'module' object has no attribute 'escape'
DEBUG:Thread-7:progress._next_step: Entering step 4 of 3 in task InstallTask.

@mjn.mixael & ngld :
I'm not sure if knossos uses mod.ini or dependencies from the JSON data in this case. I have already have "mediavps_3612".

@m!m :
I'm not strictly enforcing dependency validity in Nebula, but the good practice would be to rely on the dependency autocompletion to add these, just to make sure the ID and packages are right.
I think it would be a bad practice not to register dependencies in the right order (thus temporarily registering non existing deps), but to me that's modders decision, and should be possible as long as they know what they do (and add the missing deps quickly).
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on October 11, 2014, 09:44:20 am
@Hellzed: What do you think about adding the capability to check the dependencies of a mod and let the modder decide if they want to use it? This is obviously a feature that isn't required right now.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on October 11, 2014, 09:50:14 am
@Hellzed: Are you using Python 2? Knossos tries to use the function "escape" of the built-in module "glob" which is missing in your case.
I'm going to rewrite that part, anyway. I'll use the "filelist" instead to ensure consistency.

Knossos still parses the mod.ini but that might change in the future. Once I add support for multiple versions of the same mod, the dependencies listed in the mod.ini are bogus.

@m!m: That's a great idea.
@Hellzed: Maybe you could add a warning like "You specified an invalid dependency. Are you sure you want to submit?". I'm could re-write the dependency resolution algorithm in PHP if that helps (you'll need it if you want to check for version conflicts).
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 11, 2014, 10:11:13 am
I'm adding optional dependency enforcement to my TO DO list.
My next work items are :
- Allowing to copy a build across branches (if you want to copy the last release candidate from the unstable branch to the stable one, under a stable version number)
- Complete team logic (this is a big change, but a big chunk of it is already dormant the code, so ideally we wouldn't need to rebuild the database from scratch to activate it)
- Basic admin panel to pick which mods can publish directly to the global ("all.json") repository.
- Expand image uploads (add a Nebula specific high res logo/"cover art" and screenshots).
- Small things like list pagination on the all mods and modders pages.

These are the basic steps needed to build the public mod catalogue.

@ngld :
* Yes, I started ./launcher.py with default python binary in Ubuntu 14.10. It happens to be python2.
* Having a php version of the algorithm could be useful, as a custom validation constraint in the form. I'll tell you when I need it, but that could be in a little while, since it's not absolutely necessary yet.
I'll advise modders to test their builds in a separate branch before publishing to the default branch (once we have the build transfer feature).


EDIT : Could you add instructions to get the dependencies for the virtualenv too ? Once activated in the knossos-develop folder, python can't find basic dependencies installed from APT, like "six" and "PySide".
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on October 11, 2014, 10:50:12 am
Knossos still parses the mod.ini but that might change in the future. Once I add support for multiple versions of the same mod, the dependencies listed in the mod.ini are bogus.
I have put some though into that issue as well and here is what I implemented:
The mods that are downloaded are stored in folders that are uniquely identified by the mod ID and version (it looks like <FS root>/mods/packages/<id>/<version>.
I also decided that mod.ini files are just too cumbersome to handle so I switched to using JSON to store the information about the installed mods locally. When a new mod (or package) is installed the installer updates a file named mod.json that then includes all information that is needed later. I store the general meta data about a mod (title, version, ...) and the data about the installed packages in that file.
Here is an example of the file:
Code: (mod.json) [Select]
{
  "description": "The day is July 25, 2336. Sixteen months have passed since the Lucifer was destroyed, mysteriously locking the Sol system away. It's barely been thirteen months since the Hades superdestroyer was destroyed in orbit of Deneb, and the rogue GTI reduced to silence. There are many wounds to lick for the GTA, as the weakened fleet struggles to maintain order in its systems. Although the cease-fire treaty with their new Vasudan allies is still holding, the PVE is still busy dealing with the terrorist HoL faction.\r\n\r\nNow is our chance to get past them. We no longer need to so carefully hide. Now we can make our deal with the devil.",
  "title": "Between the Ashes: Mefistofele",
  "version": "1.1.2",
  "id": "BTA_Demo",
  "packages": [
    {
      "notes": "",
      "name": "BtA Demo",
      "status": "Required",
      "dependencies": [
        {
          "id": "BTA_Demo",
          "version": "*",
          "packages": []
        }
      ],
      "files": []
    },
    {
      "notes": "",
      "name": "Low End Compatibility",
      "status": "Optional",
      "dependencies": [
        {
          "id": "BTA_Demo",
          "version": "*",
          "packages": []
        }
      ],
      "files": []
    }
  ]
}
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 11, 2014, 11:08:50 am
What about scripts (iirc some mods write data to their folders during missions) ?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on October 11, 2014, 11:14:51 am
As far as I know scripts can't write into the mod folder as FSO doesn't allow that but that isn't a problem anyway as FSO doesn't care where a mod is located as long as the path it is given is right.
Using this system we might even be able to install total conversion without special handling and if I add the right commandline option to FSO we could even handle retail FS2 as just another total conversion but that isn't relevant now, first we need to get the core features running :p
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 11, 2014, 12:29:44 pm
Looks like I ran into outdated docs about python3 in ubuntu. I had to use pip3 instead of pip, no need for a virtualenv...
I can now run Knossos with python3.

Your latest commit broke mod install : BtA fails to download but instantly appears as installed in the mods list.

Code: [Select]
INFO:Thread-7:util.get: Retrieving "http://dev.tproxy.de/knossos/version"...
DEBUG:Thread-5:progress._next_step: Entering step 1 of 2 in task InstallTask.
ERROR:Thread-5:progress.run: Exception in Thread!
Traceback (most recent call last):
  File "/home/loris/knossos-develop/lib/progress.py", line 120, in run
    task[0].work(*task[1])
  File "/home/loris/knossos-develop/lib/progress.py", line 380, in work
    self._steps[self._cur_step][1](arg)
  File "/home/loris/knossos-develop/lib/tasks.py", line 298, in work1
    for item, info in mfiles.items():
AttributeError: 'list' object has no attribute 'items'
DEBUG:Thread-5:progress._next_step: Entering step 2 of 2 in task InstallTask.
DEBUG:Thread-5:progress._next_step: Entering step 3 of 2 in task InstallTask.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on October 11, 2014, 12:33:48 pm
As I said I'm rewriting the InstallTask and I accidentally pushed broken code. :sigh: Once I'm finished it'll work again.

EDIT: It should work now.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: mjn.mixael on October 11, 2014, 01:25:00 pm
As far as I know scripts can't write into the mod folder as FSO doesn't allow that but that isn't a problem anyway as FSO doesn't care where a mod is located as long as the path it is given is right.
Using this system we might even be able to install total conversion without special handling and if I add the right commandline option to FSO we could even handle retail FS2 as just another total conversion but that isn't relevant now, first we need to get the core features running :p

Scripts can write to the mod folder and should. For example, the shipsave script from Admiral MS writes files to mod/data/config. Actually, I'm pretty sure Admiral MS created optional stuff to change what folder the script saves files to. I have also used a script to copy and rename files from within FSO in order to unlock missions.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on October 11, 2014, 02:03:01 pm
I wasn't aware of that until Axem told me that it was possible but even with scripts it shouldn't be a problem to use the proposed mod path template.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 11, 2014, 02:44:27 pm
On Ubuntu, it looks like scripts write to a separate folder, for example, BtA writes to :
Code: [Select]
.fs2_open/BtA Demo/data/scripts
Code: [Select]
FreeSpace 2/BtA Demo/datastays empty. I haven't fully replayed it yet, so i'm not sure.

I have no idea what fs2_open scripting actually does on the filesystem.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on October 11, 2014, 02:49:12 pm
It depends on the command line. If a -mod flag is passed, then FSO will pick the first path and write the files to "<FSO folder>/<mod path>/<whatever>" on Windows and to "~/.fs2_open/<mod path>/<whatever>" on Unix.
If the flag isn't passed, it writes the files to "<FSO folder>/<whatever>" or "~/.fs2_open/<whatever>".
Since the path depends on the "-mod" flag it won't cause any problems for us. (Just like m!m said.)
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: mjn.mixael on October 11, 2014, 03:01:59 pm
On Ubuntu, it looks like scripts write to a separate folder, for example, BtA writes to :
Code: [Select]
.fs2_open/BtA Demo/data/scripts
Code: [Select]
FreeSpace 2/BtA Demo/datastays empty. I haven't fully replayed it yet, so i'm not sure.

I have no idea what fs2_open scripting actually does on the filesystem.

I think we had it writing to data/scripts/ in the demo release and changed it to config later. Can't remember.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 11, 2014, 03:06:41 pm
@mjn.mixael : You might want to run a "copy & update" on BtA Demo. JSON generation has changed, and current BtA build won't download in the latest version of Knossos.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 12, 2014, 02:24:04 pm
I cleaned up the repository, and added the "alpha" warnings.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on October 14, 2014, 06:12:35 am
I've got a few bugs left to fix and add a few things to the UI but I've nearly finished support for multiple mod versions. This allows a user to install two versions of the same mod at the same time.
I have some question for everyone reading this:

@Hellzed: I've updated the test server with your changes yesterday.
We should add a note to the "New build" form that tells the modder that the download links should always point to exactly this version. We should also allow to change the links later on (The server might have changed or maybe they want to add another mirror). If too many modders complain about this, we might be able to add a system which automatically creates a backup of all builds in the Nebula. I wonder how much disk space that would consume, though.  :blah:

This should make the FSCRP's job much easier (you would be able to download every version of a mod).
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on October 14, 2014, 06:31:06 am
  • How should updating be handled? As long as the user only has one version installed it's easy, just delete the old version and replace it with the new one. The best idea I can come up with is: If the mod's version was the latest one when it was installed, replace it with the newest version available in its branch.
Why not just keep all version installed (maybe add an optional "clean" action to delete old versions). For some mods you will need to keep some older versions anyway (e.g. the MediaVPs) so you can just apply that to all mods.

  • I'm saving the mod metadata in a file called "knossos.json" in the mod's directory. I didn't use the "mod.json" name since m!m's launcher is already using it and the schema's not compatible.
What makes it incompatible? It would be great if all launchers/installers could use the same data.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on October 14, 2014, 06:36:51 am
Why not just keep all version installed (maybe add an optional "clean" action to delete old versions). For some mods you will need to keep some older versions anyway (e.g. the MediaVPs) so you can just apply that to all mods.
I'm trying to save disk space, duh.  :P  For dependencies (like the MediaVPs) I'll check if they're still needed and keep them only in that case.

What makes it incompatible? It would be great if all launchers/installers could use the same data.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on October 14, 2014, 06:54:04 am
Why not just keep all version installed (maybe add an optional "clean" action to delete old versions). For some mods you will need to keep some older versions anyway (e.g. the MediaVPs) so you can just apply that to all mods.
I'm trying to save disk space, duh.  :P  For dependencies (like the MediaVPs) I'll check if they're still needed and keep them only in that case.
Hard disk space is cheap :p

What makes it incompatible? It would be great if all launchers/installers could use the same data.
  • You're not storing the filelist, I'm saving it for integrity checks.
  • Why did you capitalize the package's status ("Required", "Optional", ...)? I've kept the same case from the schema.
  • How do you store which packages are installed? I'm only saving installed packages in the packages property but I suspect you're doing the same.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 14, 2014, 12:12:27 pm
We should add a note to the "New build" form that tells the modder that the download links should always point to exactly this version. We should also allow to change the links later on (The server might have changed or maybe they want to add another mirror). If too many modders complain about this, we might be able to add a system which automatically creates a backup of all builds in the Nebula. I wonder how much disk space that would consume, though.  :blah:

Yeah, we should end this habit of "sneaky updates". I'll add the note to the form.
Wouldn't allowing to change the links after a build has been validated go against strict semver enforcement ?
To me, the appropriate way to change links would be to run a "copy & update" on the build, replace the broken links, and validate the new build under the same version number with a changed "version metadata" field (for example 1.0.0 to 1.0.0+serverchange1).
I'm thinking about writing a Nebula "good practices" guide, now :p

This leads me to think we would still have a problem with outdated or broken links in older builds. We should use some automated testing. I'm thinking about a cron task, that would run Knossos-server from Nebula once in a while, and if a single file is unavailable from all urls, mark the build as "obsolete" and remove its cached JSON data from the database.
(that would make 4 states for a build : "success", "failed", "processing" and "obsolete")

Since many most files are big, we can't handle them directly from the build form, so I'm not sure how Nebula could handle file hosting.
However, as Knossos-server downloads the files before generating the checksums, maybe it could cache them and add the cache URLs to the lists before generating final JSON. (I'm not proposing this because that's less work for me, I actually think it's a good solution before we come up with a better file hosting system).

If my cloning code keeps behaving as expected, we should have a simple way to transfer a build to from a branch to another tonight. I'll also add a graphical clue to see which branch is the default one (in the mod/branches list view).
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on October 14, 2014, 01:12:09 pm
Yeah, we should end this habit of "sneaky updates". I'll add the note to the form.
Wouldn't allowing to change the links after a build has been validated go against strict semver enforcement ?
To me, the appropriate way to change links would be to run a "copy & update" on the build, replace the broken links, and validate the new build under the same version number with a changed "version metadata" field (for example 1.0.0 to 1.0.0+serverchange1).
I'm thinking about writing a Nebula "good practices" guide, now :p
It wouldn't go against semver (http://semver.org) as long as the archives' checksums stay the same. If you only change the links and not the archives' contents, the actual mod data (content) stays the same and thus is still the same version. If you create a new build, that would mean there'd been an update. The client would then tell the user to update the mod since a new version means new/different content.

Since many most files are big, we can't handle them directly from the build form, so I'm not sure how Nebula could handle file hosting.
We can't? Why? Is there a browser limitation? :P

However, as Knossos-server downloads the files before generating the checksums, maybe it could cache them and add the cache URLs to the lists before generating final JSON. (I'm not proposing this because that's less work for me, I actually think it's a good solution before we come up with a better file hosting system).
I've already thought about this and it seems like a good solution. I'll add a mediafire downloader, so that modders can also paste normal mediafire links in addition to direct links.

If my cloning code keeps behaving as expected, we should have a simple way to transfer a build to from a branch to another tonight. I'll also add a graphical clue to see which branch is the default one (in the mod/branches list view).
Yay!

EDIT: @m!m: Here's a schema for you: mod_json_schema.txt (https://github.com/ngld/knossos/blob/develop/converter/mod_json_schema.txt) What do you  think?

EDIT2: @m!m: Did you implement the "-mod" flag, yet? We can generate the -mod list by checking the dependencies but that only replaces the "secondarylist" in mod.ini. What about the "primarylist"? I guess we'll need to add another field to the repo schema.
@Hellzed: Can you allow the modder to reorder the dependencies? If we generate the -mod list using these, the order becomes important.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on October 15, 2014, 12:52:45 pm
The Nebula also supports http://fsmods.net/download.php... and http://mediafire.com/... links, now. MediaFire might ask you to solve a captcha, though, depending on whether they feel like the nebula server downloads too much.
The Nebula's mirror works but it doesn't delete old files, that's still on the TODO list.

I've implemented a solution to the -mod flag problem but it only works as long as all dependencies are in the Nebula. Once one dependency disappears from the Nebula, it might not be loaded correctly.
The mod tree in Knossos now looks like this:
(http://dev.tproxy.de/mirror/kn_s1.png)

If anyone has a better idea how I should display the other versions, I'd be glad to implement it. Hellzed already suggested dropping the "Other versions" node and displaying the versions on the same level as the features. He also suggested an option to hide the other versions.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: mjn.mixael on October 15, 2014, 01:06:43 pm
No, definitely keep the Other Versions node. It makes it clear that it's not part of the main mod download.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 15, 2014, 04:04:56 pm
I'll see what I can do regarding dependencies dynamic re-ordering, but I may go with an optional "priority" field at the moment (this field would be a matter of minutes to add to the dep object, while reordering implies changing the JS and make sure it doesn't interact in a bad way with the current field index management, which is already a bit trickier than in looks).

Last night, I was stuck with an annoying form binding issue (all down to a typo that didn't cause a crash right away, so it went unnoticed and just filled one of the fields with a null when it shouldn't have...), so the build transfer commit was delayed. It's now fixed, I still have to push it.

@ngld : You might check the JSON converter version number handling, it breaks if i add it some metadata :
Code: [Select]
INFO: Parsing repo...
ERROR: Failed to parse version for Mod "MediaVPs 2014 (Nebula test)"! (Invalid version string: '3.7.1+nebula_test2')
ERROR: No mods found!

EDIT : Well, that's what i get when I try to convert MediaVPs 2014 (check my branch to for the data), even without the metadata.
Code: [Select]
INFO: Parsing repo...
ERROR: Failed to perform conversion!

Found it ! If i pass a null in an action "dest", it fails instead of assuming we are using the mod folder root. I can fix that on my side by changing it to whatever doesn't cause Knossos server not to fail (empty string ?)
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on October 15, 2014, 05:49:41 pm
The dependency re-ordering isn't the most important thing right now because we still have to figure out how we're going to replace the primarylist part of the mod.ini.

@ngld : You might check the JSON converter version number handling, it breaks if i add it some metadata :

The version number handling is correct according to semver.org:
Quote
Build metadata MAY be denoted by appending a plus sign and a series of dot separated identifiers immediately following the patch or pre-release version. Identifiers MUST comprise only ASCII alphanumerics and hyphen [0-9A-Za-z-]. Identifiers MUST NOT be empty. Build metadata SHOULD be ignored when determining version precedence. Thus two versions that differ only in the build metadata, have the same precedence. Examples: 1.0.0-alpha+001, 1.0.0+20130313144700, 1.0.0-beta+exp.sha.5114f85.
Only [0-9A-Za-z-] are allowed, no underscores. If you remove the underscore and replace it with a hyphen, it works.

EDIT : Well, that's what i get when I try to convert MediaVPs 2014 (check my branch to for the data), even without the metadata.
Code: [Select]
INFO: Parsing repo...
ERROR: Failed to perform conversion!

Found it ! If i pass a null in an action "dest", it fails instead of assuming we are using the mod folder root. I can fix that on my side by changing it to whatever doesn't cause Knossos server not to fail (empty string ?)
Yes, you should pass either an empty string or no value at all.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 15, 2014, 06:27:45 pm
About semver, sorry, my bad ! I'll change my regexp.
EDIT : the regex was compliant with semver, but the line was commented. XD
EDIT 2 : Commented and in the wrong place. I must have been tired. It will be fixed with next push.

Can you take a look at this build ? http://nebula.tproxy.de/nebula/hellzed/mediavps_2014/master/3.7.7/show_data
It failed, and I don't understand why.

NEBULA UPDATE : Build transfer (it's actually a copy across branches of the same mod, and we don't remove the source build) ! And some small fixes.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on October 16, 2014, 07:55:18 am
I've fixed a few bugs in Knossos-server which might've been related to your problem. Can you try again, please?
I've also updated the server to the newest nebula verison. I've got one smal nit-pick: When someone transfers a build, could you please exclude the source branch from the destination list?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 16, 2014, 11:46:17 am
Latest git commit excludes current branch from the build transfer form. :)

I' will try to run the conversion again later tonight.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 16, 2014, 03:46:35 pm
I'm running into a bug while trying to launch mediavps_2014 with the latest version of Knossos : an empty
Code: [Select]
-mod '' argument is added each time I use "Launch" or "Settings", so the game fails to launch and complains about the empty argument.

If I add some random string as a value for this flags, the game launches with mediavps_2014.

EDIT : I think Knossos should add the -keyboard_layout "qwertz" flag when a German locale is detected, and -keyboard_layout "azerty" when a French locale is detected.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: niffiwan on October 16, 2014, 06:50:07 pm
EDIT : I think Knossos should add the -keyboard_layout "qwertz" flag when a German locale is detected, and -keyboard_layout "azerty" when a French locale is detected.

I'm don't know if auto-enabling this without warning is a great idea. Maybe have it "appear" as a suggestion/recommendation when the appropriate locale is detected? Or at least provide an obvious way to disable it just in care someone has that locale set but has a different keyboard?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 16, 2014, 11:25:05 pm
@niffiwan : Yes, changing this has to be easy enough.
I'm not sure about this on Windows, but without this flag on Linux, "qwertz" and "azerty" layouts are broken in fs2_open. Keys "1" to "0" and some others can't be used (no comms !).
People from GSA and Belgium/France actually use these layouts, and it's unlikely to find people using a "qwerty" layout in these areas.

(interestingly, French speaking parts of CH mostly use "qwertz" layouts, while "qwerty" is used QC.)
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: niffiwan on October 16, 2014, 11:43:09 pm
Yes it would be good to have the -keyboard_layout flag be more widely known in order to fix those keyboard issues, and it seems that you'd already thought about my concern so its all good :) (the flexibility is something important to me since I use the non-standard Colemak layout on a qwerty keyboard, so I like to be able to tweak those sort of settings)
 
By the way, what would the French speaking parts of Canada normally have as a keyboard layout?

edit: typo
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on October 17, 2014, 02:42:04 am
As far as I know -keyboard_layout isn't used on windows as that switch is only used to initialize the SDL to FS2 key map on linux. When SDL2 is eventually used on all platforms then it will probably be supported but when that happens it shouldn't be a problem to implement automatic detection of the keyboard layout.
I guess enabling it for Linux/MacOS is probably a good idea.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on October 17, 2014, 07:27:46 pm
I'll add a combo box to the "FSO Settings" tab where you can select either "qwerty", "qwertz" or "azerty". The combo box will be hidden if the user isn't on Linux.
I wonder if I should use the "-keyboard_layout" or just run "setxkbmap -layout us" to change the layout. (If I use setxkbmap, I'd change the layout back after FSO exits.)
The setxkbmap method actually changes the system's keyboard layout to qwerty. All keyboard layouts should be supported that way. If I'd use "-keyboard_layout", at least the qwertz layout is broken. (The mapping and labling of the Z and Y keys are confusing.)

The bug with "-mod ''" has been fixed.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 19, 2014, 02:18:16 pm
I've just implemented some basic teamwork features, and I can confirm : no database issues so far, team owned mods behave exactly like modder owned mods.
I still have to make a few more changes to some templates, ensure everything is ok with security and team deletion, and I'll push the changes, probably on Tuesday.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on October 20, 2014, 05:07:42 am
@ngld: Do you have a list of commands that should be supported by the fso:// handler?
I looked through your code and I have found the following:
Did I miss something?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on October 20, 2014, 11:37:37 am
@m!m: I didn't have a list, so I wrote this documentation (https://github.com/ngld/knossos/wiki/fso%20Links) just for you! :P
No, you didn't miss anything. Did you read the mod.json (https://github.com/ngld/knossos/blob/develop/converter/mod_json_schema.txt) schema?

I've implemented both the "-keyboard_layout" flag and the "setxkbmap" method for Linux.
I'll probably implement a proper "Update available!" notice and Windows integration tonight. The integration will display the progress bar in the task bar.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on October 21, 2014, 06:30:16 am
@m!m: I didn't have a list, so I wrote this documentation (https://github.com/ngld/knossos/wiki/fso%20Links) just for you! :P
No, you didn't miss anything. Did you read the mod.json (https://github.com/ngld/knossos/blob/develop/converter/mod_json_schema.txt) schema?
Thanks for the documentation!
The mod.json schema looks good, one small point I would like to add is the ability to also specify an URL instead of a local path for the logo specification. I have implemented a caching mechanism that simplifies a lot of things for me but that requires to have the URL to the image.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 22, 2014, 02:23:21 pm
I still have a few issues in the "Teams" code, but I'll update the repository really soon !
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 23, 2014, 02:07:11 am
@ngld : I don't know if you still need the information, but for the Unity integration to work, you need a .desktop file in ~/.local/share/applications

The current (broken) one looks like :
Code: [Select]
[Desktop Entry]
Name=fs2mod-py
Exec=/usr/bin/python3 /home/loris/knossos-develop/lib/api.py %U
Icon=/home/loris/knossos-develop/lib/hlp.png
Type=Application
Terminal=false
MimeType=x-scheme-handler/fso;

Also, most of our settings windows don't fit on my 768px height screen ! I have spent some time reading Gnome/Qt/Windows official guidelines, and found some interesting things to help reorganize the UI.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 24, 2014, 12:43:44 am
I found how to make Knossos look native on Ubuntu (with Unity), or any GTK3 window manager ! Not exactly easy, though, because Qt uses GTK2 themes, while some of the widget styles we need are found in GTK3...

1) The window has to be borderless (we have to get rid of the 9px margins around "centralwidget"). It will probably look better on OS X, and Windows adds its own window borders anyway !
2) The Horizontal Layout which contains "settingsButton", "listControls", "pageControls" and a horizontal spacer has to be converted to a QFrame. By default, that will not change anything on other platforms.
3) Now, If we detect a GTK3 WM (Unity or Gnome-shell) with a known theme (having Gnome-shell default theme, Radiance and Ambiance would be good enough), we load a Qt CSS stylesheet and apply it to the QFrame to mimic GTK3 "Primary Toolbar" widget.

I'm still searching for the most simple and reliable way to detect the theme in 3). I'm also searching for the exact CSS values I need, and GTK3 themes are not exactly the easiest kind of CSS to read.

EDIT : Nailed it ! I have a Qt style sheet that gives Knossos a native look on Ubuntu's default Ambiance theme. (see 2nd screenshot)

EDIT 2 : Updated to Symfony 2.5.6 and latest FoSUB... I had to fix user management because they made some changes to their namespaces, and moved classes around...

[attachment kidnapped by pirates]
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on October 27, 2014, 01:40:23 pm
Updated "Ambiance" style sheet (combobox contrast fix, border fix, button height):
Code: [Select]
#toolbar {
    padding:3px;
    padding-bottom: 4px;
    max-height: 50px;
    background: qlineargradient(spread:pad, x1:0, y1:0, x2:0, y2:1, stop:0 rgba(63, 62, 58, 255), stop:1 rgba(88, 86, 78, 255));
}

#toolbar QPushButton {
    height: 20px;
    color: #DFDBD2;
    background-color: transparent;
    border-image: url(./button-toolbar.png) 8;
    border-top: 8px;
    border-bottom: 8px;
    border-right: 8px;
    border-left: 8px;
}

#toolbar QPushButton:hover {
    border-image: url(./button-toolbar-hover.png) 8;
}

#toolbar QPushButton:pressed {
    border-image: url(./button-toolbar-pressed.png) 8;
}

#toolbar QPushButton:focus {
    border-image: url(./button-toolbar-focused.png) 8;
}

#toolbar QLineEdit {
    height: 20px;
    background-color: transparent;
    border-image: url(./entry-toolbar.png) 8;
    border-top: 8px transparent;
    border-bottom: 8px transparent;
    border-right: 8px transparent;
    border-left: 8px transparent;
    color: grey;
}

#toolbar QLineEdit:focus {
    color: #4C4C4C;
    border-image: url(./entry-toolbar-focused.png) 8;
}

#toolbar QComboBox {
    color: #DFDBD2;
    height: 20px;
    padding: 0px;
    min-width: 6em;
    background-color: transparent;
    border-image: url(./button-toolbar.png) 8;
    border-top: 8px;
    border-bottom: 8px;
    border-right: 8px;
    border-left: 8px;
}

#toolbar QComboBox:hover {
    border-image: url(./button-toolbar-hover.png) 8;
}

#toolbar QComboBox:focus {
    border-image: url(./button-toolbar-focused.png) 8;
}


#toolbar QComboBox::drop-down {
    border: none;
}

#toolbar QComboBox::down-arrow {
    image: url(./combobox-arrow.png);
    subcontrol-origin: padding;
    subcontrol-position: top right;
    padding-left: 4px;
    width: 20px;
    height: 20px;
    border-left-width: 1px;
    border-left-color: #353531;
    border-left-style: solid;
    border-top-right-radius: 3px;
    border-bottom-right-radius: 3px;
}


#toolbar QComboBox QAbstractItemView {
    background-color: #353531;
}
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on November 07, 2014, 06:54:36 am
This will hopefully be the last delay in the team update. I ditched most of the previous team logic, and I'm rebuilding it nearly from scratch.

I have to manage the user-team relation, and there were 3 options :
- hardcode team permissions (admin, member, invited modders...)
- use standard Symfony ACL
- do something else (some kind of custom ACL+ relation attributes)

I experimented with the first one (hardcoded permissions), but it's not flexible enough. It doesn't really cause any issue yet, but maintainability would end up being a problem.
The second one (Symfony ACL) would be ok, except I still may have to store relation attributes that are not permission related (i.e. : join date). Managing the user-team relation with 2 completely different Symfony methods (ACL+Doctrine2 mapping) feels like a bad idea in the end.

That's why I'm going to create a small class that works a bit liks ACL (with lexx complexity), but can also store any other type of data about user-team relation. It should be good enough for now, and shouldn't break if it's extended.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on November 10, 2014, 06:00:26 pm
@m!m: I've changed the mod.json (https://github.com/ngld/knossos/blob/develop/converter/mod_json_schema.txt#L12) documentation. Is this better?
I'll add a "filesize" property to mod archives (https://github.com/ngld/knossos/blob/develop/converter/schema.txt#L130-142). That way you can determine the download size without using any HTTP requests. What do you think?

@Hellzed: I'm curious how your permission system works / looks in the web interface once it's finished. If I have enough time, I'll try to implement detection of obsolete links and I'll make the Nebula delete old files from the mirror.



I've finally finished the packages for Knossos. It runs pretty well on Windows, Linux (tested on Arch, Ubuntu and Xubuntu) and Mac OS X.

(https://dev.tproxy.de/knossos/images/screen6.png)(https://dev.tproxy.de/knossos/images/screen7.png)(https://dev.tproxy.de/knossos/images/screen8.png)

On Windows and Ubuntu the current download progress is displayed in the Taskbar.

Installation instructions (https://dev.tproxy.de/knossos/)

If you encounter any problems or bugs, you can post here or create a new issue on GitHub (https://github.com/ngld/knossos/issues).

EDIT: Updated.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on November 10, 2014, 06:45:52 pm
Awesome !

Permissions are managed through a bitmask system associated with human-readable "roles", and these roles will be displayed in a html "select" element for each member of the team (editable by team admin).
I'm still figuring some implementation choices since this is not a standard many-to-many Doctrine2 ORM relation, and if i do something wrong it will drown the SQL server under queries (well, there's not that much data, but 50 queries to generate a 10 member team page would be awful). It also makes the code to access a team's members and a member's teams much more complicated, so I'll need to write some helper functions.
In the end, the added code will not be that long (even quite short), but it's a matter of doing everything right.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: niffiwan on November 10, 2014, 07:03:22 pm
Very cool  :yes:

I have just one question at this point; how do you select a new FSO dir? I originally selected my "real" FSO dir, but now I want to switch to my test FSO dir before I try installing any mods. Just in case...

ps. I tried the pip install to start with; I think you need the --pre flag in order for this to work correctly.  And maybe other stuff as well; but I didn't troubleshoot for too long before just using the PPA instead.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on November 10, 2014, 07:04:41 pm
Click on "Settings", select "Game settings" on the left and click the "Browse" button on the right.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: niffiwan on November 10, 2014, 07:11:22 pm
Thanks for that - I tried that and was confused because the dropdown still listed the exec's from the other directory (and my test dir didn't have any execs in it at that point which may have contributed to the issue  :nervous:)
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on November 10, 2014, 07:20:57 pm
Oh, that's a bug. I forgot to update the dropdown if you change your FS2 path.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on November 13, 2014, 03:16:53 am
@m!m: I've changed the mod.json (https://github.com/ngld/knossos/blob/develop/converter/mod_json_schema.txt#L12) documentation. Is this better?
Yes that's better, thank you!

I'll add a "filesize" property to mod archives (https://github.com/ngld/knossos/blob/develop/converter/schema.txt#L130-142). That way you can determine the download size without using any HTTP requests. What do you think?
I think that's a great idea! With that the UI could display an estimated download size and maybe an estimated completion time without the added overhead of sending HTTP requests to all URLs.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on November 17, 2014, 01:38:42 am
@ngld:
I kept reading advanced doc for Symfony. Going back to the drawing board with the whole Nebula security implementation (again, but this time not juste teams. The whole security !).

I really have to stop thinking of Nebula as a "small" project. I have made some implementation choices (security, but not only that part, also some database management) because I read some things (namely Symfony default ACL, and Symfony paginator) would be overkill for a "small" project. Now I figure it's not, because we actually need features like administration granularity...
Our talk about Nebula admin helped a lot ! :)

1) I'm going to drop our custom security voters (I always planned to rewrite this part anyway, because it lead to some terrible variable naming in some of the html templates), replace it with standard ACL, and enable an early version of admin access to mods. Creating the admin panel will be much easier once it's done.
2) Going back to http://www.hard-light.net/forums/index.php?topic=86364.msg1768398#msg1768398 , I'm keeping the special "attributes" class I was writing to store data about a relation between a modder and a team, but instead of writing a custom ACL, I will use the standard ACL system so Nebula will have a single big authorization system.
(at first, I thought this would add complexity, but when I drop the voters, it will get much more simple)

Dropping custom security voter classes is also important because Symfony devs will make deep changes starting with Sf 2.6 that may completely break our security system. ACL won't change any time soon.

Less custom, more standard code will also make my code more readable to any other Symfony dev, in case someone wants to contribute.

EDIT : Good news ! It's working (already rewritten the "mod owning" part... Still have to do the branches and builds...).
Bad news : database reset coming (we can keep mods created with the old voter system, but their owners will be unable to make changes).

EDIT 2 : Since I'm changing a lot of things in the db, I'll simplify a few more things to play better with the new security system:
- No more FSMod, Team objects ! Team + Owner interface (the "repository") + FSMod will become a unified object, some kind of "work group".
- Users will own the work groups in a standard many-to-many relation, permissions will be managed through the ACL system.
- Aggregated fields (mod, branch, build counters) will be handled with more efficiency through the work groups.

---> For users, it means they won't have to worry about team management : they create a mod, and if they need to give rights on this mod to other users, they can do it through the mod admin page. Every mod will retain a "team" page, showing the current authorized members.
---> For me, that means : better top level templates and better top level controllers (no need to check if a mod is owned by a user or by a team), less duplicated code (no need to handle mod owning in team objects), easier checks when an object (user, mod) is created...
It's actually close to making teamwork a "default" on Nebula.

These changes will have no impact on the lower structure (build content).

EDIT 3 : This refactoring will go even deeper, with a better build validator abstraction !
Currently, I use a service to query knossos-server api from Controllers. The Controllers then 1) read the answer, 2) compare with the database, and 3) take appropriate action.
After the new changes, the "read the answer" and "compare with the database" steps will be included in the service, so the Controllers will only need to send the right page depending on the aggregated answer.
This will lower the risk of redirection bugs in Nebula build validation pages (in other words, I will be able to understand my own redirection code, which is currently barely the case, because it's a nested conditions hell).
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on November 20, 2014, 04:00:08 pm
I've updated knossos-server (the converter part of the Nebula). It can now queue build requests if all workers are busy and you can watch running tasks (http://s1.nebula.tproxy.de/tasks).

Knossos also received an update. I've (hopefully) fixed all the known bugs and added a new feature which allows the user to install and uninstall packages for installed mods.
Windows and Mac users should reinstall using the links in the release post. Linux users can user their package manager or if they installed knossos with "pip install", they can use "pip install -U --pre knossos".
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on November 30, 2014, 07:51:59 am
I spent the last week getting ready to relocate (new city, new country, new job)... the big refactoring is still going on though, and I'm currently waiting for an answer from Symfony devs about one of the built-in components.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on December 17, 2014, 06:10:35 pm
Well, I guess I've implemented most of the important features. Once my current TODO list is finished, I'll release Knossos 0.1.0.

Recent Changes:

@m!m: I added a new "executables" property to packages. It's not documented, yet, because I wanted your feedback on it. It allows a package to list all included executables. FSO's Windows package would have something like this:
Code: [Select]
"executables": [
  {
    "name": "3.7.2 RC4",
    "file": "fs2_open_3_7_2_RC4.exe"
  },
  {
    "name": "3.7.2 RC4 (Debug)",
    "file": "fs2_open_3_7_2_RC4_DEBUG.exe"
  }
]
This way it's easier to put together a list of FS2 Open executables and it gives the packager the opportunity to specify nice labels. What do you think?

I've also added a new type to the packages' "environment" check. It's called "bool" and allows the packager to write down an expression like "(macos || linux) && X86_64 && sse".
@m!m: The converter automatically parses that expression. The above expression would appear in the repo like this:
Code: [Select]
"environment": [
  {
    "type": "bool",
    "value": ["and", ["and", ["or", ["var", "macos"], ["var", "linux"]], ["var", "X86_64"]], ["var", "sse"]]
  }
]
If you're interested, I can give you the full list of available variables.

Things still left TODO:

Things I plan to do after I release Knossos 0.1.0:
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on December 18, 2014, 12:55:02 am
@m!m: I added a new "executables" property to packages. It's not documented, yet, because I wanted your feedback on it. It allows a package to list all included executables. FSO's Windows package would have something like this:
Code: [Select]
"executables": [
  {
    "name": "3.7.2 RC4",
    "file": "fs2_open_3_7_2_RC4.exe"
  },
  {
    "name": "3.7.2 RC4 (Debug)",
    "file": "fs2_open_3_7_2_RC4_DEBUG.exe"
  }
]
This way it's easier to put together a list of FS2 Open executables and it gives the packager the opportunity to specify nice labels. What do you think?
I like that but it would be even better to replace "name" with something like this:
Code: [Select]
  "version": {
    "major": 3,
    "minor": 7,
    "release": 2,
    "debug": true,
    "build": 42,
    "tags": [
      "RC4"
    ]
  }
The launcher can then decide how to display that executable.

I've also added a new type to the packages' "environment" check. It's called "bool" and allows the packager to write down an expression like "(macos || linux) && X86_64 && sse".
@m!m: The converter automatically parses that expression. The above expression would appear in the repo like this:
Code: [Select]
"environment": [
  {
    "type": "bool",
    "value": ["and", ["and", ["or", ["var", "macos"], ["var", "linux"]], ["var", "X86_64"]], ["var", "sse"]]
  }
]
If you're interested, I can give you the full list of available variables.
TBH I don't think we need this level of fine control for environment flags. It is only needed for executables for which our existing system should be sufficient.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on December 18, 2014, 06:04:54 am
I like that but it would be even better to replace "name" with something like this:
Code: [Select]
  "version": {
    "major": 3,
    "minor": 7,
    "release": 2,
    "debug": true,
    "build": 42,
    "tags": [
      "RC4"
    ]
  }
The launcher can then decide how to display that executable.

Can we agree on the semver (http://semver.org) standard and shorten it to something like this?
Code: [Select]
  "version": "3.7.2-RC4+42",
  "debug": true
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on December 18, 2014, 07:28:31 am
Yes, semver is much shorter and it contains all the necessary information.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: LaineyBugsDaddy on December 22, 2014, 11:27:30 am
Just submitted a bug report on this tool. I cannot make it work. See report at https://github.com/ngld/knossos/issues/15
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on January 26, 2015, 06:39:49 pm
Oh, look an update!  :nervous:

The nebula now has its own domain! You can reach it at https://fsnebula.org/nebula. The complete site is served over SSL.
However, until Hellzed publishes the next version, it won't get any new features.

Recent changes:

I still have to fix two bugs:

I'll try to finish the 0.1.0 release this week.
The test repository contains a few mods and FSO 3.7.2 RC4. Once the release is out, I'll add RC5, Nightlies and Swifty's "Deferred Lighting and soft shadows" build. If anyone wants a specific mod added, just post a link to the mod's release post or download page and I'll try to add it ASAP.

@m!m: The link is https://fsnebula.org/repo/test.json. Your launcher has to support the "includes" field and TLS >= 1.0 to access that repo.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: chief1983 on January 28, 2015, 08:54:37 pm
The amount of work put into this project in such a short time span astounds me.  I'm watching this project closely.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: X3N0-Life-Form on January 29, 2015, 12:33:44 am
Indeed. This is impressive.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on January 29, 2015, 03:04:15 am
@m!m: The link is https://fsnebula.org/repo/test.json. Your launcher has to support the "includes" field and TLS >= 1.0 to access that repo.
Thank you very much for setting up that domain and the server. I just tested my client an apparently .NET supports SSL/TLS out of the box :D
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on January 29, 2015, 06:23:37 pm
The amount of work put into this project in such a short time span astounds me.  I'm watching this project closely.

Thanks! However, most of these changes are UI changes which expose the functionality. I've been working on this new version for several months now (https://github.com/ngld/knossos/commit/c1d3f40f6f408e0fa95f1a9c634b02b82b1e90cd).
The first commit was over a year ago but that back then I was still trying to use Goober's textfiles. m!m later convinced me to use a new schema. Hellzed wrote the website (fsnebula.org) which (once it's finished) will allow modders to add/publish new mods and updates.
I guess I just want to finally release a stable version.  :lol:

Thank you very much for setting up that domain and the server. I just tested my client an apparently .NET supports SSL/TLS out of the box :D
Nice!

I've just finished the last changes. Once the packages are done, I'll edit this post and add a link to the installation instructions.
EDIT: Done.

Here are the installation instructions (https://dev.tproxy.de/knossos/). If you have problems installing Knossos or your OS is missing, feel free to post here or on the bugtracker (https://github.com/ngld/knossos/issues).
It would be nice to get some feedback regarding the UI (Is it too complicated? Is it self-explanatory or should I add more text?).

I'm looking forward to your bug reports  :).
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: chief1983 on January 29, 2015, 06:42:13 pm
Oh I know.  I was referring to everything since page 1.  Has it been a year?  Still seems fast to me.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Goober5000 on January 29, 2015, 08:14:32 pm
back then I was still trying to use Goober's textfiles

They're not really mine; Turey invented the format and I continued to use it.  It would probably be a good idea to make the FSO Installer compatible with the Knossos spec.  Do you have a spec definition somewhere?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on January 29, 2015, 08:33:31 pm
They're not really mine; Turey invented the format and I continued to use it.
I didn't know. I just saw that your installer used it and never actually wondered how Turey's installer worked.

It would probably be a good idea to make the FSO Installer compatible with the same spec.  Do you have a spec definition somewhere?
Here's the spec (https://raw.githubusercontent.com/ngld/knossos/develop/converter/schema.txt), it's split into two parts. The top part is the format my converter expects. The final JSON file contains all the fields listed in the first section but the "files" property is replaced with the format documented in the second section and the "filelist" property is added which contains the file layout of the installed mod.
As an example: This is the JSON file for FSO 3.7.2 RC5 (https://fsnebula.org/repo/fso_3.7.2_RC5.pretty.json).
The filelist describes which files should be in the mod folder, which archive contains the file and the path in the archive. The "files" property lists all archives and their URLs.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Goober5000 on January 29, 2015, 08:53:22 pm
Thanks.  I'll put this on my lengthy to-do list. :nervous:
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: niffiwan on January 29, 2015, 10:49:36 pm
Here are the installation instructions (https://dev.tproxy.de/knossos/). If you have problems installing Knossos or your OS is missing, feel free to post here or on the bugtracker (https://github.com/ngld/knossos/issues).
It would be nice to get some feedback regarding the UI (Is it too complicated? Is it self-explanatory or should I add more text?).

Nice work! The latest version showed up as an update in my package manager (on Mint 17) so super-simple to upgrade  :yes:

I also noticed a possible memory leak (https://github.com/ngld/knossos/issues/16) during the install; it's on the bugtracker.

Regarding the UI, it mostly seems fine to me. The dropdown + checkbox for the exec flags seems a little bit counter-intuitive.  That could be just because I'm used to seeing all the checkboxes in one big list (like wxLauncher).
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on February 03, 2015, 04:09:42 pm
I've released 0.1.1. The new version should fix the memory leak and an issue which might cause a crash.
The installation instructions (https://dev.tproxy.de/knossos/) now also contain update instructions.

Nice work! The latest version showed up as an update in my package manager (on Mint 17) so super-simple to upgrade  :yes:
I've tried to make installation and updating as painless as possible on all supported platforms. Seems like the effort was worth it.  :)

Regarding the UI, it mostly seems fine to me. The dropdown + checkbox for the exec flags seems a little bit counter-intuitive.  That could be just because I'm used to seeing all the checkboxes in one big list (like wxLauncher).
Well, I copied that from YAL because that's what I've used until now. :nervous: I might change that in the next version.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: chief1983 on February 03, 2015, 04:31:47 pm
Out of curiosity, how does the installer support .dmg files?  Is it shelling out to the native OSX command line tools or is it using some sort of native library for extracting them?  Or does it just leave the .dmg files in place, like the Java installer did for a while?  :P
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on February 03, 2015, 04:38:08 pm
I'm using 7zip (that's why I said that it shouldn't be a problem for the installer).
I just launch the 7z command in the background to extract the archives. It can even extract DMG files. You have to call it twice, though. The first time it decompresses the DMG file and extracts the contained sections. Then I call 7z a second time to extract the 3.hfs file which contains a HFS image to get the DMG's content.

If you're using Linux, it pretty much works like this:
Code: [Select]
td="$(mktemp -d)"
7z x -o"$td" somefile.dmg
7z x "$td/"*.hfs
rm -r "$td"
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: chief1983 on February 03, 2015, 04:59:00 pm
Ah, I think the problem with the java one is that the java bindings for 7zip don't seem to support dmg extraction or something like that.  So is 7zip made available?  Is it a prerequisite or a provided dependency?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on February 03, 2015, 05:02:23 pm
The Windows and Mac builds both bundle 7z. The Linux packages have 7zip listed as dependency which means that the user never has to manually install 7zip.

EDIT: I've added FSPort to the package repo but I'm pretty sure the generated "-mod" flag is wrong so you probably won't be able to play it with my launcher.

@m!m: I use dependencies to generate the "-mod" flag but that gets tricky with FSPort since it uses "primarylist" and "secondarylist" in mod.ini. I'll try to think of a solution but I guess we'll have to add a new field to the mod object for that.

EDIT2: I've added nightlies to the repository. The list is updated once a day. If you want to install a specific nightly version, install the "FSO Nightly" mod, then click on the cog icon next to it, switch to the "Versions" tab and click on "Edit". A new window with all available versions/revisions will show up. Select the versions you want and click on "Apply".
After you installed a nightly, you should click on "Settings", select "Game settings" and select the nightly from the dropdown.

Inspired by this thread (http://www.hard-light.net/forums/index.php?topic=89140.0), I've thought about creating deltas/patches from one mod version to the next.
A patch would be a 7z archive which contains all added files, deltas (generated by xdelta (http://xdelta.org)) and a list of deleted files. I'll probably also add a list of checksums for all files from the old and new mod version.

I can implement the patch generation on the Nebula server which means that after a new mod version has been added to the Nebula, a patch would be generated automatically. However, I'm not sure if this is worth the effort...
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on February 12, 2015, 01:03:58 pm
And.... I'm back ! Resuming work on Nebula right now.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: chief1983 on February 12, 2015, 02:16:03 pm
One dream I had was that mods would include a list of supported binaries, and the launcher/installer would simply keep enough binaries available to cover any given mod.  There'd never have to be any manual binary downloading then.  And the binaries could theoretically live outside of the program data home folder, and just be told where to look for their game data...
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on February 12, 2015, 03:02:48 pm
@ngld: Sorry I missed your updates, apparently edits aren't shown as new in the thread list.

@m!m: I use dependencies to generate the "-mod" flag but that gets tricky with FSPort since it uses "primarylist" and "secondarylist" in mod.ini. I'll try to think of a solution but I guess we'll have to add a new field to the mod object for that.
Do you happen to know why fsports needs this special treatment? Maybe FSO can be changed so no special treatment is needed.

Inspired by this thread (http://www.hard-light.net/forums/index.php?topic=89140.0), I've thought about creating deltas/patches from one mod version to the next.
A patch would be a 7z archive which contains all added files, deltas (generated by xdelta (http://xdelta.org)) and a list of deleted files. I'll probably also add a list of checksums for all files from the old and new mod version.

I can implement the patch generation on the Nebula server which means that after a new mod version has been added to the Nebula, a patch would be generated automatically. However, I'm not sure if this is worth the effort...
Patching sounds like a great idea but FSO already has limited support for VP patching. If a mod needs to distribute an update it could publish a new version which depends on the old version. The launcher could then handle that just like any other dependency without needing to implement complicated patching mechanisms.

One dream I had was that mods would include a list of supported binaries, and the launcher/installer would simply keep enough binaries available to cover any given mod.  There'd never have to be any manual binary downloading then.  And the binaries could theoretically live outside of the program data home folder, and just be told where to look for their game data...
When the repositories contains all the relevant executable definitions it would be easy to determine a minimum executable version for a mod (a mod could even specify which executable versions to use).
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: AdmiralRalwood on February 12, 2015, 03:13:03 pm
@m!m: I use dependencies to generate the "-mod" flag but that gets tricky with FSPort since it uses "primarylist" and "secondarylist" in mod.ini. I'll try to think of a solution but I guess we'll have to add a new field to the mod object for that.
Do you happen to know why fsports needs this special treatment? Maybe FSO can be changed so no special treatment is needed.
It has nothing to do with FSO; primarylist and secondarylist are launcher-side features that just change how the -mod flags are ordered. Instead, it has everything to do with how FSPort works (or, to be more precise, Silent Threat: Reborn, since FSPort itself doesn't have a "[multimod]" section in its mod.ini). ST:R puts fsport-mediavps (or now fsport-mediavps_2014) in the primarylist and FSPort and the MediaVPs in the secondarylist; this is so that the model upgrades in the FSPort MediaVPs override the POF file definitions in ST:R's data files (because primarylist entries go before the selected mod).

The problem could probably also be fixed via a separate ST:R MediaVPs mod, but presumably the FSPort team decided that would be too much of a hassle when the primarylist feature exists.

EDIT: Corrected a silly brainfart.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on February 13, 2015, 07:16:59 pm
One dream I had was that mods would include a list of supported binaries, and the launcher/installer would simply keep enough binaries available to cover any given mod.  There'd never have to be any manual binary downloading then.  And the binaries could theoretically live outside of the program data home folder, and just be told where to look for their game data...
When the repositories contains all the relevant executable definitions it would be easy to determine a minimum executable version for a mod (a mod could even specify which executable versions to use).
Well, we're doing that already...  :p
If you run Knossos on a fresh retail install and tell it to install a mod, it'll automatically download the latest FSO build (which is 3.7.2 RC5 right now). It even uses the right version for the current OS and processor. The user still has to select a build in the settings window but I'm planning to automate that, too.

@ngld: Sorry I missed your updates, apparently edits aren't shown as new in the thread list.
I guess I'll just double-post next time.

@m!m: I use dependencies to generate the "-mod" flag but that gets tricky with FSPort since it uses "primarylist" and "secondarylist" in mod.ini. I'll try to think of a solution but I guess we'll have to add a new field to the mod object for that.
Do you happen to know why fsports needs this special treatment? Maybe FSO can be changed so no special treatment is needed.
Even if the problem with FSPort could be solved otherwise, it'd still be nice to have this field to make the modline generation easier. Resolving a mod's dependencies yields a tree which is troublesome (at least for me) because I'm not quite sure in which order the mods are supposed to be. If two mods in the dependency tree depend on the same mod, the exact order is unclear which makes it impossible to generate a proper mod flag. A simple, linear list could solve this.
IMHO, the best solution would be if the Nebula interface generated this list from the dependencies but allowed the modder to reorder it.

Inspired by this thread (http://www.hard-light.net/forums/index.php?topic=89140.0), I've thought about creating deltas/patches from one mod version to the next.
A patch would be a 7z archive which contains all added files, deltas (generated by xdelta (http://xdelta.org)) and a list of deleted files. I'll probably also add a list of checksums for all files from the old and new mod version.

I can implement the patch generation on the Nebula server which means that after a new mod version has been added to the Nebula, a patch would be generated automatically. However, I'm not sure if this is worth the effort...
Patching sounds like a great idea but FSO already has limited support for VP patching. If a mod needs to distribute an update it could publish a new version which depends on the old version. The launcher could then handle that just like any other dependency without needing to implement complicated patching mechanisms.
The idea was to conserve bandwidth and disk space. Although using VPs for patching would save bandwidth when a user updates, it'd use more disk space than a complete replacement. Another problem is that you can't remove any files that way and a fresh installation / download would use up more disk space and more bandwidth.
The patching mechanism isn't that complicated and the launcher wouldn't be required to support it since the normal downloads would still be available. For now, though, I'll concentrate on finding bugs and improving Knossos' interface.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Goober5000 on February 15, 2015, 01:26:18 pm
@m!m: I use dependencies to generate the "-mod" flag but that gets tricky with FSPort since it uses "primarylist" and "secondarylist" in mod.ini. I'll try to think of a solution but I guess we'll have to add a new field to the mod object for that.
Do you happen to know why fsports needs this special treatment? Maybe FSO can be changed so no special treatment is needed.
Even if the problem with FSPort could be solved otherwise, it'd still be nice to have this field to make the modline generation easier.

This is not a "problem".  FSPort is using a launcher feature that has been around ever since RandomTiger's original launcher release.  If you want a mod to be loaded before the active mod, use primarylist.  If you want a mod to be loaded after the active mod, use secondarylist.  You need two lists in order to specify all possible mod orderings.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on February 15, 2015, 03:47:52 pm
I'm sorry if my wording was misleading. The fact that Knossos isn't generating a proper mod flag for FSPort is a problem. I didn't want to say that FSPort had a problem.

You need two lists in order to specify all possible mod orderings.
Not necessarily. The list I was thinking of includes the current mod. The list for ST:R would look something like "fsport-mediavps,fsport-str,fsport,mediavps_2014".
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Goober5000 on February 15, 2015, 04:42:27 pm
I'm sorry if my wording was misleading. The fact that Knossos isn't generating a proper mod flag for FSPort is a problem. I didn't want to say that FSPort had a problem.
Ah, fair enough then.  I suppose I should have read that more carefully.

Quote
You need two lists in order to specify all possible mod orderings.
Not necessarily. The list I was thinking of includes the current mod. The list for ST:R would look something like "fsport-mediavps,fsport-str,fsport,mediavps_2014".
Well, the Launcher always omitted the active mod (in this case fsport-str) because it was inferred from the selection.  If you include the current mod, then yes, only one list is needed, but this wouldn't be compatible with existing mod.inis.  Unless you're thinking of just the internal representation.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on February 24, 2015, 11:54:42 am
@ngld: I encountered a problem with current configuration of the FSPort package. It specifies a dependency of ">=3.7.2" for the FSO package which obviously cannot be satisfied as the "-RC5" indicates a pre-release version which is not bigger or equal to 3.7.2. How do you handle this case or is it just a configuration error?

I would also propose creating separate threads for the different projects (nebula, knossos and the mod specification) that are being discussed in this thread. That should help keeping the discussion focused on the respective projects.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Hellzed on February 28, 2015, 05:43:11 am
@ngld : I created a new branch called hlp-nebula-next on GitHub. You can take a look at how extensive changes are... And this is only the original refactoring from december. I'm still working on new stuff, and have 2 other copies of the project on my hard drive (one for testing "team" features, the other for the better knossos server handling). Please do not switch to this branch for now, it's broken !
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: chief1983 on March 04, 2015, 01:45:59 pm
So, if a standalone game was interested in using this setup to manage an installation, how would one go about that?

Also, the FSO json file (https://fsnebula.org/nebula/ngld/FSO/default/current.json) seems overly complicated, why all the duplicated hashes and file references?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on March 04, 2015, 02:33:11 pm
Standalone games aren't really supported right now as the current mod definitions assume that FreeSpace is required by all packages.

The JSON schema is meant to be generated so having a few duplicated hashes is better as it makes parsing the files easier for the launcher.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: chief1983 on March 04, 2015, 03:31:57 pm
So would it theoretically be feasible to either try to support installing standalone games like Diaspora, TBP, or FotG from within one entire launcher/installer management interface, or to 'clone' the entire system for each standalone game environment?  There aren't many but I can see how having a common (if cloned) environment for all the FSO-based standalones out there would be useful.  I especially was interested in the mention about xdelta patching.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on March 04, 2015, 05:47:12 pm
Um... sorry for the late response...  :nervous:

@Hellzed: Thanks! I'll take a closer look at it once I've got more time.

@chief1983: Well, that's not the current FSO JSON file. Here's the current (prettified) one. (https://fsnebula.org/repo/fso_3.7.2_RC5.pretty.json)
That file is generated from this config (https://fsnebula.org/repo/fso.cfg.json).

I'm not sure what you mean by duplicated hashes. The checksums should be different as long as the relevant file isn't duplicated.
Regarding the file references: I guess you mean the "filelist" attribute. It always lists the path inside an archive and the path relative to the mod folder where the file should be stored. In many cases those two paths are the same but if you look at the Mac OS package for FSO, you'll see that the paths are different.

m!m's right, we haven't considered TCs, yet, but with a few changes to the launchers it should work fine. Actually, it might already work with Knossos because it doesn't really check whether FS2 is installed or not (maybe I should work on that?).
We won't have to clone the system since we (hopefully) can treat TCs like mods (as long as the retail VPs don't cause any problems).

An installer for TCs could work like this:
You could even do this without an installer: Tell people to install either m!m's launcher or Knossos and then click on a link like this (https://dev.tproxy.de/fs2/launch.php?install/blueplanet) to install the TC.

xdelta patching was inspired by this thread (http://www.hard-light.net/forums/index.php?topic=89140.0). I've only got a rough concept, right now. I'm not sure if it's worth the effort to implement it. However, if there's enough interest, I can build a prototype.

@m!m: Sorry about that! You're right, it should be ">=3.7.2-RC5". I haven't noticed because my SemVer implementation was bugged.  :banghead:
Creating separate release threads is a great idea! I'll write a proper specification and some implementation examples for the JSON schema and post it once it's done.
I'll create the Knossos thread once I'm done with the next release.

Hellzed suggested on IRC that we should keep using this thread for discussion. I agree because most topics discussed here are relevant to all three topics.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: chief1983 on March 05, 2015, 09:21:58 am
Ah, I only asked because I found that one listed under FSO on the Nebula page.  If there's a newer one now, I didn't see a way to find it when I looked on the website from page 14 (https://fsnebula.org/nebula), from which I found the FSO page (https://fsnebula.org/nebula/ngld/FSO/details).  So I guess I got a little confused about where the most current stuff might be found.  In that file there is a filelist array, which  has all the binary files within the archives and their hashes, and then later a files array, which has all the binary files within the archives and their hashes as well as more metadata about them, and a different organization.  So that's the redundancy I was talking about.  Also, anyone viewing JSON in Chrome doesn't need a prettified link, it does it for you now, as well as syntax highlighting :)

The current RC5.pretty.json still has the similar syntax, with md5 hashes seemingly duplicated across the file, but since that's just generated I guess that's not really an issue.  And since the md5s aren't in the cfg.json at all I take it the generator is actually downloading and extracting these files to generate the final json itself?  Seems risky to trust the generator to get a correct download instead of having hashes generated by the same end that created and uploaded the files.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on March 05, 2015, 09:55:48 am
Ah, I only asked because I found that one listed under FSO on the Nebula page.  If there's a newer one now, I didn't see a way to find it when I looked on the website from page 14 (https://fsnebula.org/nebula), from which I found the FSO page (https://fsnebula.org/nebula/ngld/FSO/details).  So I guess I got a little confused about where the most current stuff might be found.
I had to manually write the .cfg.json file for FSO because the current Nebula website doesn't support some of the newer fields. This is just a temporary workaround until the Nebula catches up.

In that file there is a filelist array, which  has all the binary files within the archives and their hashes, and then later a files array, which has all the binary files within the archives and their hashes as well as more metadata about them, and a different organization.  So that's the redundancy I was talking about.
That's not quite correct. The filelist lists the files within the archives while the files list contains the archives. The FSO binaries are only in the filelist.

The current RC5.pretty.json still has the similar syntax, with md5 hashes seemingly duplicated across the file, but since that's just generated I guess that's not really an issue.  And since the md5s aren't in the cfg.json at all I take it the generator is actually downloading and extracting these files to generate the final json itself?  Seems risky to trust the generator to get a correct download instead of having hashes generated by the same end that created and uploaded the files.
I wouldn't call it "risky" since the generator verifies the archives. If a download were corrupted, it would most likely fail to extract which in turn would lead to an error message.
I've designed the generator this way because it's supposed to be as simple as possible. Since the modder has to include download links anyway, I thought it would be better to download the files (and check that those links actually work) instead of requiring the modder to generate MD5 checksums.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on March 06, 2015, 12:06:10 pm
An installer for TCs could work like this:
  • Check if a launcher capable of handling fso:// links is installed. If not, install one.
  • Open the fso://install/<TC name> which would start the actual TC download/installation in the user's launcher. (This works just like the steam:// links.)
You could even do this without an installer: Tell people to install either m!m's launcher or Knossos and then click on a link like this (https://dev.tproxy.de/fs2/launch.php?install/blueplanet) to install the TC.
How about adding a commandline option to FSO that tells it not to use the root directory as a data directory? With that option the executables could be located anywhere and the launcher could specify all the mod directories using -mod, including the location of the retail/TC data. Mods would then add normal dependencies for TC/retail data. The launcher would need to supply the "retail" package if a retail installation is present as that is the only "mod" that isn't managed by Nebula.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: chief1983 on March 06, 2015, 12:13:32 pm
I've been wanting to be able to specify the root data directory as being different from the binary's current directory for a long time.  Would have many uses, although not sure if it's exactly what we'd need here.  In this case, the standalone would need its own root folder, separate from FS2's, and the binaries would have to be installed to that folder right now, the way they currently expect to find data.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on March 07, 2015, 05:58:53 am
I've been wanting to be able to specify the root data directory as being different from the binary's current directory for a long time.  Would have many uses, although not sure if it's exactly what we'd need here.
It's already possible on Linux and Mac OS. You just have to set the current working directory (CWD) to the root directory of your choice. On Windows it doesn't work because FSO automatically changes the CWD to the executable's directory. I wonder why this was done only on Windows...

How about adding a commandline option to FSO that tells it not to use the root directory as a data directory?
As a temporary workaround, we can copy the executable to an empty directory on Windows and change the CWD on other OSes. If you copy the executable somewhere, you should add the old directory to the PATH variable to make sure that FSO still loads the DLLs from there.

With that option the executables could be located anywhere and the launcher could specify all the mod directories using -mod, including the location of the retail/TC data. Mods would then add normal dependencies for TC/retail data. The launcher would need to supply the "retail" package if a retail installation is present as that is the only "mod" that isn't managed by Nebula.
That's a great idea! I'll add a note about this to the JSON spec.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on March 29, 2015, 05:43:29 pm
I've finally finished and posted the JSON spec (http://www.hard-light.net/forums/index.php?topic=89434.new#new). What do you think?

@m!m: Can you please take a look at the "notes on launching a mod"? This should solve our problems with mods which use primarylist (like FSPort). I've also added your suggestion regarding the "retail" dependency.

@ngld: I encountered a problem with current configuration of the FSPort package. It specifies a dependency of ">=3.7.2" for the FSO package which obviously cannot be satisfied as the "-RC5" indicates a pre-release version which is not bigger or equal to 3.7.2. How do you handle this case or is it just a configuration error?
I've started fixing the broken dependencies on fsnebula.org, if you still find something, let me know!
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on April 01, 2015, 05:14:36 am
Thank you for putting that post together, I have some points I'd like to discuss but I will post that in the thread.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on June 29, 2015, 06:32:16 pm
After the recent silence, I've started working on Hellzed's Nebula (https://github.com/ngld/hlp-nebula). The results look quite promising.
I'll hopefully be able to release a new version on fsnebula.org in a few weeks.

My current goals:

Nebula

Knossos

Once I'm done with the Nebula and added TC support, I'll open a new release thread here for Nebula and Knossos. Well, we'll see how that goes...
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: chief1983 on July 21, 2016, 12:46:32 am
I'm really sad to see that this was the last post, over a year ago.  Was any progress made?  I just managed to get what was in the master branch up and running (http://nebula.cliftongordon.com/nebula/) on my VPS that runs Linux nightlies.  Next will be to see how hard it is for me to get the Knossos side running.  Not wanting to get any further into tearing into it myself until I at least see how much uncommitted code is laying around.  I may be able to help with some of those tasks if need be.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on July 22, 2016, 09:26:11 am
I'm glad to see that there's still interest in this project. I stopped working on this a while after that post because I didn't have enough time to keep working on it alone. Since Hellzed stopped working on the Nebula, I forked it (https://github.com/ngld/hlp-nebula), updated a few dependencies and fixed a few issues. I'll commit a few missing fixes later on.

I've repaired fsnebula.org (https://fsnebula.org) which means you can register once again (https://fsnebula.org/account/register).
I'll write proper setup instructions for the server and post them later tonight.

I'd be happy to work on this project again. I'll post more on the current state and unfinished tasks once I have the time (either today or tomorrow).
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: chief1983 on July 22, 2016, 02:05:01 pm
Just noticed that I can't find a register link anywhere on the site itself.  I might see if I can find a good place or two to shoehorn that in.  Maybe the Sign In page for starters, and a Modders section under Getting Started?

Edit:  Also the email verification went to my spam, wonder if we can try to reduce the likelihood of that somehow.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Goober5000 on July 22, 2016, 04:01:07 pm
There were extended periods of time I stopped working on the FSO Installer due to lack of time, lack of motivation, or other priorities.  Keep at it.  This has solid potential.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: jr2 on July 22, 2016, 05:56:42 pm
Edit:  Also the email verification went to my spam, wonder if we can try to reduce the likelihood of that somehow.

I think individual users marking e-mails "not spam" should help, but that's per e-mail provider, so..
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on July 23, 2016, 10:16:39 am
Just noticed that I can't find a register link anywhere on the site itself.  I might see if I can find a good place or two to shoehorn that in.  Maybe the Sign In page for starters, and a Modders section under Getting Started?
Sounds good. I think we intentionally didn't include that link because we wanted to be sure that it works before opening registration to everyone but I don't think that matters anymore.
It's also missing a password reset function as well as a way to delete users. The user management is implemented using the FOSUserBundle (https://symfony.com/doc/current/bundles/FOSUserBundle/index.html) so it shouldn't be too hard to add.

Edit:  Also the email verification went to my spam, wonder if we can try to reduce the likelihood of that somehow.
I'll move the mail delivery to a proper mail server, hopefully that helps.

There were extended periods of time I stopped working on the FSO Installer due to lack of time, lack of motivation, or other priorities.  Keep at it.  This has solid potential.
Thanks for the kind words! It means a lot to me.


@chief: Apparently your Nebula installation can't send mails. I tried to register but didn't get the activation email.
To actually add mods to the Nebula, you'll also need the Knossos server. You can either use my installation (should be the default) or run your own. I've added documentation to Knossos server's readme (https://github.com/ngld/knossos-server/tree/develop#installation).
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on August 09, 2016, 04:36:30 pm
I think I've finally fixed all the bugs and issues I found.

Just noticed that I can't find a register link anywhere on the site itself.
The Nebula site now has a register link, users can reset their password if they have forgotten it and they can always change the password once they're logged in.

Edit:  Also the email verification went to my spam, wonder if we can try to reduce the likelihood of that somehow.
I've now changed the email delivery. The mails are now sent through a mail server with regular mail traffic, proper SPF records and all outgoing mails are signed with DKIM. I hope this fixed that problem.

@niffiwan: You tried to add a mod with files hosted on FSFiles. That should work now. The server on FSFiles always returns the HTTP code 206 "Partial content" which caused an error in my downloader. It expected 200 "OK" instead.

I've also implemented file uploads which should make it easier to add new mods which don't have any download links.



I think the Nebula (and the Knossos client) are now ready for beta testing. AFAIK all features have been implemented all that's left is improving the UI and fixing any bugs that are found.
The next steps are:


I think that's all for now. If you want to check out the source code, you can go to GitHub: Nebula (https://github.com/ngld/hlp-nebula) and Knossos (https://github.com/ngld/knossos).
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: niffiwan on August 09, 2016, 06:57:36 pm
@niffiwan: You tried to add a mod with files hosted on FSFiles. That should work now. The server on FSFiles always returns the HTTP code 206 "Partial content" which caused an error in my downloader. It expected 200 "OK" instead.

:yes:  Awesome, fixed an issue I didn't even report :D
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on August 09, 2016, 07:24:22 pm
That's why I love having server logs.  :)

I've toyed with the idea of having Knossos upload error reports automatically but I never actually implemented that because I think that people will complain about the possible privacy issues.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: jr2 on August 10, 2016, 11:18:09 am
Have it prepare the error report and ask the user if they wish to contribute to troubleshooting by uploading?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Goober5000 on January 29, 2017, 10:18:51 pm
:bump:

ngld, how are things going here?  There appears to be significant user interest in an alternative to the FSOInstaller for mod downloading, so it would be pretty cool to get this production-ready.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on January 31, 2017, 03:46:43 pm
EDIT: Please head over to the new thread (http://www.hard-light.net/forums/index.php?topic=93144.0).

Sorry for the late reply. I've updated fsnebula.org (https://fsnebula.org) and fixed a few issues I've found.
Right now, I'm fixing a few issues in the client which are caused by newer Qt versions...

Have it prepare the error report and ask the user if they wish to contribute to troubleshooting by uploading?
Once I'm done I'll finish this and release new versions.

After that I (only) need testing and uploaded mods on fsnebula.org (https://fsnebula.org) to get production-ready.
Everyone can register and add/upload new mods there. If you see your own mod on there and want to update it yourself, edit or remove it, feel free to PM me and I can give you access.

Other things on my wishlist:
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: niffiwan on January 31, 2017, 05:55:53 pm
great to see you back ngld!:)

One thing  which I thought could be useful based on comments in some other threads, would be some kind of feedback or rating system. Similar to what you see in (e.g.) the Play Store or on GOG, 1-5 star rating, short feedback paragraph. My only concern is noise... which makes me wonder about restricted access to "HLP veterans" to provide reviews, or something like that, with a short bio per reviewer. Maybe "FrikgFeek's recommended list - he likes really challenging campaigns", or "Battuta's must play list - he loves a great story" (sorry for putting possibly wrong words in yours mouths guys...).  Or maybe just make the feedback open... with a simple moderation system to prevent spam.

The aim would be to make it easier for newcomers to find new & interesting campaigns that would interest them; from among the large list of available campaigns out there.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Axem on January 31, 2017, 07:43:27 pm
I tried to add Vassago's Dirge to Nebula as a test and it said the validation server was down. :(

Might I suggest that you start a brand new thread so Nebula and Knossos are super easy to find and point people to? The first post is pretty confusing to new people (and even me!). And could you post a tutorial of sorts on how the fields should be filled out and how modders should organize their downloads? I did a lot of guessing and I have no idea if I actually did it right...
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on January 31, 2017, 08:52:22 pm
great to see you back ngld!:)
Thanks  :)

One thing  which I thought could be useful based on comments in some other threads, would be some kind of feedback or rating system. Similar to what you see in (e.g.) the Play Store or on GOG, 1-5 star rating, short feedback paragraph. My only concern is noise... which makes me wonder about restricted access to "HLP veterans" to provide reviews, or something like that, with a short bio per reviewer. Maybe "FrikgFeek's recommended list - he likes really challenging campaigns", or "Battuta's must play list - he loves a great story" (sorry for putting possibly wrong words in yours mouths guys...).  Or maybe just make the feedback open... with a simple moderation system to prevent spam.
This sounds like a good idea but it's something I'll probably implement after the basics (mod installing & updating) are working and stable.
I'm not sure about the feedback / comments since it seems that it would be better suited to a thread here on HLP. Your list idea reminded me of GOGmixes (Example (https://www.gog.com/mix/gogs_starter_mix)). Every user would be able to make their own list of recommended mods with comments and ratings which they could then link to from other sites.

I tried to add Vassago's Dirge to Nebula as a test and it said the validation server was down. :(
Sorry, fixed it. (One part of the project was moved to a different server and I forgot to change the domain.)

Might I suggest that you start a brand new thread so Nebula and Knossos are super easy to find and point people to? The first post is pretty confusing to new people (and even me!).
I'm planning to do that but I wanted to wait until I've released the next version of Knossos (see my last post). The first post is completely obsolete by now since it's still talking about Hellzed's shell script which served as the starting point for all this.

And could you post a tutorial of sorts on how the fields should be filled out and how modders should organize their downloads? I did a lot of guessing and I have no idea if I actually did it right...
I will do that (probably tomorrow) though I hope to make the UI self-explaining enough that a tutorial isn't necessary. I just checked and you filled everything out correctly.

For now I'll just write down a few notes which might help others:
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: Axem on January 31, 2017, 09:09:45 pm
Okay, that's cool. I was just confused by things like 'file destination'. I wasn't sure if it'd be relative to the root directory or the mod directory previously specified or what.

Also I noticed when trying to use the Knossos launcher, I couldn't get any nebula mods to show up. It also said there was an update available, but when I tried to, it failed to install.

And if this really takes off, do you think there could be something like categories or tabs for mods? Like Pre FS1, FS1 era, Reconstruction, FS2 era, Post FS2, Total Conversion, Full Game, Comedy, Experimental... or something. It might help people find what they're looking for.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on January 31, 2017, 09:52:40 pm
Okay, that's cool. I was just confused by things like 'file destination'. I wasn't sure if it'd be relative to the root directory or the mod directory previously specified or what.
It's relative to the mod directory. During validation the nebula checks your archives and if it finds the mod.ini in a subdirectory then it automatically adds a "Post-install scripting" action to move your files from that subdirectory into the mod directory. Whenever this happens you'll see "WARNING: Found mod.ini in folder [...]. I assume you forgot the move action; I'll just add it myself." during validation.

Also I noticed when trying to use the Knossos launcher, I couldn't get any nebula mods to show up. It also said there was an update available, but when I tried to, it failed to install.
Thanks, I'm looking into it.

And if this really takes off, do you think there could be something like categories or tabs for mods? Like Pre FS1, FS1 era, Reconstruction, FS2 era, Post FS2, Total Conversion, Full Game, Comedy, Experimental... or something. It might help people find what they're looking for.
That's why there's a categories field, once I add some options you'll be able to select them. However there's no way to view categories but that shouldn't be hard to implement.
I've added your suggestions as options if you edit your mod (go to the mods list, click on your mod, go to the "Mod details" tab and click "Edit mod metadata") you should be able to select them now.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: niffiwan on February 01, 2017, 03:53:50 am
:(

Code: [Select]
$ sudo add-apt-repository ppa:ngld/knossos
[sudo] password for mememe:
'This PPA does not support xenial'
Cannot add PPA: ''This PPA does not support xenial''.

Hopefully this is as simple to fix as rebuilding the ppa?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on February 01, 2017, 06:04:53 pm
Hopefully this is as simple to fix as rebuilding the ppa?
Yes.

I'm done with the next release, I'm currently building and testing the packages for Arch Linux, Ubuntu and Windows.

So far the only serious issue that's left is the way the "-mod" flag is generated since it differs from the way the other launchers work. If you're interested, here's the full explanation:

All other launchers (AFAIK) read the parameter from the mod.ini file but Knossos generates it by looking at a mod's dependencies which means that if an uploader didn't list all dependencies or in the wrong order, the mod most likely won't work. The most important reason for this difference is that Knossos allows a user to install multiple versions of a mod at the same time. To make this work the different versions have different folder names which means that the settings from the mod.ini won't work anymore.
I'm not sure if I'm going to keep this feature because it makes simple things like the -mod flag or update checks more complicated than they should be and I don't think many would use it. However, this system is essential to dealing with several FSO versions (http://www.hard-light.net/forums/index.php?topic=86364.msg1776937#msg1776937).
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: niffiwan on February 01, 2017, 06:24:50 pm
FWIW I think the feature sounds cool.. I really like the way you have a mod config ~per installed mod~, it makes switching between mods super, super easy, just click on launch for each mod & you've switched mods :)  wxLauncher's profiles are good, but not quite a simple, plus certain bits of config aren't carried over, like resolution, which makes switching between full-screen "i'm playing for fun" and window'd "i'm coding or modding" more difficult than they could be.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: jr2 on February 01, 2017, 10:03:01 pm
Hopefully this is as simple to fix as rebuilding the ppa?
Yes.

I'm done with the next release, I'm currently building and testing the packages for Arch Linux, Ubuntu and Windows.

So far the only serious issue that's left is the way the "-mod" flag is generated since it differs from the way the other launchers work. If you're interested, here's the full explanation:

All other launchers (AFAIK) read the parameter from the mod.ini file but Knossos generates it by looking at a mod's dependencies which means that if an uploader didn't list all dependencies or in the wrong order, the mod most likely won't work. The most important reason for this difference is that Knossos allows a user to install multiple versions of a mod at the same time. To make this work the different versions have different folder names which means that the settings from the mod.ini won't work anymore.
I'm not sure if I'm going to keep this feature because it makes simple things like the -mod flag or update checks more complicated than they should be and I don't think many would use it. However, this system is essential to dealing with several FSO versions (http://www.hard-light.net/forums/index.php?topic=86364.msg1776937#msg1776937).


Make it a compatibility option that can be toggled to resolve potential issues?
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: niffiwan on February 02, 2017, 05:08:09 am
How do you get Knossos to see the current list of mods on Nebula? Do you need to specific a different json file in the settings? I've current just got this: https://fsnebula.org/repo/test.json

Website @ https://fsnebula.org/workshop/mods
(http://storage3.static.itmages.com/i/17/0202/h_1486033643_6686116_aa5d522b39.png)

Knossos:
(http://storage4.static.itmages.com/i/17/0202/h_1486033586_3332562_1a6e2e3ea7.png)
(http://storage2.static.itmages.com/i/17/0202/h_1486033626_7264444_a54a2c4789.png)
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on February 02, 2017, 05:51:35 pm
EDIT: Please head over to the new thread (http://www.hard-light.net/forums/index.php?topic=93144.0).

Sorry for the long delay, I ran into some issues trying to build Knossos on Windows. Anyway, the release is done. The new version is 0.2.0 (you can click on Settings -> "Launcher settings" to check which version you have) and should be available for Windows, Ubuntu and derivates and Arch Linux. No macOS build since I don't have access to one. If a mac user reads this and wants to help, I'd be happy to provide instructions to build Knossos (it's pretty simple).

The installation instructions (https://dev.tproxy.de/knossos/) are in the usual place and the Windows install should update automatically.

An important change in the new version is that it now automatically reports errors and crashes to me. If you want to opt-out, you can go to "Settings" > "Launcher settings" and disable the "Automatically report errors" option.



Also I noticed when trying to use the Knossos launcher, I couldn't get any nebula mods to show up. It also said there was an update available, but when I tried to, it failed to install.
How do you get Knossos to see the current list of mods on Nebula? Do you need to specific a different json file in the settings?
Can you please check if this still happens with the latest version?

FWIW I think the feature sounds cool.. I really like the way you have a mod config ~per installed mod~, it makes switching between mods super, super easy, just click on launch for each mod & you've switched mods :) 
Thanks :)  I implemented the mod config feature because I often saw mod authors recommending specific flags for their mods and didn't want to remember which flags belonged to which mod.

Make it a compatibility option that can be toggled to resolve potential issues?
That sounds like a good idea... I'll add an option "Use -mod flag from mod.ini" to the "Troubleshooting" tab for each mod.


Here's the tutorial for uploading new mods. I hope it's not too long-winded.

Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: chief1983 on February 02, 2017, 05:56:30 pm
If you can help me out tonight I can give mac building a go.  Eating dinner, will be available in about an hour or so.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on February 02, 2017, 06:28:39 pm
Great! I'm on IRC so feel free to contact me there.
I assume you already have the Developer Tools installed. If you use Homebrew (http://brew.sh/), then the following steps will get you started.
EDIT: Forgot to mention that the build.sh script (globally) installs the Homebrew packages for 7-Zip, Qt and SDL2. Also it uses pip to install virtualenv globally, everything else is done inside the current working directory.
Code: [Select]
$ git clone -b master https://github.com/ngld/knossos.git
$ cd knossos/tools/mac
$ sh ./build.sh

If everything works, it should build a Knossos.dmg file in the mac folder though it's quite likely that it fails.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: m!m on February 03, 2017, 05:45:53 am
I am trying to get the latest version to run on Fedora 25 and I am having problems with the settings menu.

Whenever I try to open the settings dialog the application crashes with a segmentation fault. The last few lines of the output are:
Code: [Select]
dbus[28021]: arguments to dbus_connection_unref() were incorrect, assertion "connection->generation == _dbus_current_generation" failed in file ../../dbus/dbus-connection.c line 2822.
This is normally a bug in some application using the D-Bus library.

QObject::~QObject: Timers cannot be stopped from another thread

Where is the debug log? Maybe that contains more useful information.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on February 03, 2017, 09:12:15 am
There is no debug log but if you launch Knossos in debug mode ("KN_DEBUG=1 knossos"), the debug messages are printed on the console.

The dbus error happens on my machine, too, but it doesn't cause a crash. I haven't figured out what causes it since I'm not using it directly.
The second message is most likely the cause for the crash. In case it helps, here's the code that creates the settings window (https://github.com/ngld/knossos/blob/develop/knossos/windows.py#L342).

EDIT: Managed to reproduce the bug and track it down. If I call SDL_InitSubSystem(SDL_INIT_VIDEO), a segmentation fault occurrs in QPen::QPen() but the stacktrace leading to the crash contains no calls from my code nor SDL. I guess something weird happens in Xlib or XCB because both Qt and SDL2 are trying to use it.
This isn't the first time that this code leads to crashes. Since I only need SDL2 once to get a list of supported resolutions and joysticks, I'll launch seperate process for that which doesn't use Qt.
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: jr2 on February 03, 2017, 07:53:04 pm
<snip>
Great! I'm on IRC so feel free to contact me there.
<snip>

IRC is so last decade (https://discord.gg/YNVP3WR)  :p
Title: Re: [Cross-platform] FS2 GOG/Mod installer
Post by: ngld on February 07, 2017, 07:25:29 am
Posted a new thread (http://www.hard-light.net/forums/index.php?topic=93144.0) which will hopefully make it easier for people new to the project. I've also released 0.3.1 which fixes the bug mentioned before and another bug on Ubuntu which caused an ImportError because QtWebEngine was missing.