Author Topic: [Cross-platform] FS2 GOG/Mod installer  (Read 72200 times)

0 Members and 1 Guest are viewing this topic.

Offline Hellzed

  • 28
Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Vassago's Dirge is now available ! https://www.dropbox.com/s/b23gkitwj66pqtb/vassago.fs2mod
Installer has been updated to 0.0.30, a bug of download verification if the archive has spaces in its name has been fixed. Some harmless cleaning errors have been silenced too.

Transcend is now available too ! https://www.dropbox.com/s/be2c6xf24gvoeie/Transcend.fs2mod

Sync : https://www.dropbox.com/s/g6u5xqznfh0o2oo/sync.fs2mod
« Last Edit: January 06, 2014, 02:31:48 pm by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Hmm.... Wouldn't it make sense to make this script compatible with Goober's installer configs?
That way you don't have to write your own mod installation scripts... (the .fs2mod) files.

I'd do it but I'm wondering if it would be more useful to implement this in python. It would certainly make the coding easier but it would add python as an additional dependency.
What do you think? I'd like to try...

 

Offline Hellzed

  • 28
Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
I thought about it, and I like the idea. At first, I made fs2mod files this way because they are easy to generate with the md5sum command line tool and some very light editing, and they are easy to parse with tools like read/grep/cut/sed/awk. Goober's installer configs look a bit harder to parse in bash, and the logic behind them is really different (they are meant to be stored in a repository and somehow "invisible" to the end users, while fs2mod files are meant to be acquired and updated from various locations).

Porting this script to python would be nice, as it would be the first step to make it a cross-platform tool. I'm a beginner in python, so right now I have no idea how dependencies to innoextract, zip, 7zip, rar and some other command line tools are handled.

Did you try "apt" protocol links on Ubuntu ?
I have an idea, currently on hold (because Goober's installer is not stable yet, and we will have a talk with Goober and probably a few others about it once they have more spare time) :
a "fs2://" protocol to perform "one click" mod installs in FreeSpace 2, similar to "apt://" links in Ubuntu, and add these links to the mod database (and maybe make this database prettier). It's fairly easy to do, secure (because users only install what's available in their repository) but we would have to agree on the tools we want to use to do it (modify the new installer, or do something else...).

 

Offline m!m

  • 211
Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Trying to parse the installer configs is probably not a good idea if we have the chance to switch it to a better format like JSON which would also make parsing the configs quite easy. Instead of having to write a parser yourself in your language you could just use an existing JSON parser.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Thanks for the note m!m but I already finished the parser. The current format actually is pretty easy to parse since it uses linebreaks to separate values and uses only a few tokens.
JSON would surely make it easier to work with the files but I guess Goober has other priorities right now.

I've run into a small problem now. Goober's files don't contain all the information your script needs, Hellzed.
In particular the filelist (the vp file) and dependencies are missing.
I can write a script which automatically parses Goober's installer files and generates the needed information. I would be willing to host it, too.

Python scripts can run normal shell commands so it should be no problem to use innoextract, 7zip or other tools.

Your idea with the fs2:// protocol sounds nice! I think it would be nice if you could also add repositories using that protocol but with a question like "Do you want to trust this repository?".
That way every modder could use it without having to get their mod into a central repository. What do you think?

 

Offline Hellzed

  • 28
Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
I really haven't studied how Goober's installer works... but how can it perform install/upgrades/checks without using a VP files list ? Isn't it supposed to handle dependencies ?
Correct me if I'm wrong, but I'm under the impression that Goober's installer is "action oriented" (download that file, add this folder, remove that stuff) while my script is "description oriented" (this is how a functioning mod looks like, then the script does whatever is needed to get to this state)

Now I'm worried I did something completely inefficient... I already knew it from a disk space point of view, as Goober's installer downloads and unzip files at the same time, while my script creates a cache of archives.

My primary concern with auto adding new repositories, even with a nice warning window, is that people say yes to nearly anything. I'd prefer a safer process, like people having to go to some settings screen -> add repository -> copy and paste the url there.

Anyway, to advance towards a "One-click" mod install/launch process, we would need 3 pieces of software client side :
- a fs2 protocol handler, knowing which mod is installed and which one is not (and start the appropriate action).
- a mod installer : either some modified version of Goober's installer (it would need to get a flag passed by the fs2 protocol handler, with the name of target mod), or some cross platform version of my script connected to a default repository of fs2mod files, and able to connect to user added repositories.
- a mod launcher : either a modified version of wxlauncher, skipping all the GUI part, and using the last options profile used with the mod called by fs2 protocol handler, or a new lightweight launcher (only managing fs2_open flags, per mod).

Server side :
- a new HTML5+CSS3 GUI viewable from any browser, connected to FreeSpace mod database, with the appropriate fs2:// links. I wonder if optional extras for mods (compatibility packages, advanced graphics) would have to be handled by the protocol on the web GUI side (checkboxes on the mod page, a bit like DLCs on Steam) or by the mod installer script GUI (checkboxes in the mod installation confirmation window opening after clicking on a fs2:// link)

And, as optional but interesting extras :
- a standalone, offline, mod browser. A simple Webkit/python window showing installed mods even offline, without using a full blown browser, in order not to be dependent from HLP being online to launch FreeSpace 2 and its mods.
- a dedicated installer, bundling all previous pieces of software together.

It's not that much work, for example, for example, a basic version of the standalone browser I dashed was 15 lines of python code, the protocol handler is only a few lines too, the online and offline HTML5+CSS3 GUIs would be based on the same template (I already have a working example, although incomplete and not ready for production), and if we can use Goober's installer and wxlauncher stripped from parts of their GUI, it would only be small patches.

The biggest parts would be the preprocessor (PHP, Java, Python, whatever else) to generate the mod catalogue from the mod DB using enhanced versions of my templates, porting my script to a cross platform python utility if we chose this solution, and creating a lightweight launcher/settings window.
« Last Edit: January 27, 2014, 10:23:55 pm by Hellzed »

 

Offline Goober5000

  • HLP Loremaster
  • 214
    • Goober5000 Productions
Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
I really haven't studied how Goober's installer works... but how can it perform install/upgrades/checks without using a VP files list ? Isn't it supposed to handle dependencies ?
Correct me if I'm wrong, but I'm under the impression that Goober's installer is "action oriented" (download that file, add this folder, remove that stuff) while my script is "description oriented" (this is how a functioning mod looks like, then the script does whatever is needed to get to this state)
You're correct.  The installer lists files to download, and zips to extract.  The VPs are not explicitly specified, but they are obtained from unzipping the files.

Dependencies are not explicitly specified, but they are inferred from the installation hierarchy.  If a mod is listed as a child of another mod, it is assumed to depend on its parent.  (Of course, this can only be done if all mods are specified in the same config file.)

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Well... I'm mostly done with my Python port by now. I'll publish it once I've solved the last Windows bug.

Here are some quick screenshots:


BTW, is it fine to use the HLP logo as icon of the windows exe?

 

Offline Goober5000

  • HLP Loremaster
  • 214
    • Goober5000 Productions
Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
BTW, is it fine to use the HLP logo as icon of the windows exe?
I would think so.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
I've finished the first version of the python port. The code is on Github.

For Windows I've built a package which should work out-of-the box: fs2mod-py.exe

For everyone else: It runs on Python 2 and 3, needs either PySide or PyQt4 as well as six (another python module) and 7zip. It doesn't matter where 7zip is as long as it's in your PATH.
The mod list should be the same Goober's installer shows.

Click (or double-click if you're on Windows) to see details for each entry. The installer should automatically fix the problem with the MediaVPs folder name (all files will be placed in one mediavps_3612 folder and not in mediavps_3612 and MediaVPS_3612) on Linux. It should also automatically detect installed mods (if they are listed) and should be able to detect corrupted files.

Screenshots:

« Last Edit: February 08, 2014, 12:09:29 pm by ngld »

 

Offline Hellzed

  • 28
Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
I followed your instructions (installing dependencies and trying to run manager.py) but there is an error :
Code: [Select]
$ python manager.py
Traceback (most recent call last):
  File "manager.py", line 1011, in <module>
    main()
  File "manager.py", line 930, in main
    os.chdir(os.path.dirname(__file__))
OSError: [Errno 2] No such file or directory: ''

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Argh! Thanks, fixed it.

 

Offline Hellzed

  • 28
Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
I'm not sure how exactly it mixes fs2mod files and Goober's installer files management, if it doesn't bother you, could you explain that part a bit ?
(is it possible to add our own fs2mod files yet ?)

EDIT : Anyway, this is very good work, I just installed Windmills, no issue.
« Last Edit: February 08, 2014, 11:43:35 am by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
The code uses two classes to represent the two file types: ModInfo in fso_parser.py and ModInfo2 in fs2mod.py.
ModInfo2 extends ModInfo and adds a file list as well as a dependency list. Those are the only things present in your fs2mod files but not in Goober's installer files.

Right now I'm using converter.py to fetch and parse Goober's files and generate add the additional infos needed by inspecting the mods, the result is a JSON file which is downloaded and parsed by manager.py .
The contained information is mostly the same except for the dep and update parts. I've left the update part out because the program (right now) uses a central repository to find the JSON files.
I'm generating the dependency list from the mod's mod.ini file so it's actually a list of folders the mod needs. The manager figures goes through its list of known mods to find matching mods (i.e. it would see mediavps_3612 in a dependency list and see that the MediaVPS contain a "mediavps_3612" folder).

You can actually use the converter to generate fs2mod files:
  • Run "python converter.py list" to get a list of available mods.
  • Type "python converter.py convert MODNAME FILENAME.fs2mod" to generate a fs2mod file. You'll need to change the update and dep files.

Right now the manager (nor the converter) can read fs2mod files. I've only implemented my special JSON version, yet, but I can add support for fs2mod files.
Should I add a button "Open fs2mod file" or something else to the manager? I'm thinking of letting the user add/remove repos as well.

EDIT: Fixed a small bug in converter.py, please update if you run into that.
« Last Edit: February 08, 2014, 12:11:37 pm by ngld »

 

Offline Hellzed

  • 28
Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
You could add a "Mod sources" tab, default source being the main Hard Light Project repository, other sources could be the different fs2mod files. I'm still thinking of a good way to  chose which source should have priority.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
The manager can load mods from multiple sources and display all of them at them same time. The priority will only matter if the installer encounters the same mod in multiple sources.
In that case it would make sense to pick the most recent version but what should happen, if the versions are the same? (i.e. The mod manager might find the Sync mod int the HLP sources and in an fs2mod file the user added.)

As long as the official versions are the most recent ones I'd say that they should have a higher priority but if the user adds a source with some newer version, that source sould be picked.

 

Offline Hellzed

  • 28
Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Maybe we could add explicit versioning in fs2mod files (a "version" text file). I would say, if users added a fs2mod file, we can safely assume that this is the version they want.

(Something is broken in the latest version of the installer : if I try to update the mod list, the progress bar window shows up and sits there, doing nothing)
Code: [Select]
INFO:Thread-4:util.get: Retrieving "http://dev.tproxy.de/fs2/all.json"...
ERROR:Thread-4:manager.work: Failed to decode "('json', 'http://dev.tproxy.de/fs2/all.json', "ngld's HLP Mirror")"!
Traceback (most recent call last):
  File "/usr/lib/python3.3/json/decoder.py", line 367, in raw_decode
    obj, end = self.scan_once(s, idx)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "manager.py", line 97, in work
    data = json.loads(data)
  File "/usr/lib/python3.3/json/__init__.py", line 316, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.3/json/decoder.py", line 351, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.3/json/decoder.py", line 369, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded


 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
Sorry, my repository was broken. It should work again.

I've finished my implementation of fs2mod files. Now that it's working I see another problem: There is no way to detect if two mod entries actually refer to the same mod.
For example:
  • Goober's files list the MediaVPs as "FSU MediaVPs, version 3.6.12" your fs2mod list them as "Media VPs 3612". The title is different but the subdirectory is mediavps_3612.
  • That won't help either because Derelict and the Derelict Voice Pack both install to derelict.

To load a fs2mod file (in the current dev version) a user can now go to "Settings", click "Add", select the file and press "OK". The next time the user clicks on "Update List" te selected file and all dependencies will appear in the mod list.
This means that that you will see both "FSU MediaVPs, version 3.6.12" and "Media VPs 3612" in your list. The script should mark both either as installed or not as they refer to the same files.
If you don't have the MVPs installed it should show something like "Not installed (10 shared files)" which means that the manager found 10 files this package shares with others.

Maybe comparing the filelists will help finding duplications...
Right now this version is on the develop branch on Github. I'll merge it into the master later.

 

Offline niffiwan

  • 211
  • Eluder Class
Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
This is looking good.. unfortunately I haven't been able to get it running on Linux so far :)

I've tried on Mint 13, but the version of six seems to be old and it doesn't have six.PY2 available. Do you know what the the minimum version requirement for six is?

I've also tried on Centos 6 (w/ EPEL).  It runs with python2, but the button "Select installed FS2 (Open)" doesn't seem to do anything.  I get the same results from both the master and develop branches...
Creating a fs2_open.log | Red Alert Bug = Hex Edit | MediaVPs 2014: Bigger HUD gauges | 32bit libs for 64bit Ubuntu
----
Debian Packages (testing/unstable): Freespace2 | wxLauncher
----
m|m: I think I'm suffering from Stockholm syndrome. Bmpman is starting to make sense and it's actually written reasonably well...

 

Offline Hellzed

  • 28
Re: [Linux (& Mac ?)] FS2 GOG/Mod installer, first alpha !
I have updated the fs2mod file for Derelict : it now has a "version" file, with a version string. It should make version management easier.
Other fs2mod files will be updated soon.

Vassago's Dirge fs2mod file has been updated with a version string too, and now compatible with media VPs 2014.

EDIT : I think there's an issue with the update process, though, since it doesn't get rid of the outdated VPs.

EDIT 2 : it's around line 1294 in the original bash script, a saved version of the old fs2mod file is compared with the updated fs2mod file, and files that don't belong there any more are removed.
« Last Edit: February 09, 2014, 10:49:51 pm by Hellzed »