Author Topic: [Cross-platform] FS2 GOG/Mod installer  (Read 72258 times)

0 Members and 1 Guest are viewing this topic.

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
You published some new code right after I have rebased my repository. If I send a pull request, what happens ? Can you solve the conflict on your side ? (my files are missing the reset code)

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
It should work alright... You actually pulled in the reset code according to the commit graph.
When you create the pull request, Github should show you a list of changed lines. If it only shows your changes it's all right.

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
I posted the pull request, but I didn't see any conflict warning about the reset code. It's only shown in red, as deleted code, so I guess you will have to be careful about this one.

EDIT : Now the installer is almost in a working state (except for the double download bug), I will resume work on the HTML5 + CSS3 template for a mod browser.. Hopefully, most users won't have to directly use the mod browsing/sources interface (it has to be there for advanced users), but will use their web browser or a dedicated webkit/Qt browser (with a local mod list cache if people are offline) as the main hub for playing FreeSpace 2.

EDIT : oh, I see the changes you have made in my commits (indentation, spacing). I will try to respect PEP 8 next time. I'm still reading tutorials about python.
« Last Edit: February 11, 2014, 04:28:31 pm by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
I merged your changes and pushed my fix for that bug. I've made some changes. Is that alright?

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
Thanks ! That's a good way to learn !

On a side note, we may need a higher resolution icon, since it renders upscaled (and blurred) in Unity dash, probably in Gnome-shell and Windows 8 too, but i'm not sure.

Edit : the error I get if I try to launch fs2_open : "Starting FS2 Open (/home/loris/FreeSpace 2 beta/fs2_open_3.7.1_llvm_BP) failed! (return code: -11)"

Edit 2 : I think fs2mod format needs a new evolution to support optional VPs.

- A keyword (like "opt") may be added at the very end of the line for some VPs (packagers will have to be careful about the number of ";" separators, because they have to take into account the *subdirectory* optional argument).
- An "opt" file listing the optional VPs : easier to do, cleaner. *All* VPs are still in the "vp" file, and this list only have optional VP names. This way, some of these VPs can be excluded selectively from the install process.

These not ideal solutions, and I recognize fs2mod is not that a great format (I just wanted something easy to parse in bash and to generate from command line md5sum tool when I wrote the first one) but it has the advantage not to break the legacy bash script (I may modify it... someday).

What fs2mod format does not need is support for mod description and picture, since description, pictures, download size, maybe user comments are to be included on the "launcher" web page.

What do you think ?
« Last Edit: February 11, 2014, 05:26:58 pm by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Well here's the HLP icon, I just scaled it down because it's too big.

I think it's easier to list the optional VPs in a separate file like you suggested. However, the packager should be able to provide a description of the optional VP files similiar to what the Media VPs have. Without a description the user wouldn't be able to tell if he wants that file or not.

The description and picture would be nice to have but I might implement a way to automatically read those once a mod is installed.
BTW, what do you think about a wizard to create a fs2mod file? It would ask you to enter all URLs, then download the archives and generate the hashes (for the download file). Afterwards it can look inside the archives to generate the vp file. That would leave the title, update and dep files. (And, of course, the opt file). Title and update URL can be simple input fields, I guess. For the dep file I'd list all folders referenced by the mod.ini and allow the packager to enter a fs2mod link for each dependency or leave it empty (in which case it would be ignored). For the opt file, the wizard could simply show a checklist.

The fs2mod format is a simple format which makes it easy to parse and extend (just look how short read_zip() is...) IMHO. If you wanted to implement a parser for VP files, it would be much more work.

I guess I'll create issues over on GitHub for some of the current TODOs. For example, fs2mod-py still doesn't have a proper mod update routine, the user can't specify the mod source order (aka priority), I haven't fixed your FSO crash (might check wxLauncher's source on that) and it can't check for a new version (which can be important if the mod source / fs2mod format changes again).

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
256x256 hlp.png icon will be in my next pull request. It already looks better in Unity.

A wizard is a good idea, of course.

I'm searching for a way to trigger an event if the main window is raised, what's the best way to it ? (it has to do with a last bit of Unity integration : highlight the icon and make it wriggle once all tasks are finished (for example, firefox and my torrent software do that), don't do it if the window is already raised, stop doing it if the user raises it). It will use libnotify too.

I'll study the install/update task code, at first for better debugging purpose, and maybe contribute later.

[attachment deleted by an evil time traveler]
« Last Edit: February 12, 2014, 05:38:13 am by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Here's the documentation for the Qt classes and this a list of members of main_win.

You can check if the main window is active (raised) by calling main_win.isActiveWindow(). There's no signal like "clicked" or "itemActivated" that is triggered but the focusInEvent is raised once the window gets focus. If you want to use it just write
Code: [Select]
main_win.focusInEvent = your_handler_function

The install/update code is run by the InstallTaskin tasks.py. Most of the actual code is in fso_parser.ModInfo and fs2mod.ModInfo2.

EDIT: That looks nice!

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
Actually,
Code: [Select]
main_win.focusInEvent = your_handler_function wherever I put it in manager.py (I mostly tried to put it in the block of unity specific code in main() ), the handler function never gets called. There is no error in the console. Did I misunderstand something ?

(Apparently, it has to be reimplemented to be used)
« Last Edit: February 12, 2014, 09:36:46 am by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
You're right, my bad. Apparently you have to use changeEvent. So the complete solution would be:
Code: [Select]
def focus_handler(event):
  if event.type() == QtCore.QEvent.Type.ActivationChange:
    if main_win.isActiveWindow():
      print('I\'m active!')
    else:
      print('I\'m inactive!')
 
  return super(QtGui.QMainWindow, main_win).changeEvent(event)

[...]
  main_win.changeEvent = focus_handler

Since we are overriding a method of the main window a cleaner solution would be to actually subclass QMainWindow, i.e.:
Code: [Select]
class MainWindow(QtGui.QMainWindow):
  def changeEvent(self, event):
    if event.type() == QtCore.QEvent.Type.ActivationChange:
      if main_win.isActiveWindow():
        print('I\'m active!')
      else:
        print('I\'m inactive!')
   
    return super(MainWindow, self).changeEvent(event)
Then you would have to replace
Code: [Select]
main_win = util.init_ui(Ui_MainWindow(), QtGui.QMainWindow()) with
Code: [Select]
main_win = util.init_ui(Ui_MainWindow(), MainWindow()).
The end result is the same but the latter solution is cleaner.

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
It works perfectly, now. Though, I had to access the "unity_launcher" global var from inside the new class. Is it considered ok, or is there any cleaner way to do it ? (it should not cause any issue we would never use unity_launcher without initialising main_win first)

I prefer to wait before adding notifications, since better notifications (2 cases : mod install successful or failed) will depend on the changes you plan to make to the tasks (cancelling task from progress_win, timeout on downloads, a good way to know if a mod install ha succeeded or failed...).

EDIT : Found a bug in the sources reordering code : when you reorder things, sources duplicate.

EDIT 2 : Also, I found a problem with dependencies handling (at least with fs2mod) : if you mark if you mark a (not installed)  mod dependecy for install and its parent mod (installed) for uninstall, the installer won't complain.

EDIT 3 : I'm not sure what exactly is happening, it could also be an issue caused by broken re-ordering, under some conditions, mods dependent on others are marked as broken because there is a source with higher priority than the one of their parent.

A mockup of a new mod browsing site relying on fso:// links (the cart is for redirecting to Good Old Games):


Bonus video : https://www.youtube.com/watch?feature=player_detailpage&v=LpzkGHFkEpo
« Last Edit: February 13, 2014, 01:12:07 pm by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Most likely I'll have to re-implement the re-ordering mechanism because I haven't found a way to reliably read the new order of the sources after the user finishes the drag & drop.
Right now I'm not forcing any dependencies on the user because most of those are automatically detected. In that case that this auto-detection generates wrong results I want to allow the user to correct the program.
Maybe I'll add a new option on the Settings tab called "Enforce dependencies" which will be on by default.

The broken re-ordering shouldn't affect the dependency tracking. A mod is only marked as broken if one of the files has a different checksum than listed in the mod source. Can you post the the check messages in the notes area of the broken mod, please?

I just watched your video... it looks nice! :D
What do you think about adding some kind of splash screen to the mod manager while it's starting FS2? At first you see the "Working..." window but after that you don't know if the manager crashed or if it just disappeared because it's starting FS2... Some kind of window that says "Starting FS2..." (or whatever else fits) would be nice.
I also want to change the automatic list update: If the manager finds the mod and it's installed it can simply launch FS2 otherwise it could update the list and ask the user if he wants to install the mod. What do you think?

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
So, I really don't know what's happening. I'm missing something in this bug pattern.
Somehow, one time it worked : the repository had a low priority, Silent Threat Reborn (fs2mod) had a higher priority. As fsport and fsport-mediavps are dependencies for Silent Threat Reborn, fsport and fsport mediavps were tagged as higher priority than the repo, and marked both as installed (no warnings, fs2mod version number).
Then I removed both the repo and STR fs2mod, re-added them (first, the repo, then STR fs2mod). Now only fsport and fsport-mediavps say they are  installed (!) and Silent threat reborn says it's missing files. Also, STR only shows in the FreeSpace Port tree, and doesn't show a second time as an independent mod in the global list. fsport-mediavps shows as the fs2mod version "3.4" in the global list, but as in the repo "version 3.4" in the fsport tree.

And of course here's the (predictable: files from the fsmod are checked against the repo reference) error message
Code: [Select]
Check messages:
* File "ToggleMediaVPs.bat" is missing.
* File "mod.ini.without-mediavps" is missing.

Dependencies:
* FSPort Media VPs (installed)
* FSU MediaVPs, version 3.6.12 (installed)
* FreeSpace Port (installed)

Contents:
* fsport-str/ToggleMediaVPs.bat
* fsport-str/fsport-str.bmp
* fsport-str/fsport-str.vp
* fsport-str/mod.ini
* fsport-str/mod.ini.without-mediavps
* fsport-str/readme.txt

A splash screen is a good idea, and not auto-updating a mod in a working state if driectly launching it was requested is in my opinion a good behaviour ! (if it's not broken, don't fix it !).

Bug : I can add the same fs2mod file more than one time.
« Last Edit: February 14, 2014, 06:45:06 am by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
I've fixed the mod source order now, could you check if this resolves your bug?
I also added a splash screen. It's simple but I think it's enough. I also added an error message in case the user tries to add the same fs2mod file twice. It only works if you try to add the same file twice. If you copy a fs2mod file and add the copy the script won't complain (yet).

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
Currently it doesn't run :
Code: [Select]
Traceback (most recent call last):
  File "./manager.py", line 42, in <module>
    from ui.splash import Ui_MainWindow as Ui_Splash
ImportError: No module named splash
(looks like a typo)

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Forgot to add a file to git (ui/splash.py). Should be uploaded now.

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
Bug fixed ! One strange thing though : is it intended behaviour that optional VPs from lower priority sources are appended to higher priority sources in the mod list ? (for example Silent Threat Reborn Voice pack (from the main repo) is appended to Silent Threat Reborn (from the fs2mod file) in the main list. And STR doesn't appear any more in the FreeSpace Port Tree) This is something rather interesting. I'm not sure, but it may even be useful in some rare cases. Or break things.
If optional VPs are implemented in fs2mod files, maybe it would be better not to mix optional files from 2 different sources.

You should add a hint that mods can be reordered, and where is the higher priority (bottom). Actually, it's the contrary of most pieces of UI I know (top priority on top, for example languages in Gnome language properties, keyboard maps in Gnome control center...).

The "Cancel" button doesn't seem to do anything at the moment. I guess it's only added to the UI and not connected to any callback yet ?
EDIT : I just checked the code, it is connected to a function which is supposed to abort tasks. So, this is a bug.
« Last Edit: February 14, 2014, 03:53:15 pm by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Actually it does something: It calls the "abort" method on all currently running tasks but as noted in progress.by (where this method is implemented):
Code: [Select]
# Empty the work queue, this won't stop running workers but it will
# stop calls to the work() method.
It won't stop the current work but it won't attempt to process any queued work (i.e. if it's still downloading files, it won't abort running downloads but it won't attempt to start a new download).
Also "self.aborted" is set to True which can be handled in the "finish()" method.
I'll be able to add support for aborting running downloads and innoextract but I'm not sure if I'll be able to interrupt 7z when it's extracting files.

You're right about the priority order, I'll change it.
The problem with the mod tree currently is that every mod has a given parent and that parent is identified by its name. Now the STR Voice Pack has STR listed as its parent. The STR listed isn't the STR from the STR Voice Pack source but the manager doesn't detect this edge case currently. I guess I'll have to remove all optional parts of a mod (aka all children of that option) from the mod list when a package is overriden by another package from another source.
« Last Edit: February 14, 2014, 04:05:36 pm by ngld »

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
In the current code, where is the best place to know the final state of the tasks ? I didn't see a clear test to check if something has failed. (from a UI point of view, when the "Working" window closes -> this question has to do with freedesktop.org linux notifications/Unity notifications)

[About the overridden packages : overriding all optional parts of a mod (said mod should still show optional parts coming from the same source as proposed version) should be done only if the mod is from an explicit source (a source that appears in the sources list).
For mods coming from implicit sources (sources not appearing in the sources list because they have been set by the dependencies system) they should still show up. ]*

Imagine you install STR from a fs2mod file, which will have higher priority than STR from the main repo :
-> FreeSpace Port will be updated from the fsport fs2mod file which url is specified in STR fs2mod. That's ok.
-> If all fsport children (main repo version) are replaced by fsport children (fs2mod version), and if Awakenings fs2mod file hasn't been installed... well, we can't install Awakenings now : it's not any more in the list.

*We can reasonably assume if some users only installed STR fs2mod, and *not* fsport fs2mod, they may still actually want to install a fsport child from the main repo. Which, by chance, could happen to be compatible with his fs2mod version new parent.

*If they installed both STR and fsport fs2mod files (or at least fsport fs2mod has higher priority), there are chances they don't want to use anything from the main repo. So we don't have to worry about not showing, for example, Awakenings.

Maybe(or maybe not)  all this cross-sources dependencies and optional files thing should be part of the "enforce dependencies" setting :
- if it's "on" we disallow cross-sources dependencies (aka a mod can never "adopt" optional parts from another source, and by the way, we make sure that no broken mod will be installed)
- if it's "off" we allow cross-sources dependencies (and broken mods could be installed, if their parent is from a version which is to different to allow them to work properly)
As it's on by default, it's safer. We could name this setting "Safe mode" (with a tooltip explaining that without this option, mods that may miss dependencies, or broken by other (updated or older) mod parents may be installed).

Any thoughts on the issue ?
*EDIT : I just noticed I actually proposed 2 different default behaviours in what i just wrote. You can ignore the paragraph between brackets.

EDIT :
New screenshot (no slider yet in this one, but i'm working on it !)
New video, refined html and css, with an HLP nav bar.

EDIT : second screenshot, daily update on the web slideshow.

Another edit : I have attached a ui file for a simple settings window ( https://www.dropbox.com/s/kk15ermb0j8g4bo/settings.ui ).  I'm working on adding support for fso://settings/mod links in protocol.
The right behaviour would be (assuming we will have a unique settings profile per mod):
-> if a fso://run/mod link is accessed AND there is no existing settings profile for this mod, the launcher duplicates the default profile and open a settings window.
-> if a fso://run/mod link is accessed AND there is an existing settings profile for this mod, we launch the game directly.
-> if a fso://settings/mod link is accessed : we launch the settings window for the mod (this is what the "settings" icon next to each "Play" button on the web page is for)
-> if the fso://settings link is accessed : we launch the settings window for the default profile
-> if a fso://settings/<invalid string> : we display a warning about the mod not being found, BUT we launch the settings window for the default profile.

And of course, this window (either default settings or mod profile settings) should be available from somewhere in the main window.

I'm still not sure if tabled settings should be common to all mods or set per profile. The command line should be set per profile, since different mods may require different features or lighting settings. We should save the string stored in cmdline.cfg in the installer settings, for each mod, and the last command line, to restore once the game is launched.

I can start implementing it myself, but I will need some help understanding how to call the same settings window both as a main window (if accessed from a link)
and a modal window (if the ful mod manager is launched).

[attachment deleted by an evil time traveler]
« Last Edit: February 16, 2014, 07:57:08 pm by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Sorry for the late reply!  :nervous:

Most tasks have a "finish" method which is called once the task completes. I guess that would be the best place to check if it suceeded or failed. You can also check if the user aborted the task by looking at self.aborted (read the FetchTask's finish method for an example).

On the issue of cross-source dependencies: Your ideas are nice but I'm seeing a problem here. If the user installs STR using the fs2mod pacakge and we disallow cross-source dependencies, it will try to install the fsport fs2mod pacakge to fulfill its dependency. That, in turn, might break already installed fsport mods because the fsport fs2mod package might install different vp files. If we allow cross-source dependencies the STR fs2mod package might break because it isn't compatible with the installed fsport mod.

I propose a different solution:
  • The mod manager should compare checksums and filenames of all packages it finds (all HLP and fs2mod packages). If two packages contain the same set of vp files and their checksums match, we can assume that they refer to the same mod. I only want to compare the vp files because changes to a readme.txt and mod.ini should be irrelevant to mod compatibility.
  • If two packages refer to the same mod and one of them is installed, automatically mark the other one as installed.
  • If two packages share files but refer to different versions of those files (different checksums) install them in seperate subdirectories. IMPORTANT: The mod manager will have to check and correct all mod.ini references if a mod is installed into a different subdirectory! And it will have to remember the new subdirectory somehow.

Let's assume the most recent FSPort version is 1 and both the HLP package and the fs2mod package contain this version. The user installs Awakenings and the mod manager will install HLP's fsport to fulfill its dependency. Afterwards the user installs STR using your fs2mod file. Now the mod manager will see the fsport dependency but since the checksums for all vp files match up it will assume that the fsport dependency is fulfilled and simply install STR.

Now, the FSPort project releases a new version (2) and you update your fs2mod file. If the user installs Awakenings, the mod manager will of course use the HLP package (version 1) to fulfill the dependency. If the user installs STR afterwards, the mod manager will detect the conflict between the HLP package and the fs2mod package. To resolve the conflict, it installs the new FSPort (version 2) into fsport_2_fs2mod (or whatever makes more sense) and change STR's mod.ini to load fsport from fsport_2_fs2mod.

What do you think? This should avoid any possible conflicts and still allow the user to install mods from different sources even if their dependencies aren't compatible.

I think it only makes sense to make the command line flags mod-specific. You could also make the fs2 build mod-specific, if someone does the same thing BP did but most of the time mods should work with the most recent fs2_open release.
The wxLauncher even lets mod authors specify cmdline flags in their mod.ini files.

To use your settings window in the mod manager you need to place the .ui file in the ui folder and run "compile_ui.sh". It generates a .py file which you can then import using
Code: [Select]
from ui.settings import Ui_Dialog as Ui_Settings
To open the window in scheme_handler do
Code: [Select]
settings_win = util.init_ui(Ui_Settings(), QtGui.QDialog(splash))
settings_win.setModal(True)
settings_win.destroyed.connect(app.quit)
settings_win.show()
otherwise use
Code: [Select]
settings_win = util.init_ui(Ui_Settings(), QtGui.QDialog(main_win))
settings_win.setModal(True)
settings_win.show()

I added the extra line which connects the destroyed signal to app.quit() so that fs2mod actually quits once the user closes the settings window.