Hard Light Productions Forums

Modding, Mission Design, and Coding => FS2 Open Tools => Topic started by: ngld on June 18, 2017, 03:53:31 pm

Title: [Preview] Knossos 0.5.3
Post by: ngld on June 18, 2017, 03:53:31 pm
There's a new thread for version 0.6.0! (http://www.hard-light.net/forums/index.php?topic=94068.0)

(http://i.imgur.com/hGXAsii.png)

Here's the next release. The main purpose of this release is to test the new UI designed by Mjn.Mixael.

What works

What doesn't

Download and package links (https://github.com/ngld/knossos/releases)

During the first launch, Knossos will ask you to select a directory which it will use to store mods. Please select an empty directory.

I'll work on the Development tab next. Once that's done most of the functionality I want Knossos to have will be implemented. Here's a complete list of unfinished tasks (https://github.com/ngld/knossos/issues?q=is%3Aopen+is%3Aissue).
Title: Re: [Preview] Knossos 0.5.1
Post by: m!m on June 18, 2017, 04:40:04 pm
I hate to start this thread with a bug report but whenever I try to launch the application on Windows it crashes with the error "Failed to execute script __main__". When I launch it from a terminal, this is the output:
Code: [Select]
Traceback (most recent call last):
  File "C:\knossos\knossos\__main__.py", line 122, in <module>
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "C:\Python35\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module
  File "C:\knossos\knossos\launcher.py", line 70, in <module>
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "C:\Python35\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module
  File "C:\knossos\knossos\util.py", line 31, in <module>
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "C:\Python35\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module
  File "C:\Python35\lib\site-packages\requests\__init__.py", line 89, in <module>
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "C:\Python35\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module
  File "C:\Python35\lib\site-packages\requests\utils.py", line 42, in <module>
  File "C:\Python35\lib\platform.py", line 1095, in system
  File "C:\Python35\lib\platform.py", line 990, in uname
  File "C:\Python35\lib\platform.py", line 571, in win32_ver
  File "C:\Python35\lib\platform.py", line 527, in _get_real_winver
  File "C:\Python35\lib\site-packages\PyInstaller\loader\pyiboot01_bootstrap.py", line 153, in __init__
  File "C:\Python35\lib\ctypes\__init__.py", line 347, in __init__
OSError: [WinError 126] The specified module could not be found
Title: Re: [Preview] Knossos 0.5.1
Post by: ngld on June 18, 2017, 05:30:46 pm
This is the first release to use a 64bit build of Python and it seems that might've caused this.
I'll make a new build using 32bit Python and we'll see if that fixes it.
Title: Re: [Preview] Knossos 0.5.1
Post by: niffiwan on June 19, 2017, 02:24:38 am
Sorry - still encountering this error despite the recent changes to include nodejs & handle fallback to PyQt5.QtWebKit :(
(Mint 18.1 / Ubuntu 16.04)

(http://pix.toile-libre.org/upload/img/1497857023.png) (http://pix.toile-libre.org/?img=1497857023.png)

(http://pix.toile-libre.org/upload/img/1497856656.png) (http://pix.toile-libre.org/?img=1497856656.png)
Title: Re: [Preview] Knossos 0.5.1
Post by: ngld on June 21, 2017, 12:58:31 pm
@m!m: I managed to track the problem down. Python tries to load a version.dll (which should be in C:\Windows\system32) which seems to fail on Windows 10 (it works on Windows 7). Updating Python to 3.5.3 fixed that.
@niffiwan: It was a simple copy&paste error.

Both are fixed in 0.5.2 which is building right now. I'll update the link once it's uploaded.
Title: Re: [Preview] Knossos 0.5.1
Post by: m!m on June 21, 2017, 04:11:26 pm
The new version works fine on Windows :yes:
Title: Re: [Preview] Knossos 0.5.2
Post by: chief1983 on June 22, 2017, 10:25:00 am
The Mac build won't start, saying it can't find 7z.  But it looks like the Contents has it included, so I'm not sure where it's trying to look.  Maybe the search path got messed up during the dmg assembly?
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on June 24, 2017, 06:58:06 pm
I've just released 0.5.3 which should fix the macOS builds.

@niffiwan: Can you please confirm that 0.5.2 fixed the Ubuntu builds?
Title: Re: [Preview] Knossos 0.5.3
Post by: niffiwan on June 24, 2017, 08:11:58 pm
yes, 0.5.2 fixed the Ubuntu LTS issues - thanks!
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on June 27, 2017, 04:01:43 pm
I've finally gotten to the development tab.

This is what I've implemented so far:
(http://i.imgur.com/X3AcHAP.png)

Here's mjn's mockup:
(http://i282.photobucket.com/albums/kk264/mjnmixael/Private/Knossos%20Development_zpsbmxd4qwu.jpg~original)
Obviously, I still have to add a few options but while I'm working on that I want some feedback since I think the mockup is still incomplete.

Ideally this is how I imagine a modder would create a mod in Knossos 1.0.0:

There are a few things missing to enable that workflow (i.e. a widget to select the dependencies). Before I start working on those parts I want your opinions on this: As a modder, tester or dev, what would you like to see here?
@mjn: I'd like to hear your take on this, too.


Eventually this interface should support TCs and adding FSO builds as well. I'm still working on the details so I'm not sure how those will look. Feel free to offer your opinions/suggestions though.

Title: Re: [Preview] Knossos 0.5.3
Post by: Cyborg17 on June 27, 2017, 05:45:38 pm
This may be beyond the scope of Knossos, but a list of files in the mod directories and dependencies, sorted by type could be pretty useful.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on June 27, 2017, 05:49:34 pm
The file listing is fairly simple to implement. Do you mean dependencies on other mods (as specified by the mod's author) or dependencies on other mod files (generated automatically)? The former is already visible in the mockup and will be implemented.
Title: Re: [Preview] Knossos 0.5.3
Post by: Axem on June 27, 2017, 06:45:54 pm
I think the way you have it makes sense to me... at least in general.

Just a few things to cover off first. In mod dependencies, are these going to be actual mods or just folders (or both)? Some larger mods (like BtA, BP and even JAD) have a complex web of folders that we use to keep things separate for later packaging to VP. For example, in mjn's mockup, he has a bunch of BTA folders, which aren't really mods, just folders, but the main mod.ini refers to all of the paths to make them into a whole functioning mod. Maybe some way to select and indicate mod paths and actual mods?

Also how would things like add on packs be handled in development? Like optional high res textures or movies?
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on June 27, 2017, 07:06:18 pm
I'm splitting dependencies into two parts. Knossos uses mods (which represent mod folders) and packages (which refer to individual files like MV_Advanced.vp).
The idea is that you tell Knossos about the mods you require (which is then used to build the -mod flag) but don't have to worry about the packages until you decide to upload.
When you want to upload your mod, you have to split it into packages. For each package you select which folders and files it contains and which packages of other mods it requires. You also have to select which package depends on which mods (and which of their packages are required). Knossos then builds an archive per package (.7z) and uploads that to fsnebula.org.

I'm not sure if it's feasible but I'd like to integrate .vp file generation into the whole process. I'm just not sure how or where to implement it (is creating one .vp file per package a good idea?).

To demonstrate how mods and packages work, here's the current install window:
(http://i.imgur.com/Xe58Wl3.png)
BtA and MediaVPs are mods every other option is a package. Each package can be required, recommended or optional. Required files are disabled because you can't deselect them. Recommended files are selected by default and optional packages are unchecked by default.


Also how would things like add on packs be handled in development? Like optional high res textures or movies?
Honestly, I haven't considered how to handle them in development. When you upload them you would put them in seperate packages and mark them as recommended or optional as explained above. Would it make sense to have packages during development which you could then disable or enable?
Title: Re: [Preview] Knossos 0.5.3
Post by: Axem on June 27, 2017, 07:29:26 pm
I think I sort of get it. So mods are like things Knossos can find, use and launch with, but aren't handled by Nebula's auto updating or versioning. Mods can be turned into packages through the magical process of *magic*. And packages are solidly handled by Nebula and come with dependencies, versions and the other fun stuff. (Correct me if I'm not getting it)

So what happens when I have a package released, but I need to work on it for patches and stuff? (Personally I have 2 FreeSpace folders, a playing one where its just released packaged mods, and a production/development one where I have a bunch of misc mod folders and svn directories, so that's how I handle that now)
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on June 27, 2017, 07:50:37 pm
Not exactly. A mod has a version, title, description, etc. A mod is what shows up in lists.
Each mod has at least one package but can have as many as a modder wants. A package always belongs to exactly one mod (and version). A package has a title, included files and dependencies.
A package serves to seperate mod files into groups as seen in the last screenshot.

Knossos currently can't install several versions of the same mod (I had implemented support for that in the past but it got pretty complicated and due to suggestions in the last thread, I've taken it out simplify the relevant code.).

However, you could have 2 folders: The "base folder" which is where Knossos installs mods and a second folder where you keep the mods you're working on. The mods in the second folder would still be able to use dependencies from the first (i.e. Media VPs) so you'll only have to install those once.
I'm not sure how I'm going to handle duplicates (i.e. if you have JAD installed in your base folder and a dev version of JAD in the second folder). I could either have the dev version replace the released version in the UI or display both but somehow mark them (different color?). I bet someone else has a better idea regarding this.

EDIT: You could also use a single folder but then you wouldn't be able to have the released version and the dev version of your mod installed at the same time.
Title: Re: [Preview] Knossos 0.5.3
Post by: Axem on June 27, 2017, 08:25:54 pm
Yeah, it's sort of complex. I guess we've all come up with our own idiosyncratic methods to organize ourselves. Maybe we need to show you our modding development environments and see if you can find some magic point to unite us all. :p

How do other systems like this work? Like steam workshop? I've never used it that much so I don't know...
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on June 27, 2017, 08:58:12 pm
I have no idea how steam workshop works since I've never used it myself. However, AFAICT it doesn't handle dependencies which makes the whole thing much easier.
Most package systems I know of only allow one version of a package to be installed.
Nexus Mods Manager / Mod Organizer don't handle dependencies, either. Hell, they can't even search for mods. You need to use the website to start a download.

The simplest option so far would be to have two mods: JAD and JAD_dev. Knossos could easily tell them apart and handle them as two seperate mods. Only when you upload would Knossos have to remember that JAD_dev is actually JAD and upload it as such. Maybe I can implement an option which does exactly that.

Title: Re: [Preview] Knossos 0.5.3
Post by: Axem on June 27, 2017, 09:48:34 pm
Maybe when you do the upload submission, you have an opportunity to finalize certain things such as "development mod" or change the name so it could go from something like "JAD - DEV" to "JAD"?

In my in-development folder, my mods are typically done like "JAD - Dev" or "Vassago-NOVP". When I export the files from SVN, and create the VPs, I take the mod.inis and modify them so they reflect the way things will look in production.

For example in \JADNOVP\ the mod.ini's secondary list is
Code: [Select]
secondarylist = JADNOVP\JAD2-2\JAD222,JADNOVP\JAD2-2\JADXA,JADNOVP\JAD2-2\JAD221,mediavps_2014;The JAD2.21 files and JAD2.22 files were separated out because of episodic releases, and each episode has its data divided into 3 VPs so patches can be applied to just the small text data files instead of the giant assets. I have a batch script that does all this for me right now.

So when I go to do a real release, \JAD2-2\'s mod.ini secondary list is just
Code: [Select]
secondarylist = mediavps_2014;Because its all meant to play together.

Blue Planet has a huge mess of svn folders everywhere. But everything important is in BP Complete, and each VP that would be made has its own mod folder (no mod.ini inside) and its secondary list looks like...
Code: [Select]
secondarylist = blueplanetSVN\BP Complete\bpc-fs2blue,blueplanetSVN\BP Complete\bpc-core,blueplanetSVN\BP Complete\bpc-visuals4,blueplanetSVN\BP Complete\bpc-visuals3,blueplanetSVN\BP Complete\bpc-visuals2,blueplanetSVN\BP Complete\bpc-visuals1,blueplanetSVN\BP Complete\bpc-audio1,blueplanetSVN\BP Complete\bpc-audio2,blueplanetSVN\BP Complete\bpc-audio3,mediavps_2014;
and likewise the final release mod.ini would just have mediavps_2014 in the secondary list.

I don't know if this helps you or not, I'm just trying to tell you how I've seen things done, so maybe a way can be found that doesn't throw all of us into confusion.
Title: Re: [Preview] Knossos 0.5.3
Post by: Cyborg17 on June 27, 2017, 09:59:15 pm
My thought would just be a list of whatever files are in the packets used in the chosen mod and the mods that the chosen mod is trying to depend on.

It would be a way to quickly check what ships, scripts, tables, etc., are in use, especially if your mod is larger and you're not sure if you got around to adding that last civilian station for that new mission you're about to fred, or its tbm, or the custom cockpit glass texture, etc.
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on June 28, 2017, 01:05:48 pm
Excellent! I don't have much more to add regarding the process over what Axem has already discussed.

I love the idea that we can essentially start mod creation right from Knossos. That's absolutely brilliant and makes modding even a little more accessible. :yes: :yes:

From a UI perspective, feel free to not fuss too much about getting things to look exactly like my mock up. Get the functionality in and I can spend time tweaking the UI to get it Just Right (tm). The idea, at least, is to free you up to work on functionality. :)
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on June 28, 2017, 01:36:48 pm
@Axem: Being able to change things immediateley before the upload sounds like a good idea.

I think I undestand your structure. You basically have a folder per .vp file and during development FSO loads files directly from that folder (which is why it's listed in secondarylist) and after release the contents will be part of a .vp file which means the directory isn't listed in the release version.
If Knossos allowed you to select a folder for each package, then you could do the same there.

@Cyborg17: Something like this (https://fsnebula.org/repository/bta_complete/master/1.0.8/files)? It should be less cluttered though... I'll think of something.

@mjn: Glad to hear! :)
One of the things I haven't fixed, yet, is that the PLAY/DETAILS/FRED buttons only show on hover because that element is essentially the same thing you see in the other lists.
I've recently restructured the HTML part and split it into multiple files because that single index.html was becoming hard to read through. Most of the HTML code is now in html/templates.
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on June 28, 2017, 05:17:19 pm
Hmm, Knossos is stuck on Loading... when I try to run the latest from github. Do I need to clear a cache somewhere?
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on June 28, 2017, 05:53:27 pm
Sorry, there was an issue with Windows paths in the template loading code. If you update your checkout, it should work again.
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on June 28, 2017, 06:41:02 pm
Aha, there we go. Also.. So glad the HTML has been split up! I was thinking the same thing as you were.
Title: Re: [Preview] Knossos 0.5.3
Post by: jr2 on June 30, 2017, 02:38:43 am
Would different lighting presets like on wxLauncher be a good idea?  It'd be nice to have a way for mods to include their own recommended settings, and have the option to use those, as well (might already have that, not sure :shrug: )

(http://i63.tinypic.com/k21ifk.jpg)
Title: Re: [Preview] Knossos 0.5.3
Post by: woutersmits on June 30, 2017, 03:21:39 am
i get this errror

and fso wont show up

while there's fso


[attachment stolen by Russian hackers]
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on July 01, 2017, 06:08:42 pm
@jr2: I've got something like that in mind but I need to make sure that Knossos can actually launch mods first.

You've found two bugs which will be fixed by the next release. FSO doesn't show up because you (probably) didn't use Knossos to install it.
Knossos unfortunately can't use your existing FS2 directory because it needs a specific folder structure. You should select an empty base directory and install the FSO package. If FSO still doesn't show up afterwards, you've found a bug.
Title: Re: [Preview] Knossos 0.5.3
Post by: woutersmits on July 02, 2017, 03:41:51 am
Fso runs fred everythime if i trying to run and if i say launch game it will run fred2 not the game
Thats bugg becouse if i set to gameexecuteble it runs fso fredding not actuel game
Title: Re: [Preview] Knossos 0.5.3
Post by: Bryan See on July 02, 2017, 03:55:38 am
Fso runs fred everythime if i trying to run and if i say launch game it will run fred2 not the game
Thats bugg becouse if i set to gameexecuteble it runs fso fredding not actuel game

You mean that FSO runs FRED everytime you tried to run. If you say launch game, it runs only FRED2, not the game. It's a bug because if you set FSO game executable, it runs FRED not the actual game itself.

Do you guys know what's the cause?
Title: Re: [Preview] Knossos 0.5.3
Post by: woutersmits on July 02, 2017, 03:57:27 am
Yes you correct
Thats why i to tell
Title: Re: [Preview] Knossos 0.5.3
Post by: Hellzed on July 03, 2017, 12:49:04 am
I've just taken some time to understand how the JS API in Knossos works.
If I haven't missed anything, I suspect it's currently lacking a bunch of elements before the web-based UI can be completed.


I haven't been doing anything focused or especially useful, but I've experimented with a few things that might come in handy.
The first was injecting more than one object through the QtWebChannel in web.py, like this:
Code: [Select]
objects = {'fs2mod': self, 'Knossos': knossos}
channel.registerObjects(objects)

Then I provided not only functions, but also object properties, through the JS API:
(and if we don't want the JS code to update values in our Python code, we can set pyqtProperty with constant=True, or even make the WebChannel ignore updates from the client script)
Code: [Select]
class MoreStuff(QtCore.QObject):
    onMyPropChange = QtCore.pyqtSignal(str)

    def __init__(self):
        super(Kn, self).__init__()
        self.__myProp = "My Value"

    @QtCore.pyqtProperty(str, notify=onMyPropChange)
    def myProp(self):
        return self.__myProp

    @myProp.setter
    def myProp(self, value):
        logging.debug('Setting myProp to: %s', value)
        self.__myProp = value
        self.onMyPropChange.emit(self.__myProp)

    @myProp.getter
    def myProp(self):
        logging.debug('Getting myProp as: %s', self.__myProp)
        return self.__myProp
       

Now I feel like trying to wrap UI interactions into Promises, and tasks such as downloads and unpacking into ProgressEvent, then see what comes out of it
(I'm thinking about validation of the user input, error handling directly in the web UI if something fails, nicer JS code, and progress bars integrated with the rest of UI...)
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise
https://developer.mozilla.org/en-US/docs/Web/API/ProgressEvent

EDIT: @ngld: If I understand correctly, "self.onMyPropChange.emit(self.__myProp)" houldn't be necessary, but somehow notify=onMyPropChange gets ignored. Any idea ?
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on July 04, 2017, 08:05:38 am
Do you guys know what's the cause?

The logic for detecting whether an .exe file is FRED or not failed. I'll look into it once I'm done with mod creation since that's more or less a requirement for launch testing.

If I haven't missed anything, I suspect it's currently lacking a bunch of elements before the web-based UI can be completed.
I usually add the required functions/methods whenever I need them. For example, I added the settings related functions when I did the settings screen and I will add more functions while working on the development tab.

It's nice that you're interested in Knossos' JS API but I'm not sure if the things you've been experimenting with are going to be useful.

The first was injecting more than one object through the QtWebChannel in web.py.
I don't think we need more than one object unless it makes sense to seperate the API methods into different namespaces. Renaming fs2mod to knossos makes sense.

Then I provided not only functions, but also object properties, through the JS API:
I'm not sure how that would be useful and I don't think it works with QtWebKit.

Now I feel like trying to wrap UI interactions into Promises, and tasks such as downloads and unpacking into ProgressEvent, then see what comes out of it
(I'm thinking about validation of the user input, error handling directly in the web UI if something fails, nicer JS code, and progress bars integrated with the rest of UI...)
The existing JS interaction is already callback based; you could use Promises by rewriting the call() function in html/js/modlist.js but that would just add more code (unless I misunderstood you).
Error handling rarely happens directly in the JS code since most errors happen in Python code and then are displayed either using a message box or once again in the web UI.
Progress bars for downloading and unpacking are already implemented and the ProgressEvent is completely unrelated to that.

EDIT: @ngld: If I understand correctly, "self.onMyPropChange.emit(self.__myProp)" houldn't be necessary, but somehow notify=onMyPropChange gets ignored. Any idea ?
I don't know since I haven't used PyQt properties before. You have two getters in your example, maybe correcting that helps?

I'm trying to keep the JS API as simple as reasonable since it reduces the chance that (hard to debug) errors related to it occur. Those errors can be especially annoying when QtWebKit is used because it tends to segfault whenever something unexpected happens.
Title: Re: [Preview] Knossos 0.5.3
Post by: Hellzed on July 04, 2017, 12:07:18 pm
If I haven't missed anything, I suspect it's currently lacking a bunch of elements before the web-based UI can be completed.
I usually add the required functions/methods whenever I need them. For example, I added the settings related functions when I did the settings screen and I will add more functions while working on the development tab.

It's nice that you're interested in Knossos' JS API but I'm not sure if the things you've been experimenting with are going to be useful.
No usefulness guaranteed here, it's all out of curiosity !

The first was injecting more than one object through the QtWebChannel in web.py.
I don't think we need more than one object unless it makes sense to seperate the API methods into different namespaces. Renaming fs2mod to knossos makes sense.
No real need for two separate objects indeed, but it was convenient for me in order to have one I can experiment with while not breaking fs2mod.

Then I provided not only functions, but also object properties, through the JS API:
I'm not sure how that would be useful and I don't think it works with QtWebKit.
I haven't tried yet, but according to the doc it's supposed to be exactly the same:
http://doc.qt.io/archives/qt-5.5/qwebframe.html#addToJavaScriptWindowObject

Now I feel like trying to wrap UI interactions into Promises, and tasks such as downloads and unpacking into ProgressEvent, then see what comes out of it
(I'm thinking about validation of the user input, error handling directly in the web UI if something fails, nicer JS code, and progress bars integrated with the rest of UI...)
The existing JS interaction is already callback based; you could use Promises by rewriting the call() function in html/js/modlist.js but that would just add more code (unless I misunderstood you).
Error handling rarely happens directly in the JS code since most errors happen in Python code and then are displayed either using a message box or once again in the web UI.
Progress bars for downloading and unpacking are already implemented and the ProgressEvent is completely unrelated to that.
That's true, that would be a bit more code probably, and a lot of what's doable with Promises is already doable with the callback based API, but there might still be some advantages (I have looked more into the Python side of things for now, a bit les the JS side).

EDIT: @ngld: If I understand correctly, "self.onMyPropChange.emit(self.__myProp)" houldn't be necessary, but somehow notify=onMyPropChange gets ignored. Any idea ?
I don't know since I haven't used PyQt properties before. You have two getters in your example, maybe correcting that helps?

I'm trying to keep the JS API as simple as reasonable since it reduces the chance that (hard to debug) errors related to it occur. Those errors can be especially annoying when QtWebKit is used because it tends to segfault whenever something unexpected happens.
Fixed. I originally added the getter to the snippet as an example and forgot about it.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on July 06, 2017, 03:47:56 pm
I've made some progress with the mod creation dialog but now I'm wondering about mod versions.
Originally I inteded to use Semantic Versioning (http://semver.org/) but now I'm not so sure because the instructions don't make too much sense in the context of mods...
I need to enforce some versioning scheme since Knossos needs to be able to compare version numbers. Otherwise detecting updates will be impossible.

Any thoughts or suggestions on that?
I'm considering going with a simple a.b scheme where a is increased for every new release and b is increased for every bugfix release. Of course, whenever a is increased, b is reset to 0. (i.e. 1.0 would be the initial release, 1.2 would be the second bugfix release and 2.0 would be the next major release).


No usefulness guaranteed here, it's all out of curiosity !
Ah, now I understand. Have fun experimenting then.  :D

Fixed. I originally added the getter to the snippet as an example and forgot about it.
Did that fix the problem with the notify signal?
Title: Re: [Preview] Knossos 0.5.3
Post by: chief1983 on July 06, 2017, 04:31:36 pm
Not so sure it doesn't make sense.  Mods themselves could have a concept of asset compatibility, kind of like ABI/API compatibility.  Other mods dependent on them should be able to rely on minor version changes not breaking anything, but more significant version number changes could involve major asset overhauls and break compatibility with mods depending on their assets.  So, you'd end up needing to have multiple versions of the mod on hand to ensure compatibility of older mods until they can be updated.

A cleanup system capable of keeping track of things only installed as dependencies, and removing them as warranted might also be useful.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on July 07, 2017, 03:47:23 pm
I'm not yet sure how I'm going to deal with multiple versions. The last iteration of Knossos did support that but it became an overcomplicated nightmare which is why the current version doesn't support it.
I have a concept in mind which could solve the problem while still being simple enough to maintain. I'm not sure if I'm going to implement it though. I guess it depends on how important multiple versions are going to be.

The cleanup system isn't just useful, it's necessary for updates.

The development tab is slowly taking shape.
(http://i.imgur.com/L9Qtbhi.png)

It's not exactly pretty and it might be a bit confusing but it's functional. The simple input fields work. You can already create new mods (and uninstall them). You can also add and delete packages. Saving and some of the more complicated input widgets (file selection, flag editor, images, dependencies) are still missing.
Title: Re: [Preview] Knossos 0.5.3
Post by: woutersmits on July 08, 2017, 07:58:20 am
My question is why if i runs youer knosses?
and if i tryien to run fso it opens fred instead fso?
Game it self.
How to fix that tell me why is that?
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on July 08, 2017, 10:42:55 am
The logic for detecting whether an .exe file is FRED or not failed. I'll look into it once I'm done with mod creation since that's more or less a requirement for launch testing.

The only way to fix this would be to fix the relevant code which I won't do until the development tab is done because then I'll have to rewrite that part anyway.
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on July 09, 2017, 04:08:04 pm
I've been traveling. I'll pretty up the dev tab when I get home! Looking good
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on July 15, 2017, 07:14:36 pm
Thanks!

Unfortunately, my hard drive died a few days ago. Nothing was lost because I had everything backed up but restoring everything took a while. I haven't made much progress since my last post, however I finally finished two concepts:
I think I'm going to rename packages because the name seems to confuse people. I just don't know what to name them instead (packs?). Anyway, Knossos will create a folder for each package you create through the dev tab. When you start a mod from there you will be able to choose which packages to enable. You can also tell Knossos to put everything in the package's folder into a VP file when you upload your mod.

As a reminder if you (the reader) don't know what packages are:
A mod has a version, title, description, etc. A mod is what shows up in lists.
Each mod has at least one package but can have as many as a modder wants. A package always belongs to exactly one mod (and version). A package has a title, included files and dependencies.
A package serves to seperate mod files into groups.


I'm also done with the concept for BitTorrent support. The communication with the tracker will go through HTTPS and communication between clients will be SSL encrypted. Due to this it should be impossible for an ISP to tell that users are using BitTorrent. They will be able to detect that P2P connections are used but that shouldn't be too surprising nowadays (see Windows updates and various game launchers).
Knossos will allow the user to completely turn BitTorrent support off in which case it falls back to normal HTTP(S) downloads. It will also allow people to limit the download and upload bandwidth. I might also add an option to stop seeding once you've uploaded as much as you downloaded.
If BitTorrent downloads are enabled, the HTTP(S) mirror will still be used so even if noone else is seeding, it should be at least as fast as normal downloads.
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on July 16, 2017, 01:15:12 am
Randomly, I just noticed tonight that the Install % does not update unless you switch tabs now.

I created a new folder for Knossos to install FS2 and mods to. Told it to download and install BtA. Had to keep switching to Explore, then back to Home to see the update bar progress and to see the percentage number increase. Pretty sure this worked before. Then after the install should have completed, the mods did not show in Home. Odd. Tomorrow I'm gonna poke around with the dev tab UI design anyway.
Title: Re: [Preview] Knossos 0.5.3
Post by: m!m on July 16, 2017, 03:31:52 am
Knossos will allow the user to completely turn BitTorrent support off in which case it falls back to normal HTTP(S) downloads. It will also allow people to limit the download and upload bandwidth. I might also add an option to stop seeding once you've uploaded as much as you downloaded.
If BitTorrent downloads are enabled, the HTTP(S) mirror will still be used so even if noone else is seeding, it should be at least as fast as normal downloads.
Does that mean that BitTorrent is enabled by default? I already voiced my concerns about enabling BitTorrent by default so how about showing a message at the first startup which explains that Knossos can use BitTorrent and what advantages that would provide? The user could then choose to enable or disable it but they will at least be notified that by enabling it they might use more of their data volume than they expected.

Also, I just tried to run Knossos from the development branch and now it doesn't start anymore. I'll try to bisect to find the commit that caused this.

EDIT: This may be a system error on my side. The last message that is shown by "ninja run" is this:
Code: [Select]
Check failed: NamespaceUtils::DenySetgroups()

It seems to be related to the chromium sandbox and I found a few references to this message in the chromium bug tracker. I also see the following message in "dmesg" every time I try to run Knossos:

Code: [Select]
[ 2245.963242] traps: python3.5[11392] general protection ip:7f7dc35ef602 sp:7ffe045cf960 error:0 in libc-2.24.so[7f7dc35b8000+1bc000]
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on July 16, 2017, 08:46:41 am
@mjn: Weird. I've only noticed that bug on Ubuntu before where it was a bug in QtWebKit which didn't properly foward the events. I can't test it right now because the Explore tab is empty. I'll look into it later.

Does that mean that BitTorrent is enabled by default? I already voiced my concerns about enabling BitTorrent by default so how about showing a message at the first startup which explains that Knossos can use BitTorrent and what advantages that would provide? [...]
We already came to that conclusion in the last thread (http://www.hard-light.net/forums/index.php?topic=93144.msg1850056#msg1850056).


EDIT: This may be a system error on my side. The last message that is shown by "ninja run" is this:
Code: [Select]
Check failed: NamespaceUtils::DenySetgroups()
Are you trying to run Knossos under a Ubuntu guest account? There Chrome/Chromium didn't work there (https://bugs.launchpad.net/ubuntu/+source/lightdm/+bug/1298021) until they added an AppArmor exception for Chrome. Since that exception is path specific I'm going to guess that they didn't add Qt's QtWebEngineProcess to that exception which in turn would cause the same bug for Knossos.

It seems to be related to the chromium sandbox [...]
You can try to disable the sandbox by running it like this:
Code: [Select]
QTWEBENGINE_DISABLE_SANDBOX=1 ninja run
Title: Re: [Preview] Knossos 0.5.3
Post by: m!m on July 16, 2017, 09:04:52 am
We already came to that conclusion in the last thread (http://www.hard-light.net/forums/index.php?topic=93144.msg1850056#msg1850056).
Ok, I wasn't sure what exactly you meant in your post.

Are you trying to run Knossos under a Ubuntu guest account? There Chrome/Chromium didn't work there (https://bugs.launchpad.net/ubuntu/+source/lightdm/+bug/1298021) until they added an AppArmor exception for Chrome. Since that exception is path specific I'm going to guess that they didn't add Qt's QtWebEngineProcess to that exception which in turn would cause the same bug for Knossos.
No, I'm on Fedora and I can't recall any significant changes to my system since I last ran Knossos. It could have been broken in a recent kernel update, I also found a few kernel commit references in the chromium bug report.

You can try to disable the sandbox by running it like this:
Code: [Select]
QTWEBENGINE_DISABLE_SANDBOX=1 ninja run
Yes, that makes it work. Although now most of the mods don't show up (I'm not sure if they are supposed to show up, maybe you are doing some refactoring and it doesn't work at the moment). Here is the stacktrace of one parse failure:

Code: [Select]
ERROR:Thread-1:tasks.work1: Failed to parse "/fso/blueplanet/mod.ini"!
Traceback (most recent call last):
  File "knossos/tasks.py", line 146, in work1
    mod.set_base(base_mod)
AttributeError: 'IniMod' object has no attribute 'set_base'
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on July 16, 2017, 09:44:52 am
I had to move fsnebula.org to a new server (the old server is outdated and is being replaced) and I didn't want to transfer the old stuff hosted there since I'll have to replace it anyway. That's why Knossos currently can't download a mod list and thus can't show any mod on the "Explore" tab.

The set_base error will be fixed with the next commit. It might cause some mods you have locally installed to not show up but shouldn't affect anything else.

Regarding the sandbox issue: The issue here is that the Chromium sandbox isn't initialized properly because it failed to create the namespace. This most of the time is a permission problem. Fedora uses SELinux which probably denied access to this API. The cause is probably a recent QtWebEngine or policy update. For reference here's Fedora's bug tracker (https://bugzilla.redhat.com/buglist.cgi?component=qt5-qtwebengine&product=Fedora). I've seen similar issues there but none seem to be the root cause for this problem.
Title: Re: [Preview] Knossos 0.5.3
Post by: m!m on July 16, 2017, 10:44:52 am
Regarding the sandbox issue: The issue here is that the Chromium sandbox isn't initialized properly because it failed to create the namespace. This most of the time is a permission problem. Fedora uses SELinux which probably denied access to this API. The cause is probably a recent QtWebEngine or policy update. For reference here's Fedora's bug tracker (https://bugzilla.redhat.com/buglist.cgi?component=qt5-qtwebengine&product=Fedora). I've seen similar issues there but none seem to be the root cause for this problem.
SELinux is disabled at the moment so it's can't be that. Just to be sure, I reenabled it.
Title: Re: [Preview] Knossos 0.5.3
Post by: CapellaGoBoom on July 24, 2017, 12:18:30 pm
Is this going to be the replacement for FSO installer for mod installation/management?
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on July 24, 2017, 05:47:07 pm
Probably, at least if enough people here like it.
My intention was to combine installer and launcher (i.e. FSO Installer and wxLauncher) into one program AND make it easier to use. Whether that works out or not is something we will see once this is finally finished. It will take a while but hopefully I'll be able to release 1.0.0 this year.

I'm finally done with the dev tab UI, now I still have to write the saving and upload code. Afterwards I'll have to test and fix the code which launches mods. I'll probably have to fix the install/update/uninstall parts, too. I guess I'll release a new beta at that point and work on UI polishing, bug fixing and eventually torrent support. Once that beta is polished and stable enough, I'll be able to release 1.0.0 at which point (I hope) more and more people will use it.

@m!m: I've built a Fedora VM and wasn't able to reproduce your issue on it. I cloned the repo, installed the neccessary packages with dnf and finally ran Knossos both with "ninja debug" and "ninja run". It worked both times without issue. Any idea what I did differently?
Title: Re: [Preview] Knossos 0.5.3
Post by: CapellaGoBoom on July 24, 2017, 11:45:14 pm
Probably, at least if enough people here like it.
My intention was to combine installer and launcher (i.e. FSO Installer and wxLauncher) into one program AND make it easier to use. Whether that works out or not is something we will see once this is finally finished. It will take a while but hopefully I'll be able to release 1.0.0 this year.

I'm finally done with the dev tab UI, now I still have to write the saving and upload code. Afterwards I'll have to test and fix the code which launches mods. I'll probably have to fix the install/update/uninstall parts, too. I guess I'll release a new beta at that point and work on UI polishing, bug fixing and eventually torrent support. Once that beta is polished and stable enough, I'll be able to release 1.0.0 at which point (I hope) more and more people will use it.

@m!m: I've built a Fedora VM and wasn't able to reproduce your issue on it. I cloned the repo, installed the neccessary packages with dnf and finally ran Knossos both with "ninja debug" and "ninja run". It worked both times without issue. Any idea what I did differently?

I like it so far. But when I tried to use the explore search bar it didn't work for me. Is that going to get fixed?
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on July 25, 2017, 08:26:37 am
Are you talking about the fact that no mods are showing up? That's going to change once I'm done with the uploading code which will allow me to add mods again.
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on July 25, 2017, 10:30:32 am
I haven't poked around with this in a little while.. has the mod.json been updated to include to additional mod images/descriptions/etc yet?
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on July 25, 2017, 10:37:18 am
Kind of... You can open the dev tab, go to "Mod details" and edit everything there (including the new fields), however you can't save that, yet.
Title: Re: [Preview] Knossos 0.5.3
Post by: CapellaGoBoom on July 25, 2017, 11:38:39 am
Are you talking about the fact that no mods are showing up? That's going to change once I'm done with the uploading code which will allow me to add mods again.

No mods showing up is precisely what I meant
Title: Re: [Preview] Knossos 0.5.3
Post by: m!m on July 26, 2017, 04:26:56 am
@m!m: I've built a Fedora VM and wasn't able to reproduce your issue on it. I cloned the repo, installed the neccessary packages with dnf and finally ran Knossos both with "ninja debug" and "ninja run". It worked both times without issue. Any idea what I did differently?
My guess is that the chromium version Qt is using is outdated and can't handle my system setup for some reason. Google Chrome is working fine so it will probably be fixed by a Qt update in the future. In the meantime I'm just going to run Knossos with QTWEBENGINE_DISABLE_SANDBOX=1.

EDIT: I'm also getting an uncaught error on startup:
Code: [Select]
Traceback (most recent call last):
  File "knossos/launcher.py", line 364, in main
    run_knossos()
  File "knossos/launcher.py", line 184, in run_knossos
    center.mods.load_json(mod_db)
  File "knossos/repo.py", line 81, in load_json
    self.set(json.load(stream))
  File "knossos/repo.py", line 92, in set
    self.add_mod(Mod(mod, self))
  File "knossos/repo.py", line 392, in __init__
    self.set(values)
  File "knossos/repo.py", line 417, in set
    if pkg.check_env():
  File "knossos/repo.py", line 630, in check_env
    logging.warn('Invalid value for environment check in mod %s (%s)!' % (self.mid, self.version))
AttributeError: 'Package' object has no attribute 'mid'
I "fixed" that by simply removing that line but you may want to take a closer look at this.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on July 28, 2017, 12:41:10 pm
@m!m: Thanks, should be fixed now.

I've finally implemented saving and the FSO settings section. You can now install retail files and create/edit/delete mods.
Launching should work now, however, I'm still working on the -mod flag. This means that you can only launch retail and TCs right now.

Knossos now allows multiple versions of a mod to be installed (since it was easier to implement than I thought). After the -mod flag is done, I'll make sure that the UI works with multiple versions of the same mod and then I'll look for problems with launching FSO and fix those.
I'm not yet sure if I'll release a new beta at that point or if I should wait until uploading is implemented. That will take a while because I'll also have to write the server part for that. Either way, I'll release a new beta after that and I think that only polishing and bug fixing will be necessary after that point to reach 1.0.0.
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on July 28, 2017, 01:11:39 pm
Updated and upon launching got..

Code: [Select]
ninja: error: 'html/templates/flag-editor.html', needed by 'knossos/data/resoures.rrc', missing and no known rule to make it
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on July 28, 2017, 01:13:55 pm
Try removing the build.ninja file and then running windows_run.bat again. I renamed a file and build.ninja still references the old location. I haven't found a better way to deal with this, yet.
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on July 28, 2017, 01:38:35 pm
That works
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on July 28, 2017, 04:30:13 pm
Working on some of the cosmetics. Here's a few thoughts.

Can this handle Steam install?
The "Install Retail" button is persistent even after I installed retail. It's that correct behavior? I would have thought that button would disappear when retail is installed? Then include a "reset/reinstall retail" button in the settings somewhere. But not a big deal. If it's permanent, then I might redesign the button a bit.

Saving mod details gives
Code: [Select]
A critical error occurred! Knossos might not work correctly until you restart it. The error has been reported and will hopefully be fixed soon. Seems like Video links saving has an error based on the CMD output window.

When we're in the Settings page, the Gear should be green and the tab links (Home/Explore/Dev) should be gray. I could not find the tab-bar elements in the html files anymore.

Still would like to see the Uninstall button moved to the dropdown for each mod, but that will require javascript to get working. I can move the button semi easily, but I could never get it to be functional from the dropdown.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on July 28, 2017, 05:13:35 pm
It should be able to handle a Steam install if the user manually selects the correct directory. Knossos can't auto-detect that, yet, since I don't know the exact directory Steam uses. I'm guessing it's "C:\Program Files (x86)\Steam\SteamApps\common\Freespace2" but I'm not sure. As soon as someone can confirm this, I'll add it.

I added the "Install Retail" button as a workaround when I realized that I couldn't trigger that window without any available mods. Hiding it when the retail files are installed makes sense. The "reset/reinstall retail" button shouldn't be necessary because you can uninstall the FS2 TC entry instead (Knossos handles the retail files as a TC). Feel free to change this button however you wish, I haven't thought much about it when I added it.

The Video links error should be fixed now. I'll fix the active states for tabs and links tomorrow. The tabs' HTML is in html/templates/kn-page.html, just below "Start the Tab Menus". The actual tab list is in html/js/modlist.js (search for "tabs:").

I'm fine with moving the Uninstall button, I'll take care of it and the dropdown problem tomorrow. That reminds me, the details page still isn't finished. I think I still need to add a field for the banner image and implement the VIDEOS button there.

BTW, I'm wondering about the mod description field. It's currently a simple plain text field. Should I add support for BBCode or replace it with a WYSIWYG editor?
Title: Re: [Preview] Knossos 0.5.3
Post by: jr2 on July 28, 2017, 05:22:30 pm
It should be able to handle a Steam install if the user manually selects the correct directory. Knossos can't auto-detect that, yet, since I don't know the exact directory Steam uses. I'm guessing it's "C:\Program Files (x86)\Steam\SteamApps\common\Freespace2" but I'm not sure. As soon as someone can confirm this, I'll add it.

I added the "Install Retail" button as a workaround when I realized that I couldn't trigger that window without any available mods. Hiding it when the retail files are installed makes sense. The "reset/reinstall retail" button shouldn't be necessary because you can uninstall the FS2 TC entry instead (Knossos handles the retail files as a TC). Feel free to change this button however you wish, I haven't thought much about it when I added it.

The Video links error should be fixed now. I'll fix the active states for tabs and links tomorrow. The tabs' HTML is in html/templates/kn-page.html, just below "Start the Tab Menus". The actual tab list is in html/js/modlist.js (search for "tabs:").

I'm fine with moving the Uninstall button, I'll take care of it and the dropdown problem tomorrow. That reminds me, the details page still isn't finished. I think I still need to add a field for the banner image and implement the VIDEOS button there.

BTW, I'm wondering about the mod description field. It's currently a simple plain text field. Should I add support for BBCode or replace it with a WYSIWYG editor?

Copy-pasting from my Steam FS2 install (obviously it would normally be C: )

Code: [Select]
D:\Program Files (x86)\Steam\steamapps\common\Freespace 2
So there's a space before 2, and lower-case steamapps (should be case in-sensitive, right?), but other than that, correct.
Title: Re: [Preview] Knossos 0.5.3
Post by: jr2 on July 28, 2017, 05:39:17 pm
Double-posting to say you may be able to get a list of Steam Library locations by looking in %SteamPath%\config\config.vdf  -- you can find SteamPath in the registry,
Code: [Select]
Computer\HKEY_CURRENT_USER\Software\Valve\Steam
Link to spec for vdf format: https://developer.valvesoftware.com/wiki/KeyValues#File_Format

Where I got this info: https://www.reddit.com/r/learnprogramming/comments/1x0amf/ctrouble_finding_where_a_user_installs_steam/

Why that's useful:  Just get a list of Steam Library locations, and search them for "FreeSpace 2" to find the install location.  (You could also use the uninstall string InstallLocation from the registry, but that doesn't change if the install location gets moved in Steam, I just checked.)

InstallLocation:
Code: [Select]
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\Steam App 273620
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on July 28, 2017, 07:07:08 pm
I humbly submit this to be used as the default FS2 Retail Install portrait. And that the mod name be changed to something a little more along the lines of "Basic FS2", "Base FS2", or "Retail FS2" instead of just "FS2"

(https://i.cubeupload.com/LlEA2h.png)

(https://i.cubeupload.com/WNc2CO.jpg)
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on July 28, 2017, 09:01:10 pm
BTW, I'm wondering about the mod description field. It's currently a simple plain text field. Should I add support for BBCode or replace it with a WYSIWYG editor?

I think some sort of basic BBCode or WYSIWYG would be nice. As it is, the Description field doesn't even respect line breaks.

Other than that, the only things that seem to be missing from the Details page mockup (which you can override, btw) is the Release Date and Last Updated to the right of the Videos/Forum buttons. (For unreleased mods, it should say something like "unreleased" or something. Last Updated could be the mod creation date, perhaps.)

I also envisioned a place that listed mod team members since Nebula supported that at one point.

Lastly, could it be made so that the Development tab uses a duplicate of the mod-node class (probably dev-mod-node) so that I can change it's appearance for the development tab only? If you want to take care of the duplication (because it looks like it needs to be done in the js files), I can take care of the actual cosmetics of the job.
Title: Re: [Preview] Knossos 0.5.3
Post by: m!m on July 29, 2017, 05:29:17 am
BTW, I'm wondering about the mod description field. It's currently a simple plain text field. Should I add support for BBCode or replace it with a WYSIWYG editor?
How about using Markdown (https://en.wikipedia.org/wiki/Markdown) for the description text? It it simple to use and pretty widely supported. There is even a Qt example  (https://doc.qt.io/qt-5/qtwebengine-webenginewidgets-markdowneditor-example.html)for implementing a Markdown editor using WebEngine.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on July 29, 2017, 01:41:40 pm
@jr2: Thanks for the detailed info! I'll implement that.

@mjn: The image is great! Knossos now uses it along with the name "Retail FS2". I thought that name fit best with the image.
I've added html/templates/kn-dev-mod.html as the duplicate of kn-mod.html. The "Release Date" and "Last Updated" fields are still unfinished because I want to use a proper date picker and I haven't gotten around to including one.

I'll make a new tab for the mod team members. It will also serve as a way to give other users access to modify the mod.
I've also merged your PR, it helped a lot! However, I think we should change the text colour to make it easier to read.

@m!m and mjn: What do you think about this editor (https://hinesboy.github.io/mavonEditor/dist/) (you can switch to english in the upper right)?
I originally wanted to use BBCode because most modders will be familiar with that unlike Markdown.
Title: Re: [Preview] Knossos 0.5.3
Post by: jr2 on July 29, 2017, 05:14:26 pm
I had a look at my GOG.com FS2, and you may be able to get fix on GOG FS2 installations using the following registry key:

Code: [Select]
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\GameUX\Games\{B14DCE92-4D55-4BBA-984D-1CCB4D48DA36}
Key name

Code: [Select]
ConfigApplicationPath
I'm not sure where GOG stores its install paths (I checked Program Files (x86)\GOG.com\ and didn't find anything).


EDIT: OK, found it, it's in

Code: [Select]
%ProgramData%\GOG.com\Galaxy\storage\index.db
 - not sure what the file storage spec is, but you can read FS2's install path in plain text.

EDIT2: The above is probably more reliable as the user might have changed the install location, but you can also find the normal GOG Library location in plain text from

Code: [Select]
%ProgramData%\GOG.com\Galaxy\config.json
EDIT3: of course that all relies on having Galaxy installed instead of using the stand-alone installer.  The registry key I initially pointed out would be better in that case then?  I dunno what's best, neither, both, or one then the other.  If you need more tests / searches done, let me know and I can help.
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on July 30, 2017, 08:55:47 am
@mjn: The image is great! Knossos now uses it along with the name "Retail FS2". I thought that name fit best with the image.
I've added html/templates/kn-dev-mod.html as the duplicate of kn-mod.html. The "Release Date" and "Last Updated" fields are still unfinished because I want to use a proper date picker and I haven't gotten around to including one.

I'll make a new tab for the mod team members. It will also serve as a way to give other users access to modify the mod.
:yes:

Quote
I've also merged your PR, it helped a lot! However, I think we should change the text colour to make it easier to read.
Which text are you talking about? It's either button or the Dev tab intro text. For the Dev tab, I went back and forth a lot on the text color. I could still go either way, but I chose a milder color because I wanted it to be more a part of the background rather than the focus of the view. But yeah, it could go full white if you think that's better.

Quote
@m!m and mjn: What do you think about this editor (https://hinesboy.github.io/mavonEditor/dist/) (you can switch to english in the upper right)?
I originally wanted to use BBCode because most modders will be familiar with that unlike Markdown.

My 2 cents is use something more common. Both of these would need to be learned by modders, which isn't a big deal. But what's the gain for using Markdown or Mavon instead of BBCode or even just (filtered) HTML that everyone already knows?
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on July 30, 2017, 09:08:42 am
@jr2: Once again, thanks for the detailed research! I'm going to guess that the index.db is a SQLite (https://sqlite.org/) database.

@mjn: I was talking about the intro text and the help text that I added to the input fields.
I want to avoid filtered HTML because that can become a headache and/or a security risk. I'd rather use BBCode or Markdown. Markdown is more modern and easier to learn than BBCode. That doesn't matter too much, however, since most modders will already know BBCode from the forum anyway (which is why I suggested it).

Maybe I should implement both and leave the decision up to the mod author? Implementing that isn't too hard since there are JS libraries available for both of them.
Title: Re: [Preview] Knossos 0.5.3
Post by: m!m on July 30, 2017, 09:13:50 am
My 2 cents is use something more common. Both of these would need to be learned by modders, which isn't a big deal. But what's the gain for using Markdown or Mavon instead of BBCode or even just (filtered) HTML that everyone already knows?
My reason for suggesting Markdown is that it is used pretty widely so it should be well supported by Python and HTML/JavaScript. It's simple enough to learn while also being powerful enough to create nice description texts.
There are a lot of existing WYSIWYG editors for Markdown out there so modders wouldn't even need to bother with the specifics of how Markdown works and Knossos wouldn't have to implement the entire editor.

IMO, filtered HTML is not a good choice in any case because it basically needs a full HTML renderer to be well supported. This is obviously not a problem for Knossos but it's something that should be considered for ensuring better compatibility in the future. Also, filtering HTML is pretty easy to get wrong so we might run into issues with HTML injections if it's not done properly.

BBCode apparently has decent support by Python (https://github.com/dcwatson/bbcode) but I'm not sure how well it's supported by JavaScript (I don't know where the HTML rendering would happen so I'm listing both options here).
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on July 30, 2017, 10:13:37 am
Ah, well in the case of there being an WYSIWYG editor, then the particular language doesn't matter.
Title: Re: [Preview] Knossos 0.5.3
Post by: m!m on July 30, 2017, 10:16:23 am
Just to clarify, I'm not sure if there will be a WYSIWYG since I'm not the one who writes the code for that. However, at least for Markdown it would make sense since there are a lot of existing editors that should be pretty easy to integrate.
Title: Re: [Preview] Knossos 0.5.3
Post by: jr2 on July 31, 2017, 12:22:26 am
@jr2: Once again, thanks for the detailed research! I'm going to guess that the index.db is a SQLite (https://sqlite.org/) database.

Looks like you're right, I opened it up successfully here: http://inloop.github.io/sqlite-viewer/  (site says it doesn't upload, just uses JavaScript HTML5 FileReader)

Don't remember how to SQLite, but you'd want the Table
Code: [Select]
Products then the
Code: [Select]
localpath of productId
Code: [Select]
5
Title: Re: [Preview] Knossos 0.5.3
Post by: jr2 on July 31, 2017, 10:50:06 am
@jr2: Once again, thanks for the detailed research! I'm going to guess that the index.db is a SQLite (https://sqlite.org/) database.

Looks like you're right, I opened it up successfully here: http://inloop.github.io/sqlite-viewer/  (site says it doesn't upload, just uses JavaScript HTML5 FileReader)

Don't remember how to SQLite, but you'd want the Table
Code: [Select]
Products then the
Code: [Select]
localpath of productId
Code: [Select]
5

If it helps, I'd also be more than happy to PM you a download link to my index.db
Title: Re: [Preview] Knossos 0.5.3
Post by: jr2 on July 31, 2017, 11:38:28 am
OK, for the GOG standalone installer, looks like you can find the path in
Code: [Select]
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\GOG.com\GOGFREESPACE2under the key
Code: [Select]
PATH
Checked with an old 2012 standalone installer, uninstalled, verified entry was gone, and tried with the latest standalone (v2.0.0.8) - both make the above entry upon install.
Title: Re: [Preview] Knossos 0.5.3
Post by: chief1983 on September 06, 2017, 08:38:49 pm
How goes the interface progress?  The Babylon Project is now being overhauled to work with 3.8+ without warnings or game-breaking bugs, and could theoretically be the first total conversion ready for public distribution via Knossos (FotG is very likely ready to test distribution via closed beta via Knossos, if it ever supports mod access controls).  That or Diaspora but TBP has a _lot_ more content available.  And now that 3.8 is released, it will hopefully be the only build we need to worry about supporting in Knossos.
Title: Re: [Preview] Knossos 0.5.3
Post by: m!m on September 07, 2017, 03:23:50 am
Also, the nightly/release script recently had an addition that allows it to generate a configuration file for the "old" installer. It should be pretty easy to adapt this code so that it also generates a manifest file compatible with Knossos so that new releases and nightly builds are automatically available as soon as they are released.
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on September 07, 2017, 10:27:27 am
I've kept my eye on the github and ngld hasn't updated in almost a month.
Title: Re: [Preview] Knossos 0.5.3
Post by: chief1983 on September 07, 2017, 10:32:27 am
I guess we were just waiting on completed integration of your interface work then?
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on September 07, 2017, 11:44:59 am
No, it needs some significant work on the backend in relation to handling additional mod data and how it works with the online portion.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on September 10, 2017, 07:43:02 pm
Sorry for the late reply. I've been busy studying for exams which is why I haven't worked much on Knossos recently. The exams are this week so I should be able to continue my work soon.

The current state is: The whole uploading part (interface & logic) is missing, parts of the settings screen (mostly joystick & network options) are missing, a few parts of the dev tab (the team screen and (I think) the screen for the -mod flag) as well as a bunch of details (menu items to launch FRED, filtering of TCs, sorting of mods, etc.) are also missing.
The uploading code is the biggest piece left. To start on that, however, I need to finish the server. I've already started working on that and will upload the code tomorrow but it will take a while to finish.
Once the server and uploading code work, we should have the basic functionality in place and could start the next beta phase.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on September 11, 2017, 10:30:26 pm
As promised, I've uploaded the server code (https://github.com/ngld/fsnebula). It's still in a very early stage but it's there if someone wants to look at it.

Something I haven't been able to decide on is whether to upload files in one piece or split them into several blocks. The latter approach is useful when uploads fail. If I upload each block in a single request, I only have to retry the block that failed. If I upload files without splitting them, it would be easier (faster) to implement but the whole file would have to be retried which can be very annoying if you're working with huge files.
Title: Re: [Preview] Knossos 0.5.3
Post by: chief1983 on September 12, 2017, 08:58:43 am
I do agree that anything that could be done to make uploading saner for mod developers, it would be helpful.  But, it can probably be iterated in later when we have something that works at all, if upgrading the upload method later would be not much harder than doing it that way first.  Seems odd though, that breaking up uploads is the only way to do what sounds like a type of resume-able upload?  Is it not possible to just resume uploads from arbitrary byte indices?
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on September 13, 2017, 05:42:28 pm
Changing the upload method later on isn't very hard.
Resuming an interrupted upload without splitting the upload is hard because normally the web application code is only run once the request is complete. An incomplete request like an interrupted upload is discarded. I could work around that but I'd rather avoid the additional complexity.
I've looked at how tus.io (https://tus.io/) and Amazon's S3 implement resumable uploads and they both split files into multiple parts which are then uploaded individually.

I'll implement simple file uploading for now and later replace it with tus.io.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on September 15, 2017, 05:27:09 am
I'm finally done with exams and started working on this. I've finished the basic functionality on the server (register, login, password reset, mod creation/updating, file uploads). The UI work on Knossos is mostly done (only the register, login and password reset buttons are missing). This only leaves the main part: The code which controls the actual upload. I should be able to finish an alpha build either this weekend or next week. It probably will be buggy and it will have a bunch of issues.

While I work on the uploading code I wanted to discuss a few issues Knossos currently has:

I think most of these issues are UI related which might be because I don't know much about UI/UX design.
I want feedback on these issues. I want to create some mockups before I start changing the UI code but before I work on those, I want to hear your thoughts. (@mjn feel free to create/share your own, I think they'll be better than mine).
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on September 15, 2017, 10:18:11 am
I think a version select dropdown is best. We don't want to the tile page to be filled up with multiple mod versions. It just ends up looking and feeling cluttered. Especially since the use-case for running older mod versions is way more niche.

Agree on dev tab after some thought.

Agree on folder structure points.

Agree on adding a version button to the dev tab. The dev tab needs some UI work in general that I'd like to get done before a release. Been waiting on feature completion.

Do people change their flags on a per-mod basis often? I would be happy with an option in settings for the user to have a set of custom flags. Then they can specify "prefer player flags" or "prefer mod flags". Knossos would override any same mod flags with player flags if the preference is selected. Maybe. I dunno, just a thought.

Keep a list of deleted mod IDs and the metadata. We don't need to do anything on Knossos for UI on this if a deleted mod is installed on a player's machine. Just warn them before they delete it. "This mod has been removed from Nebula, if you continue to delete it, you won't be able to redownload it."

Documentation is good.

I've also been waiting on feature completion of other mod details (banner image, I'd love for it to pull a list of mod staff if possible and list that on the details page, I think there were other things, but it's been a while since I worked on the UI.)
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on September 15, 2017, 12:09:02 pm
My main problem with flags is that they are specific to each FSO build. If I allow a user to globally set flags, I need to get the flag list from somewhere (maybe allow the user to select a FSO build? That might be confusing though...) and I risk passing a new flag to an older build which will then complain about the unknown flag.

Regarding mod deletion: We have to put a delete button somewhere because the Nebula doesn't have a UI which leaves Knossos as the only way to add/edit/delete mods.
Title: Re: [Preview] Knossos 0.5.3
Post by: X3N0-Life-Form on September 16, 2017, 03:32:08 am
Do people change their flags on a per-mod basis often?
Speaking for myself, I know I've done so on several times through profiles in wxLauncher, for instance I had base settings for standard FSO, profiles with darker or brighter lighting depending on a mod's mood, a profile with shadows disabled for resource intensive mods, etc.
I'm not sure that's a common use case though, and if mods can supply recommended settings, I'd probably use that 8 times out of 10.
Title: Re: [Preview] Knossos 0.5.3
Post by: chief1983 on September 17, 2017, 09:54:18 am
I'm not saying I agree one way or the other with how we allow flags to be set, but just from a technical standpoint, I don't see why allowing global flag preferences shouldn't be able to work.

The global settings (all known flags to exist across any builds) could be saved somewhere.  When configuring for a specific build, any removed flags would be hidden but still saved.  When running a build with global flags saved that don't exist yet or anymore in that build's flags, couldn't we just silently not pass them in?  You'd have to compare the global flags to the available flags for that build but that should allow to strip out any flags that aren't supported before actually passing them to the build I would think.  Then, instead of the harsh unsuppported flags popup, the launcher could also determine if or how to notify the user that some chosen global flags are unavailable with the current mod/build configuration.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on September 17, 2017, 08:50:29 pm
(https://i.imgur.com/BGEsExG.png)

So... I finished the upload code and it works! The fsnebula.org server is running again, registration, login, upload & download work. If you build the latest git version, you should be able to test it. Go to Settings > Nebula settings, enter values for all three fields and click "Register". You should now receive an email with a confirmation link. Click it, switch back to Knossos and then click Login. Once that worked, go to your dev tab, select a mod and click "Upload". That's it, your mod is now available to everyone using Knossos.

I'll make an alpha build sometime this week so more people can test this but I want to fix a few issues first:

On another note, I'm confident that private mods (i.e. for beta testing) will be part of the stable release.

@chief: I like your concept. Any idea where I could get a list of flags for all existing builds? On second thought... Only 3.8.0 is currently supported, that makes this easy. I'll just have to edit the list whenever the SCP removes a flag but that doesn't happen too often from what I can tell.
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on September 18, 2017, 12:07:54 am
Excellent. I'll have to poke around to make sure the UI is solid.

Are the additional mod fields supported yet? (details banner, etc.)

Can't the launchers ping builds for available flags? Could that be used silently when selecting mods?
Title: Re: [Preview] Knossos 0.5.3
Post by: Cyborg17 on September 18, 2017, 04:24:03 am
Just a note, since it's so easy to upload a mod, you may want to have a report mod button, so that we can report anyone who uploads stuff that isn't a mod.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on September 20, 2017, 02:16:34 am
@mjn: The additional fields haven't been added, yet, I want to focus on fixing the bugs in the upload and install code, first.
Knossos can ask the builds for flags but the global flags aren't associated with any build.

@Cyborg17: That's a great idea. I need to add moderation functions while I add that (I think moderators should be able to hide or delete mods. I'm not yet sure how to handle this for private mods).

During testing I also realized that I need to add a delete function soon because I accidentally uploaded broken metadata.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on September 22, 2017, 11:45:25 am
Since I still haven't finished the alpha build, I wanted to update you on the progress so far.

I've added the upload checks (prevents upload of the same or an older version), added a report button and a delete button. The version selector and a button to create new versions have also been finished.
You can now also reset your password. Now I'm only missing a way for users to change their passwords. The appropriate parts have already been implemented on the server and in Knossos but I'm not sure how to put it in the UI so I've left it out for now (the Nebula section of the settings screen feels already pretty crowded).

I still have to deal with the directory structure issue and the staff list. I might delay the staff list a bit and release the alpha after I solve the first issue.

@mjn: Well... the UI's a bit of a mess right now. I think we can convert some of the buttons on the sidebar into tabs which might make their purpose clearer.
(https://i.imgur.com/rAZvg74.png)

EDIT:
The code should now be aware of the new directory structure and refuses to edit mods which aren't in "dev mode". I've also finished the -mod editing tab and fixed the launching logic. So far everything seems to be working fine. I can create, edit, upload, install and launch mods.
There are only a few issues remaining (apart from the unfinished staff page):
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on September 24, 2017, 09:05:12 am
What sidebar area were you thinking for tabs? Your post wasn't clear.

I'll spend some time this afternoon working on the UI in places.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on September 24, 2017, 09:13:04 am
The area which contains the "Edit Mod Details", "Upload" buttons, mod path and the list of packages.

The code's now in a state where I could release an alpha build. Creating mods, uploading, installing & launching should all work. Now I'm wondering, should I release the current build or should I wait until I've put some more polish in (i.e. finish the dropdown menus on the "Home" tab, the sorting menu (the one on the upper right) and the staff list)?
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on September 24, 2017, 09:21:58 am
I think we can give the UI a bit of polish. It won't take a whole lot of work comparatively.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on October 01, 2017, 06:12:41 am
Here's an example of what I meant regarding the button / tab placement:
(https://i.imgur.com/vqmBtWI.png)

I've also changed a few other things: I think the current CREATE button in the mod list doesn't fit there so I replaced it with the one you see. I've also removed the "INSTALL RETAIL" button because the Knossos styled buttons have a static width and the text wouldn't fit with that. Not sure what to do about that.
The middle column has been cleaned up: All buttons use the proper style and a few of them have been converted to tabs on the right side. I just realized that I forgot the "Add Package" button...  :banghead: Not sure where I can put that. The tab bar listing all packages seems to be the most logical place. Maybe I can just add a tab labeled "+"?

This is obviously just a sketch but I want to finish the new UI concept before I start implementing it.

@mjn: Can you give me a list of missing mod fields and how they should look? I forgot...


EDIT: Here's the staff page:
(https://i.imgur.com/AT3WiQk.png)
The access field allows you modify who can edit/upload/delete a mod and who can install it (for private mods). The checkbox at the top allows the user to switch the mod between public and private mode.
Title: Re: [Preview] Knossos 0.5.3
Post by: chief1983 on October 01, 2017, 08:18:45 am
Private mods should probably not be seen by those without access either.  Although there may be mods that want themselves to be visible but not installable, like unlockable content maybe?  I dunno if there's a good case for that or not.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on October 01, 2017, 11:33:52 am
Knossos hides mods without releases so that wouldn't be possible right now. I guess I should rename the checkbox "Private mode".

Would it make sense to mark only specific versions as private? You could then have a publicly available version 1.0 (or something) and beta test a new version without having to hide the whole mod.
Title: Re: [Preview] Knossos 0.5.3
Post by: AdmiralRalwood on October 01, 2017, 03:04:11 pm
Would it make sense to mark only specific versions as private?
Yes, definitely.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on October 04, 2017, 11:02:20 am
Then I'll add a "Mark this release as private" checkbox to the upload screen. That should be clear enough.

I've turned the version number on the mod details pages into a select box. You should now be able to see all released versions and install/launch them.
I already mentioned that it's currently impossible to delete an uploaded mod if you lose your dev files. A possible solution would be to add a delete button to the details page if you're authorized to delete a release... that might be confusing though since we already have an uninstall button for installed mods. Any ideas for better labeling?

Most form elements currently use Bootstrap's default style which doesn't really fit with the rest (and I don't like how much space the form elements need). I'll experiment with this and see if I can improve it. In the meantime I'm going to fix debug builds and FRED (the way I originally wanted to handle them doesn't work) and then implement sorting on the home and explore tab.

@mjn I'd really like to hear your opinion on the mockups.
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on October 11, 2017, 07:55:59 am
Sorry, ngld.. I ended up having a bit of a medical procedure which took me out of everything for a while. Those mockups look good. I think that's the best way to handle it, too. If you have time to get the basics on the tabs in place, i'll make it look pretty.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on October 14, 2017, 04:52:43 pm
Alright, I've implemented the basics and finished some other things (The dropdowns on the home tab are working, finally!).
There are only a few issues with the new design:

(https://i.imgur.com/raF17nq.png)

@mjn: Knossos has a few new dependencies. The windows_run.bat script should take care of the most stuff (Node.js, etc.) but it might complain about a missing ply or requests_toolbelt package. If it does, launch a command line and enter "py -3 -mpip install ply requests_toolbelt" to install the missing packages.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on October 15, 2017, 04:26:07 pm
I've done some testing by adding/uploading/installing/launching MediaVPs 3.6.12 & 2014 as well as BP Complete. I've found and fixed a bunch of bugs. It's working pretty well so far.
I've experimented a bit with BP Complete; I split the content in only two packages, one containing the required files and a second one containing the optional files. The first package resulted in a .7z file which is over 1GB big but the upload (and download) went fine. I don't think it's a good idea to have such huge archives and now I'm wondering if Knossos should automatically build more than one archive per package if it's as big as this (currently it builds one .7z per package during upload)...
Mod authors can work (and probably should) work around this by creating multiple packages similar to how mods already use multiple VP files.

I've also managed to display 7z's progress which means that the progress bar doesn't freeze anymore during the "Packing..." step when uploading. The debug builds can now be launched directly through the dropdown menu. FRED should work the same but I haven't checked, yet.

Updating, sorting, debug log upload are still missing. I also completely forgot about the description field. I think the last consensus was that it should use BBCode for formatting.
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on October 15, 2017, 04:31:14 pm
Excellent. I'll have time tomorrow night to get to work on it.
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on October 16, 2017, 10:14:54 pm
Where'd the install FS2 button go?

I was finally prompted to install FS2 when trying to install BP complete. I'm assuming you don't have any sort of "first run" prompt right now?

I also noticed that it did not fetch the mod list upon first boot. I had to hit Refresh. Might want to adjust that to make sure it updates the list when it's run?
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on October 16, 2017, 10:18:53 pm
The "install retail" button? I removed it because I didn't think fit in there. You should be able trigger the same dialog by installing the MediaVPs.
If you want to restore the button, open html/templates/kn-devel-page.vue, search for "INSTALL RETAIL" and remove the comment markers ("<!--" and "-->") in that line.
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on October 17, 2017, 01:26:22 am
Submitted a PR.

The only other thing I'd like to do, but my brain wasn't figuring it out... I'd like to move the Version and Package buttons adjacent to Upload and Delete.

We'd have...

<DELETE> <UPLOAD>
<+PACKAGE><+VERSION>

instead of...

<UPLOAD>
<DELETE>
<+VERSION>
<+PACKAGE>

That would also free up some room there at the bottom of that "column" to help things feel less cluttered.

EDIT: Thanks. Got it!
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on October 17, 2017, 12:19:03 pm
Tried to setup a dev environment on my laptop. It complains that it can't find RCC. I installed QT Creator, but not sure if that includes the development packages like your installation guide suggests.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on October 17, 2017, 12:24:14 pm
You shouldn't have to install anything from Qt. Check your releng\windows\support directory. If it doesn't contain an rcc.exe file but an support.json.chk file, delete the latter and run windows_run.bat.
If neither of those exsists, launching windows_run.bat should fix the problem unless it's stuck somewhere...  Do you get any error messages?

Regarding the installation instructions: The README.md file only gives a quick overview. The detailed instructions (https://github.com/ngld/knossos/wiki/Building-instructions) are on the Wiki. I guess I should link them there...
Title: Re: [Preview] Knossos 0.5.3
Post by: m!m on October 18, 2017, 05:57:55 am
I tested the newest development version and encountered a few issues. I fixed one of the issues and submitted a PR with the fix but I have another issue where I don't know how I would fix it.

Basically, I see a lot of "checksum" KeyErrors, both on startup and when installing a mod:
Code: [Select]
ERROR:Thread-9:progress.run: Exception in Thread!
Traceback (most recent call last):
  File "knossos/progress.py", line 114, in run
    task[0].work(*task[1])
  File "knossos/progress.py", line 393, in work
    self._steps[self._cur_step][1](arg)
  File "knossos/tasks.py", line 516, in work2
    if util.check_hash(archive['checksum'], arpath):
KeyError: 'checksum'
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on October 18, 2017, 08:29:37 am
In poking around this morning, I noticed a few issues...

Playing or Launching any exec fails.
Retail says it can't find an exec at all.
Clicking the Mod Path on the dev tab suggests it will open a shell window to that file location, but that also fails.
Increasing Max Downloads gives an error and won't save.

EDIT: It also strikes me as odd that the FSO release is treated as a mod you can play. It's not really a huge problem UI wise, though, as long as all FSO releases use the versioning system rather than new "mods" like they do in Goober's installer. We want to prevent the Home and Explore tab from being filled with FSO versions.

Nebula doesn't have an web UI yet, right? (Obviously some functions are working, so that's why I ask.)
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on October 19, 2017, 12:06:00 am
@m!m: I'll merge the PR today when I'm back at my computer.

I apparently made a stupid mistake when I uploaded the FSO build. The 32bit and 64bit files are extracted into the same folder and overwrite each other. This leads to the checksum errors (the files can't match both the 32bit and 64bit checksums) and the launch errors (a 32bit .exe can't load 64bit libraries and vice versa).

I'll upload a fixed version later today and add a check to the upload code to prevent this from happening in the future.

Retail says it can't find an exec at all.
Take a look at the mod.json file, I think it depends on FSO >= 3.8.0-RC3 which doesn't work because the current FSO build is 3.8.0-1. I fixed the code which generates the mod.json a while back and it should work if you redo the retail install section.

Clicking the Mod Path on the dev tab suggests it will open a shell window to that file location, but that also fails.
Increasing Max Downloads gives an error and won't save.
No idea why this happens, I'll look into it.

EDIT: It also strikes me as odd that the FSO release is treated as a mod you can play. It's not really a huge problem UI wise, though, as long as all FSO releases use the versioning system rather than new "mods" like they do in Goober's installer. We want to prevent the Home and Explore tab from being filled with FSO versions.
FSO is displayed the same as mods because Knossos mostly treats them the same, that said I want to add a filter which hides builds and make it the default. I think the sorting menu would be a good place for this.
All FSO releases will be displayed as one entry. Custom builds like WoD builds would be seperate.

Nebula doesn't have an web UI yet, right? (Obviously some functions are working, so that's why I ask.)
No, haven't gotten around to it. I want to add at least the account management (i.e. register, login and password reset) stuff and maybe a mod list. Once that's done I think we should replace the Nebula section in Knossos' settings with an iframe displaying Nebula's login screen / account management screen.

Some other issues I noticed are that the progress bar isn't displayed if you install retail from GOG's installer (the taskbar displays the progress correctly but there should also be a progress bar in Knossos' window) and when launching a mod FSO always asks you to pick a build. That screen should only display for mods in dev mode since I don't think it's something users should be bothered by. It's displayed in this case because Knossos doesn't know which build to pick. On Linux this isn't a problem since there no 32bit build exists.

BTW, do we need a ToS or DMCA page since we're hosting user-generated content?
Title: Re: [Preview] Knossos 0.5.3
Post by: m!m on October 19, 2017, 03:20:33 am
@m!m: I'll merge the PR today when I'm back at my computer.
Thank you. I also forced knossos to reload the mod data and now the checksum errors don't occur anymore.

I have another suggestion. When I checked if the checksum error still occurs I installed just the root files of the MediaVPs to avoid downloading all the files but now I can't add the optional files to the installed mod. I could uninstall it and reinstall the mod but that seems a bit unintuitive. Could it be possible to add a "change" entry to the drop-down menu? It could also be used to remove installed components again but that probably requires more work but the installation GUI already exists so simply installing missing optional components may be easier to implement.

EDIT: I found another issue while testing some stuff. I uninstalled my retail "mod" in case that was also affected by the checksum bug I had but now I can't "reinstall" it anymore and when I try to start the MediaVPs it throws this exception:
Code: [Select]
Traceback (most recent call last):
  File "knossos/web.py", line 301, in runMod
    runner.run_mod(mod)
  File "knossos/runner.py", line 315, in run_mod
    run_mod_ex(mod, exes[0]['file'], [path for path, label in mod_flag])
  File "knossos/runner.py", line 323, in run_mod_ex
    basepath = mod.get_parent().folder
  File "knossos/repo.py", line 530, in get_parent
    return self._repo.query(self.parent)
  File "knossos/repo.py", line 248, in query
    raise ModNotFound('Mod "%s" wasn\'t found!' % (mid), mid)
knossos.repo.ModNotFound: Mod "FS2" wasn't found!

Maybe a button for manually installing retail could be added?

EDIT2: Actually, the "Uninstall" button only removed the mod.json from the retail install but left the files intact. That means that WebBridge.install thinks that there is a retail install but when something tries to use that mod it doesn't exists because the mod.json file is missing (at least I think that is the cause).
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on October 19, 2017, 06:57:41 am
Honestly, uninstalling retail probably shouldn't be allowed. Outside of TCs, every mod requires it.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on October 19, 2017, 08:15:57 am
The PR is merged.

Modifying mod installs has been implemented a while back and currently only the UI for it is missing (it got lost during the UI rewrite). I want to add it back but I'll have to think about the UI. The modify button sounds fine but I'm not sure how the dialog after that should look. The simplest case would be a list of checkboxes (one for each package). The user could then check or uncheck the packages they want to (un)install.

The uninstall issue consists of two bugs: a) I need to implement a check which disallows uninstallation of any mod which is required by others and b) your uninstallation failed. I think b) is due to the fact that the mod.json doesn't actually contain a list of files. Since this is a special case and I agree with mjn, I should just hide the uninstall button for the retail mod.
The retail prompt (the popup that asks you to select a FS2 folder or exe) should always show up if the FS2 mod doesn't exist. I guess I need to correct the condition for that.
Title: Re: [Preview] Knossos 0.5.3
Post by: chief1983 on October 19, 2017, 08:33:40 am
I'm not 100% we need to require FS2 to be available for this to be useful.  There are more than a handful of people who have ventured into the community only for total conversions such as The Babylon Project or Diaspora, and will likely remain the case when those are updated or more are released.  Steam for instance does not require the Half-Life engines to be installed until you install a Source or GoldSrc based gamed.  Sure, most people end up doing that, but not everyone.  I will admit, uninstalling FS2 should probably include a heavier prompt, although giving it special behavior like that would probably have to be some kind of kludge.  Or, when the dependency stuff is figured out, just treat it like any other dependency.  I still feel like Knossos should not prompt about FS2 at all though, until you request it to install an FS2-based mod.  It gets you into the experience faster for one thing, on first run.  But even if we want it to prompt for FS2 immediately, I think it should at least be skippable, and not make future prompts.

I am hoping that any dependency management system will keep track of mods installed only as dependencies vs those specifically installed by the user.  This way during an uninstall, any orphaned dependencies can be prompted to be removed as well, or left behind by the user.  Any orphaned deps elected to remain should probably be added to the requested mods at that point.  An exception could be mods that only exist to be dependencies, something we don't really have yet, but maybe could at some point in the future with this new system, similar to how the Source Engine is broken out.  Those things could be cleaned up without prompting if they ever existed.  This is mostly just to keep in mind for potential future iterations, but tracking requested vs dependencies is probably something to get right out of the gate.  It's a feature of most of the better package managers out there that some of the less mature ones commonly lack, which can be frustrating.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on October 19, 2017, 08:41:17 am
FS2 is only required if you install a mod which depends on FS2 (like the MediaVPs). The retail prompt only shows up once you try to install a mod like that. You can install TCs just fine without retail.
I was referring to the fact that the retail prompt didn't show up when he tried to re-install the MediaVPs. I'm sorry if that wasn't clear. The check previously checked if the saved retail directory existed, now I want to check for the existence of the FS2 mod instead.

The difference between user-installed mods and auto-installed dependencies isn't tracked right now but should be easy to add. Especially since we already have mods that are only dependencies: the FSO builds.
Title: Re: [Preview] Knossos 0.5.3
Post by: chief1983 on October 19, 2017, 08:43:15 am
Ah, that makes more sense lol.
Title: Re: [Preview] Knossos 0.5.3
Post by: m!m on October 19, 2017, 08:48:34 am
I am hoping that any dependency management system will keep track of mods installed only as dependencies vs those specifically installed by the user.  This way during an uninstall, any orphaned dependencies can be prompted to be removed as well, or left behind by the user.  Any orphaned deps elected to remain should probably be added to the requested mods at that point.  An exception could be mods that only exist to be dependencies, something we don't really have yet, but maybe could at some point in the future with this new system, similar to how the Source Engine is broken out.  Those things could be cleaned up without prompting if they ever existed.  This is mostly just to keep in mind for potential future iterations, but tracking requested vs dependencies is probably something to get right out of the gate.  It's a feature of most of the better package managers out there that some of the less mature ones commonly lack, which can be frustrating.
Distinguishing between requested mods and dependency mods would be nice to have but is not really necessary right now. As you said, all mods that are currently "actual" mods that can be played. If we ever add support for mods that are only dependencies then it would also require UI changes (since those mods should not show up in the Explore or Play sections by default).

Modifying mod installs has been implemented a while back and currently only the UI for it is missing (it got lost during the UI rewrite). I want to add it back but I'll have to think about the UI. The modify button sounds fine but I'm not sure how the dialog after that should look. The simplest case would be a list of checkboxes (one for each package). The user could then check or uncheck the packages they want to (un)install.
Could the existing installation dialog be reused? It already has a list of all packages with checkboxes so after the user made their choice Knossos could figure out which packages should be deleted and which should be installed.

The uninstall issue consists of two bugs: a) I need to implement a check which disallows uninstallation of any mod which is required by others and b) your uninstallation failed. I think b) is due to the fact that the mod.json doesn't actually contain a list of files. Since this is a special case and I agree with mjn, I should just hide the uninstall button for the retail mod.
The retail prompt (the popup that asks you to select a FS2 folder or exe) should always show up if the FS2 mod doesn't exist. I guess I need to correct the condition for that.
I agree. Uninstalling retail FSO is not a very common task so disallowing that completely shouldn't cause any issues.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on October 19, 2017, 09:02:50 am
If we ever add support for mods that are only dependencies then it would also require UI changes (since those mods should not show up in the Explore or Play sections by default).
Especially since we already have mods that are only dependencies: the FSO builds.
FSO is displayed the same as mods because Knossos mostly treats it the same, that said I want to add a filter which hides builds and make it the default. I think the sorting menu would be a good place for this.

On an unrelated note, Knossos currently hides installed mods from the explore tab. This makes it impossible to install more than one version, it also makes it impossible to see which versions are available. I want to show all mods on the explore tab but mark the installed mods either with an icon or maybe by dimming them.
Title: Re: [Preview] Knossos 0.5.3
Post by: m!m on October 19, 2017, 09:11:53 am
FSO is displayed the same as mods because Knossos mostly treats it the same, that said I want to add a filter which hides builds and make it the default. I think the sorting menu would be a good place for this.
The same filter could be applied to these "dependency mods" so that they aren't shown by default but the user can chose to show them.

Can you actually "Play" an FSO executable? I remember that I saw a Play button on an FSO entry at some point but I'm not sure if that is still the case.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on October 19, 2017, 09:50:39 am
Recently I changed the Play button to only show on mods and TCs, so the answer is no. It was there before though (and it would still work if shown now).
The filter I had in mind was a simple if(mod.type != 'engine'), if we assign a special type to dependency mods then we could filter those out the same way but I'm not sure if that makes sense. I guess we'll have to wait and look at it once a use case for that appears.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on October 19, 2017, 11:50:56 am
I've uploaded the fixed FSO version, made it impossible to uninstall mods which others depend on, added a check to the upload code to prevent file conflicts and I've fixed the error on the settings screen (I forgot a string to integer conversion).
That should hopefully fix most of the reported errors.
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on October 19, 2017, 02:13:48 pm
On an unrelated note, Knossos currently hides installed mods from the explore tab. This makes it impossible to install more than one version, it also makes it impossible to see which versions are available. I want to show all mods on the explore tab but mark the installed mods either with an icon or maybe by dimming them.

Good idea. I'll toy with some options to see what looks best and is clearest from a UI standpoint.
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on October 22, 2017, 04:22:16 pm
Whatever you changed recently fixed a lot of functionality! :yes:

A few things to look at from my poking around this afternoon.

I had BP, FSO, and MVPs already installed. They showed on the Explore tab with "Install" as an option. I clicked it and it really just did an update of some sort? Might want to make sure that was intended functionality and if so, when that's the case, the button should say "Update" instead of "Install".

Additionally, during the install, I still had the option to click "Install" on the Explore tab even though just under that button was another button with the install progress.

I still can't launch Retail.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on October 22, 2017, 05:48:23 pm
Yeah, I fixed a few bugs which I found thanks to the automatic error reporting. The update button should work as intended now if you use the home tab (you can try it by uninstalling FSO, going to the explore tab, clicking details for FSO, selecting version 3.8.0-1 and then clicking install. When you now go to the home tab, the FSO block should have an update button).

You're right, the explore tab should show an update button where appropriate. The reason that Knossos doesn't do it, yet, is that for this to work I need to query both lists (local and remote mods). As mentioned above, this already works on the home tab but I should also implement it for the explore tab.

The duplicated install button is a simple oversight and should be easy to fix.

To fix your retail issue: open the FS2 folder which contains your retail files. There should be a mod.json file, open it and then replace ">=3.8.0-RC3" with ">=3.8.0-1". Restart Knossos (or click refresh) and try launching retail again.

I've changed the explore tab to also display installed mods, they should be marked by the little green icon in the upper-right corner. It's easy to miss but once you know it, it should be enough of an indicator.
The install button in your case installed the new version but didn't touch your old version; now you have both! The update button actually removes the previous version. Beside the fact that it doesn't prompt which packages to install (it always installs the packages you had before) it behaves the same as the install button.

I've also implemented banners. Feel free to test them.
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on October 22, 2017, 07:17:54 pm
Yay! Banners!

:yes: on the other issues.

Separately, when I created a new mod and selected a build, I got the "web cursor not found" issue.
Title: Re: [Preview] Knossos 0.5.3
Post by: ngld on October 22, 2017, 07:32:49 pm
Argh. I never tested that on Windows. FSO doesn't use a window for that error message on Linux which is why it didn't show up there. If you just click "Exit", then everything still works as intended since FSO is just run to get the flag list. In that situation FSO shouldn't even load anything since I'm already passing -parse_cmdline_only.
This is annoying because that means I need either the retail files or a TC to get the flag list.

I can add a workaround which checks if either of those is installed and use that when retrieving the flag list. That should cover 99% of cases but the error would still show up if you created a mod before installing anything. =/
I'd rather fix this in FSO.

EDIT: I thought of a better workaround. Since FSO just checks if it can load an image called "cursorweb", I could create an empty image and save it as data/interface/cursorweb.png in Knossos' folder and start FSO there. Then it could load the cursor thus suppressing the error message. Now I just need to find out if this works...
Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on October 26, 2017, 06:32:38 pm
Knossos Alpha Build

I may have helped design the UI, but Ngld has done the real work here. Like seriously.. check this out.


Few bugs aside, and the footage is pretty heavily sped up in places, this thing is already working quite well. Last night I created and uploaded the MediaVPs version 3.6.10 - 2014 and BtA smoothly. That video was on a different computer, so it's a completely fresh install of FS2 and mods. So easy!
Title: Re: [Preview] Knossos 0.5.3
Post by: AdmiralRalwood on October 26, 2017, 06:44:40 pm
Very nice work! :yes:
Title: Re: [Preview] Knossos 0.5.3
Post by: jr2 on October 27, 2017, 05:02:09 am
Knossos Alpha Build

I may have helped design the UI, but Ngld has done the real work here. Like seriously.. check this out.


Few bugs aside, and the footage is pretty heavily sped up in places, this thing is already working quite well. Last night I created and uploaded the MediaVPs version 3.6.10 - 2014 and BtA smoothly. That video was on a different computer, so it's a completely fresh install of FS2 and mods. So easy!

Sweet.  Now we just need a torpedobeats or other cool type soundtrack and we can make a promo video.  ;)

Hidden Text: Show
Title: Re: [Preview] Knossos 0.5.3
Post by: Spoon on October 27, 2017, 07:02:33 am
Aaah, holy**** that looks amazing. Good work you turbonerds  :yes: :yes: :yes:

(Hah, mjn.mixael you fool, now everyone knows your password is ⚫⚫⚫⚫⚫⚫⚫⚫⚫⚫)

Title: Re: [Preview] Knossos 0.5.3
Post by: mjn.mixael on October 28, 2017, 03:10:13 am
Aaah, holy**** that looks amazing. Good work you turbonerds  :yes: :yes: :yes:

(Hah, mjn.mixael you fool, now everyone knows your password is ⚫⚫⚫⚫⚫⚫⚫⚫⚫⚫)

 :lol: