Hi, it's... been a while
I'm sorry for not being around as much as I probably should've been. Either way, I'm finally (slowly) working on Knossos again. However, it's probably going to take quite some time before I can release anything substantial.
Let's recap and take a look at the current issues with Knossos:
- The way Knossos manages mod metadata (both within Knossos itself as well as downloading from Nebula) is being pushed way past the limits it was designed for. .json files aren't databases.
- Basing most of Knossos' UI around a Qt5-wrapped Chromium was probably a mistake (especially since most of the non-UI code is written in Python and has to go through the PyQt5 bindings as well).
- There are a bunch of subtle bugs which I never figured out: Issues with the joystick setup preventing FSO from launching, Knossos hanging/crashing on launch
- The transfer of data from Python -> PyQt5 -> Qt5 -> Chromium -> JavaScript worked better than it had any right to but only at first. With the current amount of mods and releases, bugs start to appear which seem to mostly result in things not showing up in the UI although they really should.
- The current dependency resolution is far too messy and mostly the result of chasing an ideal that probably isn't realistic/practical for mods.
- Building new Knossos releases is a pain since it involves bundling Qt5/PyQt5 with PyInstaller which can fail in fascinating ways.
And that's just what I can recall right now. I'm pretty sure there are a few more bugs around uploading mods and metadata processing which I don't recall right now.
First of all, I want to make a few changes to how mods work (this will make mods more reliable and simpler in the long run):
- Dependencies always point to a fixed version of each dependency. Users will always get exactly the release specified by the uploader. I've tried to avoid this because it means that someone has to update all mods depending on MVPs after each MVPs release. However, this will prevent mods from breaking after a dependency updates.
- To balance this change, users will get a screen where they can easily edit (add/remove/change) dependencies. I'll put a disclaimer on that page that those changes aren't supported and might cause breakage. Hopefully, that's enough to prevent people from messing up their mods.
- FSO builds are linked directly to each release instead of (indirectly) through mod packages. Uploaders can select a specific or minimum FSO version. This should fix the various issues where Knossos couldn't find an executable to run.
- Transitive dependencies aren't allowed. Whenever you add a dependency, all required mods for that dependencies are added as dependencies to your mod as well. This makes dependency resolution much simpler and gives you complete control over the mod versions Knossos users.
I think those are the most important changes. The next section will talk about more technical aspects and the different implementations of the current Knossos / Nebula implementation and what I'm planning instead. If you're not interested in that, skip to the end.
With that said, my goal has been to rewrite Knossos (and Nebula) from scratch. My current priorities are to make a) maintanance easier (to make sure Knossos stays stable even if I can't actively work on it) b) make bundling / compiling easier and c) to make working on the code fun again. All of these bugs are very frustrating to deal with especially since most of them don't have a clear or straightforward solution.
So here's the current plan and what I've done so far to get there:
I'm rewriting Nebula first. This time it's a modern web application (SPA). The benefit here is that I'm more comfortable with that since web dev is more or less my day job. It also ensures that you can always manage any mods you have uploaded to Nebula regardless of whether you can currently use Knossos or install the mods. The downside is that managing mods requires an active internet connection. Having a proper web application also simplifies account management (registration, login, password, reset).
The current Nebula implementation is a bare-bones Flask application implemented in Python which likes to waste memory for seemingly no reason (three processes are currently sitting at 1.3 GiB each).
The new implementation is done in
Go. There are many trade-offs involved but for me the benefits are: Vastly simpler to deploy (a single static binary vs installing Python, dependencies and a WSGI server) and easier background tasks (goroutines vs threads). Aside from that, the type checking is nice and the implementation will probably be faster but that doesn't really matter for Nebula since the current Python implementation usually processes requests in less time than a TCP handshake takes (0-3ms for DB-only stuff and 30-500ms for upload processing; mod releases take way longer but that's mostly because Nebula has to export a 150MiB .json file).
The current database is MongoDB but I'll replace it with PostgreSQL. This means I'll have to conform to a proper schema but that's not a bad thing. The main reason for this move is that Nebula is the only stuff I'm running that uses MongoDB while I have several applications that need PostgreSQL. Having less databases to maintain simplifies things. By now I also have far more experience troubleshooting and optimizing PostgreSQL.
On the client side, we currently have a Python application that uses PyQt5 to render the UI and Qt5's Chromium implementation (QtWebEngine) to render most of the UI. As I pointed out before, this lead to a bunch of issues. What makes it worse is that Knossos kind of grew into this and was never designed around the problems this kind of UI brings with it. Here are a few screenshots from ancient Knossos versions:
You can see how more and more of the UI was replaced with the Chromium widget. That's not necessarily a bad thing. However, the current Python code is still designed around the assumption that it has direct access to the UI widgets and that it has to filter mods, etc. For example, performing the mod search and filtering in JavaScript would avoid quite a few problems Knossos currently has to deal with.
With that said, my goal for the new Knossos implementation is to embrace the web UI completely by replacing Qt5 with CEF (Chromium Embedded Framework). I'll implement most of the desktop-specific code (archive extraction, downloads, etc.) in Go (since I'm more comfortable with it than C++) and then write a fairly simple launcher in C++ which initializes CEF, opens the main window and loads the UI. Additionally, it will give JavaScript access to the Go-code (CEF allows us to define additional browser APIs. Through this we can call C++ functions from JavaScript and C++ functions can call Go functions through cgo). I know this sounds complicated (and probably error-prone) but it's far simpler than the current Python -> PyQt5 -> Qt5 -> Chromium -> JavaScript interface. To avoid type conversions (which caused most of the current issues with the Python -> JS interface), I'll pass raw bytes encoded in Protocol Buffers, Flat Buffers or a similar protocol. The goal is to pass statically typed struct-like objects between both languages with little overhead.
In case anyone's wondering why I'm not using Electron: In my opinion it's overly complicated and uses too much memory. Some of that is caused by Chromium itself (I suppose we all know how memory hungry Google Chrome is) but Electron seems to make it worse. I'm hoping this implementation is goind to be more efficient. I've also used CEF several times before but haven't used Electron for anything.
For file uploads, I can use
TUS to replace the current resumable uploads. TUS should be more reliable since a) I've had pretty good results with it in a different project b) it's a mature and active project.
To extract archives, I'll use libarchive. This means Knossos won't have to move files around after extraction, instead it'll create the files at the final location.
I'm not entirely sure which format I'll use for metadata sync between Knossos and Nebula. Currently, Nebula exports the list of all public mods with all releases as a single .json file which Knossos downloads and parses. To put it mildly, this is a bad idea and leads to several issues (including running out of memory).
My currently favored solutions are either splitting the metadata into several files (probably one per mod) and encode them with Protocol Buffers since it's fairly compact and easier to parse than JSON. An alternative would be to use an embedded database like SQLite, Bolt, ... to store the metadata and have Knossos download and load that database.
The benefits of the latter approach are that we won't have to worry about limits since these databases can handle several GiBs worth of data and we'll probably never reach that with metadata alone. However, it'd require Knossos to always download the full DB dump which seems like a waste of bandwidth.
Having one Protocol Buffer file per mod would allow us to only download metadata for updated mods (a list of mods and last change date / revision would be stored in an index to make that possible). The problem with Protocol Buffers is that you always have to load the whole file in memory before you deserialize it (you can't stream it). This essentially puts an upper limit on the file size. I'm not sure if a single mod can ever reach a problematic size but I suppose it's better to assume it might happen and think of a way to split the files further.
With that huge info dump out of the way, I can finally talk about how far I am. I've already started the Nebula rewrite a while ago and user registration, login and password reset are already functional (including the relevant mails). I'm currently working on importing the old Nebula database into the new DB schema. Once that's done, I'll have to finish the mod UI (view, edit, download for people without Knossos). After that, I'll try to get the desktop client into a state where it can download, install and launch mods.
We'll see how long this takes. I'm hoping that once I reach that milestone, the new client will be more reliable and will allow people to spend more time actually playing mods and less time troubleshooting Knossos.
As I understand it, they're supposed to already be in, but there might be some bugs there. As an example, I just updated BP from 1.1.1 to 1.1.2. The first thing the update process did was go over every installed file looking for ones that haven't changed.
This is correct. Knossos compares the checksums to find identical files, however this doesn't always work. Sometimes the reason is that while the VP contents haven't changed, the VPs were repacked which caused the file checksums to change as well.
The empty folders (and sometimes unused mods) are left behind because Knossos generally errs on the side of caution and only deletes stuff if it's sure you want to remove it. Specifically, uninstalling mods or updating one installed version to the next. If you have multiple versions of the same mod installed, the older versions are never deleted.
Empty folders are sometimes left behind because Knossos usually only looks at individual packages and never at a whole mod. Once the last package is removed, the mod isn't installed anymore as far as Knossos is concerned. Knossos doesn't touch anything not part of a package because everything else is was probably created by the user.
This is for Ubuntu. (and in German... )
I tried the Ubuntu package, unsucessfully. I guess that I have no choice but try to compile Knossos from source.
I will give it a try.
Thanks.
I've made the last release available to the new distros as well. However, I can't rebuild the package right now so I have no idea if it even works. It's not even the latest version (the PPA still has 0.13.3 instead of 0.14.3) because I never got around to updating the Ubuntu build scripts. You'll probably be better of building from source.
Found it. Gosh, it is python-based and some web-stuff on-top (or is Node-JS used for Python today as well?) Anyway, not my cup of tea.
Which does not mean that the community, including myself, does not appreciate your work, ngld .
It's even worse: Python, PyQt5, Qt5, Chromium (QtWebEngine) and Vue (UI framework in JavaScript). The Node.JS stuff is only used to compile the web assets.
Dare I say it, as cool as Knossos is, it seems to focus too much on the fresh install and not on maintaining one.
I think a better way to put it would be that Knossos is paranoid about accidentally deleting something you might want to keep. Having to manually delete old mod versions seemed the lesser evil at the time.
It should be possible to compute a dependency graph for all the currently installed games/mods and to remove everything not referenced by that graph anymore. Thoughts?
Knossos doesn't track whether mods were installed by the user or to satisfy a dependency. That would have to be added first but aside from that it sounds like a good idea.
It would be wonderful if someone could get Knossos installable again by simply getting it from a PPA, but I gather that is quite a bit of work and requires a decent amount of maintenance to keep it functioning. Praseodym and Anagram have both provided instructions for successfully building Knossos 0.14.3.
Here's the build script for Ubuntu. Essentially, you have to create an archive with Knossos' sources (see the tar -czf line in the linked script), create a new empty folder, place that archive and the releng/ubuntu/debian folder inside, update the changelog and run "dpkg-buildpackage -us -uc". If anyone can make that work (and the built .deb file works), send me that folder (with the .orig.tar.gz and debian folder) and I'll publish it on the PPA.
If an SCP dev wants access to the automated error reports, I can give them an account. I can also publicly post the most common errors if anyone's interested in fixing them.
TL;DR: I'm still alive, a new Knossos version might be coming... maybe even before the end of the year but who knows.