[Vidmaster]

I have switched from my manual managed Freespace install to Knossos about 2 years ago now, a time span long enough to start this thread. The purpose of this thread is not to credit or discredit the hard work that went into Knossos but instead have a discussion of what how launcher has affected the distribution of Freespace content. Both the good stuff and the bad stuff needs to be pointed out and I look forward to a fruitful discussion with all of you 

PROs:

* let's get the big one out of the way first, accessibility: thanks to Knossos, accessing the work of the HLP community for the newcomer has never been easier. Buy the game on Steam or GoG, download the MediaVPs and enjoy a beautiful Freespace . The importance of this cannot be overstated 

* ease-of-patching: receiving fixes for problems with content, be it a campaign, a mod or a full TC, has never been easier for the average user. Just click on the button. We actually have hard(-light ) data to back this up, the number of fixes released in this community has dramatically increased since Knossos is a thing

* abstraction: while computer-stuff has become ever more complicated, the possibilites of the laymen user to interact with that technology has actually increased over the last decades (just look at the mobiles). While this comes with drawbacks, there is no denying the obvious fact that more people than ever before have access to e.g. PC games thanks to simple and abstract interfaces like Steam. Knossos is very similar in that regard, it takes away all the complexity of managing an installation, of configuring something. It boils down to "click on the logo", just like Steam or any other modern front-end, while hiding all the technical details.

CONs:

* file-amount overkill: having clean and stand-alone local copies that "just work" comes with a lot of redundancies, as Knossos has to keep seperate copies of versions, mods, VPs, packs, etc (let's call them packages in an obvious Linux comparison). Dozens of almost-the-same different versions of a package to ensure the dependencies of a particular mod in a particular state are satisfied. My Freespace folder's size regularly grows until I start weeding it out again by hand . I have not looked at the source code, hence I do not exactly understand where this issue comes from...
An example: MjnMixael's HD mainhalls was updated today (that is at 2021-12-08) from 1.4.5 to 1.4.6 as far as I can see. Now, I got a "MjnMHs-1.4.5" and a "MjnMHs-1.4.6" in my folder but the former is empty (which is weird but okay , no space wasted). The Freespace Port, still on 4.5.1 and not updated does seem to make use of "MjnMHs-1.4.6" according to the manifest so everything is nice even if I fail to understand how it detected that. But on the other hand, I got a BtA-1.6.6, BtA-1.6.8 and BtA-1.6.9 packages in my folder, each more than 7GB big . Apparently, we either have an issue with distributing some things (or we fail to communicate how to properly distribute things to the people uploading this stuff) or Knossos has a bug or I simply fail to understand the concept here. Could somebody shed some light on this?

* overwhelming amount of packages: now that we have everything in one place, we have a new issue: oversaturation , which absurdly goes against the biggest PRO of Knossos. Much acclaimed mods like Blue Planet sit right next to things like Lafiel's HUDIcon Script or Machina Terra DUMP. From my perspective, there is no issue here but for a newcomer to this community, this is a gigantic issue: distinguishing a playable, polished experience from purely technical (aka unplayable) packages is hard to impossible without all the background knowledge. Personally, I believe we need to establish some sort of high-level categorization soon before that list grows longer or risk confusing the newcomers.

Now that this little essay is over, I would very much like to hear your thoughts on the matter  .

[mjn.mixael]

Tags, categories, and filtering were always planned. The full feature has yet to be implemented. Knossos has a base filtering option. I'm hoping new knossos will solve this, especially since some creators want to split their mods up into many individual downloadable releases.

[jadeddragoon]

Pros:

• Enables a shift away from the problematic, existing status quo of "just use the latest" to actually using Known Working Versions based on dependency resolution. This can enable unmaintained mods to remain playable indefinitely. It leads to less overhead for mod maintainers who no longer need to fix problems introduced by newer builds and upstream mods. It moves this community closer to conventions that have been successfully used throughout the oss world for decades.

Cons:

• It doesn't currently work, not completely. Dependency resolution is unreliable. Settings seem to reset themselves at random. Just a litany of bugs and problems that make using knossos something of a chore. I personally believe this has done a lot to damage knossos' image in the community and has turned some people against it.
• Is affected by the systemic mindset in the community of "just use the latest". A mindset that made sense before knossos came along but significantly less so now that a package manager like knossos exists."

A couple thoughts:

* file-amount overkill: having clean and stand-alone local copies that "just work" comes with a lot of redundancies, as Knossos has to keep seperate copies of versions, mods, VPs, packs, etc (let's call them packages in an obvious Linux comparison). Dozens of almost-the-same different versions of a package to ensure the dependencies of a particular mod in a particular state are satisfied. My Freespace folder's size regularly grows until I start weeding it out again by hand . I have not looked at the source code, hence I do not exactly understand where this issue comes from...
An example: MjnMixael's HD mainhalls was updated today (that is at 2021-12-08) from 1.4.5 to 1.4.6 as far as I can see. Now, I got a "MjnMHs-1.4.5" and a "MjnMHs-1.4.6" in my folder but the former is empty (which is weird but okay , no space wasted). The Freespace Port, still on 4.5.1 and not updated does seem to make use of "MjnMHs-1.4.6" according to the manifest so everything is nice even if I fail to understand how it detected that. But on the other hand, I got a BtA-1.6.6, BtA-1.6.8 and BtA-1.6.9 packages in my folder, each more than 7GB big . Apparently, we either have an issue with distributing some things (or we fail to communicate how to properly distribute things to the people uploading this stuff) or Knossos has a bug or I simply fail to understand the concept here. Could somebody shed some light on this?

I'd like to point out that this is not really an issue with knossos so much as an issue with the very concept of software dependencies. Knossos, ultimately, is a simple package manager for Freespace Open and the HLP Community. And this same issue shows up in every package manager for every linux distro out there. Check carefully and you will find numerous packages with multiple versions installed in any linux distro. And there are literally thousands of library files on every modern linux system that the average user has no idea why it's there except that "something else needed it". The old solution to this, prior to knossos was to tell new users "just use the latest"... which often didn't work correctly and led to bugs. There was no real alternative because no one had a list on hand of what versions of what mods worked with what versions of other mods. Knossos has that list. It's the foundation of it's dependency resolution system. But knossos, due to "just use the latest" being a systemic mindset after decades of it being the norm, doesn't really take advantage of it's own abilities here. When building packages dependencies default to "latest" and build requirement default to "X.X.x or newer". Then to make matters worse, knossos' current implementation is prone to errors in it's dependency resolution attempts. This has been a plague for my modpacks which have the most complicated dependency structure i'm aware of on knossos.

Now imagine the inconsistent dependency resolution is fixed, as it's expected to be in nuknossos. Imagine nuknossos takes better advantage of the ability to pin mods to Known Working Versions of their dependencies, as I've lobbied for in nuknossos. And then imagine an automated cleanup routine that uses this improved dependency resolution to accurately identify what versions of which mods you don't need and allow you to have all the unneeded version removed at the press of a button. In my own tests, I was able to remove 6 versions of FSU MediaVPs leaving only 4 behind. That lets me continue playing mods old and new without all the extra wasted space. With a working dependency resolution system automating such cleanup should be rather simple I would think. And automatically removing all but the latest version of campaign mods shouldn't be too much beyond that.

* overwhelming amount of packages: now that we have everything in one place, we have a new issue: oversaturation , which absurdly goes against the biggest PRO of Knossos. Much acclaimed mods like Blue Planet sit right next to things like Lafiel's HUDIcon Script or Machina Terra DUMP. From my perspective, there is no issue here but for a newcomer to this community, this is a gigantic issue: distinguishing a playable, polished experience from purely technical (aka unplayable) packages is hard to impossible without all the background knowledge. Personally, I believe we need to establish some sort of high-level categorization soon before that list grows longer or risk confusing the newcomers.

I'd also like to mention that nuknossos is planned to have a filter in place that hides purely technical mods like Lafiel's HUDIcon Script by default. Allowing those mods to exist on the nebula for package maintainers to use but without confusing novice users. As a maintainer of a few such mods (including the listing on nebula for Lafiel's HUDIcon Script) I actually brought this up to ngld myself months ago. It's absolutely a valid concern but hopefully the fix is already on the way. :-)

[Vidmaster]

I'd like to point out that this is not really an issue with knossos so much as an issue with the very concept of software dependencies. Knossos, ultimately, is just a simple package manager for Freespace Open and the HLP Community.

I am a software engineer myself and totally understand what you are trying to say here  but this is why the BtA-example I gave in my OP is so puzzling to me. Nothing in my installation has any dependency on BtA. Updating BtA, aka selecting a new version to replace the old, which is the default behavior a user expects when updating (although your point about not necessarily using the newest version is true but Knossos does enable that too), does not replace my older version and instead places a new version besides the old one. And when those are 7GB in size, stuff gets ugly fast.

I actually also lobbied for the "automated cleanup routine" you are mentioning, which should be possible to implement quite easliy. After all, it is a acyclic graphic by its very nature and if outdated versions of something are detected which are not required by any other node in the graph, they choice whenever to eliminate them can be put to the user.

Granted, I have not been very active on HLP in the last years (bar donating a few dollars and lurking around, only posting something here and there) but I have actually not heard of nuknossos before. In short, I was not aware that Knossos had been forked or there were plans to replace it. Tell me more.


Another interesting effect of Knossos I neglected to mention in my OP is the fact that combining mods (e.g. Cockpit Mod with e.g. Derelict) is actually quite a lot harder than in a manually managed install. It is by design and not a bad thing, since the newbie user would not know how to combine this stuff by him-, her or themselves. If the modpacks combining multiple things were easier to access and filter, I even think this would not count as a problem at all.

[jadeddragoon]

I'd like to point out that this is not really an issue with knossos so much as an issue with the very concept of software dependencies. Knossos, ultimately, is just a simple package manager for Freespace Open and the HLP Community.

I am a software engineer myself and totally understand what you are trying to say here  but this is why the BtA-example I gave in my OP is so puzzling to me. Nothing in my installation has any dependency on BtA. Updating BtA, aka selecting a new version to replace the old, which is the default behavior a user expects when updating (although your point about not necessarily using the newest version is true but Knossos does enable that too), does not replace my older version and instead places a new version besides the old one. And when those are 7GB in size, stuff gets ugly fast.

I actually also lobbied for the "automated cleanup routine" you are mentioning, which should be possible to implement quite easliy. After all, it is a acyclic graphic by its very nature and if outdated versions of something are detected which are not required by any other node in the graph, they choice whenever to eliminate them can be put to the user.

You're right, I went more general with it and then kinda meandered off on a tangent. However, I do think the problem you mentioned with (for example) BtA is a relatively small fix away. Simply identify that it's not the newest version installed and present the option to automatically uninstall it. Probably see what dependencies will be freed up for removal at the same time.

Granted, I have not been very active on HLP in the last years (bar donating a few dollars and lurking around, only posting something here and there) but I have actually not heard of nuknossos before. In short, I was not aware that Knossos had been forked or there were plans to replace it. Tell me more.

Nuknossos is a complete re-write of knossos from the ground up using lessons learned from the original and implementing many quality of life fixes along the way. It's being worked on by the dev of the original knossos, ngld. It will also come with a revamp of the nebula servers to work with the new functionality. The general goal is to making it faster and much more reliable while addressing a number of concerns by the community. Early alpha builds are starting to show up in the discord channel.

Another interesting effect of Knossos I neglected to mention in my OP is the fact that combining mods (e.g. Cockpit Mod with e.g. Derelict) is actually quite a lot harder than in a manually managed install. It is by design and not a bad thing, since the newbie user would not know how to combine this stuff by him-, her or themselves. If the modpacks combining multiple things were easier to access and filter, I even think this would not count as a problem at all.

While I agree that it's more complicated than simply dropping the cockpit mod files into the derelict folder or modifying a mod.ini or bat/sh file and making a few other tweaks... i think from a maintenance perspective it's easier. And I think it's less intimidating for pc novices (not that we seem to have a many of those). I have asked for a "modpack" tag both to enable filtering them out and selecting for them, but not sure what ngld's plans are yet for tags beyond "modders resource".

[mjn.mixael]

Tags should include all the common stuff by default.

-Model/Asset
-Mod Dump
-Campaign
-Single Mission
-Script
-Total Conversion
-Freespace Mod
-Not Freespace Mod

In addition, probably things like timeline eras because that's also a very, very common ask by new players.

-Pre FS1
-Great War
-Reconstruction
-Second Incursion
-Post FS2

Also genres

-Classic Freespace
-Arcade
-Action
-Mystery
-Parody/Comedy
-Whatever else

Probably also a way to add custom tags because you never know what people are going to come up with next. Tags should also not be mutually exclusive.

[Cyborg17]

Love the categories.  Is

-Total Conversion

distinct from

-Not Freespace Mod ?

[mjn.mixael]

Love the categories.  Is

-Total Conversion

distinct from

-Not Freespace Mod ?

I mean, I guess in practice it is. But in theory Total Conversions do not have to be non freespace. It just means it doesn't require FS2 installed to run.

[Nightmare]

Love the categories.  Is

-Total Conversion

distinct from

-Not Freespace Mod ?

Well you can take a mod featuring non FS2 ships and run them in FSO. I think there're a number of modpacks (ships + weapons) that are running but not as standalone. Most are from canceled TCs which got functional to a degree but none of them have actually "playable" content (in the sense of you get a fun campagin)

[Mongoose]

I'd like to point out that this is not really an issue with knossos so much as an issue with the very concept of software dependencies. Knossos, ultimately, is a simple package manager for Freespace Open and the HLP Community. And this same issue shows up in every package manager for every linux distro out there. Check carefully and you will find numerous packages with multiple versions installed in any linux distro. And there are literally thousands of library files on every modern linux system that the average user has no idea why it's there except that "something else needed it". The old solution to this, prior to knossos was to tell new users "just use the latest"... which often didn't work correctly and led to bugs. There was no real alternative because no one had a list on hand of what versions of what mods worked with what versions of other mods. Knossos has that list. It's the foundation of it's dependency resolution system. But knossos, due to "just use the latest" being a systemic mindset after decades of it being the norm, doesn't really take advantage of it's own abilities here. When building packages dependencies default to "latest" and build requirement default to "X.X.x or newer". Then to make matters worse, knossos' current implementation is prone to errors in it's dependency resolution attempts. This has been a plague for my modpacks which have the most complicated dependency structure i'm aware of on knossos.

Now imagine the inconsistent dependency resolution is fixed, as it's expected to be in nuknossos. Imagine nuknossos takes better advantage of the ability to pin mods to Known Working Versions of their dependencies, as I've lobbied for in nuknossos. And then imagine an automated cleanup routine that uses this improved dependency resolution to accurately identify what versions of which mods you don't need and allow you to have all the unneeded version removed at the press of a button. In my own tests, I was able to remove 6 versions of FSU MediaVPs leaving only 4 behind. That lets me continue playing mods old and new without all the extra wasted space. With a working dependency resolution system automating such cleanup should be rather simple I would think. And automatically removing all but the latest version of campaign mods shouldn't be too much beyond that.

I know that the concept of "known working version" is the proper way to do things from a software development standpoint, and it has obvious advantages in terms of ensuring that a mod always works, but speaking strictly from an end-user standpoint there are also some substantial disadvantages to it. I think having to keep around "only 4" versions of the MediaVPs is asking a lot, especially when one or two of them may be required for only a single mod or two. That's a whole lot of disk space wasted on what are largely duplicates of the same upgraded content. More pressingly, locking players into an older version of a dependency by default means that they don't get the benefits of what can be many years' worth of improvements, whether it's new models in the MediaVPs or bugfixes and performance gains in FSO builds. I know that things are very different now than a decade-plus ago, when mods were much simpler as a rule, and the MediaVPs largely consisted of upgraded content that had the exact same filenames as the retail originals. I know that you couldn't get away with doing that in many cases now. But I also know that there are cases where you still can, where mods are still locked to very old upgrades or builds when they could get away with running on the latest releases just fine, and everyone could ditch one or two of those older MediaVP installs.

I don't really know what a good solution to this looks like. I wondered some time ago if we could recruit players to fire up finished/unmaintained mods on each new major release, to see if there were any obvious non-starters that would prevent them from using those new versions. I know that ngld talked about attempting to implement some sort of automatic compatibility checker that could at least pick up warnings and the like. (Obviously detecting things like balance regressions goes well beyond any sort of automated tool...unless someone wants to sit down and code a full FS-playing AI, which if so then by all means.) Whatever the case, I feel like there has to be a better answer than needing to keep 5 or 6 versions of Common Dependency X taking up a good 15-20 GB of drive space just because one or two old mods haven't been given a stamp of approval.

[jadeddragoon]

I know that the concept of "known working version" is the proper way to do things from a software development standpoint, and it has obvious advantages in terms of ensuring that a mod always works, but speaking strictly from an end-user standpoint there are also some substantial disadvantages to it. I think having to keep around "only 4" versions of the MediaVPs is asking a lot, especially when one or two of them may be required for only a single mod or two. That's a whole lot of disk space wasted on what are largely duplicates of the same upgraded content. More pressingly, locking players into an older version of a dependency by default means that they don't get the benefits of what can be many years' worth of improvements, whether it's new models in the MediaVPs or bugfixes and performance gains in FSO builds. I know that things are very different now than a decade-plus ago, when mods were much simpler as a rule, and the MediaVPs largely consisted of upgraded content that had the exact same filenames as the retail originals. I know that you couldn't get away with doing that in many cases now. But I also know that there are cases where you still can, where mods are still locked to very old upgrades or builds when they could get away with running on the latest releases just fine, and everyone could ditch one or two of those older MediaVP installs.

I don't really know what a good solution to this looks like. I wondered some time ago if we could recruit players to fire up finished/unmaintained mods on each new major release, to see if there were any obvious non-starters that would prevent them from using those new versions. I know that ngld talked about attempting to implement some sort of automatic compatibility checker that could at least pick up warnings and the like. (Obviously detecting things like balance regressions goes well beyond any sort of automated tool...unless someone wants to sit down and code a full FS-playing AI, which if so then by all means.) Whatever the case, I feel like there has to be a better answer than needing to keep 5 or 6 versions of Common Dependency X taking up a good 15-20 GB of drive space just because one or two old mods haven't been given a stamp of approval.

In a perfect world we would use de-duplication... in the real world that would mean extracting the .vp files, identifying which files are identical, then using symlinks (Windows has had them since Vista) or hardlinks to rewire the file table so that only one copy of each file that is a duplicate needs to be physically stored on disk. On linux this could be achieved even easier with something like btrfs which has block level deduplication. Conceivably de-duplication scripts could be written for windows and *nix to do the job. Better still would be knossos handling all of this and never even downloading duplicates.  The big problem is having to extract the files from the vps... there are some potential load order shenanigans involved in that. And knossos's file validation would need to account for the extracted files somehow. Then there's the maintenance of the web of sym/hardlinks as versions of mods are installed/removed.

I fear we are dragging the topic off course a bit though. I mainly wanted to posit What Could Be if the capabilities Knossos presents were fully utilized. The specifics of how to get from here to there (or if we even should) may not quite fit the OP's intent for the thread. I'd be glad to continue the discussion elsewhere, however.

[Vidmaster]

I am happy to read all the responses here, there was no dedicated intent other than to get a discussion going and learn about people's opinions.