[EatThePath]

Every mod I've looked at follows a very similar structure, one I'm sure you all know but I'll summarize anyway so it's clear what I'm talking about: One single mod encompassing assets, missions and tables, sometimes split into separate VPs for organizational or practical reasons, and typically loading one other mod, the FSU MVPs. It's a reasonable approach, and before Knossos probably the only reasonable one, but I think with Knossos and the promise of Knussos there's another path people should consider.

Let me explain some example situations I think this new method would be helpful in first.

Scenario:
Your mod loads FSU, and FSU releases a content update.
Currently:
You can probably retarget and release an update with minimal work. Despite Knossos' attempts to avoid it, people will likely need to download a new version of the FSU in full and store two copies if they use any other mods that haven't been updated. If you have to make changes to adapt your mod besides changing version numbering then users may have to redownload some or all of your mod.
Potentially:
You can probably retarget and release an update with minimal work. Knossos will download only the new content added to the FSU, and the only redundant copies stored will be of the changed content and perhaps some text tables.

Scenario:
Another mod has an ship you want, but it wouldn't make sense to load that mod in full
Currently:
You dig out that ship's files, or get them off an asset release post, and incorporate them into your mod. Now if anyone plays both mods they download and store that ship twice.
Potentially:
You include the ship in your mod with an addition to your mod config and a table with any adjustments you need to make.

Scenario:
As the last, but the source mod discovers some bugs with the model and releases a bugfix update.
Currently:
You have to find out about the update somehow, then manually incorporate it and release a new version of your mod that people need to redownload in full.
Potentially:
As soon as the bugfix is released users can receive it with a download of only the ship's files, and no action from you.

From those scenarios you may have already guessed the proposal, but if not it is this; break mods out into a core file with your tables and missions and so forth, and then a separate mod for each asset or logical collection of assets( a ship, a set of ships that share many textures, a set of weapon effects, and so on). To my mind these packages would have minimal tables that make them functional enough to drop in freespace but be tuned primarily in the core mod table, but there's some cases to be made for other approaches there. If any part of the community does head in this direction then there probably needs to be a conversation about the best convention to adopt there.

This concept has a few caveats. A big one is that it needs new features from Knossos to work on large scale, ones that have been discussed but aren't there yet. That includes a system of categorization in mod lists to separate out the playable mods from the 'component' mods, and ways to keep from exceeding the limits of the current launch flag feeding system that could be hit with the huge mod lists this would end up with. This also depends on everyone putting out components to properly use Knossos' version numbering system for best effect.

I'll admit this does add some overhead to releasing and developing a mod, and the benefits will be limited by the amount of legacy content that does not follow this model and would be very likely never be updated to it. But I honestly think that if people begin moving to it with new mods or updates to existing ones, it could be a subtle but significant improvement to the community and the user experience on Knossos. It would obviously have the biggest impact if adopted by FSU, and in a perfect world done retroactively, perhaps releasing an 'alternate history' MVPs mod that old campaigns can be retargeted to, but that's such a titanic and error-prone task that I wouldn't ever expect it to actually be undertaken. But in the case of the FSU changes like Morguy's sounds and Kestrellius's reworked effects could have been developed as separate component mods and then added to the FSU component list once accepted.

I think this also potentially offers help in some ways on the development end: if you use version control in your mod development(and you should) with this system you can keep your big fat images outside of the repository and let knossos handle versioning of models, textures, and possibly even sounds and music for you. That seems like it's a little more dubious in its merits, but potentially useful.

All and all I intend to experiment with this model in Warmachine when we begin releasing versions with significant new art assets, and hope others will give it consideration. I would to hear your thoughts, and please tell me if this post is unclear in any way.

[Goober5000]

Very interesting thoughts.  The same thing has occurred to me, and we have a lot of the necessary tools already - the most important of which being extensible modular tables.

We're sort of moving in that direction already, with MJN's main halls, the cockpit mod, the railgun mod, and the other mods you mentioned.  However, something as significant as breaking up the MVPs into hundreds of individual ship and weapon mods would be a monumental task.

As you said, this would need some sort of comprehensive organizational system in Knossos to be even remotely tractable.

[jadeddragoon]

As someone already releasing "mods" that do not fit the standard pattern you describe, I agree that Knossos has opened the door for much more. In fact, it was Knossos that finally convinced me to stop lurking and begin releasing and fixing up content... because the things I want to accomplish are now viable. I think there is a lot of "inertia" in this community that has kept mod developers thinking and doing things the same way. That a mod can only be two things... a new campaign or a set of assets. I had to add AdmiralMS's screenscam script and Svedalrain's Radar Icons Script (both with permission) to knossos because no one had apparently considered doing so. And I've been accused of trying to take credit for other's work via my mod packs... as if the act of packing those mods together for the convenience of non-tech savvy gamers has no value (and in spite of some of the original mod creators even being involved in my efforts to make the mod packs). This is a case of institutionalized thinking in this community. It's not "wrong" to think of mods only in terms of a new campaign or a bunch of new assets... but it is unnecessarily restrictive. And that's the first hurdle that needs to be overcome for the ideas you describe here.

I also agree that, in an ideal world, the creators of new assets would release each asset or set of assets as it's own mod to the nebula. And then campaign mods or modpacks that wish to include them would depend on them... making adjustments as needed. In fact, I implement this to a lesser degree in my mod packs. Svedalrain's radar icon script as a stripped down version of of a common config parsing script. This conflicts with mods that need the full version. Since I typically need to load radar icon script after these mods, I place the full version of parsecfg-sct.tbm in the data folder for the modpack, overriding the one in RIS. Other examples include adding hotkeys for turning the hud icons on and off. Tweaking the mission file for FreeSpace Blue to use the improved Aquitane mainhall from mjn's mainhalls. And more. Going back to adding screencam script and radar icon script to knossos, I could have just bundled them into my mod packs directly... but I agree that the _correct_ way to handle this, now that it's possible, is for these scripts to be available on nebula and use dependencies to have them pulled in and used in other mods... and for exactly the reasons you mention.

But, and this is a huge BUT, the community needs to get on the same page about semver for this to work. Semver allows for intelligent dependency resolution that doesn't result in mods being broken by updates to assets they depend on. How often have changes to MVPS broken campaigns? I've spoken with the FSU team... they already use semver to try to reduce that problem. But the campaigns depending on them always set the version of MVPS they depend on to "latest" which is an end-run around semver... when they should depend on "~<version the campaign was tested with>". This means that when a breaking change happens to MVPS... because they use semver versioning... the campaign in question will continue using the older version and thus avoid the compatibility issues. You mention multiple versions of mvps being installed. This is actually a good thing for exactly the reason I just mentioned. But it would be better if you didn't need a new copy of every asset in the MVPS when the only asset that changed was... say... the GTFr Poseidon. Instead of having two copies of the entire MVPS to support a campaign that needs the older Poseidon you could have two copies of the Poseidon and one copy of every other FSU asset. At the same time if a small texture change is made to the Poseidon... which I don't believe would be able to break a campaign... then the campaign can benefit from the updated version without the campaign's creator needing to lift a finger.

Very interesting thoughts.  The same thing has occurred to me, and we have a lot of the necessary tools already - the most important of which being extensible modular tables.

We're sort of moving in that direction already, with MJN's main halls, the cockpit mod, the railgun mod, and the other mods you mentioned.  However, something as significant as breaking up the MVPs into hundreds of individual ship and weapon mods would be a monumental task.

As you said, this would need some sort of comprehensive organizational system in Knossos to be even remotely tractable.

Agreed on most points. I still think splitting up MVPS would be worth it if, and only if, maintaining the resulting legion of mods was not untenable. I'd also like to hear your thoughts on what kind of organization changes (generally speaking) would be needed.

[mjn.mixael]

Below is a mostly negative view on this idea. I apologize. I applaud thinking outside the box, but I'm not on board with the idea. Let me explain.

Is there a limit to the length of the -mod line that FSO will accept? Because if so, this is gonna find it really quickly. BtA has 222 ship classes; about 60 are from FSU. Let's round down to 150 custom classes and generously suggest that they are mostly in asset packs of 2 on Knossos. That's 75 dependencies. My FSO install is 20 characters in + mod folder name. Knossos uses short names mostly, so let's set that at 15 for a total of 35 characters. My -mod line is now 2,250 characters long. But maybe that's not a thing. I don't really know.

Maybe I'm old hat... but I would not want to do this at all with a mod the size of BtA. The amount of debugging alone that would have to be dedicated to tbm load order is terrifying. Not to mention just mod organization and debugging. What happens when I find a bug in an asset pack my mod relies on, but I don't own? Instead of just fixing it in my modpack, I submit the bug and hope it's fixed, I guess. Until then I have to roll my own asset. It's not really overhead.. but it's side...head...? It's just more to do during the creative process that can already be demotivating.

Let's take Solaris as another example. Darius would have to upload an entire game's worth of ship assets individually or as small packs... then support them. Ouch. BtA2 uses a lot of assets from Solaris. Many of them have been changed/updated/recolored to better fit our storyline. So would Knossos have individual versions of these similar models as asset packs or do I just carry the changes in BtA?

What about fan updates to existing assets (meaning higher poly versions)? Someone uploads the Kestral asset. I'm working on BtA and decide I want a higher quality Kestral, so I make one. Does that go as yet another asset pack on Knossos or does the old one get updated? What about downstream mods that are affected by a new asset? Do they need to then test and update just for this one ship? I can see this process getting annoying really quickly depending on how majorly a given asset is changed. I know in Knossos you could just set your dependency as "Newest". This can cause bugs either way. New modders, I suspect, would miss the nuance. "Newest" means potential bugs in your mod that you have no control over. Not using "Newest" means you don't get any updates for that asset.

This all seems like a lot of stuff just to save a little hard drive space. Most FS fan assets are used a handful of times here and there with a few more popular exceptions. The biggest contender here would be retail asset upgrades and we already have a modpack that cleanly handles all of those assets and provides players a simple way to play updated FS2 at the same time to boot. FSU currently clocks in at just around 20gb. That's on the low side of the median AAA modern game size. So even someone who's rocking two versions of FSU is still only taking up about the size of one modern PC game. We're even seeing it be more common to have major modern games releasing at 80+gb. Hard drive space is cheap. Spart_N downloads just about everything on Knossos. IIRC he reported his entire FSO install size as a few hundred GB. Call of Duty Modern Warfare was a single game over 200gb. Red Dead Redemption 2 was over 100gb.

Furthermore, it is my observation that many FS mods on Knossos quickly update to the latest MediaVPs. In addition, the mods that generally do not update usually rely on a much older FSU version... usually in the 3.6.10-3.8.2 range. Those versions range from 1.7gb to 7.5gb in size. Literally small potatoes in today's storage terms. Can you even buy flash drives smaller than 10gb anymore? I find it uncommon to see a mod that requires, for example, 4.2.x when 4.4.x is the latest.

I think this idea mistakes players frustrations with re-downloading unchanged files as players being frustrated with having duplicate files. I hold out hope that the new Knossos will be much better at detecting unchanged packages and solve that problem. The reality is that many FS mods just aren't that big. Outside of the handful of tentpoles we have, most mods have only a couple custom assets and constitute a few hundred mb. Hard drive space just isn't the problem and having to download two versions of the High poly Cairo model at 330mb just isn't really a problem for most.

In the end, this makes most sense for absolutely huge assets like the mainhalls that take up multiple GB. But every fighter it's own package? I say no thanks.

[karajorma]

This all seems like a lot of stuff just to save a little hard drive space.

I'm not even certain it will do that. There are a lot of assets that get shared in a mod. Thruster plumes, muzzle flashes, explosions for certain weapons. So what do you do with those? Include them multiple times?


I like the general idea but I don't see this as a problem that should be solved by Knossos. If you want to have this as a feature, it should be something FS2 itself does by being allowed to use anything from the entire knossos directory and having a table of stuff that is used out of all the files present. When you picked a mod, you'd be picking that table as the first thing FS2 reads. FS2 then loads in the mod by simply going down the table and parsing in the assets.

Doing things this way would also solve the issue of precedence (if you needed something to be parsed earlier, just move it higher on the table) and would also solve issues caused by needing to read in everything else in a mod just cause you want one ship.

Doing things this way might not be possible or practical but I tend to agree with mjn that expecting modders to do it all by hand is likely to be more trouble and error-prone than teaching Knossos and FS2 how to do it. And if we can't teach FS2 to do it, we're not going to be able to do it ourselves without a lot of problems.

[jadeddragoon]

Is there a limit to the length of the -mod line that FSO will accept? Because if so, this is gonna find it really quickly. BtA has 222 ship classes; about 60 are from FSU. Let's round down to 150 custom classes and generously suggest that they are mostly in asset packs of 2 on Knossos. That's 75 dependencies. My FSO install is 20 characters in + mod folder name. Knossos uses short names mostly, so let's set that at 15 for a total of 35 characters. My -mod line is now 2,250 characters long. But maybe that's not a thing. I don't really know.

There is a limit. The limit actually depends on the OS/shell. But command line flags are also not the only way a list of folders could be passed to FSO... just (afaik) the only current way.

Maybe I'm old hat... but I would not want to do this at all with a mod the size of BtA. The amount of debugging alone that would have to be dedicated to tbm load order is terrifying. Not to mention just mod organization and debugging.

This would actually remove tbm load order as a concern entirely as the mod load order supersedes that... and the mod load order is entirely controllable.

What happens when I find a bug in an asset pack my mod relies on, but I don't own? Instead of just fixing it in my modpack, I submit the bug and hope it's fixed, I guess. Until then I have to roll my own asset. It's not really overhead.. but it's side...head...? It's just more to do during the creative process that can already be demotivating.

You you would implement a patch in your mod until the upstream mod is fixed. Same thing you would have to do anyway. I already do this in my mod packs to fix conflicts between mods. This is also how this is handled in every single package manager for every single flavor of linux. The ubuntu package, for example, often includes fixes on top of the upstream package from the developer.

Let's take Solaris as another example. Darius would have to upload an entire game's worth of ship assets individually or as small packs... then support them. Ouch.

I'm not sure it would actually be more work. I some ways it would be more... in others it would be less. But yes, this is a very legitimate concern.

BtA2 uses a lot of assets from Solaris. Many of them have been changed/updated/recolored to better fit our storyline. So would Knossos have individual versions of these similar models as asset packs or do I just carry the changes in BtA?

No. You would depend on the original and only include the changed assets in your own... letting unchanged assets be loaded from the original. So if all you changed was a texture... then you wouldn't include the mesh. You would let FSO use the mesh from the original and override the texture with your own.

What about fan updates to existing assets (meaning higher poly versions)? Someone uploads the Kestral asset. I'm working on BtA and decide I want a higher quality Kestral, so I make one. Does that go as yet another asset pack on Knossos or does the old one get updated?

It would be another mod, likely depending on the asset pack and changing only the assets for that one craft/structure.

What about downstream mods that are affected by a new asset? Do they need to then test and update just for this one ship? I can see this process getting annoying really quickly depending on how majorly a given asset is changed. I know in Knossos you could just set your dependency as "Newest". This can cause bugs either way. New modders, I suspect, would miss the nuance. "Newest" means potential bugs in your mod that you have no control over. Not using "Newest" means you don't get any updates for that asset.

This is where semver comes in. No one should ever be setting their dependencies to "newest", frankly. Doing this would get someone laughed out of the room in most GNU/BSD circles. Instead one should use "~a.b.c". What this means is that, so long as "a" and "b" remain the same you get updates from upstream. It is the responsibility of the upstream developer to know if the changes they are making are a potentially breaking change and increment "b" instead of just "c". Changes to "c" are usually for bug fixes or very minor tweaks. And new modders not understanding how to use semver properly is a problem they need to work on... honestly it seems like not many people in this community implement semver well. If an upstream mod's dev is being irresponsible with their versioning, then the options are to confront the dev about it, not use that mod, or to fix the version by setting the dependency to "=a.b.c" instead. There's not much that can be done when an upstream developer is irresponsible. But to be clear... these particular issues have nothing to do with splitting big mods into individual assets. These are problems that already exist because this community doesn't adhere well to semver. If anything, having the assets seperated into individual mods makes things more manageable cause when a breaking change is introduced into a mod you would know exactly what asset is the potential source of trouble. Also, one shouldn't be updating dependencies just because updates exist... if a change to "b" happens developers of downstream mods need to feel comfortable continue using the old asset unless the change is to fix a serious bug or they actively want to benefit from the changes.

This all seems like a lot of stuff just to save a little hard drive space.

Many of the things you yourself have brought up as issues would be solved or reduced in severity by this change. Not just the use of hard drive space.

Most FS fan assets are used a handful of times here and there with a few more popular exceptions.

I would argue that this is an effect, not a cause. Things have been done that way cause it was hard to do it any other way. Whats being suggested would make it possible to change that.

The biggest contender here would be retail asset upgrades and we already have a modpack that cleanly handles all of those assets and provides players a simple way to play updated FS2 at the same time to boot. FSU currently clocks in at just around 20gb. That's on the low side of the median AAA modern game size. So even someone who's rocking two versions of FSU is still only taking up about the size of one modern PC game. We're even seeing it be more common to have major modern games releasing at 80+gb. Hard drive space is cheap. Spart_N downloads just about everything on Knossos. IIRC he reported his entire FSO install size as a few hundred GB. Call of Duty Modern Warfare was a single game over 200gb. Red Dead Redemption 2 was over 100gb.

Actually, almost every mod on knossos could benefit from this. Especially campaigns. And, again, reduced hard drive space is only one of the benefits

Furthermore, it is my observation that many FS mods on Knossos quickly update to the latest MediaVPs. In addition, the mods that generally do not update usually rely on a much older FSU version... usually in the 3.6.10-3.8.2 range. Those versions range from 1.7gb to 7.5gb in size. Literally small potatoes in today's storage terms. Can you even buy flash drives smaller than 10gb anymore? I find it uncommon to see a mod that requires, for example, 4.2.x when 4.4.x is the latest.

They shouldn't. That's one of the problems this community has. It's a common source of many many issues. Issues which stricter adherence to semver would alleviate significantly.

I think this idea mistakes players frustrations with re-downloading unchanged files as players being frustrated with having duplicate files. I hold out hope that the new Knossos will be much better at detecting unchanged packages and solve that problem. The reality is that many FS mods just aren't that big. Outside of the handful of tentpoles we have, most mods have only a couple custom assets and constitute a few hundred mb. Hard drive space just isn't the problem and having to download two versions of the High poly Cairo model at 330mb just isn't really a problem for most.

You're right, hard drive space _isn't_ the problem. The problem is dozens upon dozens of different version of the same assets being maintained by dozens of different people and having to rip assets out of someone else's mod and directly include them in your own to use them (which leads to more versions of the asset floating around). This would significantly reduce that. The idea is called "modular design" and is already considered best practice in software development circles... including games development. In this case assets or asset packs would be equivalent to libraries with campaigns being equivalent to end-user software.

[EatThePath]

I think Jadeddragoon's response to MJN covers a lot of what I would say, but it's too late in the evening right now for me to be sure. However, I do want to respond to the space savings issue. My solid state drive is used for operating system and select games. At the moment I have 186 GB not taken up by system stuff on that drive, and thus usable for games. 131 GB of that is being used by FSO mods, and sure the obvious duplicate old MVPs versions is only about 25 GB of that, but it's hard to say how much more things might go down if something like I am proposing was universally adopted.

To be clear, I don't propose this purely to save space, I propose it mainly because if semver is followed will it should make life overall easier for everyone. Someone could have put out a bugfix (.c) version-incrementing update of the demon pof to fix it's wiggy turrets in an afternoon and everyone would have gotten it without needing to go through the hoops of waiting for enough fixes to build up to warrant building a full MVPs version update. Yeah, if people use the tools badly(not following semver right, using newest, etc) then it could cause problems. I think there's a lot that could be done to make people aware of what the right thing to do is and why before throwing the baby out with the bathwater though. I also think that if this is done widely and well it could be a big boon for asset reuse and community awareness of what's out there, as Knossos would essentially end up with a built in library of cool ships you could add to your mod with a few clicks.

But, aside from making those things easier, it's also a matter of saving bandwidth. Unless everyone starts releasing their mods purely as loose files rather than VPs, which has it's own problems, then there's only so much Knossos is ever going to be able to do. If an MVP update only touches a couple of VPs then it's still a few GB of downloading. It's not murder, but it could be better, and I'd rather it be better.

This all seems like a lot of stuff just to save a little hard drive space.

I'm not even certain it will do that. There are a lot of assets that get shared in a mod. Thruster plumes, muzzle flashes, explosions for certain weapons. So what do you do with those? Include them multiple times?


I like the general idea but I don't see this as a problem that should be solved by Knossos. If you want to have this as a feature, it should be something FS2 itself does by being allowed to use anything from the entire knossos directory and having a table of stuff that is used out of all the files present. When you picked a mod, you'd be picking that table as the first thing FS2 reads. FS2 then loads in the mod by simply going down the table and parsing in the assets.

Doing things this way would also solve the issue of precedence (if you needed something to be parsed earlier, just move it higher on the table) and would also solve issues caused by needing to read in everything else in a mod just cause you want one ship.

Doing things this way might not be possible or practical but I tend to agree with mjn that expecting modders to do it all by hand is likely to be more trouble and error-prone than teaching Knossos and FS2 how to do it. And if we can't teach FS2 to do it, we're not going to be able to do it ourselves without a lot of problems.

I'm having trouble fully grasping your meaning here. To the initial questions: If a bunch of ships all need a set of shared textures to be used, then they can either all require a base texture pack, or all be grouped into one package. Similar logic applies to the other scenarios.

To the rest, I'm really confused. "If we can't teach FS2 to do" what exactly? Manage an asset library? Being able to control the context of a mod is paramount, if FS2 just loaded everything in the mods directory then wouldn't there be massive possibility of mods accidentally stepping on each other and injecting unexpected content? This to me feels like proposing the unreal engine should load the content of every Unreal game I've ever downloaded and then dynamically load the right parts, I can't see how it could be a good idea at all.

[mjn.mixael]

There is a limit. The limit actually depends on the OS/shell. But command line flags are also not the only way a list of folders could be passed to FSO... just (afaik) the only current way.

Then this is a non-starter right here.

This would actually remove tbm load order as a concern entirely as the mod load order supersedes that... and the mod load order is entirely controllable.

This is not true. Dependency tbms can affect main mod data.

I'm not sure it would actually be more work. I some ways it would be more... in others it would be less. But yes, this is a very legitimate concern.

Please explain what would be less work. I'm genuinely curious. Because at the very least creating additional modpacks and setting up the dependency tree is more work.

No. You would depend on the original and only include the changed assets in your own... letting unchanged assets be loaded from the original. So if all you changed was a texture... then you wouldn't include the mesh. You would let FSO use the mesh from the original and override the texture with your own.

You misunderstand. My changes affect the POF, texture, and tbl data. There's nothing left to rely on of an original ship asset. So my question is still unanswered. However I assume based on your responses you want it again as a dependency.

It would be another mod, likely depending on the asset pack and changing only the assets for that one craft/structure.

So when does a new model become an updated asset, because this seems like you're arguing to have both Old Kestrel and New Kestral as their own 'mods' for people to use. I thought the issues was too many versions of assets maintained by too many people. This just moves the problem from the Wiki to Knossos.

semvar stuff

This has yet to solve all the issues people have with the FSU updates and I've been versioning FSU that way since it first got on Knossos... it's not going to solve all the issues here.

Many of the things you yourself have brought up as issues would be solved or reduced in severity by this change. Not just the use of hard drive space.

As per above.. most of what I bring up remains an issue. It's just the location of the issue is migrated.

I would argue that this is an effect, not a cause. Things have been done that way cause it was hard to do it any other way. Whats being suggested would make it possible to change that.

In my ten years modding this game, people's inability to install a custom asset has hardly been insurmountable by a quick post explaining how to it. I really don't think people are making small mods just because they don't know how to find the most recent version of an asset. Especially after MatthTheGeek's incredible work on the wiki recently.

Actually, almost every mod on knossos could benefit from this. Especially campaigns. And, again, reduced hard drive space is only one of the benefits

As a creator of a large campaign with a large modpack, I can tell you I absolutely do not see this as a benefit. There's a reason it's called dependency hell.

They shouldn't. That's one of the problems this community has. It's a common source of many many issues. Issues which stricter adherence to semver would alleviate significantly.

The reasons many of these campaigns rely on older MediaVPs has been explained to death. People are free to update those campaigns at any time. No one has taken the time.

You're right, hard drive space _isn't_ the problem. The problem is dozens upon dozens of different version of the same assets being maintained by dozens of different people and having to rip assets out of someone else's mod and directly include them in your own to use them (which leads to more versions of the asset floating around). This would significantly reduce that. The idea is called "modular design" and is already considered best practice in software development circles... including games development. In this case assets or asset packs would be equivalent to libraries with campaigns being equivalent to end-user software.

I understand what you're going for. But I have never seen this be an issue here. It's painfully easy to go to the Wiki, search for an asset and click the links below to download the asset or the mod the asset is located in. Being able to do that is FSO modding 101 and I'll gladly teach anyone who is unfamiliar with the process of extracting or importing asset data from/into a mod.

And again. I've been following semvar with FSU for years and it has not solved all the problems you suggest it will solve when translated to hundreds of asset packs.

[0rph3u5]

The OP proposal runs into an issue when you stop taking assets "as is". Mjn.mixael already pointed this out and no sufficent response as been given so far - I think because he was being a bit vague about those Solaris assets; Jadeddragon's response did not adress changes in functionality.

Let's run one hypothetical here to make clearer:

Take the GVB Petbe - it is a very cool design that lends itself well to an evolution of bomber design towards strafing runs using thrust or glide. However as the most recent version of the model of the model, its "straight from the box"-version, comes tooled for Inferno: Nostros - and you might not be inclined to to carry parts of the Inferno designs over into your mod just to use it, e.g. its 6 (wiki)/8 (.pof) primary firepoints.

Setting aside that on BrotherBryon's version of the Petbe the primary cannon mounts are modelled, there is nothing stopping you from just reducing the number of firing points and switching the composition of the banks around, e.g. give it 3 primary banks with a 2+2+1 speard and disable primary linking via the tables. But as soon as you do it, you are no longer with an "straight from the box"-asset that can be managed in a way you describe (or at least how I understand it after one reading) because its functional divergence. You cannot just import it from an outside dependency unless the creator/curator of that asset will keep up with your divergences - which is a pretty big ask even if the creator that asset is positively disposed to going along with your wishes in the first place.



Same result if you start taking community made ships and reduce their number of turrets to move them to a different role, e.g. the versions of the PVFg Cleopatra (PVM Naunet) and PVCv_Pnepheros (PVCv Kauket) that come with Memories of the Great War have significantly less turrets than their originals (the PVM Naunet only has 3 turrets, down from 9; the PVCv Kauket has 7, down from 27).

[DefCynodont119]

I would foresee a lot of issues from doing this, partly due to edited versions of table files, texture map files, model files, effect files, interface files, Ect. being present in a lot of mods.

Shadow Genesis for example has tons of re-textured ships and weapons from other mods, so does Exile and BtA has a few as well, the duplicates would outweigh the original assets so much that the returns from using one original version would be negligible.


Plus it seems like way more work for the modder and user TBH, not everyone is using Knossos.


EDIT: I also want to note the amount of one-use models that only are used in one mission and special case versions of assets, in FS modding, the assets serve the mission outlines, and are often edited to do so.

[EatThePath]

This post is getting big, it's late, and if it touches off the kind of response the first one did there's no way I'm going to keep up with that already. There's more I could and still intend to reply to, and I've already split off a couple paragraphs as seed for the next post once I started fading, but I think there's value to trying to take things a few at a time rather than tackle everything at once; I find the sentence by sentence quote wars exhausting. So if it seems I've ignored an important point you made, please give me a chance to come back to it in the future.

Alright, so first off the technical angle. I spoke to ngld about the mod count issue in the past and as I understand it there are ways around it, ones that Knossos isn’t using right now but could. But beyond that, if there’s a problem like that in the way of this idea, then I do not consider that a flaw with the concept but rather an unfortunate and almost certainly correctable flaw with reality. If nobody else is interested in correcting it, I’ve submitted code to the engine before and I intend to do so again for a number of things already.
 
On the practical issues of converting a mod like BtA or Solaris: Yes, absolutely, it’s totally impractical to get out the knives and carve up 200+ ships and parcel them all out manually, simply due to sheer tedium. But, I really don’t think setting up a component mod for a ship as you make or add the ships, spread out over the course of development, is that big of a deal. The Knossos interface definitely isn't made for a mod like that right now, either in explore or in installing and launching a mod, and a mod with 200-300 component mods would definitely make a mess of things, but these are again solvable problems. I think some of my own modding biases come into play here, as I am used to doing incremental releases of things as assets get finished rather than holding out till a campaign is all done and unleashing a glorious wave of new ships all at once.

Now, the issue of asset modification. I admit, the landscape of mods as has been expressed to me by community veterans in the couple days since i posted this is somewhat different than I had thought it leading up to that, and if tweaking textures and pofs is as common as has been expressed to me then yes the space and time savings are reduced. But this isn't an all or nothing thing. I would still expect that while a mod will frequently tweak one or more of the big files in a ship, it'd be rare for them to tweak all, especially in the PBR era. A practical example:
The SD Vritra in Warmachine is a ravana with recolored glows. What we do now, putting our new glow textures in the mod and loading the MVPs, is highly equivalent to how I would handle this in my proposed model if you just replace the MVPs with a Nyxvana component mod. The new textures go in a package in the main mod, or if they grow into a big enough thing(say for example recolored glows for all shivan ships, that would qualify in my book) then they might get pared into a component mod. In either case, the glows for the Ravana add up to about 63 MB, with the remaining texture layers and pof coming to ~525MB if I didn't miss any files. So by loading the MVPs rather than putting a Nyxvana in our files directly that's 90% of the download size of the ship removed for anyone who has played any other mod with the same mvps version, despite us doing the thing that supposedly kills this. And no, we don't modify the pof to modify the textures, because tabled texture replacement works.
And yes, 500MB is chump change, but 500MB times dozens of ships times every update of a mod over its lifetime adds up, both in drive space and in time. I can't tell you how many times I've started Knossos, started an update for a mod(because if I don't do it then I'll not remember to do it later), and then got distracted doing something else by the time it finished downloading, and I know there are users out there with worse connections than mine. Knossos improvements can help here yes, but I do not believe it'll be able to get the same results unless Knossos takes truly heroic measures or a modder divides things up into packages as deeply as I'm suggesting.

The other question raised is what to do if the asset is entirely changed, or comes close enough to obviate any advantages of loading the original. In that case I don't at all think it's a bad thing to have two variants of the ship uploaded to knossos as components, so long as things are clearly labeled. Coming into the community as a new modder, I had no idea how easy it was to swap around turrets between different models, and I think most people coming in with any experience with other mod communities would not realize right away how done a thing it is here to have repainted and returreted ships in your mod that were made for a different mod. Having a library of ships that include variants of ships side by side with each other would be a strong message about what can be done and how things are done here in general. I do think there's a lot that could be done on the knossos interface side to make this smoother, many of those things are nice-to-haves, not need-to-haves.

But to step away from the nitty gritty point by point, there's two threads or themes in the discussion that frustrate me, and I want to give a little voice to it before I pack it in for the night. And this is perhaps more addressed at people on discord who had a nice lively discussion about it mostly at times when I couldn't keep up with it, so apologies if it's kind of general.
First is the idea that if the thing doesn't work perfectly and deliver 100% of it's potential maximum benefits then it isn't worth doing at all. This comes up in the asset modification stuff mostly, and it really baffles me. I'll just point back to my Ravana example there. Some points have been made that take a little of the shine off my original dream, but far from totally tarnished it. Partial benefit is still benefit.
Second is pointing to how modding as it is presently doesn't need or perfectly mesh with the proposed process, or how people don't do the things the proposed process would enable so it's pointless. To me that's circular logic, modding developed in a situation without any automated package management or distribution infrastructure so of course it is adapted well to not taking full advantage of one, and patterns of behavior that require one to function don't exist for the same reason. That doesn't mean those patterns have no value.
That's not to say it's not possible for the benefits to be too small to be worth the trouble, but it's felt to me that for some people these fuel a black-and-white dismissal of the thing that isn't warranted.

[mjn.mixael]

True enough. Let's do away with the hypotheticals and talk big picture.

I am failing to see how this is actually a beneficial change. At best, to me, it's a side grade. It's different but not necessarily better. It has it's own drawbacks and annoyances that are just different from the ones that I think you're trying to solve. However it still isn't entirely clear either what you're trying to solve. I think it has to do with asset file sizes as it relates to storage space and bandwidth (the former I've talked at length about already and the latter I hope new Knossos is better at managing), but I was also told that's not the case. It also seems that it might be about clearer access to the most recent versions of an asset but with all the many changes mods make to assets, you'll end up with the same issue but now it's on Knossos instead of the Wiki. Or perhaps it's about lowering the modding barrier of entry? Maybe a bit of all three?

What this sounds like to me is that you prefer this style of mod organization and think everyone else should follow suit. That's fine. Mod how you want. If you prefer this style, then absolutely go for it. All my years of experience modding this game is screaming at me to avoid this because of all the obvious pitfalls. To me, this is dependency hell. This is a debugging nightmare. This is added busy-work. It's going to be more administrative things for me to do when all I really want to do is make my mod.

A few additional thoughts...

I think there's a communication disconnect here. Yes, as a community we use the word 'mod' to mostly mean campaigns. But not always. I think this kind of asset release on Knossos makes sense for large or otherwise independent/stand-alone additions like mainhalls, scripts, or other bolt-on features. The thing is that this game isn't Skyrim or Subnautica. It's not really the kind of game where you can bolt on new features and then play over and over again. The linear mission-based structure and the rigidity in which missions flow makes that uninteresting pretty quickly. Is having screencam available really going to change my experience of FS2? Probably not. Even the mainhalls, for all the hours upon hours I poured into them, aren't going to alter the experience that much. From the very beginnings, FS modding has been structured around mission-making for this very reason. That fancy new user-made ship is useless to FS unless there's a mission that makes it do stuff. As such, we've mostly become storytellers. People, almost universally, show up to HLP asking not what ships they can add to FS2.. but what campaign they should play next. So yes, our entire modding culture has centered on that for a reason.

For the record.. the days of mods creating entire new fleets and keeping them sekret until the campaign is released are long gone as far as I can tell (Maybe Inferno still, to some small extent?). Most modellers here release stuff publicly as soon as the asset is done, even if it was made for a particular campaign.

Lowering the barrier-to-entry of modding is a worthy cause. But let's be clear what we're talking about here. Adding a ship into FSO is usually as easy as copy/pasting files into the right folders. POFs to data\models. Textures to data\maps. Tables to data\tables. That's an incredibly low barrier. So low, in fact, that you can't even add a retail mission to FSO without understanding, at least in part, that same folder structure. Missions to data\missions. I do not think the person exists that wants to mod but doesn't want to figure out the folder structure and the very next step basic table structure (easy ones like ships or weapons). In my opinion the biggest boon here would be taking Matth's work on the wiki one step further. Instead of a ship page linking to the modpack or asset dump an asset is in... it would be worth a community effort to make sure each ship has a downloadable archive of just that ship with the data in the appropriate folders already. This removes the step of new modders having to hunt for a ship and its assets in a modpack. (Some more modern ship releases are already done like this.)

Having a section or tag of mods on Knossos for add-ons was always in my original vision back when I helped out with the original UI. I do see some use-cases for that kind of thing. I personally would hate to see that theoretical section in new Knossos swamped with hundreds upon hundreds of individual ships or ship packs, making it difficult to browse. But the cool thing is you don't need me to give it the A-OK. If you want to make your mod that way, have at it.


As I tell the BtA team when they have an idea I don't see the benefit of... give it a shot. I'm happy to be proven wrong by ideas that turn out to be good.

[ngld]

I think the bandwidth / space issue is completely separate from how mods should be organized. The bandwidth issue can be easily adressed by Knossos itself, the space issue is more complicated but not as severe, either. The current Knossos release tries to skip unnecessary downloads but that's still a very simple approach that was implemented badly. There's a lot I can do to improve upon that. Saving disk space is harder since I'm limited by what FSO supports on that end but with some work that might change. Either way, a technical solution within Knossos itself would equally apply to all mods regardless of organisation / structure which IMO would lead to more bandwidth / space saving overall.

The mod list limit is a non-issue since Knossos can pass the parameters through a file to FSO which would circumvent any OS limits. I think there are some limitations in FSO itself (i.e. the amount of CFile roots you can have) but those are either high enough that they're hard to hit or easy enough to change once they're hit.

I think overall this is an interesting concept that is worth trying but I wouldn't try to convert any existing mod over to this concept due to the work involved. It would be much more interesting to test this concept with a new mod IMO since that way you'd run into fewer issues and could design everything to take advantage of the modular concept from the start.
That said, I'm not sure if the benefits are worth the additional burden long dependency lists bring with them (and I say that because I have a few Node.JS projects with 2000+ dependencies... >_>). In general, dependencies introduce new error sources, more ways a mod could break. In software projects, you can usually avoid this using unit tests, e2e testing, CI, etc. but we have none of those concepts for mods (yet).
However, right now we don't know whether this will even become a problem or not since it depends a lot on the way releases are made, how mods depend on each other and how well Knossos' own dependency handling will deal with this. That's why I'd say it's worth testing this concept once NuKnossos is stable (enough). I wouldn't try with the current Knossos release because it has known issues around dependency resolution which will most likely any non-trivial dependency tree annoying to deal with.

Regarding the motivations: I think this might be a good way to handle scripts like the ship save/load script. I think a dedicated tool (which may or may not be integrated with Knossos) would be better suited to dealing with downloading and updating ships. For example, it could either update the ship files or (if you modified the pof file) show you the detailed changes made in the ship update so you could update your modified pof to match (if you want to).
That said, I don't know how common ship updates are and how important it is to have the latest version of each ship.

[FrikgFeek]

This is just dependancy hell waiting to happen and when it breaks it's going to be a ***** and a half to find out why.

Let's go for a simple example. You want to include some ships from Solaris, basically as-is. To do that you also need their tabled weapons and if you don't want to edit the tables and want this to be a simple-drop in you need ALL of their weapons otherwise FSO will scream bloody murder as it finds out you have non-existent weapons listed for primary or secondary compatibility. So, OK, the ship you want is availible as a mod and has its necessary weapons either included with it or as dependencies. So now you're already downloading more stuff than you really need.

Let's say you also want some UEF ships from Blue Planet. You repeat the same process, getting the ship and then getting the weapons with their files and tbms. You go in game and.... your weapons are broken. You launch debug and it spews out line after line of "expected #End, found <whatever> instead".  So what happened? Well, Solaris has a weapon called the Rapier. So does Blue Planet. With the same $Name and with both tbms loading they're gonna mess each other up and your "final" entry is going to have essentially garbage data in it.

Now imagine you don't know this. Debug is telling you the problem is with UEF_Rapier-wep.tbm, you look up and down the table and find nothing wrong. You're also loading 500 other tbms from dependancies because each ship pulled like 10 weapons with it as dependencies and any one of them could be causing a conflict with the Rapier. Good luck debugging that.

Or maybe the tables don't cause a fatal conflict but just change the $Damage or $Firewait of your weapon or some other things, depending on load order. You see nothing wrong, debug sees nothing wrong, but your testers are complaining that the weapon is ludicrously OP. Or even worse, it's an enemy weapon and testers are complaining that all the player ships are way too fragile. Now you're compiling comparison excels of player ship health and shield numbers and calculating TTKs and trying to find out why the **** people are getting bursted down. And while nicely laid out like this the answer might seem obvious(because for the sake of this example I had to tell you the answer), it's gonna be much less obvious in an actual dev scenario.

In the above Rapier example if the Solaris rapier loads first it's gonna keep its 3 shot burst, balanced around lower base damage and fire wait, then the later tbm will change its damage and firewait to a weapon that's meant to fire a single shot. Different mods also use different numbers for damage and health, it's often somewhat OK with mods based in the FS universe but not with TCs. So all these assets you pulled will need to be retabled for your mod, meaning your barrier of entry still requires retabling everything and these really aren't just drop-in.

And now think about the ****ery that can happen if instead of $name in tables you have some texture or weapon effect with the same name in your hundreds of spread-out dependencies. Modders don't really adhere to super strict naming standards, you'll see a lot of flashes named "Mflash1" or effects named "BoltGreen" or whatever.

[EatThePath]

Right, so based on the posts by ngld and mjn, and some related conversations in discord, I think the best way to continue is to shift the discussion to how it could best be done rather than if it should be done. A lot of the disagreement at this point is about expected practical outcomes and I don't think any amount of hypotheticals is going to convince anyone to change their expectations. But even if it is a side-grade as mjn suggests I still prefer to work with assets divided into individual packages, and so long as it's not flooding Knossos in an ugly fashion I see advantages to my development workflow in having these as separately uploaded mods instead of packages. So, I'm going to be doing this anyway and everyone benefits if it's done as intelligently as possible. As Frikg so astutely deduced if you try to hammer the last 20 years of square pegs into this proposed round hole without any changes, it will of course go poorly.

Possibly the most important one would be file naming conventions. If people prefix their filenames with useful identifiers, then the odds of two different mflash1.dds files smooshing each other goes down a lot. "WMA_[filename]" is where I'm leaning towards for warmachine assets, but if anyone has compelling arguments for different approaches I'm happy to hear them.

There were other things I wanted to raise along those lines for discussion but they fell out of my head at some point over the week. I'll hopefully remember some later and post more. But the main one I want to ask right now is of mjn: are there any of the MVPs dependency/compatibility issues you've had to troubleshoot that weren't traceable to someone using 'newest' as their version target? If so I'd like to hear about them.

A bit of pie in the sky: In my ideal hypothetical version of this system, even if mod makers chose to fully incorporate assets into their mod, they could use knossos to link back to the original asset. This would do two things. First, it would show users looking at the mod where that original comes from in some sort of detail panel. Second, the developer/maintainer of the mod would have a panel that shows them if an asset has been updated since the version they're using. This would be where something ngld suggested on discord could appear

The asset store idea is great and IMO could be expanded to also offer warnings for known issues (i.e. "You're using the old particle script which is known to cause performance issues! Click here to find out more.").