[m!m]

@ngld: I encountered a problem with current configuration of the FSPort package. It specifies a dependency of ">=3.7.2" for the FSO package which obviously cannot be satisfied as the "-RC5" indicates a pre-release version which is not bigger or equal to 3.7.2. How do you handle this case or is it just a configuration error?

I would also propose creating separate threads for the different projects (nebula, knossos and the mod specification) that are being discussed in this thread. That should help keeping the discussion focused on the respective projects.

[Hellzed]

@ngld : I created a new branch called hlp-nebula-next on GitHub. You can take a look at how extensive changes are... And this is only the original refactoring from december. I'm still working on new stuff, and have 2 other copies of the project on my hard drive (one for testing "team" features, the other for the better knossos server handling). Please do not switch to this branch for now, it's broken !

[chief1983]

So, if a standalone game was interested in using this setup to manage an installation, how would one go about that?

Also, the FSO json file seems overly complicated, why all the duplicated hashes and file references?

[m!m]

Standalone games aren't really supported right now as the current mod definitions assume that FreeSpace is required by all packages.

The JSON schema is meant to be generated so having a few duplicated hashes is better as it makes parsing the files easier for the launcher.

[chief1983]

So would it theoretically be feasible to either try to support installing standalone games like Diaspora, TBP, or FotG from within one entire launcher/installer management interface, or to 'clone' the entire system for each standalone game environment?  There aren't many but I can see how having a common (if cloned) environment for all the FSO-based standalones out there would be useful.  I especially was interested in the mention about xdelta patching.

[ngld]

Um... sorry for the late response... 

@Hellzed: Thanks! I'll take a closer look at it once I've got more time.

@chief1983: Well, that's not the current FSO JSON file. Here's the current (prettified) one.
That file is generated from this config.

I'm not sure what you mean by duplicated hashes. The checksums should be different as long as the relevant file isn't duplicated.
Regarding the file references: I guess you mean the "filelist" attribute. It always lists the path inside an archive and the path relative to the mod folder where the file should be stored. In many cases those two paths are the same but if you look at the Mac OS package for FSO, you'll see that the paths are different.

m!m's right, we haven't considered TCs, yet, but with a few changes to the launchers it should work fine. Actually, it might already work with Knossos because it doesn't really check whether FS2 is installed or not (maybe I should work on that?).
We won't have to clone the system since we (hopefully) can treat TCs like mods (as long as the retail VPs don't cause any problems).

An installer for TCs could work like this:

• Check if a launcher capable of handling fso:// links is installed. If not, install one.
• Open the fso://install/<TC name> which would start the actual TC download/installation in the user's launcher. (This works just like the steam:// links.)

You could even do this without an installer: Tell people to install either m!m's launcher or Knossos and then click on a link like this to install the TC.

xdelta patching was inspired by this thread. I've only got a rough concept, right now. I'm not sure if it's worth the effort to implement it. However, if there's enough interest, I can build a prototype.

@m!m: Sorry about that! You're right, it should be ">=3.7.2-RC5". I haven't noticed because my SemVer implementation was bugged. 
Creating separate release threads is a great idea! I'll write a proper specification and some implementation examples for the JSON schema and post it once it's done.
I'll create the Knossos thread once I'm done with the next release.

Hellzed suggested on IRC that we should keep using this thread for discussion. I agree because most topics discussed here are relevant to all three topics.

[chief1983]

Ah, I only asked because I found that one listed under FSO on the Nebula page.  If there's a newer one now, I didn't see a way to find it when I looked on the website from page 14, from which I found the FSO page.  So I guess I got a little confused about where the most current stuff might be found.  In that file there is a filelist array, which  has all the binary files within the archives and their hashes, and then later a files array, which has all the binary files within the archives and their hashes as well as more metadata about them, and a different organization.  So that's the redundancy I was talking about.  Also, anyone viewing JSON in Chrome doesn't need a prettified link, it does it for you now, as well as syntax highlighting

The current RC5.pretty.json still has the similar syntax, with md5 hashes seemingly duplicated across the file, but since that's just generated I guess that's not really an issue.  And since the md5s aren't in the cfg.json at all I take it the generator is actually downloading and extracting these files to generate the final json itself?  Seems risky to trust the generator to get a correct download instead of having hashes generated by the same end that created and uploaded the files.

[ngld]

Ah, I only asked because I found that one listed under FSO on the Nebula page.  If there's a newer one now, I didn't see a way to find it when I looked on the website from page 14, from which I found the FSO page.  So I guess I got a little confused about where the most current stuff might be found.

I had to manually write the .cfg.json file for FSO because the current Nebula website doesn't support some of the newer fields. This is just a temporary workaround until the Nebula catches up.

In that file there is a filelist array, which  has all the binary files within the archives and their hashes, and then later a files array, which has all the binary files within the archives and their hashes as well as more metadata about them, and a different organization.  So that's the redundancy I was talking about.

That's not quite correct. The filelist lists the files within the archives while the files list contains the archives. The FSO binaries are only in the filelist.

The current RC5.pretty.json still has the similar syntax, with md5 hashes seemingly duplicated across the file, but since that's just generated I guess that's not really an issue.  And since the md5s aren't in the cfg.json at all I take it the generator is actually downloading and extracting these files to generate the final json itself?  Seems risky to trust the generator to get a correct download instead of having hashes generated by the same end that created and uploaded the files.

I wouldn't call it "risky" since the generator verifies the archives. If a download were corrupted, it would most likely fail to extract which in turn would lead to an error message.
I've designed the generator this way because it's supposed to be as simple as possible. Since the modder has to include download links anyway, I thought it would be better to download the files (and check that those links actually work) instead of requiring the modder to generate MD5 checksums.

[m!m]

An installer for TCs could work like this:

• Check if a launcher capable of handling fso:// links is installed. If not, install one.
• Open the fso://install/<TC name> which would start the actual TC download/installation in the user's launcher. (This works just like the steam:// links.)

You could even do this without an installer: Tell people to install either m!m's launcher or Knossos and then click on a link like this to install the TC.

How about adding a commandline option to FSO that tells it not to use the root directory as a data directory? With that option the executables could be located anywhere and the launcher could specify all the mod directories using -mod, including the location of the retail/TC data. Mods would then add normal dependencies for TC/retail data. The launcher would need to supply the "retail" package if a retail installation is present as that is the only "mod" that isn't managed by Nebula.

[chief1983]

I've been wanting to be able to specify the root data directory as being different from the binary's current directory for a long time.  Would have many uses, although not sure if it's exactly what we'd need here.  In this case, the standalone would need its own root folder, separate from FS2's, and the binaries would have to be installed to that folder right now, the way they currently expect to find data.

[ngld]

I've been wanting to be able to specify the root data directory as being different from the binary's current directory for a long time.  Would have many uses, although not sure if it's exactly what we'd need here.

It's already possible on Linux and Mac OS. You just have to set the current working directory (CWD) to the root directory of your choice. On Windows it doesn't work because FSO automatically changes the CWD to the executable's directory. I wonder why this was done only on Windows...

How about adding a commandline option to FSO that tells it not to use the root directory as a data directory?

As a temporary workaround, we can copy the executable to an empty directory on Windows and change the CWD on other OSes. If you copy the executable somewhere, you should add the old directory to the PATH variable to make sure that FSO still loads the DLLs from there.

With that option the executables could be located anywhere and the launcher could specify all the mod directories using -mod, including the location of the retail/TC data. Mods would then add normal dependencies for TC/retail data. The launcher would need to supply the "retail" package if a retail installation is present as that is the only "mod" that isn't managed by Nebula.

That's a great idea! I'll add a note about this to the JSON spec.

[ngld]

I've finally finished and posted the JSON spec. What do you think?

@m!m: Can you please take a look at the "notes on launching a mod"? This should solve our problems with mods which use primarylist (like FSPort). I've also added your suggestion regarding the "retail" dependency.

@ngld: I encountered a problem with current configuration of the FSPort package. It specifies a dependency of ">=3.7.2" for the FSO package which obviously cannot be satisfied as the "-RC5" indicates a pre-release version which is not bigger or equal to 3.7.2. How do you handle this case or is it just a configuration error?

I've started fixing the broken dependencies on fsnebula.org, if you still find something, let me know!

[m!m]

Thank you for putting that post together, I have some points I'd like to discuss but I will post that in the thread.

[ngld]

After the recent silence, I've started working on Hellzed's Nebula. The results look quite promising.
I'll hopefully be able to release a new version on fsnebula.org in a few weeks.

My current goals:

Nebula

• A proper interface for mod searching and downloading.
     
  • Mods now have pages which show their description, logo and download links. There's also a page which lists all files which belong to a mod and their MD5 checksum.
  • There's currently only a simple mod list. I want to expand it (display the mod's logo, a short excerpt from the mod's description, etc.) and add a search function.
   
• Mod submission/uploading should be easier.
     
  • The version field is now only a single text field (instead of 5 as it was before).
  • The converter performs a few additional checks which should make it easier to add a new mod (i.e. it automatically detects whether the mod's archives contain the mod folder or only the files within that folder).
  • I want to add the ability to upload files directly which would remove the need for DDLs.
   
• TC support!
  Well, we have to figure out how to handle TCs in the JSON spec (see that thread). Once that's done, I'll add the necessary fields.

Knossos

• Add support for TCs
• Improve troubleshooting capabilites:
  
  • Knossos can already verify and repair installed mods, remove old files and open the debug log for the user. However, some of those functions are hard to find. This came to mind after I read this thread.
  • Once I'm done with the other tasks (Nebula, TCs), I'll probably start a new thread on the FSO Support board (is there a better place for this?) where I'll ask what kind of troubleshooting functions the supporters would like to see in a launcher.

Once I'm done with the Nebula and added TC support, I'll open a new release thread here for Nebula and Knossos. Well, we'll see how that goes...

[chief1983]

I'm really sad to see that this was the last post, over a year ago.  Was any progress made?  I just managed to get what was in the master branch up and running on my VPS that runs Linux nightlies.  Next will be to see how hard it is for me to get the Knossos side running.  Not wanting to get any further into tearing into it myself until I at least see how much uncommitted code is laying around.  I may be able to help with some of those tasks if need be.

[ngld]

I'm glad to see that there's still interest in this project. I stopped working on this a while after that post because I didn't have enough time to keep working on it alone. Since Hellzed stopped working on the Nebula, I forked it, updated a few dependencies and fixed a few issues. I'll commit a few missing fixes later on.

I've repaired fsnebula.org which means you can register once again.
I'll write proper setup instructions for the server and post them later tonight.

I'd be happy to work on this project again. I'll post more on the current state and unfinished tasks once I have the time (either today or tomorrow).

[chief1983]

Just noticed that I can't find a register link anywhere on the site itself.  I might see if I can find a good place or two to shoehorn that in.  Maybe the Sign In page for starters, and a Modders section under Getting Started?

Edit:  Also the email verification went to my spam, wonder if we can try to reduce the likelihood of that somehow.

[Goober5000]

There were extended periods of time I stopped working on the FSO Installer due to lack of time, lack of motivation, or other priorities.  Keep at it.  This has solid potential.

[jr2]

Edit:  Also the email verification went to my spam, wonder if we can try to reduce the likelihood of that somehow.

I think individual users marking e-mails "not spam" should help, but that's per e-mail provider, so..

[ngld]

Just noticed that I can't find a register link anywhere on the site itself.  I might see if I can find a good place or two to shoehorn that in.  Maybe the Sign In page for starters, and a Modders section under Getting Started?

Sounds good. I think we intentionally didn't include that link because we wanted to be sure that it works before opening registration to everyone but I don't think that matters anymore.
It's also missing a password reset function as well as a way to delete users. The user management is implemented using the FOSUserBundle so it shouldn't be too hard to add.

Edit:  Also the email verification went to my spam, wonder if we can try to reduce the likelihood of that somehow.

I'll move the mail delivery to a proper mail server, hopefully that helps.

There were extended periods of time I stopped working on the FSO Installer due to lack of time, lack of motivation, or other priorities.  Keep at it.  This has solid potential.

Thanks for the kind words! It means a lot to me.

@chief: Apparently your Nebula installation can't send mails. I tried to register but didn't get the activation email.
To actually add mods to the Nebula, you'll also need the Knossos server. You can either use my installation (should be the default) or run your own. I've added documentation to Knossos server's readme.