Author Topic: [Cross-platform] FS2 GOG/Mod installer  (Read 26442 times)

0 Members and 1 Guest are viewing this topic.

Offline ngld

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
I think I've finally fixed all the bugs and issues I found.

Just noticed that I can't find a register link anywhere on the site itself.
The Nebula site now has a register link, users can reset their password if they have forgotten it and they can always change the password once they're logged in.

Edit:  Also the email verification went to my spam, wonder if we can try to reduce the likelihood of that somehow.
I've now changed the email delivery. The mails are now sent through a mail server with regular mail traffic, proper SPF records and all outgoing mails are signed with DKIM. I hope this fixed that problem.

@niffiwan: You tried to add a mod with files hosted on FSFiles. That should work now. The server on FSFiles always returns the HTTP code 206 "Partial content" which caused an error in my downloader. It expected 200 "OK" instead.

I've also implemented file uploads which should make it easier to add new mods which don't have any download links.



I think the Nebula (and the Knossos client) are now ready for beta testing. AFAIK all features have been implemented all that's left is improving the UI and fixing any bugs that are found.
The next steps are:

  • Testing: I could use some help here. Just register over on fsnebula.org, add a new mod then download Knossos and try to install your mod. If you encounter any bugs or have problems with the UI, feel free to report them here or over on GitHub (Nebula / Knossos)
  • UI improvements: The forms on the Nebula site take up too much screen space. Some fields are still missing help texts.
  • Knossos first time setup: The first time you start Knossos, you have to tell it where your FS2 directory is or let it install FS2 using an exe from GOG. It works but the UI for this feels clunky.
  • Add proper documentation to the Nebula: The "Get Started" page is only a placeholder. I should write proper step-by-step instructions for uploading your first mod.
  • Feedback: I want to know what you think about this project. Anything you want to change or improve?
  • Cleanup scripts: The server should remove the associated files if someone deletes a mod.

I think that's all for now. If you want to check out the source code, you can go to GitHub: Nebula and Knossos.

 

Offline niffiwan

  • 211
  • Eluder Class
Re: [Cross-platform] FS2 GOG/Mod installer
@niffiwan: You tried to add a mod with files hosted on FSFiles. That should work now. The server on FSFiles always returns the HTTP code 206 "Partial content" which caused an error in my downloader. It expected 200 "OK" instead.

:yes:  Awesome, fixed an issue I didn't even report :D
Creating a fs2_open.log | Red Alert Bug = Hex Edit | MediaVPs 2014: Bigger HUD gauges | 32bit libs for 64bit Ubuntu
----
Linux OBS Packages: FSO 3.7.0 | FSO BP Build | wxLauncher (?) | PCS2 (?) | wxVPView (?)
Debian Packages (testing/unstable): Freespace2 | wxLauncher
----
m|m: I think I'm suffering from Stockholm syndrome. Bmpman is starting to make sense and it's actually written reasonably well...

 

Offline ngld

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
That's why I love having server logs.  :)

I've toyed with the idea of having Knossos upload error reports automatically but I never actually implemented that because I think that people will complain about the possible privacy issues.

 

Offline jr2

  • The Mail Man
  • 212
  • It's prounounced jayartoo 0x6A7232
    • Steam
Re: [Cross-platform] FS2 GOG/Mod installer
Have it prepare the error report and ask the user if they wish to contribute to troubleshooting by uploading?

 

Offline Goober5000

  • HLP Loremaster
  • Administrator
  • 214
    • Goober5000 Productions
Re: [Cross-platform] FS2 GOG/Mod installer
:bump:

ngld, how are things going here?  There appears to be significant user interest in an alternative to the FSOInstaller for mod downloading, so it would be pretty cool to get this production-ready.

  

Offline ngld

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
EDIT: Please head over to the new thread.

Sorry for the late reply. I've updated fsnebula.org and fixed a few issues I've found.
Right now, I'm fixing a few issues in the client which are caused by newer Qt versions...

Have it prepare the error report and ask the user if they wish to contribute to troubleshooting by uploading?
Once I'm done I'll finish this and release new versions.

After that I (only) need testing and uploaded mods on fsnebula.org to get production-ready.
Everyone can register and add/upload new mods there. If you see your own mod on there and want to update it yourself, edit or remove it, feel free to PM me and I can give you access.

Other things on my wishlist:
  • Actual moderation interface on fsnebula.org (so I don't have to use SQL queries to edit ACLs)
  • Cleanup scripts which remove outdated (unreferenced) files from the mirror
  • A better UI: The forms on fsnebula.org seem bulky and too complicated
  • A refactored, cleaned up codebase (mostly fsnebula and knossos-server).
« Last Edit: February 07, 2017, 08:26:54 am by ngld »

 

Offline niffiwan

  • 211
  • Eluder Class
Re: [Cross-platform] FS2 GOG/Mod installer
great to see you back ngld!:)

One thing  which I thought could be useful based on comments in some other threads, would be some kind of feedback or rating system. Similar to what you see in (e.g.) the Play Store or on GOG, 1-5 star rating, short feedback paragraph. My only concern is noise... which makes me wonder about restricted access to "HLP veterans" to provide reviews, or something like that, with a short bio per reviewer. Maybe "FrikgFeek's recommended list - he likes really challenging campaigns", or "Battuta's must play list - he loves a great story" (sorry for putting possibly wrong words in yours mouths guys...).  Or maybe just make the feedback open... with a simple moderation system to prevent spam.

The aim would be to make it easier for newcomers to find new & interesting campaigns that would interest them; from among the large list of available campaigns out there.
Creating a fs2_open.log | Red Alert Bug = Hex Edit | MediaVPs 2014: Bigger HUD gauges | 32bit libs for 64bit Ubuntu
----
Linux OBS Packages: FSO 3.7.0 | FSO BP Build | wxLauncher (?) | PCS2 (?) | wxVPView (?)
Debian Packages (testing/unstable): Freespace2 | wxLauncher
----
m|m: I think I'm suffering from Stockholm syndrome. Bmpman is starting to make sense and it's actually written reasonably well...

 

Offline Axem

  • Administrator
  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
I tried to add Vassago's Dirge to Nebula as a test and it said the validation server was down. :(

Might I suggest that you start a brand new thread so Nebula and Knossos are super easy to find and point people to? The first post is pretty confusing to new people (and even me!). And could you post a tutorial of sorts on how the fields should be filled out and how modders should organize their downloads? I did a lot of guessing and I have no idea if I actually did it right...

 

Offline ngld

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
great to see you back ngld!:)
Thanks  :)

One thing  which I thought could be useful based on comments in some other threads, would be some kind of feedback or rating system. Similar to what you see in (e.g.) the Play Store or on GOG, 1-5 star rating, short feedback paragraph. My only concern is noise... which makes me wonder about restricted access to "HLP veterans" to provide reviews, or something like that, with a short bio per reviewer. Maybe "FrikgFeek's recommended list - he likes really challenging campaigns", or "Battuta's must play list - he loves a great story" (sorry for putting possibly wrong words in yours mouths guys...).  Or maybe just make the feedback open... with a simple moderation system to prevent spam.
This sounds like a good idea but it's something I'll probably implement after the basics (mod installing & updating) are working and stable.
I'm not sure about the feedback / comments since it seems that it would be better suited to a thread here on HLP. Your list idea reminded me of GOGmixes (Example). Every user would be able to make their own list of recommended mods with comments and ratings which they could then link to from other sites.

I tried to add Vassago's Dirge to Nebula as a test and it said the validation server was down. :(
Sorry, fixed it. (One part of the project was moved to a different server and I forgot to change the domain.)

Might I suggest that you start a brand new thread so Nebula and Knossos are super easy to find and point people to? The first post is pretty confusing to new people (and even me!).
I'm planning to do that but I wanted to wait until I've released the next version of Knossos (see my last post). The first post is completely obsolete by now since it's still talking about Hellzed's shell script which served as the starting point for all this.

And could you post a tutorial of sorts on how the fields should be filled out and how modders should organize their downloads? I did a lot of guessing and I have no idea if I actually did it right...
I will do that (probably tomorrow) though I hope to make the UI self-explaining enough that a tutorial isn't necessary. I just checked and you filled everything out correctly.

For now I'll just write down a few notes which might help others:
  • The categories and keywords on the "New mod" page aren't used anywhere, yet.
  • Packages are parts of your mod which the user can choose to install or ignore (unless you set it to "Required"). Most mods will have exactly one package which is set to required and might have several optional or recommended packages.
  • "Client requirements" and "Executables" only make sense if you're uploading a build of FSO and can be ignored by most modders.
  • "File destination" and the "Post-install scripting" section can be used to move your mod's files around but can be left empty if all your files are either in a subfolder or in the root of their archives.
  • The description is displayed on your mod page and when a user selects your mod in the client and applies to all versions. The notes on the "New build" page are specific to that version and are displayed above the download links and once a player clicks "Install" for your mod.
  • For version numbers Semantic versioning is recommended but you can use a simple incrementing number with ".0.0" appended just as well (1.0.0, 2.0.0, ..., 24.0.0, ...).
  • Every file which is listed on the nebula is mirrored on dl.fsnebula.org so your mod will stay online as long as I can keep the server running and don't run out of disk space (which shouldn't be a problem).

 

Offline Axem

  • Administrator
  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
Okay, that's cool. I was just confused by things like 'file destination'. I wasn't sure if it'd be relative to the root directory or the mod directory previously specified or what.

Also I noticed when trying to use the Knossos launcher, I couldn't get any nebula mods to show up. It also said there was an update available, but when I tried to, it failed to install.

And if this really takes off, do you think there could be something like categories or tabs for mods? Like Pre FS1, FS1 era, Reconstruction, FS2 era, Post FS2, Total Conversion, Full Game, Comedy, Experimental... or something. It might help people find what they're looking for.

 

Offline ngld

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
Okay, that's cool. I was just confused by things like 'file destination'. I wasn't sure if it'd be relative to the root directory or the mod directory previously specified or what.
It's relative to the mod directory. During validation the nebula checks your archives and if it finds the mod.ini in a subdirectory then it automatically adds a "Post-install scripting" action to move your files from that subdirectory into the mod directory. Whenever this happens you'll see "WARNING: Found mod.ini in folder [...]. I assume you forgot the move action; I'll just add it myself." during validation.

Also I noticed when trying to use the Knossos launcher, I couldn't get any nebula mods to show up. It also said there was an update available, but when I tried to, it failed to install.
Thanks, I'm looking into it.

And if this really takes off, do you think there could be something like categories or tabs for mods? Like Pre FS1, FS1 era, Reconstruction, FS2 era, Post FS2, Total Conversion, Full Game, Comedy, Experimental... or something. It might help people find what they're looking for.
That's why there's a categories field, once I add some options you'll be able to select them. However there's no way to view categories but that shouldn't be hard to implement.
I've added your suggestions as options if you edit your mod (go to the mods list, click on your mod, go to the "Mod details" tab and click "Edit mod metadata") you should be able to select them now.

 

Offline niffiwan

  • 211
  • Eluder Class
Re: [Cross-platform] FS2 GOG/Mod installer
:(

Code: [Select]
$ sudo add-apt-repository ppa:ngld/knossos
[sudo] password for mememe:
'This PPA does not support xenial'
Cannot add PPA: ''This PPA does not support xenial''.

Hopefully this is as simple to fix as rebuilding the ppa?
Creating a fs2_open.log | Red Alert Bug = Hex Edit | MediaVPs 2014: Bigger HUD gauges | 32bit libs for 64bit Ubuntu
----
Linux OBS Packages: FSO 3.7.0 | FSO BP Build | wxLauncher (?) | PCS2 (?) | wxVPView (?)
Debian Packages (testing/unstable): Freespace2 | wxLauncher
----
m|m: I think I'm suffering from Stockholm syndrome. Bmpman is starting to make sense and it's actually written reasonably well...

 

Offline ngld

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
Hopefully this is as simple to fix as rebuilding the ppa?
Yes.

I'm done with the next release, I'm currently building and testing the packages for Arch Linux, Ubuntu and Windows.

So far the only serious issue that's left is the way the "-mod" flag is generated since it differs from the way the other launchers work. If you're interested, here's the full explanation:

All other launchers (AFAIK) read the parameter from the mod.ini file but Knossos generates it by looking at a mod's dependencies which means that if an uploader didn't list all dependencies or in the wrong order, the mod most likely won't work. The most important reason for this difference is that Knossos allows a user to install multiple versions of a mod at the same time. To make this work the different versions have different folder names which means that the settings from the mod.ini won't work anymore.
I'm not sure if I'm going to keep this feature because it makes simple things like the -mod flag or update checks more complicated than they should be and I don't think many would use it. However, this system is essential to dealing with several FSO versions.

 

Offline niffiwan

  • 211
  • Eluder Class
Re: [Cross-platform] FS2 GOG/Mod installer
FWIW I think the feature sounds cool.. I really like the way you have a mod config ~per installed mod~, it makes switching between mods super, super easy, just click on launch for each mod & you've switched mods :)  wxLauncher's profiles are good, but not quite a simple, plus certain bits of config aren't carried over, like resolution, which makes switching between full-screen "i'm playing for fun" and window'd "i'm coding or modding" more difficult than they could be.
Creating a fs2_open.log | Red Alert Bug = Hex Edit | MediaVPs 2014: Bigger HUD gauges | 32bit libs for 64bit Ubuntu
----
Linux OBS Packages: FSO 3.7.0 | FSO BP Build | wxLauncher (?) | PCS2 (?) | wxVPView (?)
Debian Packages (testing/unstable): Freespace2 | wxLauncher
----
m|m: I think I'm suffering from Stockholm syndrome. Bmpman is starting to make sense and it's actually written reasonably well...

 

Offline jr2

  • The Mail Man
  • 212
  • It's prounounced jayartoo 0x6A7232
    • Steam
Re: [Cross-platform] FS2 GOG/Mod installer
Hopefully this is as simple to fix as rebuilding the ppa?
Yes.

I'm done with the next release, I'm currently building and testing the packages for Arch Linux, Ubuntu and Windows.

So far the only serious issue that's left is the way the "-mod" flag is generated since it differs from the way the other launchers work. If you're interested, here's the full explanation:

All other launchers (AFAIK) read the parameter from the mod.ini file but Knossos generates it by looking at a mod's dependencies which means that if an uploader didn't list all dependencies or in the wrong order, the mod most likely won't work. The most important reason for this difference is that Knossos allows a user to install multiple versions of a mod at the same time. To make this work the different versions have different folder names which means that the settings from the mod.ini won't work anymore.
I'm not sure if I'm going to keep this feature because it makes simple things like the -mod flag or update checks more complicated than they should be and I don't think many would use it. However, this system is essential to dealing with several FSO versions.


Make it a compatibility option that can be toggled to resolve potential issues?

 

Offline niffiwan

  • 211
  • Eluder Class
Re: [Cross-platform] FS2 GOG/Mod installer
How do you get Knossos to see the current list of mods on Nebula? Do you need to specific a different json file in the settings? I've current just got this: https://fsnebula.org/repo/test.json

Website @ https://fsnebula.org/workshop/mods


Knossos:

Creating a fs2_open.log | Red Alert Bug = Hex Edit | MediaVPs 2014: Bigger HUD gauges | 32bit libs for 64bit Ubuntu
----
Linux OBS Packages: FSO 3.7.0 | FSO BP Build | wxLauncher (?) | PCS2 (?) | wxVPView (?)
Debian Packages (testing/unstable): Freespace2 | wxLauncher
----
m|m: I think I'm suffering from Stockholm syndrome. Bmpman is starting to make sense and it's actually written reasonably well...

 

Offline ngld

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
EDIT: Please head over to the new thread.

Sorry for the long delay, I ran into some issues trying to build Knossos on Windows. Anyway, the release is done. The new version is 0.2.0 (you can click on Settings -> "Launcher settings" to check which version you have) and should be available for Windows, Ubuntu and derivates and Arch Linux. No macOS build since I don't have access to one. If a mac user reads this and wants to help, I'd be happy to provide instructions to build Knossos (it's pretty simple).

The installation instructions are in the usual place and the Windows install should update automatically.

An important change in the new version is that it now automatically reports errors and crashes to me. If you want to opt-out, you can go to "Settings" > "Launcher settings" and disable the "Automatically report errors" option.



Also I noticed when trying to use the Knossos launcher, I couldn't get any nebula mods to show up. It also said there was an update available, but when I tried to, it failed to install.
How do you get Knossos to see the current list of mods on Nebula? Do you need to specific a different json file in the settings?
Can you please check if this still happens with the latest version?

FWIW I think the feature sounds cool.. I really like the way you have a mod config ~per installed mod~, it makes switching between mods super, super easy, just click on launch for each mod & you've switched mods :) 
Thanks :)  I implemented the mod config feature because I often saw mod authors recommending specific flags for their mods and didn't want to remember which flags belonged to which mod.

Make it a compatibility option that can be toggled to resolve potential issues?
That sounds like a good idea... I'll add an option "Use -mod flag from mod.ini" to the "Troubleshooting" tab for each mod.


Here's the tutorial for uploading new mods. I hope it's not too long-winded.

  • Go to fsnebula.org and login or register a new account.
  • Click on the big green button "Upload a new mod".
  • Enter the mod title and use the mod's folder name as the "Unique mod ID". You can also enter a description, specify categories and tags. Finally, you can also upload an image (which must be exactly 255px wide and 122px high).
    Please note: The categories and keywords aren't used anywhere yet but please enter them so we don't have to add them later on.
  • Click on "Submit". On the next page, click on "New build".
  • If you have any notes which only apply to this release (i.e. "Known issues") please write them into the Notes field.
  • Click on "Add package". Call the first package something like "Required files" or "Core". This name will only appear on the download page and as a disabled option in the launcher.
  • If you want you can add further notes to this package, this field is meant to describe what this package contains and makes more sense for optional packages.
  • We will now add the archives (.7z,.zip,.rar,...) that are required for your mod.
    • Click "Add file". If you have a direct download link to your files, enter it into the "Download URLs" field. If you have multiple URLs, click on "Add mirror" to add more text fields. (Note: If your file is on Dropbox, you can append "?dl=1" to the end of the link to get a direct download link. For example: https://www.dropbox.com/s/q7jivmvp7z2nyq7/Lost_v.1.2.zip?dl=1)

      If you don't have a direct download link, leave the text field empty and click on "Upload a file". Your browser will now ask you to select a file. Please select only one file.
      Ignore the "File destination" and "Unpack" fields.
    • If you have more than one archive with required files, click on "Add file" and repeat the above step for each file.
  • Ignore the "Client requirements" and "Executables" sections.
  • If your mod has any dependencies (i.e. Media VPs), click on "Add dependency". If you're not sure, open your mod.ini and look the line with "secondarylist".
    Enter the mod's name in the "Mod ID" field (the name for the Media VPs 2014 is mvps_2014... sorry). I'd recommend to only type the first few characters because after that a autocomplete list will pop up. You must select the name from this list. If the mod you want to enter is not on this list, it's not uploaded (yet) which means that someone (you?) has to add it before you can add it as a dependency.
  • Enter "*" in the "Version" field. If your mod has more than one dependency, click on "Add dependency" and repeat the above steps.
  • If you have any optional parts (packages) for your mod, repeat the above steps to add more packages but set the "Status" field to "Optional" or "Recommended" (if you select recommended, the package will be selected by default).
  • Skip the "Post-install scripting" section and click on "Submit". If everything went well, the server should now be downloading your mod and verifying everything. Once this step is complete, the "Continue" button at the top will become active. Before you click it, take a look at the log below.
  • If the server approved your build, the progress bar will be green and you can now install the mod with the launcher. Congratulations, you're done.
    If the bar is red, an error occurred and you (or I) have to fix it. If you're not sure what caused a certain error (or warning), feel free to ask here.
  • If you know how to solve the error, click on "Continue" and then on "Edit". You'll see the same screen as before and can now fix whatever caused the error and submit your build again. If you're sure that everything was correct and the issue happened on the server, you can click on "Continue" and "Reprocess" to have the server verify the build again.
« Last Edit: February 07, 2017, 08:27:08 am by ngld »

 

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️À➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: [Cross-platform] FS2 GOG/Mod installer
If you can help me out tonight I can give mac building a go.  Eating dinner, will be available in about an hour or so.
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline ngld

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
Great! I'm on IRC so feel free to contact me there.
I assume you already have the Developer Tools installed. If you use Homebrew, then the following steps will get you started.
EDIT: Forgot to mention that the build.sh script (globally) installs the Homebrew packages for 7-Zip, Qt and SDL2. Also it uses pip to install virtualenv globally, everything else is done inside the current working directory.
Code: [Select]
$ git clone -b master https://github.com/ngld/knossos.git
$ cd knossos/tools/mac
$ sh ./build.sh

If everything works, it should build a Knossos.dmg file in the mac folder though it's quite likely that it fails.
« Last Edit: February 02, 2017, 09:56:44 pm by ngld »

 

Offline m!m

  • 210
Re: [Cross-platform] FS2 GOG/Mod installer
I am trying to get the latest version to run on Fedora 25 and I am having problems with the settings menu.

Whenever I try to open the settings dialog the application crashes with a segmentation fault. The last few lines of the output are:
Code: [Select]
dbus[28021]: arguments to dbus_connection_unref() were incorrect, assertion "connection->generation == _dbus_current_generation" failed in file ../../dbus/dbus-connection.c line 2822.
This is normally a bug in some application using the D-Bus library.

QObject::~QObject: Timers cannot be stopped from another thread

Where is the debug log? Maybe that contains more useful information.