Author Topic: [Cross-platform] FS2 GOG/Mod installer  (Read 72024 times)

0 Members and 2 Guests are viewing this topic.

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Uh, that was my fault. I accidentally added a bug to the converter when I implemented the web frontend. It's fixed now.

I've added a bunch of new checks to the converter (which should find most errors now) and simplified the output. The online converter now tells you if the conversion was a success ("Done!" message and a green "Continue" button) or a failure ("Failed!" and a red "Continue" button).

The online converter works like this:
  • The nebula sends a POST request to http://s1.nebula.tproxy.de/api/converter/request with the following fields: passwd set to the API key and data containing the repo JSON.
    The response is a JSON object with the following fields: ticket and token
  • The browser connects a WebSocket to http://s1.nebula.tproxy.de/ws/converter and sends the ticket. The status updates are then transmitted through the WebSocket in real-time. Once the converter is finished, it sends a "done" message containing the result (did it fail?).
  • The nebula sends a second POST request to http://s1.nebula.tproxy.de/api/converter/retrieve with the following fields: ticket and token, both fields containing the values returned by the first request.
    The response is the generated JSON. You don't have to worry about capturing it.  :D

If you look at the source code of the online converter, you'll see that the JavaScript is doing exactly that. However, the two POST request are supposed to be handled by the server to make sure the API key is kept secret and because the JSON response has to be stored on the server anyway.

The source code is on GitHub. It's written in Python because it's easier to integrate the converter and handle WebSockets that way.
To setup the server, you'll have to run the following commands (assuming you have python3 and virtualenv installed):
Code: [Select]
git clone -b develop https://github.com/ngld/knossos-server
cd knossos-server
git submodule update --init
python3 /usr/bin/virtualenv py-env
source py-env/bin/activate
pip install -r requirements.txt
python setup_db.py

To run the server, you should run:
Code: [Select]
source py-env/bin/activate
uwsgi --master -p 4 --http localhost:8080 --http-websockets --wsgi server:app --enable-threads

I'll think about a way to solve the mods issue but I don't understand why you don't want to just change the JSON in PHP.

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
Ok, I'll change the JSON in the PHP script ! No need to change anything on your side.
(to me it's only a matter of removing the "mods" container, concatenating the right builds of mods, and adding a new "mods" container. I only wanted to know if things could be made one step simpler on both our sides, but that's really a tiny detail.)

I'll start writing the new build validation page in Nebula tomorrow. Thanks to your work on Knossos-server, it should go pretty fast.

Have you tried running Nebula ?

EDIT :
I'm still pretty new to all these things ^^
I never needed before to send a request to an external web service from php. I've read i'll need to use cURL.

From the 3 steps you described, I get the impression that someone has to try and access the "build in progress" page in Nebula so that the second POST request is sent, and the generated data registered in the database.
What happens if the modder who started the validation process quits the page and doesn't come back ?
I can circumvent this issue in Nebula by accessing http://s1.nebula.tproxy.de/api/converter/retrieve whenever a mod known to be configured is requested but has no cached final JSON. (to do this, I only need to save tickets and tokens in the database and use them when needed, right ? I mean Knossos server doesn't loose track of mods if there's more than one at the same time ?)

I'm thinking of another solution : maybe Knossos server could call the right URL in Nebula once it's done, to trigger the database registration ?
To avoid the security risk of getting an arbitrary JSON entry in the database, we could either :
- use a simple public/private key mechanism.
- only accept requests coming from Knossos-server's domain.
« Last Edit: October 02, 2014, 05:11:37 pm by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Sorry, I didn't get to test the Nebula but I'll probably have time tomorrow.

I've already thought of that problem but I didn't get to implement a solution (yet). My idea is that I'll add an optional field to the /api/converter/request endpoint called webhook. The Nebula server fills a URL in this field and once the conversion is finished, the Knossos Server will send a POST request to that URL containing the ticket ID. The Nebula can then access the retrieve endpoint to get the actual JSON. Since the JSON will be fetched from the Knossos Server, no one can insert malicious JSON. What do you think?
The Knossos server forgets a mod once the generated JSON is retrieved to avoid wasting space. I haven't found a better solution. There's also a problem with the mod logos: I haven't figured out a good way to include them in the result so I simply embedded them using data: URLs but there's something wrong with the base64 encoding which breaks the images...

Have you had a look at Guzzle? It should make accessing the API much easier.

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
Guzzle looks nice but i'll probably won't need it. I already have all the JSON and incoming request classes I need in Symfony.
Sending the requests is actually only a few lines of php with cURL, and looks like this (I've just tried it for the first request, it works) :
Code: [Select]
<?php
$url 
'http://s1.nebula.tproxy.de/api/converter/request';
$fields = array(
'passwd' => urlencode($passwd),  //$passwd is the API key
'data' => urlencode($data), //$data is the original JSON
);

//url-ify the data for the POST
foreach($fields as $key=>$value)
{
  
$fields_string .= $key.'='.$value.'&';
}
rtrim($fields_string'&');

//open Knossos-server connection
$ksconn curl_init();

//set the url, number of POST vars, POST data
curl_setopt($ksconn,CURLOPT_URL$url);
curl_setopt($ksconn,CURLOPT_POSTcount($fields));
curl_setopt($ksconn,CURLOPT_POSTFIELDS$fields_string);

//execute post
$result curl_exec($ksconn);

//close connection
curl_close($ksconn);

//show the raw JSON token+ticket response
echo $result;

?>

As it will happen only twice each time a mod is validated, I'll put that code in a Symfony "service" that can be called from anywhere in the controllers, and the urls settings in a Symfony YML config file...
I already did that with the custom JSONBuilder.

Webhook sounds good to me.

Mod logos will be handled by Nebula, probably as a special item in the mod image gallery (3 optional special images : the logo, the large background image for the mod page, the "collapsed" background image for Knossos mods list ; and of course the screenshots).
« Last Edit: October 03, 2014, 02:56:25 pm by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
I'd move the urlencode() call in the foreach loop. You can use parse_str() to parse the response into an array.
I've installed the Nebula on the server. You can reach it at http://nebula.tproxy.de/nebula and register at http://nebula.tproxy.de/nebula/register .

I installed it with Composer which was nice because the install script already created the app/config/parameters.yml for me. Everything else worked without problems.
I've found a small bug: The "Modders & Teams" tab (/nebula/owners) doesn't work because Symfony tries to call BaseController::ownersAction which doesn't exist. I guess the OwnerController's index method should be called instead?
I've got a couple of suggestions:
  • Could you add a validation email to the registration process to make sure everyone registers with a real email address?
  • Could you create a default branch whenever a modder creates a new mod and redirect to the "New build" page to make the process of adding a new mod easier and faster?
  • The post-install actions are meant to give the package creator more control over where the extracted files are put.
    This is necessary for some mods where the archive contains a folder (i.e. MediaVPs). The actions are a great way to change the file structure without having to change any of the archives.
  • Why is the unique branch URL empty?

To make the integration of the online converter easier for you I'm writing a simple client. To display the progress page you could do something like this:
Code: [Select]
<div id="progress-container"></div>
<script type="text/javascript" src="http://s1.nebula.tproxy.de/static/converter.js"></script>
<script type="text/javascript">
var cv = new Converter('http://s1.nebula.tproxy.de/ws/converter', <?php echo json_encode($ticket); ?>);
cv.bootstrap_ui($('#progress-container'));
cv.on('done', function (success) {
  // Do something to tell the user that the conversion is finished.
});
cv.connect();
</script>

The JS client isn't finished, yet. I'll release it together with the new webhook feature once I'm done.

EDIT: Oh, I didn't know Hard-Light had syntax highlighting. That PHP code is reaaally legible.  :lol:

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
Nice job with the js client ! I was reaching the point where I would have needed to implement this !

Could you render the widget  (content of #progress-container) as :
Code: [Select]
      <h4><span class="glyphicon glyphicon-cog"></span> Progress</h4>
      <hr/>
      <div class="progress">
          <div id="master-pg" class="progress-bar progress-bar-success" style="width: 0%;"></div>
      </div>
      <pre id="pg-well"></pre>
      <br/>
      <h4><span class="glyphicon glyphicon-list"></span> Log</h4>
      <hr/>
      <pre id="logging"></pre>

I already have the template for this page, and with this it will blend perfectly with the theme.

Quote
Could you add a validation email to the registration process to make sure everyone registers with a real email address?
FoSUB has a simple switch to activate this, IIRC. But at least for the early tests, I'll make the registration "invite only" (and maybe keep it this way since FS2 modders are only a small number.). I have to copy a few templates from FoSUB wiki to do this, because this feature is not included in their code base.

Quote
Could you create a default branch whenever a modder creates a new mod and redirect to the "New build" page to make the process of adding a new mod easier and faster?
That's smart. I'll add this too, at the same time as the "default branch" logic.

Quote
The post-install actions are meant to give the package creator more control over where the extracted files are put.
This is necessary for some mods where the archive contains a folder (i.e. MediaVPs). The actions are a great way to change the file structure without having to change any of the archives.
Oh, ok ! I didn't knew that (I thought Knossos tried to guess the actual archive root, like my old sh script did, and this was only intended for post-install/update cleaning. I'll change the form help-text ).

Quote
Why is the unique branch URL empty?
As "default branch" and "default build" logics are not in yet, I can only generate URLs for a single build at a time.


I'll add a temporary fix for the owners list error. (the definitive fix will come when I'll finish implementing modders teams, the objects are in but there's no control panel yet).

I've also figured I'd have to refactor the count logic to make it a bit more efficient. According to Symfony and Doctrine devs, SQL COUNT()  should be avoided, and instead, a count field should be created in the parent object and updated whenever a sub-object is created or deleted.


EDIT : I thought about the player UI. I originally planned to do a dynamic "tags" system like the new Steam tags. Freespace has a lot of mods, but that's still a few dozens, not thousands like Steam games. The risk is a catalogue "fragmentation", where mods end up being alone in their too specific categories, making navigation harder and wasting UI space.
Instead, I'll do 2 completely separate things : a static list of categories (think netflix-like), and a list of custom search keywords for each mod. A mod could be a part of multiple categories (some of these will be taken from this page http://www.hard-light.net/wiki/index.php/Campaign_List, but I'll add a few others, a single dynamic "New !" category and a "virtual" category "Related mods", depending on the mod you are viewing).
Keywords will be used by the search engine to bring better results, and not directly shown to the end user.

@ngld : I noticed your test server is offline. did something wrong happen (critical bug, security issue, spam bot) ? I only gave the link to MjnMixael and Axem so they could see the forms and comment on the usability.
(I'll read how to fine tune Symfony to switch it to production mode)

EDIT 2 : I noticed you've made changes to the api, can I start sending requests ?
« Last Edit: October 05, 2014, 04:06:41 pm by Hellzed »

 

Offline pecenipicek

  • Roast Chicken
  • 211
  • Powered by copious amounts of coffee and nicotine
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • PeceniPicek's own deviantart page
Re: [Cross-platform] FS2 GOG/Mod installer
All i have to say is good job guys :)





also, ew, symfony :p
Skype: vrganjko
Ho, ho, ho, to the bottle I go
to heal my heart and drown my woe!
Rain may fall and wind may blow,
and many miles be still to go,
but under a tall tree I will lie!

The Apocalypse Project needs YOU! - recruiting info thread.

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
@pecenipicek : Thanks for your support !
By the way, I chose Symfony/PHP because I'm not even a real programmer, but there was a good MOOC about it on my usual e-learning website :P (IRL, I'm a political science/European history student, but I've been doing small stuff with php since I was in high school)
If you are worried about Symfony's performance, yes, it's slow in debug mode and on first generation of a page, but it will get much better in production mode, once the cache is warm.


Here's my current list of mod categories :
- Essentials : Media VPs, FSPort, Silent Threat : Reborn
- New : the latest 4 or 5 mods
- Highlights : mods matching some criteria like being voice-acted, latest Media VPs compatible...
- Classics : FSCRP mods
(next ones are from the wiki)
- Pre-Great War Era
- Great War Era
- Reconstruction Era
- Capella Era
- Post-Capella Era
- Alternate timelines/parallel universes
- Parodies
- Total conversions

Imagine a Netflix like UI : each category is a line, with a short (less than one line) presentation text. You can either scroll to the right or left to see more mods in the same category, or click the category title to get a full page of these mods.

 

Offline AdmiralRalwood

  • 211
  • The Cthulhu programmer himself!
    • Skype
    • Steam
    • Twitter
Re: [Cross-platform] FS2 GOG/Mod installer
I'm not sure "Capella Era" and "Post-Capella Era" are the most spoiler-free category names... suppose a brand-new user, never having played FS2, buys it from GOG or Steam and decides to check out FSO with mods. Sure, they may only be grabbing the Essentials... but from one look at the category list, they might now spend the entire campaign waiting for something to happen to Capella.
Ph'nglui mglw'nafh Codethulhu GitHub wgah'nagl fhtagn.

schrödinbug (noun) - a bug that manifests itself in running software after a programmer notices that the code should never have worked in the first place.

When you gaze long into BMPMAN, BMPMAN also gazes into you.

"I am one of the best FREDders on Earth" -General Battuta

<Aesaar> literary criticism is vladimir putin

<MageKing17> "There's probably a reason the code is the way it is" is a very dangerous line of thought. :P
<MageKing17> Because the "reason" often turns out to be "nobody noticed it was wrong".
(the very next day)
<MageKing17> this ****ing code did it to me again
<MageKing17> "That doesn't really make sense to me, but I'll assume it was being done for a reason."
<MageKing17> **** ME
<MageKing17> THE REASON IS PEOPLE ARE STUPID
<MageKing17> ESPECIALLY ME

<MageKing17> God damn, I do not understand how this is breaking.
<MageKing17> Everything points to "this should work fine", and yet it's clearly not working.
<MjnMixael> 2 hours later... "God damn, how did this ever work at all?!"
(...)
<MageKing17> so
<MageKing17> more than two hours
<MageKing17> but once again we have reached the inevitable conclusion
<MageKing17> How did this code ever work in the first place!?

<@The_E> Welcome to OpenGL, where standards compliance is optional, and error reporting inconsistent

<MageKing17> It was all working perfectly until I actually tried it on an actual mission.

<IronWorks> I am useful for FSO stuff again. This is a red-letter day!
* z64555 erases "Thursday" and rewrites it in red ink

<MageKing17> TIL the entire homing code is held up by shoestrings and duct tape, basically.

 

Offline niffiwan

  • 211
  • Eluder Class
Re: [Cross-platform] FS2 GOG/Mod installer
How about "2nd Incursion" and "Post 2nd Incursion"? Hardly any spoiler in that as you already expect the Shivans to be back :)
Creating a fs2_open.log | Red Alert Bug = Hex Edit | MediaVPs 2014: Bigger HUD gauges | 32bit libs for 64bit Ubuntu
----
Debian Packages (testing/unstable): Freespace2 | wxLauncher
----
m|m: I think I'm suffering from Stockholm syndrome. Bmpman is starting to make sense and it's actually written reasonably well...

 

Offline AdmiralRalwood

  • 211
  • The Cthulhu programmer himself!
    • Skype
    • Steam
    • Twitter
Re: [Cross-platform] FS2 GOG/Mod installer
That sounds better than "FS2 Era" and "Post-FS2 Era", which was all my brain was suggesting.  :yes:
Ph'nglui mglw'nafh Codethulhu GitHub wgah'nagl fhtagn.

schrödinbug (noun) - a bug that manifests itself in running software after a programmer notices that the code should never have worked in the first place.

When you gaze long into BMPMAN, BMPMAN also gazes into you.

"I am one of the best FREDders on Earth" -General Battuta

<Aesaar> literary criticism is vladimir putin

<MageKing17> "There's probably a reason the code is the way it is" is a very dangerous line of thought. :P
<MageKing17> Because the "reason" often turns out to be "nobody noticed it was wrong".
(the very next day)
<MageKing17> this ****ing code did it to me again
<MageKing17> "That doesn't really make sense to me, but I'll assume it was being done for a reason."
<MageKing17> **** ME
<MageKing17> THE REASON IS PEOPLE ARE STUPID
<MageKing17> ESPECIALLY ME

<MageKing17> God damn, I do not understand how this is breaking.
<MageKing17> Everything points to "this should work fine", and yet it's clearly not working.
<MjnMixael> 2 hours later... "God damn, how did this ever work at all?!"
(...)
<MageKing17> so
<MageKing17> more than two hours
<MageKing17> but once again we have reached the inevitable conclusion
<MageKing17> How did this code ever work in the first place!?

<@The_E> Welcome to OpenGL, where standards compliance is optional, and error reporting inconsistent

<MageKing17> It was all working perfectly until I actually tried it on an actual mission.

<IronWorks> I am useful for FSO stuff again. This is a red-letter day!
* z64555 erases "Thursday" and rewrites it in red ink

<MageKing17> TIL the entire homing code is held up by shoestrings and duct tape, basically.

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
I'm liking "Second Incursion" and "Post-Second Incursion" ! :)

Here are some early screenshots of the highlights and mod browsing pages. Sorry ngld, I haven't decided yet where to put Knossos download link in the navbar...

- The navbar needs a better nebula logo
- I used the default mod logos, so they are a bit stretched... I'll ask modders to upload better ones.
- I'll add a mouseover effect on the mod items, to show a few lines of their presentation text and the play button, and probably place the mod title on top of the mod image, with some transparency.
- The items are not huge, my screen is really small. Expect more items per line on bigger screens.


[attachment kidnapped by pirates]

 

Offline pecenipicek

  • Roast Chicken
  • 211
  • Powered by copious amounts of coffee and nicotine
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • PeceniPicek's own deviantart page
Re: [Cross-platform] FS2 GOG/Mod installer
@pecenipicek : Thanks for your support !
By the way, I chose Symfony/PHP because I'm not even a real programmer, but there was a good MOOC about it on my usual e-learning website :P (IRL, I'm a political science/European history student, but I've been doing small stuff with php since I was in high school)
If you are worried about Symfony's performance, yes, it's slow in debug mode and on first generation of a page, but it will get much better in production mode, once the cache is warm.
i'm a php dev by trade these days apparently :p

my beef with symfony is that its this big massive beast which pretends to do everything right in the php world :p

and dont worry about "real programmer" stuff. i think that those days are gone.

i'm just curious, on what will you guys host this?
Skype: vrganjko
Ho, ho, ho, to the bottle I go
to heal my heart and drown my woe!
Rain may fall and wind may blow,
and many miles be still to go,
but under a tall tree I will lie!

The Apocalypse Project needs YOU! - recruiting info thread.

 

Offline Goober5000

  • HLP Loremaster
  • 214
    • Goober5000 Productions
Re: [Cross-platform] FS2 GOG/Mod installer
and dont worry about "real programmer" stuff. i think that those days are gone.

None of us are Real Programmers anymore...
http://www.catb.org/jargon/html/story-of-mel.html

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
@pecenipicek : I don't know yet. ngld is hosting the current test server, so we'll discuss this when we're ready.

@Goober5000 : Awesome story !

@ngld :
« Last Edit: October 08, 2014, 10:39:56 am by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
The new API (with the webhook feature) and the JS client are now on GitHub and the test server. I'll later add some documentation for it to the GitHub wiki.

@pecenipicek: Are you asking about the hardware or software?
The server is a dedicated box. I'm running Apache, PHP (in FastCGI mode) and MySQL on it. The converter is written in Python and runs inside a uwsgi server.
HAProxy forwards all requests depending on the domain name (nebula.tproxy.de is forwarded to Apache and s1.nebula.tproxy.de is forwarded to uwsgi). It also handles idle connections to ease the load on the Apache server (similiar to what lingerd does).

@Hellzed: Looks nice! :) I'd suggest using "mod manager" instead of "modding client", though.
I'm working on the auto updater function for Knossos next.
I already have a working installer for windows, creating an updater should be easy for this case.
I'm not so sure about Linux and Mac OS.

EDIT: The whole API / schema documentation is now linked here: https://github.com/ngld/knossos/wiki
« Last Edit: October 08, 2014, 03:01:24 pm by ngld »

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
I found a small mistake in the documentation, the websocket address is wrong :
Code: [Select]
http://s1.nebula.tproxy.de/ws/converterit should be replaced by (like in the online converter, that replaces http and https by ws):
Code: [Select]
ws://s1.nebula.tproxy.de/ws/converter
I will try the webhook tomorrow.

EDIT: i have no idea how to distribute a software package for OS X.
Have you ever tried building .deb and .rpm packages ?
« Last Edit: October 09, 2014, 08:10:11 am by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Yeah, I missed the ws protocol. Should be fixed now.

I'm not sure about OS X but you essentially have to distribute a "Knossos.app" folder which the user would copy to their Applications folder.
I've built deb and rpm packages before but it's been a while. I can write PKGBUILD files and will definitively add Knossos to Arch's AUR.

The windows installer and updater work now. I just have to add an update check to Knossos and the auto update feature will be done.

 

Offline pecenipicek

  • Roast Chicken
  • 211
  • Powered by copious amounts of coffee and nicotine
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • PeceniPicek's own deviantart page
Re: [Cross-platform] FS2 GOG/Mod installer
@pecenipicek: Are you asking about the hardware or software?
The server is a dedicated box. I'm running Apache, PHP (in FastCGI mode) and MySQL on it. The converter is written in Python and runs inside a uwsgi server.
HAProxy forwards all requests depending on the domain name (nebula.tproxy.de is forwarded to Apache and s1.nebula.tproxy.de is forwarded to uwsgi). It also handles idle connections to ease the load on the Apache server (similiar to what lingerd does).
Wouldnt nginx on its own be perfectly enough for all the "balancing"? also, if by FastCGI you dont mean PHP-FPM, you are in big trouble :p

(reading up on HAProxy, it looks like another one of those "ITS NEW SO IT MUST BE BETTER THAN ANYTHING THAT CAME BEFORE!" solution -.- i'm sure it does the job well, i'm just personally on the front of "keep the software stack as small as possible so you dont have to remember 14 different config conventions and at least 6 different 'languages'")


@Goob point well made :p
Skype: vrganjko
Ho, ho, ho, to the bottle I go
to heal my heart and drown my woe!
Rain may fall and wind may blow,
and many miles be still to go,
but under a tall tree I will lie!

The Apocalypse Project needs YOU! - recruiting info thread.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Wouldnt nginx on its own be perfectly enough for all the "balancing"? also, if by FastCGI you dont mean PHP-FPM, you are in big trouble :p

(reading up on HAProxy, it looks like another one of those "ITS NEW SO IT MUST BE BETTER THAN ANYTHING THAT CAME BEFORE!" solution -.-
Uh, HAProxy is pretty old actually... the first release was in 2001. I had to install it a while back because nginx didn't support WebSockets, yet. It does now but there's no point in replacing HAProxy with nginx.

And what's wrong with mod_fcgid ?  :wtf: