Author Topic: [Cross-platform] FS2 GOG/Mod installer  (Read 72018 times)

0 Members and 1 Guest are viewing this topic.

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
Quote from: ngld
@Hellzed: Whenever I think of emebedding your nebula web interface in fs2mod-py, I think of Steam. I'd like to do something similiar. (Not a simplified UI but the full web interface with Install / Play / Settings buttons.)
Providing the same level of functionality in the client and on the website would be good indeed. I'm just worried the client would feel a bit cluttered if it features the whole website UI without any adaptations. I kinda liked the "lighter" feeling of a smaller window retaining the "launcher" spirit, where the essential information (mods that are installed and ready to play) is immediately available.
Compared to the full website, the only things missing from the simple UI would be the advanced mod search and the current highlights (this one can be added easily).
I'll think again about it, my mind is not set yet.

Anyway the more I think about, the more I like the idea of using JS API.

Today has been a good day :
- The base logic for modder teams is working, with single user and teams both being considered valid owners for a mod. (This feature will be disabled on first release because I'd have to write a simple team control panel, and this is not the current priority).
- I have made mod/branch/build removal safer, with better confirmation pages.
- Some small fixes went into the mod/branch forms.
- Most secondary objects have been re-added to the database structure. I still have to do the "package" objects, though.

What's left to do :
- Re-add packages, and fix their subforms.
- Render fs2mod-py usable JSON.
- Make registration invite-only.
« Last Edit: September 21, 2014, 07:29:20 pm by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
I'm trying to get rid of the FS2 tab which looks kind of awkward IMO. That leaves us with three tabs: The mod tree, FSO settings and fs2mod-py settings.
I'll add a "nebula / steam" mode which opens the web view by default. That mode will have three buttons above the web view, one for each tab. If you click on one of those buttons, the current window is opened and the appropriate tab is selected.
If you use the "traditional" mode, it will open the current window and offer a button to open the nebula window. The mode can be changed in fs2mod-py's settings at any time.
I'll keep the "simple mod list" in both modes and make it adapt to the different screen sizes.

What's left to do :
- Re-add packages, and fix their subforms.
- Render fs2mod-py usable JSON.
- Make registration invite-only.

Yay! Any idea on when you're going to release it?  :D

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
Really soon ! :p

Packages (and package sub-objects) are now correctly attached to their parent build. But forms are a mess with no validation (except if you consider "qsfdcqdqzddsc" being a valid server mirror). I'm currently writing some validation rules. Next, JS fixes !

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
I am also making good progress with my implementation. Downloading and installing a mod works fine but now I need to implement the handling of installed mods and I would also like to add some JSON caching.

Just to make sure we don't do too much duplicated work, what does fs2mod-py actually do? Is it a full FSO launcher or is it just the package installer?

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
@m!m : Both a package installer and a launcher !

@ngld : here's a small change request, could we change this :
Code: [Select]
"environment": [ // optional
{
"type": "cpu_feature", // required
"feature": "<SSE|SSE2|AVX|...>" // required if type == "cpu_feature"
},
{
"type": "os", // required
"os": "<windows|linux|macos>" // required if type == "os"
}
into this :
Code: [Select]
"environment": [ // optional
{
"type": "cpu_feature", // required
"value": "<SSE|SSE2|AVX|...>" // required
{
"type": "os", // required
"value": "<windows|linux|macos>" // required
}

To me, these environment vars are always type:value pairs, so it makes sense to always store the value as the same property (value) instead of having 2 properties (os and feature).

Also, I think the "dependencies" field is missing the "packages" list. (your post http://www.hard-light.net/forums/index.php?topic=86364.msg1761165#msg1761165 )

(And yes, If I'm asking these questions, that means I'm overhauling the last bits of the old storage logic)
« Last Edit: September 23, 2014, 07:31:59 am by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
@m!m: It's a full FSO launcher and package installer. It can even install FS2 from GOG's installer (it's the same method Goober's installer uses because Hellzed developed that feature).
You can change FSO's settings and cmdline flags. fs2mod-py saves a global ("default") cmdline which every mod uses by default and can be changed by the user. You can also set cmdline flags for a specific mod, then that cmdline will be used instead of the global default.
It can reliably detect installed mods using the repo information and by checking the files in a user's FS2 folder.
Are you planning to implement somehting like that, too? It's really useful if you already have a big FS2 installation and you're using fs2mod-py for the first time.

@Hellzed: Sure. Any objections, m!m?

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
@m!m: It's a full FSO launcher and package installer. It can even install FS2 from GOG's installer (it's the same method Goober's installer uses because Hellzed developed that feature).
You can change FSO's settings and cmdline flags. fs2mod-py saves a global ("default") cmdline which every mod uses by default and can be changed by the user. You can also set cmdline flags for a specific mod, then that cmdline will be used instead of the global default.
It can reliably detect installed mods using the repo information and by checking the files in a user's FS2 folder.
Are you planning to implement somehting like that, too? It's really useful if you already have a big FS2 installation and you're using fs2mod-py for the first time.
I am doing the smae thing. Currently is is a standard launcher with multiple profiles and support for multiple game roots. I want to implement a new type of mod management using this package management.
My plan is to not install mods like they are installed now. Instead my installer will install them into subfolders according to their Ids and version. By doing that mod.ini files are not needed anymore and having two mods that depend on different versions of the same mod doesn't cause major problems anymore.

@Hellzed: Sure. Any objections, m!m?
No, I will have to change my enum handling but that should simplify my JSON classes.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Also, I think the "dependencies" field is missing the "packages" list. (your post http://www.hard-light.net/forums/index.php?topic=86364.msg1761165#msg1761165 )
Nice catch! Fixed it.

I added some comments to the paths in  the schema ("folder" for mods, "dest" for files, "paths" and "dest" for actions).

@m!m: Is this fine or does it break your launcher?
As far as I understood it, you just have to force the mod's "folder" property to "mod/<id>/<version>" and get exactly the paths you wanted.

@Hellzed: I hope I didn't break anything in the Nebula with this change.  :nervous:

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
Actually, your comments match exactly the behaviour I expected from fs2mod-py, except for the mod "folder" defaulting to mod title. Shouldn't it be the mod id instead ?
As I understand it, the mod id is a url friendly canonical string (no spaces, no special chars...), while mod title is a full utf-8 compatible string. For example, for a mod with title "Blue Planet : Age of Aquarius", id is "blueplanet".

EDIT : A few months ago, someone said fs2mod-py was not a "catchy" name. We're all used to it now, but we must admit it's true :p
The "Nebula" is a big reference to FreeSpace 2, as this is where the campaign really takes off, and has a "cloud" feeling, as this is a server side software... What do you think about "Nereid", a tribute to the GTD Nereid (the GTD Orion that should have ended the war in Capella), or "Nephthys" (Vasudan support ship) ?
I think Nebula/Nereid would convey a sense of integration between both tools...
« Last Edit: September 23, 2014, 10:23:37 am by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Well, the mod ID can be pretty much anything and isn't shown to the user. It's main purpose is to uniquely identify a mod (in my first version it was supposed to be a UUID).
I'll add your explanation to the schema and change the comment for the mod folder.

EDIT: @Hellzed: "Knossos" would also fit since it's the portal that leads into the nebula. Nereid sounds nicer, though, and ties better with Nebula.

@m!m: You know that your path change will most likely break every single mod that uses cf.something functions in its LUA scripts, don't you?
« Last Edit: September 23, 2014, 04:26:37 pm by ngld »

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
Small things left to do :
- last bits of form validation (mostly matching relative paths)
- recycle the old form theme, add it to the new forms (biggest chunk left actually).
- clean the JSON builder, make sure Nebula doesn't send NULLs to fs2mod-py (Nereid ? Knossos sounded cool too)
- copy-paste the "invite only" script into the security bundle.
- add licensing, write a quick start configuration guide (there's a fair bit of configuration so that NebulaBundle and FOSUB are registered by Symfony).
- run the converter from php to generate final json, and cache the result

Then, it goes straight to github ! Probably on tuesday. We should have a server running this week-end.

I'll have plenty of room for improvement :
- public activity tracking, for modders, teams, mods, branches...
- show more info about the objects (fill the "details" tabs)
- fix the object counts (always showing how many mods, branches, builds a modder has)
- write Nebula first level pages, like "Getting started", and add the modders, mods and activity common public lists.
- optimise DB queries
- activate modding teams
- start working on the "public" part of the website (mod public pages, default branch and build selection).

[attachment kidnapped by pirates]

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
@ngld : how strict are we regarding the relative paths ("folder", "dest"...) ? what about trailing slashes ? (I know an initial slash would be totally wrong for a relative path, but what happens if fs2mod-py gets one ? Should Nebula quietly remove these slashes while saving data, or fail to validate and give a warning ?)

I also figured that changing a mod/branch unique IDs would break dependencies in cached generated files, so I'll lock this field on mod/branch edit and add a warning on mod/branch creation.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Trailing slashes actually don't matter. You can strip them if you want. You should strip the leading slash (if there is one) because it makes no sense (just as you said). A warning shouldn't be necessary since it's pretty obvious that this is a relative path and that the software is going to interpret it like one.
I never did a security check on my repo parser. Right now, fs2mod-py installs all mods with a leading slash in their "dest" path in the root of your harddrive (i.e. C:\MediaVPs_2014).  :banghead: I'll fix this right away and see if I can find similar problems.

BTW, I merged the packager branch back into the develop branch and will push all new commits on that branch. The name Nereid is already used by a webframework for python so I guess Knossos would be a better choice...

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
To anyone wondering what happened to the "first release on tuesday", I noticed in the last tests that if the form was resent after failing to validate, the small piece of javascript powering the dynamic form would go completely mad (overwriting form fields "name" and "id" properties, compromising final mod integrity)...
Button placement has always been ugly with this script anyway.

So I had to rewrite it from scratch to handle more things on php side, and play better with form themes...
Good news : now, it works !

Last things to do : change the registration setting to "invitation only", licensing, uploading to github.

Edit : I followed Symfony guide instructions to upload the code to github, but i noticed some of the most used bundles follow different guidelines (they ship less bits of Symfony kernel).
https://github.com/Nepenthes/hlp-nebula
« Last Edit: September 26, 2014, 07:12:40 pm by Hellzed »

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
@ngld (there's no hurry to answer, I have a lot of other things to add to Nebula) :
What is the current state of converter.py ? (I read it has been updated 2 days ago)
When I try to run it,  I get :
Code: [Select]
INFO:Thread-4:util.download: Downloading "http://www.mediafire.com/?zmh14h7o2122
 444"...
Unhandled exception in thread started by
                                         Unhandled exception in thread started by Unhandled exception in thread started by
                                           Unhandled exception in thread started by

    Unhandled exception in thread started by


The file was :
Code: [Select]
{
    "mods":  [
        {
            "id": "bta-demo-copy",
            "title": "Between the Ashes : Mefistofele (copy)",
            "version": "1.1.0-demo+nebula_test",
            "description": "The BtA Team is proud to release this small 2 mission demo. Consider it more like a prequel to BtA: Slaves of Chaos. In Mefistofele, you'll fly with local criminals and smugglers as they seek to make a simple profit, but they might have gotten in over their heads this time.",
            "notes": "MOD NOTES :\\nThis is a copy of Between the Ashes : Mefistofele, a mod by Mjn.Mixael.\\n\\nBRANCH NOTES :\\nThis is a stable branch.\\n\\nBUILD NOTES :\\nLet's register a build to test the database !",
            "folder": "BtA Demo",
            "packages": [
                {
                    "name": "BaseInstall",
                    "notes": "Packages are the base elements of your mod. They can be \"required\", \"recommended\" or \"optional\".\r\nThis is one \"BaseInstall\" the bare minimum to run this mod.\r\n\r\n- \"Client requirements\" are designed to distribute builds of fs2_open : our client knows about the OS, architecture and instruction sets !\r\n- \"Dependencies\" can be other mods, other mod packages, and other packages inside the same mod.\r\n- \"Files\" are the archives the client will download to install your mod. They can have multiple mirrors.\r\n\r\nAll checksums will be calculated by a server side script. (but we don't do full hosting (yet?))",
                    "status": "required",
                    "dependencies": [
                        {
                            "id": "mediavps_3612",
                            "version": ">=3.6.12",
                            "packages": [
                                "core"
                            ]
                        }
                    ],
                    "files": [
                        {
                            "filename": "BTA_DEMO_Root.7z",
                            "is_archive": true,
                            "urls": [
                                "http://www.mediafire.com/?sxayb4bhxkd22vf",
                                "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_Root.7z"
                            ]
                        },
                        {
                            "filename": "BTA_DEMO_Assets1.7z",
                            "is_archive": true,
                            "urls": [
                                "http://www.mediafire.com/?zmh14h7o2122444",
                                "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_Assets1.7z"
                            ]
                        },
                        {
                            "filename": "BTA_DEMO_Assets2.7z",
                            "is_archive": true,
                            "urls": [
                                "http://www.mediafire.com/?sfxkg4zqccdod6i",
                                "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_Assets2.7z"
                            ]
                        },
                        {
                            "filename": "BTA_DEMO_Patch_1_1.vp",
                            "is_archive": false,
                            "urls": [
                                "http://www.mediafire.com/download/cm0uu5qt9fww3n4/BTA_DEMO_Patch_1_1.vp",
                                "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_Patch_1_1.vp"
                            ]
                        }
                    ]
                },
                {
                    "name": "LowEnd",
                    "notes": "Use this package if you have a low end computer.",
                    "status": "optional",
                    "files": [
                        {
                            "filename": "BTA_DEMO_00LowEnd.7z",
                            "is_archive": true,
                            "urls": [
                                "http://www.mediafire.com/?7rsur95dvv6n07e",
                                "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_00LowEnd.7z"
                            ]
                        }
                    ]
                }
            ]
        }
    ]
}

Any idea what's going on ?


EDIT 27/09 : I've just finished implementing build cloning (php has a useful "clone" keyword to trigger specific code on object copy, but it doesn't apply to sub-objects).
So this was mostly a matter adding a __clone() method to the build object and its sub-objects, to make sure the whole tree gets cloned and we don't accidentally update sub-objects that belong to the original object (that should stay unchanged).

It will be useful in two cases :
- when modders push a small update and they want to keep content from a previous build, they can now click "Copy and update" on any existing build.
- there will be a possibility to copy a build from one branch to another. Lets say the last release candidate in the unstable branch was good enough, it can be copied to the stable branch under a stable version number.

It also helps enforcing semver since a published build can never be edited. Instead, clones are edited, and then published under a new version number.

EDIT 28/09 : I will definitely clear the current github repository, and upload everything again in a better way (easier to distribute/install with composer).
Now, mods/branches/builds counters work fine. I also have an early version of dependency autocompletion for mods.

EDIT 29/09 : Small changes in forms :
- added missing server side validation constraints and slash trimming (mostly on path lists)
- added more client side validation, for HTML5 compatible browsers.
- mod ID and branch ID are now locked after mod or branch creation (no more risk of breaking deps.)
- mod unique ID has now to be unique *across the whole mod table* instead of *for a single owner*, for easier administration of public mod access
- more help text in the build form.
- autocompletion on dependency mod id.

@ngld : what is the exact syntax of the "environment" item ? I don't see the architecture flag, only "os" and "cpu_feature". If the syntax is strict and small enough, i can handle that with dynamic lists in the form.
(I've just seen the converter fix on github ! thanks !)
« Last Edit: September 29, 2014, 05:12:24 pm by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
I'm not sure how you ran the converter but somehow your PHP script didn't capture the whole output (the stacktrace is missing).
What do you think about this thing?  :D
During my tests I found two errors:
  • You used mediafire.com links. Knossos needs direct links (i.e. if you access them with your browser, it will prompt you to save the file). I'll add proper warnings for this case.
  • There was a bug in my JSON generation code, it's fixed now.

Your progress sounds great! I had a look at the PHP code in your GitHub repo but I didn't have a chance to test it (yet).
However there are two things I want to add: Real-time display of the converter's progress (see the link above) and uploading. Since direct-link hosting (for free) is rare, I want to allow modders to upload the files directly to the Nebula.

Quote from: Hellzed
@ngld : what is the exact syntax of the "environment" item ? I don't see the architecture flag, only "os" and "cpu_feature". If the syntax is strict and small enough, i can handle that with dynamic lists in the form.

Here's the relavant section from the schema.txt:
Code: [Select]
"environment": [ // optional
    {
        "type": "cpu_feature", // required
        "value": "<SSE|SSE2|AVX|...>" // required
    },
    {
        "type": "os", // required
        "value": "<windows|linux|macos>" // required
    },
    ...
],

There's no architecture flag, yet, but I can add it (I'm assuming you mean x86_64 / x86_32). I'm also thinking about extending this. In the current version you only see the builds which fit your platform. However, if you're using Windows and have a processor which supports SSE2, you'll see both the SSE and SSE2 builds. If we extend the logic for the environment section (i.e. add a "invert": true property), we can fix that.

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
I'm not sure how you ran the converter but somehow your PHP script didn't capture the whole output (the stacktrace is missing).
I actually ran it from the command line, and it became unresponsive. No way to get the output past that point. I haven't tried the new version yet.

What do you think about this thing?  :D
During my tests I found two errors:
  • You used mediafire.com links. Knossos needs direct links (i.e. if you access them with your browser, it will prompt you to save the file). I'll add proper warnings for this case.
  • There was a bug in my JSON generation code, it's fixed now.

I currently can't do a full test of the online converter, but I did a quick test :
- first one failed because my json was corrupted by an annoying bug in Nebula (when there's only one url for a file, the "urls" field becomes an object with a single key named "1" instead of a list...)
- second one went better, with actual md5 generation.

Code: [Select]
INFO:MainThread:__init__.main: Parsing repo...
INFO:MainThread:__init__.main: Updating checksums...
INFO:Thread-4:util.download: Downloading "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_Assets2.7z"...
INFO:Thread-2:util.download: Downloading "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_Assets1.7z"...
INFO:Thread-3:util.download: Downloading "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_Patch_1_1.vp"...
INFO:Thread-1:util.download: Downloading "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_Root.7z"...
INFO:Thread-5:util.download: Downloading "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_00LowEnd.7z"...
ERROR:Thread-3:util.download: Failed to load "http://www.lunardigitalproductions.com/bta/downloads/demo/BTA_DEMO_Patch_1_1.vp"!
Traceback (most recent call last):
  File "./knossos/lib/util.py", line 272, in download
    result = urlopen(Request(link, headers=headers))
  File "/usr/lib/python3.4/urllib/request.py", line 153, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.4/urllib/request.py", line 461, in open
    response = meth(req, response)
  File "/usr/lib/python3.4/urllib/request.py", line 571, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.4/urllib/request.py", line 499, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 433, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 579, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

BTA_DEMO_Patch_1_1.vp actually gives a 404.

In both case, it didn't really differentiate success from failure. Is it possible to show less python ouptput, and better error messages in case of failure, at the end of the script ?
(and make the application that called the script aware of what happened ? Build objects in Nebula have two boolean properties : isReady and isFailed. Both are false when a new build is created, and the build is marked as "validating" in the UI. I'm planning to use converter return value/output to change these vars to the right state.)
Anyway, awesome work ! Can't wait to see this page integrated into Nebula !

Your progress sounds great! I had a look at the PHP code in your GitHub repo but I didn't have a chance to test it (yet).
However there are two things I want to add: Real-time display of the converter's progress (see the link above) and uploading. Since direct-link hosting (for free) is rare, I want to allow modders to upload the files directly to the Nebula.
I was thinking about uploading too, but it will add some complexity (I can take a "snapshot" of a build to enforce semver and duplicate a few text entries, but doing so with files would be terribly inefficient. I have some ideas to optimise it, though.)
We should focus on getting a first version online without the file upload system, then it will become my #1 priority.
I was planning to use screenshots and background images uploads as a benchmark, to know exactly what to expect with file uploads.


Quote from: Hellzed
@ngld : what is the exact syntax of the "environment" item ? I don't see the architecture flag, only "os" and "cpu_feature". If the syntax is strict and small enough, i can handle that with dynamic lists in the form.

Here's the relavant section from the schema.txt:
Code: [Select]
"environment": [ // optional
    {
        "type": "cpu_feature", // required
        "value": "<SSE|SSE2|AVX|...>" // required
    },
    {
        "type": "os", // required
        "value": "<windows|linux|macos>" // required
    },
    ...
],

There's no architecture flag, yet, but I can add it (I'm assuming you mean x86_64 / x86_32). I'm also thinking about extending this. In the current version you only see the builds which fit your platform. However, if you're using Windows and have a processor which supports SSE2, you'll see both the SSE and SSE2 builds. If we extend the logic for the environment section (i.e. add a "invert": true property), we can fix that.
Good ideas indeed. I'll adapt the forms when you make the changes in Knossos.


EDIT : THAT FEELS GOOD !!!
(JSON corruption bug fixed. Not an easy one :
- finding the pattern : corruption only happened if a value was deleted from a list in the form (files urls, actions paths, dependency packages...) AND this value was *not the last* in the list AND the build was a clone from a previous build.

- what actually happened : cloning was causing items stored in lists (files urls, action paths, dependency packages) to keep their "keys" in the php arrays.
Then, when lists were stored as MySQL arrays in the database. When all keys were numbers following each other, beginning at 0, MySQL would detect a simple ("value", "value"...) array. When keys were not following each others or not starting at 0, MySQL would detect a ("key"=>"value" , "key => "value"...) arrray.
All of this would go undetected unless you are watching every entry of the database.

When it all arrives at the JSON parser, ("value", "value"...) arrays become JSON arrays.  ("key"=>"value" , "key => "value"...) become objects.
Hence the {"1": "url"} issue, instead of ["url"].

Fix was actually simple : use array_values() on lists I get from the form, to ensure arrays stay simple ("value") arrays to MySQL.
« Last Edit: September 29, 2014, 08:27:13 pm by Hellzed »

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
I cloned your repository and installed the composer dependencies but I have been unable to properly test it.
What are the steps to get it to run? I have no experience with Symfony so that might be an issue.

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
Yes, there's a bunch of things to do to get it running. You don't really want to run this version, though, because I'll do a full upload again. Instead of shipping a full Symfony project, I'd like to ship the HLP/NebulaBundle/* part only, and add a small install script. I've also changed A LOT of things in the last 3 days.

On your local install, you should make sure that :
- you rename "app/config/parameters.yml.dist" to "app/config/parameters.yml"
- inside "app/config/parameters.yml" change these options to match your local mysql install:
Code: [Select]
database_host: 127.0.0.1
    database_port: null
    database_name: symfony
    database_user: root
    database_password: yourpassword
- from Symfony folder, run :
Code: [Select]
sudo chmod 777 -R app/cache
sudo chmod 777 -R app/logs
php app/console doctrine:database:create
php app/console doctrine:schema:update --force

If Symfony folder is at the root of your local web server, you can access pages from http://localhost/Symfony/web/app_dev.php/nebula/ .
Register as a new user from http://localhost/Symfony/web/app_dev.php/nebula/register/
Once you are connected, the main pages (get started, mods, modders...) are empty, but you can access your own mod repository with a link on the right side of the nav bar.
(no theme on FoSUserBundle pages yet, sorry !)

I can't remember of anything else. If you used composer to get the deps, it should run.
If it doesn't, that means some yml config file didn't make it to github :p

If you want to start reading the code, you should know a few things :
- src/HLP/NebulaBundle holds most Nebula source code
- Resources/config/routing.yml lists all Nebula urls
- Resources/views contains the html.twig templates
- Controller contains the php controllers : they get variables from the router ({variable} in routing.yml), and return a response (generated html view, json response, redirection...)
- Request contains the ParamConverters. these classes handle variables from the route just before Controllers can use them. They make sure routing variables have a corresponding database entry.
- JSONBuilder is self explanatory :p
- Entity contains entities and repositories : every object  specific to Nebula is in there. Repositories hold custom database queries.
- Forms are not actual forms, but php form builders : Symfony call these classes to build forms with my form templates.
- Security/Authorization/ModVoter is a class that makes sure you own the page. I call it directly from the templates to show "new/edit/buttons" buttons, and from the controllers before generating forms.
- Good to know : the method to create a new branch from scratch is actually in the *mod* controller. It's logical if you think about it :) (same goes for every sub-object)

There are a few things I don't like in the current code :
- pagination, because it loads all entities from the database, and only display the right slice. I had to chose between efficient object preloading in the ParamConverter and efficient pagination. This is a limitation of Doctrine2 (the database abstraction layer above MySQL), but can be fixed by writing some complex native MySQL query. I'll do it at some point.
- ParamConverters sometimes preload too much stuff to avoid making more db queries (loading subobject lists when not needed). That's a small optimization, and not my current priority.
- The ModVoter only checks if you own the repository, and gives you the authorization to add objects. Templates use this "add" authorization even to show "delete" and "edit" buttons. I'll add a bit more granularity to this logic, and make the authorization name clearer.

@ngld  : I'd like to take a look at the server side of the online converter. Could you send me the files ?
« Last Edit: September 30, 2014, 04:11:32 pm by Hellzed »

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
@ngld : Any idea on what could cause this : (i'm trying to run converter.py from the command line to prepare the generated json storage. I must definitely do something wrong, because if I copy and paste Nebula's output to the online converter, it works.)
Code: [Select]
python converter.py checksums repo.json generated.json
Code: [Select]
{
    "mods": [
        {
            "id": "fs2_open",
            "title": "FreeSpace Open engine",
            "version": "3.7.2-rc4",
            "packages": [
                {
                    "name": "linux32",
                    "status": "required",
                    "files": [
                        {
                            "filename": "fs2_open_3.7.2_RC4.tar.bz2",
                            "is_archive": true,
                            "urls": [
                                "http://swc.fs2downloads.com/builds/LINUX/fs2_open_3.7.2_RC4.tar.bz2"
                            ]
                        }
                    ]
                }
            ]
        }
    ]
}
Code: [Select]
INFO:MainThread:__init__.main: Parsing repo...
INFO:MainThread:__init__.main: Updating checksums...
Traceback (most recent call last):
  File "./converter.py", line 26, in <module>
    converter.main(sys.argv[1:])
  File "/home/loris/knossos-develop/converter/__init__.py", line 238, in main
    prg_wrap(core)
TypeError: 'bool' object is not callable

By the way, it would be nice if the converter could do without wrapping the data inside a "mods" tag. Server side, it will only run on one build at a time.
This way, i could do the whole repository wrapping in Nebula, on demand, depending on the url accessed by the client (that's the easiest way to implement urls that reference the "default" branch and "current" build, like /nebula/modder/mod/default/current.json).
Also I'd avoid manipulating json inside Nebula, only storing it.

I'll also have to check what's the best way to grab the json output and store it into the database.
« Last Edit: October 02, 2014, 01:07:42 pm by Hellzed »