Author Topic: [Cross-platform] FS2 GOG/Mod installer  (Read 40209 times)

0 Members and 1 Guest are viewing this topic.

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
Quote
Since we aren't restricted by Goober's files anymore, I made semantic versioning a requirement for the "version" field. I suggest we use composer's syntax for versioned dependencies.
So you want to drop txt files conversion to json in fs2mod-py ? (the php server is far from ready, but It will get there eventually)

I've just read the new schema :

- I really like the new "dependencies", "environment" and "actions" handling. (single level lists of objects in json help me a lot on the database and form building side)

- I'd like to keep my object map as simple, and as close as possible to the json data structure, for obvious parsing reasons.

    *With your previous json schema, I was using file groups. File groups are custom objects that contain all files sharing the same mirrors, and these mirrors urls. For both database management and dynamic form building, I'm mapping a relation from multiple file groups to a single mod object.

    *With the new json schema, I'll have to map a relation from multiple file groups to both the mod object (because of the "files" string) and to a new type object (because of the "features" string). It would make my objects map a bit less readable, and data operations cascades (like removing orphan file groups on mod removal) a bit more complicated.

    *The goal of "required" "packages" was for me to completely drop the file group relation to the mod object, and replace it with a relation to the "package" instead.
Short version : having "files" at 2 different depth in the data structure is not ideal on my side. If the new json schema makes things easier on the python side, I'm ok with it, I will find a solution (either at the database level or parsing level).

    *Anyway, including additional/overridden "dependencies" in the "features" is a good idea. Will the fs2_open builds to install on the users system be specified through dependency (if the build is considered as a standard "mod", I guess) or environment  variables? I'm asking this because if it's set with environment, it could be useful to get "environment" inside "features" too.

- Now we nearly have a good dependency handling, isn't "submods" list going to be obsolete ?

EDIT : At first I wanted modders to manually enter md5 hashes into the mod building form. I know it can easily be automated in php, but it also looks heavily server dependent (reaching all kinds of limits : execution time, file size...) for big files like VPs. Thoughts ?
« Last Edit: August 28, 2014, 03:21:17 am by Hellzed »

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
EDIT : At first I wanted modders to manually enter md5 hashes into the mod building form. I know it can easily be automated in php, but it also looks heavily server dependent (reaching all kinds of limits : execution time, file size...) for big files like VPs. Thoughts ?
You could do it locally by using the JavaScript FileAPI and a JavaScript implementation of MD5 (for example this: https://github.com/satazor/SparkMD5).

    *The goal of "required" "packages" was for me to completely drop the file group relation to the mod object, and replace it with a relation to the "package" instead.
Short version : having "files" at 2 different depth in the data structure is not ideal on my side. If the new json schema makes things easier on the python side, I'm ok with it, I will find a solution (either at the database level or parsing level).
Dropping the file group and adding a "required" feature status seems like a good idea.

    *Anyway, including additional/overridden "dependencies" in the "features" is a good idea. Will the fs2_open builds to install on the users system be specified through dependency (if the build is considered as a standard "mod", I guess) or environment  variables? I'm asking this because if it's set with environment, it could be useful to get "environment" inside "features" too.
I would like to have the executable dependency handled by the "dependencies" array but moving the "environment" specification into the feature object sounds like a great idea. That way one version of the FSO executable "mod" could specify multiple features that would be selected based on the users environment.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
If you're generating the first version ("repo configuration"), you don't need to include the checksums. The second version ("generated configuration", I still need a better name for this. :blah:) can be generated with "python converter.py checksums".
You could simply host a "repo configuration" and I'll be running the converter every hour to update the central repo with your mod information and the appropriate checksums.

If you want to do it yourself, you can use some kind of queue and background processes.

Quote from: m!m
You could do it locally by using the JavaScript FileAPI and a JavaScript implementation of MD5
That only works if the user is uploading his files, not if he inputs links. You would also need unzip.js and similiar scripts to extract the archives.

Alright, I'll be removing the "files" property and include "required" packages with m!m's changes.


 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
That only works if the user is uploading his files, not if he inputs links. You would also need unzip.js and similiar scripts to extract the archives.
The FileAPI gives you access to the raw byte contents of the file without uploading it anywhere. I also don't understand why you would need to extract the archives to get the checksum of the file. All modern archive formats have some sort of archive integrity system in place to make sure archive corruption gets detected.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
I know about the File API. My point was that if the user doesn't have the file locally and pastes a link like http://mvp.fsmods.net/..., you won't be able to read the file using JavaScript.
The checksums of the archive's contents are necessary so that fs2mod-py can check the local files without downloading the archive. It's also a lot easier to use MD5 checksums for everything.

EDIT: I've updated the schema again.
@Hellzed: We still need the "submods" property to build a tree structure.
« Last Edit: August 28, 2014, 09:15:23 am by ngld »

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
The latest schema looks good to me. I guess everyone is ok with this version ? (if that's the case, I'll rebuild my database accordingly)

If you're generating the first version ("repo configuration"), you don't need to include the checksums. The second version ("generated configuration", I still need a better name for this. :blah:) can be generated with "python converter.py checksums".
You could simply host a "repo configuration" and I'll be running the converter every hour to update the central repo with your mod information and the appropriate checksums.

Regarding the checksums, I think I can leave the choice to the modders : manually enter all checksums, or leave it to our hosted version of fs2mod-py and wait for the next hourly refresh. From what I read about php and big files, that's the most reliable solution.

I'll add the option not to publish a mod after configuring the repository, as a way to save current work without propagating an unfinished mod/update to the users, and a few other mod management features.
With the new packager branch logic, will fs2mod-py still be able manually add a json source like it used to? (I'm asking because I can also add a "private repository" feature for modders who would want to test their mod deployment)

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
The latest schema looks good to me. I guess everyone is ok with this version ?
I'm not sure. You should probably wait for m!m's opinion.
@m!m: With the new schema it's still not possible to make the MediaVPs 3.6.12 and MediaVPs 2014 a single mod. I'm not sure if it's worth the extra effort. What do you think? FSO however, can be unified across platforms and CPU architectures. (This means that there will only be one FSO package the user can select.)

Regarding the checksums, I think I can leave the choice to the modders : manually enter all checksums, or leave it to our hosted version of fs2mod-py and wait for the next hourly refresh. From what I read about php and big files, that's the most reliable solution.
So... are you generating two json files? I'm asking because I want to add your database to the central repo.

I'll add the option not to publish a mod after configuring the repository, as a way to save current work without propagating an unfinished mod/update to the users, and a few other mod management features.
With the new packager branch logic, will fs2mod-py still be able manually add a json source like it used to? (I'm asking because I can also add a "private repository" feature for modders who would want to test their mod deployment)
The users will still be able to add additional repos but I'll drop the support for fs2mod archives (the format you invented for the original bash script).

Will you be releasing the source code to your project? I'd be interested.

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
I think the schema is pretty good now. Do you intend to write a parser for this format in python? If that is the case can you bundle it in a library so I can use that in the launcher I am writing?

I'm not sure. You should probably wait for m!m's opinion.
@m!m: With the new schema it's still not possible to make the MediaVPs 3.6.12 and MediaVPs 2014 a single mod. I'm not sure if it's worth the extra effort. What do you think? FSO however, can be unified across platforms and CPU architectures. (This means that there will only be one FSO package the user can select.)
Yeah, one "mod" definition for multiple versions is not possible but having the MediaVPs share an ID (for example "mediavps") is possible so depending on the newest version of the MediaVPs is as easy as specifying the dependency "mediavps". Basically you would have multiple mod definitions with different versions.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
I think the schema is pretty good now. Do you intend to write a parser for this format in python? If that is the case can you bundle it in a library so I can use that in the launcher I am writing?
Yes, I'll write a parser in python for this. What kind of library do you mean? A .dll file?

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
Yes, I'll write a parser in python for this. What kind of library do you mean? A .dll file?
No just the python source code that I could then execute using IronPython.

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
Regarding the checksums, I think I can leave the choice to the modders : manually enter all checksums, or leave it to our hosted version of fs2mod-py and wait for the next hourly refresh. From what I read about php and big files, that's the most reliable solution.
So... are you generating two json files? I'm asking because I want to add your database to the central repo.
In the process I was describing, my script would generate 2 json files. If the modder enters all checksums manually, a full "generated configuration" can be generated. If the modder doesn't, only a "repository configuration" file is generated. I thought your server could grab the list of repo configs and finish the job.
(I'll think of better server interactions and hosting policy when I'm done with dynamic mod forms. I guess you'll have better ideas in the meanwhile)


I'm waiting for the project to be a bit more advanced, but yes, everything will be released. :)
I've known php for years, but I'm still learning how to use symfony framework. The only thing I've done with it yet is a blog with the help of the official walkthrough, and this project uses a slightly different set of features, so that's a lot of documentation reading while I'm coding.

I think the schema is pretty good now. Do you intend to write a parser for this format in python? If that is the case can you bundle it in a library so I can use that in the launcher I am writing?

@m!m: With the new schema it's still not possible to make the MediaVPs 3.6.12 and MediaVPs 2014 a single mod. I'm not sure if it's worth the extra effort. What do you think? FSO however, can be unified across platforms and CPU architectures. (This means that there will only be one FSO package the user can select.)
Yeah, one "mod" definition for multiple versions is not possible but having the MediaVPs share an ID (for example "mediavps") is possible so depending on the newest version of the MediaVPs is as easy as specifying the dependency "mediavps". Basically you would have multiple mod definitions with different versions.
This is exactly what I plan to do inside the database. By default, you'll always get the latest version of a mod, but nothing stops you from requesting an older version if it's still in the db.
www.example.com/nebula/all.json // you get the full public repository
www.example.com/nebula/mod/<ModID>.json // you get <ModID>'s latest version repository
www.example.com/nebula/mod/<ModID>.<ModVersion>.json //  you get <ModID>'s <ModVersion> repository


 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
I think we can still improve the dependency handling: Right now a mod's packages can depend on other mods but not on specific packages (or "features"). I suggest we add a "package" option to the dependency object so that it looks like this:
Code: [Select]
{
    "id": "<ID>",
    "packages": [],
    "version": "1.2.*" // https://getcomposer.org/doc/01-basic-usage.md#package-versions
}
By default, a mod depends on another mod's required packages but if it requires specific packages, those can be specified in the packages option.

@m!m: I'll either put the parser code in a single file or an external module that you can import in IronPython.

Quote from: Hellzed
I thought your server could grab the list of repo configs and finish the job.
That was actually my intention.  ;)

Quote from: Hellzed
I'm waiting for the project to be a bit more advanced, but yes, everything will be released. :)
Thanks!

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
If I get it right, a "package" could depend on another one in the same mod ? Does it use the full current mod id, or a special keyword in that case (like "CURRENT_MOD") ?
("optional" and "recommended" packages usually only depend on the "required" packages, but are independent from each others, I know. But I've seen this type of dependency used for fsport hi-res command briefings. fsport hi-res command briefings are optional (and should be recommended), and "higher"-res  CBs are depending on hi-res CBs, not on mod core files...)

EDIT : @ngld : do you want me to store mod logos inside the db, as remote a remote ressources for fs2mod-py (just in case you need it for mods that are not installed) ?
« Last Edit: August 28, 2014, 03:53:56 pm by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
I was actually thinking of MV_Advanced which requires MV_Assets and MV_Effects. Both are otherwise optional (but should be recommended).

This feature should also allow the same for different mods: Blue Planet's bp2-adv should depend on bp-adv but BP:AoA and BP:WiH are different mods.

EDIT: @Hellzed: You can make that optional. Just as the schema says, the logos are extracted and hosted by my server whenever they aren't linked (i.e. the "logo" property is null).
« Last Edit: August 28, 2014, 04:15:12 pm by ngld »

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
Should I get ready to support more than one hashing types ? (SHA ?)
I'm not asking for it as a feature, but I was thinking maybe someone would need it, even us if someday we change default hashing type. I'm ok if we stick with MD5 as only hashing type.

@ngld :
I've decided not to implement the possibility to manually enter checksums, to avoid human error, and always use fs2mod-py. Maybe fs2mod-py could output some very simple log file during the hashing process ? Somethinglike " [timestamp] action done/failed", as hashing progresses ?
After the mod form is submitted, php would call fs2mod-py for current mod, and redirect the modder to a page showing the fs2mod-py log file as it updates (with ajax requests). When it's over, "generated" json response is stored in the database, mod is flagged as "ready", and final json download links are generated.
This would also avoid using a json parser in php at each end user request, as final json is cached in the db for a single revision of a mod.
db gets a lot bigger (we create a new entry for every new revision of a mod, and the cached final json nearly doubles the size of any entry). I could refine the logic a bit, so that patch entries would replace a previous patch, and we only create a new entry on minor/major revisions, but I'm not sure it's worth the extra work. (EDIT : this was bad idea : acocrding to http://semver.org : " 3. Once a versioned package has been released, the contents of that version MUST NOT be modified. Any modifications MUST be released as a new version.")
Is that close to how you wanted to integrate db and central repo ?
« Last Edit: August 29, 2014, 07:24:56 am by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Should I get ready to support more than one hashing types ? (SHA ?)
I'm not asking for it as a feature, but I was thinking maybe someone would need it, even us if someday we change default hashing type. I'm ok if we stick with MD5 as only hashing type.
MD5 should work well enough because we aren't hashing passwords but checking file integrity.

@ngld :
I've decided not to implement the possibility to manually enter checksums, to avoid human error, and always use fs2mod-py. Maybe fs2mod-py could output some very simple log file during the hashing process ? Somethinglike " [timestamp] action done/failed", as hashing progresses ?
The converter.py script already does that.

After the mod form is submitted, php would call fs2mod-py for current mod, and redirect the modder to a page showing the fs2mod-py log file as it updates (with ajax requests).
So you're going to run fs2mod-py on your server?

Is that close to how you wanted to integrate db and central repo ?
I wanted to put your all.json in my "includes" list but your solution is better. This way modders can get immediate feedback whether the download links work.

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
So you're going to run fs2mod-py on your server?
Yes. I'll add a way to track "failed" mod versions that didn't pass the file availability test.
Basically, modders will have a view listing all versions for the mod, including published versions, private versions, failed versions and working copies. Only working copies can be edited, according to semver recommendation.
Default new mod version will be created from the lastest available data stored in the db (public, private, failed or working copy).
Ideally, a new mod version could also be created (as a working copy) from any other version, and maybe other mods (so it would be some sort of forking mechanism).

I'll start to implement all these database operations (they're actually pretty easy, that's only multi-field sorting and entries copy) once the base template for mod dynamic form is working as intended. I'm struggling to get JS to do exactly what I want with symfony generated html (handling multiple levels of subsections, while being able to add new subsections), but I'll get there eventually. When it's done, live testing of database operations will get much easier.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Just a quick update: I've finished the new implementation in fs2mod-py. I haven't had a chance to extensively test it, yet. It might destroy your FS2 installation. </disclaimer>
For now FSO and the MediaVPs 2014 are in the repo. Here's the repo configuration and here's the file fs2mod-py uses.

I'll finish the package for m!m next.
Updated TODO list:
  • Finish standalone repo parser
  • Expand repo
  • Test new repo logic (install/uninstall) and look for possible problems/bugs
  • Improve the UI
  • Get fs2mod-py into PyPi. Installing will then be as simple as "pip install fs2mod-py".
  • New public release (1.0.0)
  • Add an auto-update feature (?)
  • Check and improve OS X support

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
I'll give it a try soon. I always have multiple FS2 installations on my hard drive ;)

I've also made good progress today (finished working on javascript for the mod form, now I can feed it any Symfony form, and it will be able to recursively add/delete nested form elements, mirroring what Sf database expects.)

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
For now FSO and the MediaVPs 2014 are in the repo. Here's the repo configuration and here's the file fs2mod-py uses.
I see you are still using the old installer like "files" syntax. Is this going to change at a later point?