Author Topic: [Cross-platform] FS2 GOG/Mod installer  (Read 72019 times)

0 Members and 1 Guest are viewing this topic.

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
I went for the current syntax when I rebuilt the database, but it can be changed (it's just a matter of automated code generation).
Good news : tonight, for the first time, a dummy mod (with its full object structure and metadata) was successfully registered from the dynamic form. :)



My current to-do list is now :
- Double check object relations to make sure the database stays clean (and is not broken) after mod removal,
- Validate user input,
- Add a JSON parser and a way to get raw JSON output and feed it to fs2mod-py,
- Add a Bootstrap theme to the dynamic form (it's currently hardly readable with 3 to 4 embedded form levels),
- Install a "user bundle" (probably FoSUserBundle) to manage multiple users and access control lists,
- Optimise DB queries,
- Fix the url rewriting.

When it's done, I'll upload the code to github.

After that, I'll keep working, not necessarily in that order :
- Integrate with the standalone repo parser,
- Refine mod administration tools,
- Add the public pages (with generated fso:// links) and an actually good looking theme (simple Bootstrap for the modders side, something more elaborate for the public side),
- Add cool features like auto-completion (I've already looked into jquery-ui) and images/screenshots preview in the dynamic form
- maybe import/export txt files.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
I could transform my files syntax into your syntax in the same step where the checksums are added. What do you think about that?
Sounds good. I would also advise to move the checksums to the file objects as those things belong together.

For now FSO and the MediaVPs 2014 are in the repo. Here's the repo configuration and here's the file fs2mod-py uses.
I see you are still using the old installer like "files" syntax. Is this going to change at a later point?

The second file actually uses your syntax. I've added the filenames as keys and changed some key names. Here's a prettier version of the second file:
Code: [Select]
{
   "generated" : 1409428735.93665,
   "mods" : [
      {
         "title" : "MediaVPs",
         "version" : "2.0.0-2014",
         "id" : "MediaVPs_2014",
         "logo" : "logofzicrk73.jpg",
         "packages" : [
            {
               "notes" : "Required: The bare minimum required for running FSO, also contains the mod.ini, FSU-MVP.bmp and Readme.txt and sets up the mod folder",
               "files" : {
                  "MV_Root.7z" : {
                     "contents" : {
                        "MediaVPs_2014/FSU-MVP.bmp" : "1c5972372b4d2a38bc1e5a7d8653f580",
                        "MediaVPs_2014/mod.ini" : "976bc289676bef464a37f70683baba92",
                        "MediaVPs_2014/readme.txt" : "4c9a122950141e828ae2f2a3aa73580b",
                        "MediaVPs_2014/MV_Root.vp" : "c7729dea15e7b23de1936cd79e5a036c",
                        "MediaVPs_2014/FSU-MVP_small.bmp" : "2fe5111a95f37f0008ad200896175035"
                     },
                     "md5sum" : "96bf9c147c7e371abde42a10064d9714",
                     "is_archive" : true,
                     "dest" : "",
                     "urls" : [
                        "http://mvp.fsmods.net/2014/MV_Root.7z",
                        "http://www.lunardigitalproductions.com/downloads/MediaVPs2014/MV_Root.7z"
                     ]
                  }
               },
               "name" : "MV_Root",
               "dependencies" : [
                  {
                     "id" : "FSO",
                     "version" : ">=3.7.1"
                  }
               ],
               "environment" : [],
               "status" : "required"
            },
            {
               "status" : "recommended",
               "notes" : "Enhanced OGG Music Package",
               "name" : "MV_Music",
               "files" : {
                  "MV_Music.7z" : {
                     "urls" : [
                        "http://mvp.fsmods.net/2014/MV_Music.7z",
                        "http://www.lunardigitalproductions.com/downloads/MediaVPs2014/MV_Music.7z"
                     ],
                     "dest" : "",
                     "is_archive" : true,
                     "md5sum" : "ec577fef3c101553ecb1bff86e7ad8ae",
                     "contents" : {
                        "MediaVPs_2014/MV_Music.vp" : "121300bbe229d0373cd731a37b205b3a"
                     }
                  }
               },
               "environment" : [],
               "dependencies" : []
            },
            {
               "notes" : "All the High Poly Models and their Maps",
               "files" : {
                  "MV_Assets.7z" : {
                     "contents" : {
                        "MediaVPs_2014/MV_Assets.vp" : "bb65f1e147c3417a86dd5464f268feb5"
                     },
                     "md5sum" : "17bc0c46a08de9753e342c445c6f9850",
                     "is_archive" : true,
                     "dest" : "",
                     "urls" : [
                        "http://mvp.fsmods.net/2014/MV_Assets.7z",
                        "http://www.lunardigitalproductions.com/downloads/MediaVPs2014/MV_Assets.7z"
                     ]
                  }
               },
               "name" : "MV_Assets",
               "dependencies" : [],
               "environment" : [],
               "status" : "recommended"
            },
            {
               "status" : "recommended",
               "dependencies" : [],
               "environment" : [],
               "files" : {
                  "MV_Effects.7z" : {
                     "is_archive" : true,
                     "dest" : "",
                     "urls" : [
                        "http://mvp.fsmods.net/2014/MV_Effects.7z",
                        "http://www.lunardigitalproductions.com/downloads/MediaVPs2014/MV_Effects.7z"
                     ],
                     "contents" : {
                        "MediaVPs_2014/MV_Effects.vp" : "cd3fef0af8308087f619d0e19cf4b620"
                     },
                     "md5sum" : "0d71c9ea7e4d620fd7902d1941161a35"
                  }
               },
               "name" : "MV_Effects",
               "notes" : "All the touched up or redone Effects (Weapons, Starfields, etc)"
            },
            {
               "notes" : "For users with higher end machines, this package contains many significantly more demanding textures and effects that may cause poor performance on low end machines",
               "name" : "MV_Advanced",
               "files" : {
                  "MV_Advanced.7z" : {
                     "md5sum" : "6f3275e66746e7997d060ee71bbd0b47",
                     "contents" : {
                        "MediaVPs_2014/MV_Advanced.vp" : "2646219d1d67250adc4bcf4ceaa0176f"
                     },
                     "urls" : [
                        "http://mvp.fsmods.net/2014/MV_Advanced.7z",
                        "http://www.lunardigitalproductions.com/downloads/MediaVPs2014/MV_Advanced.7z"
                     ],
                     "dest" : "",
                     "is_archive" : true
                  }
               },
               "environment" : [],
               "dependencies" : [
                  {
                     "packages" : [
                        "MV_Assets",
                        "MV_Effects"
                     ],
                     "version" : "*",
                     "id" : "MediaVPs_2014"
                  }
               ],
               "status" : "recommended"
            },
            {
               "name" : "MV_A-Glows",
               "files" : {
                  "MV_A-Glows.7z" : {
                     "md5sum" : "bed0e25ac2af663dd41574b458f37fd5",
                     "contents" : {
                        "MediaVPs_2014/MV_A-Glows.vp" : "f38badfdc5e3af3bacb356809a8ad8a6"
                     },
                     "dest" : "",
                     "urls" : [
                        "http://mvp.fsmods.net/2014/MV_A-Glows.7z",
                        "http://www.lunardigitalproductions.com/downloads/MediaVPs2014/MV_A-Glows.7z"
                     ],
                     "is_archive" : true
                  }
               },
               "dependencies" : [
                  {
                     "id" : "MediaVPs_2014",
                     "version" : "*",
                     "packages" : [
                        "MV_Assets"
                     ]
                  }
               ],
               "environment" : [],
               "notes" : "For users with extra texture space, contains all the Animated Glow Maps",
               "status" : "optional"
            },
            {
               "name" : "MV_RadarIcons",
               "files" : {
                  "MV_RadarIcons.7z" : {
                     "contents" : {
                        "MediaVPs_2014/MV_RadarIcons.vp" : "45b22fb976ada8c8eceb96592b940d33"
                     },
                     "md5sum" : "60a99d17d4be13b478dabdb8da23550d",
                     "is_archive" : true,
                     "urls" : [
                        "http://mvp.fsmods.net/2014/MV_RadarIcons.7z",
                        "http://www.lunardigitalproductions.com/downloads/MediaVPs2014/MV_RadarIcons.7z"
                     ],
                     "dest" : ""
                  }
               },
               "dependencies" : [],
               "environment" : [],
               "notes" : "Provides ship Icons for the 2D Radar View",
               "status" : "recommended"
            },
            {
               "status" : "recommended",
               "notes" : "Part of a two part pack that updates all of the cbanims to hi-res 660x300.",
               "environment" : [],
               "dependencies" : [],
               "name" : "MV_CB_ANI_1",
               "files" : {
                  "MV_CB_ANI_1.7z" : {
                     "md5sum" : "a521e079e0343de8a2873757b1c7e901",
                     "contents" : {
                        "MediaVPs_2014/MV_CB_ANI_1.vp" : "67ef346aba6a38e60e3517306d0dedca"
                     },
                     "dest" : "",
                     "urls" : [
                        "http://mvp.fsmods.net/2014/MV_CB_ANI_1.7z",
                        "http://www.lunardigitalproductions.com/downloads/MediaVPs2014/MV_CB_ANI_1.7z"
                     ],
                     "is_archive" : true
                  }
               }
            },
            {
               "notes" : "Part of a two part pack that updates all of the cbanims to hi-res 660x300.",
               "dependencies" : [],
               "environment" : [],
               "files" : {
                  "MV_CB_ANI_2.7z" : {
                     "urls" : [
                        "http://mvp.fsmods.net/2014/MV_CB_ANI_2.7z",
                        "http://www.lunardigitalproductions.com/downloads/MediaVPs2014/MV_CB_ANI_2.7z"
                     ],
                     "dest" : "",
                     "is_archive" : true,
                     "md5sum" : "d39066b360194429e634d01366568778",
                     "contents" : {
                        "MediaVPs_2014/MV_CB_ANI_2.vp" : "b25264ca441b654bc147c1e7da574eea"
                     }
                  }
               },
               "name" : "MV_CB_ANI_2",
               "status" : "recommended"
            }
         ],
         "notes" : "",
         "folder" : "Mediavps_2014",
         "submods" : [],
         "description" : ""
      },
      {
         "id" : "FSO",
         "version" : "3.7.2-rc4",
         "logo" : "http://www.fsoinstaller.com/files/installer/java/FS2.bmp",
         "packages" : [
            {
               "status" : "required",
               "notes" : "Windows builds for FreeSpace 2 Open, compiled with SSE optimizations.",
               "environment" : [
                  {
                     "os" : "windows",
                     "type" : "os"
                  },
                  {
                     "feature" : "sse",
                     "type" : "cpu_feature"
                  }
               ],
               "dependencies" : [],
               "files" : {
                  "fs2_open_3.7.2_RC4_SSE.7z" : {
                     "is_archive" : true,
                     "urls" : [
                        "http://swc.fs2downloads.com/builds/WIN/fs2_open_3.7.2_RC4_SSE.7z",
                        "http://scp.fsmods.net/builds/WIN/fs2_open_3.7.2_RC4_SSE.7z",
                        "http://scp.indiegames.us/builds/WIN/fs2_open_3.7.2_RC4_SSE.7z"
                     ],
                     "dest" : "",
                     "contents" : {
                        "fred2_open_3_7_2_RC4_SSE.exe" : "7de9615d5b212e2489dd59833bdbf8f5",
                        "fred2_open_3_7_2_RC4_SSE-DEBUG.exe" : "bc0d622342b6ecda22ffa17187097b26",
                        "fs2_open_3_7_2_RC4_SSE-DEBUG.exe" : "24057ff0d1ba3360e6ba164641b45727",
                        "fs2_open_3_7_2_RC4_SSE-DEBUG.map" : "d687df1a222ca0d8951819444a059a18",
                        "fs2_open_3_7_2_RC4_SSE-DEBUG.pdb" : "ccdd1b98d18580f6c0170954af92e9dd",
                        "fred2_open_3_7_2_RC4_SSE-DEBUG.pdb" : "1a764e4eac166a792bfe026a4d0db5ae",
                        "fred2_open_3_7_2_RC4_SSE-DEBUG.map" : "5a0797ebe00c8144716105e3e6be9d6a",
                        "fs2_open_3_7_2_RC4_SSE.exe" : "da4a73536061ede4e5e659b1e6bbb704"
                     },
                     "md5sum" : "70ecb06d9bf8a447d3a93d212628de37"
                  },
                  "Launcher55g.zip" : {
                     "md5sum" : "20bdf9dc2f225c30961df672be438e37",
                     "contents" : {
                        "Launcher.exe" : "798f7da18e3ce3a7caaa9c4aec2d3072"
                     },
                     "dest" : "",
                     "urls" : [
                        "http://swc.fs2downloads.com/files/Launcher55g.zip",
                        "http://scp.fsmods.net/builds/Launcher55g.zip",
                        "http://scp.indiegames.us/builds/Launcher55g.zip"
                     ],
                     "is_archive" : true
                  }
               },
               "name" : "Windows SSE"
            },
            {
               "status" : "required",
               "notes" : "Windows builds for FreeSpace 2 Open, compiled with SSE2 optimizations.",
               "dependencies" : [],
               "environment" : [
                  {
                     "os" : "windows",
                     "type" : "os"
                  },
                  {
                     "type" : "cpu_feature",
                     "feature" : "sse2"
                  }
               ],
               "name" : "Windows SSE2",
               "files" : {
                  "fs2_open_3.7.2_RC4.7z" : {
                     "md5sum" : "a865d781aad15e8503a8bf4e0d9546e0",
                     "contents" : {
                        "fred2_open_3_7_2_RC4-DEBUG.map" : "10e17002485c8b4e074cd7a833668f95",
                        "fs2_open_3_7_2_RC4.exe" : "2166833b7cf3f795316ed0fc28fe7efd",
                        "fs2_open_3_7_2_RC4-DEBUG.pdb" : "be9e90a78812527df9a18e00f8710aea",
                        "fred2_open_3_7_2_RC4.exe" : "f2968bb9dcc30bc7893637f08e54de16",
                        "fred2_open_3_7_2_RC4-DEBUG.pdb" : "89cdc982273b3ecba95f4134bff0c477",
                        "fred2_open_3_7_2_RC4-DEBUG.exe" : "0ce97a2b8fda1b9b99a0fcbff57cd5d9",
                        "fs2_open_3_7_2_RC4-DEBUG.map" : "df19268d1680729c327b87e006221424",
                        "fs2_open_3_7_2_RC4-DEBUG.exe" : "ac71a1eb4815fe514829b3b6de1c5f8c"
                     },
                     "dest" : "",
                     "urls" : [
                        "http://swc.fs2downloads.com/builds/WIN/fs2_open_3.7.2_RC4.7z",
                        "http://scp.fsmods.net/builds/WIN/fs2_open_3.7.2_RC4.7z",
                        "http://scp.indiegames.us/builds/WIN/fs2_open_3.7.2_RC4.7z"
                     ],
                     "is_archive" : true
                  },
                  "Launcher55g.zip" : {
                     "urls" : [
                        "http://swc.fs2downloads.com/files/Launcher55g.zip",
                        "http://scp.fsmods.net/builds/Launcher55g.zip",
                        "http://scp.indiegames.us/builds/Launcher55g.zip"
                     ],
                     "dest" : "",
                     "is_archive" : true,
                     "md5sum" : "20bdf9dc2f225c30961df672be438e37",
                     "contents" : {
                        "Launcher.exe" : "798f7da18e3ce3a7caaa9c4aec2d3072"
                     }
                  }
               }
            },
            {
               "status" : "required",
               "notes" : "Linux builds for FreeSpace 2 Open.",
               "files" : {
                  "fs2_open_3.7.2_RC4.tar.bz2" : {
                     "dest" : "",
                     "urls" : [
                        "http://swc.fs2downloads.com/builds/LINUX/fs2_open_3.7.2_RC4.tar.bz2",
                        "http://scp.fsmods.net/builds/LINUX/fs2_open_3.7.2_RC4.tar.bz2",
                        "http://scp.indiegames.us/builds/LINUX/fs2_open_3.7.2_RC4.tar.bz2"
                     ],
                     "is_archive" : true,
                     "md5sum" : "5fd897dd72ec0c76eb7eea8377aeffcf",
                     "contents" : {
                        "fs2_open_3.7.2_RC4_DEBUG" : "3f3db65b0f4fadd54a5fa1b269894abf",
                        "fs2_open_3.7.2_RC4" : "027cf0ffa7a96bbfb5239293feada1f3"
                     }
                  }
               },
               "name" : "Linux",
               "dependencies" : [],
               "environment" : [
                  {
                     "type" : "os",
                     "os" : "linux"
                  }
               ]
            },
            {
               "status" : "required",
               "notes" : "",
               "dependencies" : [],
               "environment" : [],
               "files" : {
                  "mod.ini" : {
                     "contents" : {},
                     "md5sum" : "d0576b36ef22b599186f67876e9b3aec",
                     "is_archive" : false,
                     "dest" : "",
                     "urls" : [
                        "http://www.fsoinstaller.com/files/installer/java/mod.ini"
                     ]
                  },
                  "scptrackir.zip" : {
                     "md5sum" : "b611edc87ef30ce57133b9721db01b59",
                     "contents" : {
                        "scptrackir.dll" : "6617fd0f9421ca6594cf2c1eebb20381"
                     },
                     "urls" : [
                        "http://scp.indiegames.us/builds/scptrackir.zip"
                     ],
                     "dest" : "",
                     "is_archive" : true
                  },
                  "FS2.bmp" : {
                     "is_archive" : false,
                     "dest" : "",
                     "urls" : [
                        "http://www.fsoinstaller.com/files/installer/java/FS2.bmp"
                     ],
                     "contents" : {},
                     "md5sum" : "faa4c82b1aa0d3cf5427207e10150711"
                  }
               },
               "name" : "Common files"
            }
         ],
         "title" : "FreeSpace Open",
         "description" : "Recent builds of FreeSpace Open (the engine)",
         "submods" : [],
         "notes" : "",
         "actions" : [],
         "folder" : ""
      }
   ]
}

Anyway, I'm adding this syntax to the converter. It'll accept both variants.

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
I think I'm going to use this syntax in the forms :
I was using a "file group" syntax, where a "file group" is defined as "one or more files sharing the same set of mirror servers".
As I expect modders will use a lot of copy-paste to fill in this part of the form, using full URLs is better (even if adds some duplicated text).

Is the "isArchive" boolean really necessary ? Can't we guess that from the mime type ? (i'm betting no mod will ever ship 7z, zip or rar files that don't need to be extracted)
« Last Edit: September 06, 2014, 05:52:13 pm by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
What you're describing is pretty much what m|m wanted. It's already in the repository syntax but I've added it as an alternative to the repo configuration (converter.py's input). Now you can simply write
Code: [Select]
{
  /* ... */
  "files": {
    "filename1.tar.gz": {
      "dest": "my_path",
      "urls": [
        "http://host.tld/path/to/file.tar.gz",
        "http://other.host.tld/path/to/other/file.tar.gz"
      ]
    },
    "filename2.7z": {
      "dest": "my_path",
      "urls": [
        "http://host.tld/path/to/file.7z",
        "http://other.host.tld/path/to/other/file.7z"
      ]
    }
  }
}

I've included "is_archive" in the schema because some archives aren't meant to be extracted (i.e. jar files).
It's actually optional. The converter and fs2mod-py assume that the given file is an archive if you don't set the "is_archive" property.
Here's the list of file extensions Goober's installer uses to decide whether a file is an archive or not: zip, tar, split, rar, lzma, iso, hfs, gzip, gz, cpio, bzip2, bz2, 7z, z, arj, cab, lzh, chm, nsis, deb, rpm, udf, wim, xar

I'll add a note to all optional properties in the schema...

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
I really don't like using the file names as keys for some data structures. The problem I see here is that you can't have a direct class <-> JSON relationship in object-oriented languages. I would prefer to use a standard array of objects that would contain a "filename" key with the file name.

Is the "isArchive" boolean really necessary ? Can't we guess that from the mime type ? (i'm betting no mod will ever ship 7z, zip or rar files that don't need to be extracted)
I have FSO builds that support 7zip and zip archives but those wouldn't be called 7z or zip :P

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
I've been working on the templates. No real php behind it yet (except including the right bits), but here's how it looks. Comments are welcome.




EDIT 11/09/2014 : Progress ! Most of the mod and branch storage logic is done. I still have to refactor my first draft of the build storage logic and integrate it with the rest, polish the forms a bit, and install a third party account management bundle. After these steps we'll be ready for a first test. Expect this to be in something like 10 days.
« Last Edit: September 10, 2014, 08:50:47 pm by Hellzed »

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
EDIT 11/09/2014 : Progress ! Most of the mod and branch storage logic is done. I still have to refactor my first draft of the build storage logic and integrate it with the rest, polish the forms a bit, and install a third party account management bundle. After these steps we'll be ready for a first test. Expect this to be in something like 10 days.
:yes: That's great news!

I have been looking at the installation process a bit and I noticed that there is no way of specifying the folder of the package file that should be extracted. New mods should have all files in the root folder of the zip file but to support mods that were created before that we need some way of specifying the folder name (for example "MediaVPs_2014" would be the value used for the MediaVPs 2014 package).

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
Make it one more day than what i thought. I've spent my daily development time trying to figure out some badly documented feature of my framework.

If you're interested in the details, it's called "ParamConverter". It safely converts a variable from the page url into an object from your database, or throws an error 404 if no corresponding object exists, just by using a single keyword (php "type hinting"). It avoids a lot of duplicated checks in the php code (or extending the class just to perform said checks).
I've started to rely a lot on this to achieve a github-like hierarchy (owner_object/branch_object/build_object), but it's terribly documented (basically, it says : go check the source...).
It works well on a single object, but fails badly to deal with a group of related objects...
I know exactly what the workaround should be, but I keep having version specific issues (many forum threads are too old for my version of the framework...).

Hopefully I'll find a solution later tonight. Or I'll switch to good old variable checks.

EDIT : Finally found how to write a custom paramconverter from scratch (and that works with Sf 2.5).
« Last Edit: September 13, 2014, 03:52:39 am by Hellzed »

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
I am also making good progress with the implementation of the client library.
It can currently handle parsing the JSON data into an object representation of the repository. I also just finished implementing dependency resolution of a package. Now I can begin implementing the actual installation backend but that shouldn't be a big problem :nervous:

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
That's great ! Hopefully the server is up when you need it.

The basic build storage is in, and we now fully support semver. That means you can use urls like : "/nebula/owner/my_mod/testing/1.1.0-beta2+cockpitspatch/show_data" or "/nebula/owner/my_mod/testing/current/show_data" (for the latest build in the branch).

I've finished reading a big tutorial on FoSUB (the user management bundle I'll be using on the server), I'll start installing it in the project tonight. There are many steps to follow carefully, but it's well documented and nothing is really tricky.
Last week, I stripped my main objects (mod, branch, build) from their secondary objects (logo, package, screenshots...) to make database testing easier. I'll probably re-add them this week-end. From there, I'll only need to pass it to the default php JSON parser, so I'm still aiming to do a first code release on github in a matter of days.

EDIT : Adding FoSUB went smoother than anything I expected :)
Now the app has login/logout/profile/password_reset pages and a "User" table in the database. I'm already writing a Sf Voter (a php class with a method to do a simple check, like verifying if the mod/branch/builds belongs to the current user). Voters are great because they can also be called directly from the template engine (to show the "edit"/"remove" buttons or not, for example).
« Last Edit: September 18, 2014, 03:30:49 pm by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Damn, I'm too slow.
@m!m: I guess you don't need the repo parser anymore, do you?

I've updated the schema to include the second files syntax (@m!m: Is this more to your liking?  :)) and document which fields are optional and which ones are required.

I should be able to finish a stable fs2mod-py release this weekend.
@Hellzed: I'm looking forward to test your Nebula. I'll implement a browser (Qt Webkit) similiar to Steam's shopping tab. I'll add a JS API (if possible) so that your web page can check which mods (and features) are installed. Any thoughts on that?

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
Damn, I'm too slow.
@m!m: I guess you don't need the repo parser anymore, do you?
Apparently not :p Writing the parser was much easier than I imagined. Actually writing the logic to get downloading and installation to work is the hard part.

I've updated the schema to include the second files syntax (@m!m: Is this more to your liking?  :)) and document which fields are optional and which ones are required.
Much better ;)
I have noticed some things while implementing my installer:
  • The "mod" object has both "description" and "notes". What's the difference? It would be helpful to know what that should be used for.
  • What do we need "submod" for? I am not sure what I should do with that.
  • Can the "environment" specification be moved into the file specification? That would simplify dependency resolution for me and also seems like the better place.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
I have noticed some things while implementing my installer:
  • The "mod" object has both "description" and "notes". What's the difference? It would be helpful to know what that should be used for.
  • What do we need "submod" for? I am not sure what I should do with that.
  • Can the "environment" specification be moved into the file specification? That would simplify dependency resolution for me and also seems like the better place.

  • The description is supposed to match the mod.ini's description. The notes should contain important things a modder wants to tell the user before installing (pretty much what release posts on the forum contain).
  • Submods are mods that are related to a given mod (i.e. ST:R is a submod of FSPort 1). I'll display the tree structure in fs2mod-py like this:
Code: [Select]
- Mod 1
+ Mod 2
L Submod 1
L + Submod 2
| L Subsubmod 1
| L Subsubmod 2
L Submod 3
- Mod 3
  • Hm... I'm not sure. In the repo example I did, the fs2_open mod has four packages: Common files, Windows, Linux and Mac OS builds. If we would move the environment specification to the files, I'd only make one package but I'd have to add the OS requirement to every file.
    How do you handle the environment checks, anyway? My repo parser checks the environment requirements whenever a repo is loaded and skips all packages which don't meed the requirements. This way you won't see the Linux or Mac OS builds of fs2_open on Windows.

I'm just wondering... how did you implement the dependency tracking? Can your resolver handle conflicts and cycles?

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
  • The description is supposed to match the mod.ini's description. The notes should contain important things a modder wants to tell the user before installing (pretty much what release posts on the forum contain).
  • Submods are mods that are related to a given mod (i.e. ST:R is a submod of FSPort 1). I'll display the tree structure in fs2mod-py like this:
Code: [Select]
- Mod 1
+ Mod 2
L Submod 1
L + Submod 2
| L Subsubmod 1
| L Subsubmod 2
L Submod 3
- Mod 3
Thanks for the clarification.

  • Hm... I'm not sure. In the repo example I did, the fs2_open mod has four packages: Common files, Windows, Linux and Mac OS builds. If we would move the environment specification to the files, I'd only make one package but I'd have to add the OS requirement to every file.
    How do you handle the environment checks, anyway? My repo parser checks the environment requirements whenever a repo is loaded and skips all packages which don't meed the requirements. This way you won't see the Linux or Mac OS builds of fs2_open on Windows.
Currently I am checking the environment specifications while resolving the dependencies of a package. That solution will not work correctly but I didn't want to spend much time implementing it.
Simply skipping the packages might work but then you will run into issues if a mod depends on the package "Windows SSE" which will not be present on other platforms. By making packages available everywhere but restricting the files that are being installed you solve that issue.

I'm just wondering... how did you implement the dependency tracking? Can your resolver handle conflicts and cycles?
For discovering the dependencies of a package I simply search through all known modifications and select the packages that the given package depends on. I then use that information to build a dependency graph which can be used to get a sorted list of all dependencies of a package.
If the algorithm detects a cycle it throws an exception as that is a case that can not be handled by FSO. For now version conflicts are not handled at all but I'll implement that at a later point.

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
@m!m :
Simply skipping the packages might work but then you will run into issues if a mod depends on the package "Windows SSE" which will not be present on other platforms.
To me, that's only a case of broken dependency. It happens in every package management system. It's not the responsibility of the package management system developers to avoid this, it's the responsibility of packagers not to do silly things (in FreeSpace 2 modding world : don't do mods that depend exclusively on platform specific packages, inside or outside your own mod.)
I agree with ngld : environment specification is in the right place at the package level.
Anyway, just in case it happens, we should warn the user at some point, and tell them to report the issue to the package manager for the broken mod.

Regarding the description/notes issue on server side : I chose not to add the possibility to override the description and other fields of mod metadata at the build level. Instead, I've added a "notes" fields to the branch and build. They will be appended to mod notes.

@ngld :
JS API on the web pages sounds cool.
Showing mod hierarchy is useful for the advanced UI (that we currently have), but I think the simple UI mod browser should present all mods on the same level. It's irrelevant to a newcomer who's just seen a youtube video of ST:R and who wants play it to know it's a submod of fsport (of course, this information will be shown before starting the install process).

To me, the simple UI should look like :
Code: [Select]
-------------------------------------
                      Quicksearch
-------------------------------------
Local mods
-------------------------------------
Mod            |play|settings|remove
-------------------------------------
Mod 2          |play|settings|remove
-------------------------------------
Available mods
-------------------------------------
Mod 3                       |install
-------------------------------------
Mod 4                       |install

-Every mod has its own background image (specific to the collapsed view, meaning modders should provide it as well).
-When you click on a mod title, you get a mod page adapted for a narrow view.
-Actions like "play"/"settings"/"remove"/"install" should only appear on mod mouseover.
-When internet connection is not available, instead of "Available mods", it shows "Go online to browse available mods !"
-When you're on a mod page, the quicksearch block should be replaced by a "Back to mods" button aligned on the left.
-If the "quicksearch" block is native Qt and not a part of the webview, to the right of the search box, we should have a button to switch to the advanced UI.

I'm not sure what the mods list cache management policy should be, but installed mods list and pages should be available while offline.
I'll do the templates if you want.

EDIT : Sf Voters are now working, meaning modders can only work in mods and branches they own.
I'm looking into adding modder teams that could also own mods (effectively allowing multiple modders to work on a single mod). The best way to do it is to add an "owner" abstraction layer, with the owner being either a "modder" or a "team of modders" object. This is a major change (since a lot of my functions revolve around the modder object), but I have to do it now, or it will turn into a massive refactoring later.

EDIT 2 : Implementing the "owner" logic (with modders and teams) will delay the initial git commit (it won't be on sunday, but early next week). I'll keep you posted.

EDIT 3 : @ngld : is JS API what https://extensions.gnome.org/ uses to know if you are running gnome-shell or not ?
« Last Edit: September 20, 2014, 11:25:29 am by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Um... this post wasn't supposed to be this big.  :nervous:

Replies

EDIT 3 : @ngld : is JS API what https://extensions.gnome.org/ uses to know if you are running gnome-shell or not ?
Yes. gnome-shell adds an object called "SweetTooth" which the site uses to interact with it. I'll add a "fs2mod" object.

@m!m :
Simply skipping the packages might work but then you will run into issues if a mod depends on the package "Windows SSE" which will not be present on other platforms.
To me, that's only a case of broken dependency. It happens in every package management system. It's not the responsibility of the package management system developers to avoid this, it's the responsibility of packagers not to do silly things (in FreeSpace 2 modding world : don't do mods that depend exclusively on platform specific packages, inside or outside your own mod.)
I agree with ngld : environment specification is in the right place at the package level.
Anyway, just in case it happens, we should warn the user at some point, and tell them to report the issue to the package manager for the broken mod.
fs2mod-py would complain with "Mod Windows SSE wasn't found!" if you tried to install a mod that depends on it on any OS other than Windows (or on a processor that doesn't support SSE instructions). The message is only logged, right now, but I'll add a message box for the user.
If a mod depends on a package that isn't available on your platform, you won't be able to install it since you can't expect it to run without that dependency.

I'm not sure what the mods list cache management policy should be, but installed mods list and pages should be available while offline.
I'll do the templates if you want.
I'm currently writing and testing the JS API by implementing the simple mod list you mentioned. It's using the JS API to retrieve fs2mod-py's mod list so that shouldn't be a problem.
I'm wondering whether this interface should be stored locally (static files in the script's folder) or served by a server and saved in the application cache...
If we'd serve the interface from a server we could change it without having to update the fs2mod-py client.


JS API

Here's an early draft of the JS API:
  • getVersion(): Returns fs2mod-py's version
  • isFsoInstalled(): Returns true if fs2_open is installed and configured.
  • getInstalledMods(): Returns a list of all installed mods (see the generated section of schema.txt)
  • isInstalled(mid, spec?): Returns true if the given mod is installed. The parameters are the same as query()'s.
  • query(mid, spec?): Allows the web page to query the local mod cache. The first parameter is the mod ID. The second parameter is optional and can specify a version requirement. (i.e. ">=3.0.*")
  • fetchModlist(): Update the local mod cache.
  • install(mid, spec?, pkgs?): Installs the given mod. The first two parameters are the same as query()'s. The third parameter is optional and contains the names of all packages which should be installed (defaults to all required).
  • uninstall(mid, pkgs?): Uninstalls the given mod. The first parameter is the mod's ID. The second parameter should be used if only some packages should be uninstalled.
  • runMod(mid): Launch fs2_open with the given mod selected.
  • showSettings(mid?): Open the settings window for the given mod or fs2_open if no mid is given.

Here's an example of what you could do with it:
Code: [Select]
<script>
$(function () {
  var list = $('<ul>').appendTo('body');
  $.each(fs2mod.getInstalledMods(), function (i, mod) {
    var row = $('<li>').appendTo(list);
    row.append(mod.title);
    row.append($('<a href="#">Run</a>').click(function (e) {
      e.preventDefault();
      fs2mod.runMod(mod.mid);
    }));
    row.append(' | ').append($('<a href="#">Settings</a>').click(function (e) {
      e.preventDefault();
      fs2mod.showSettings(mod.mid);
    }));
  });
});
</script>


Recent changes

Recent changes I've done:
  • Integrated QtWebKit to display web pages and implement the JS API.
  • fs2mod-py is only started once. If you start it again, the active window is moved to the front instead. All fso:// links now open the manager in addition to their specific action.
  • I've implemented CPU flags detection using the neat py-cpuinfo. It works on Windows, Linux and Mac OS X (didn't test).
  • Installing and uninstalling mods is stable and works AFAICT.
  • fs2mod-py has now a status bar and allows the progress window to be hidden.
  • The user can limit the number of parallel downloads. The default is 3, the script has a hard-coded limit of 5 (only 5 threads are used).
  • INTERNAL: The task system has been extended. You can now limit tasks to a specific number of threads and multiple tasks can run at once.

@Hellzed: Whenever I think of emebedding your nebula web interface in fs2mod-py, I think of Steam. I'd like to do something similiar. (Not a simplified UI but the full web interface with Install / Play / Settings buttons.)
So... any thoughts?

Once I'm done with the JS bridge / local html interface, I'll make a release.
Current TODO list:
  • Improve the UI
  • 1.0.0-alpha1 release
  • Test the Nebula once it's released
  • First public release (1.0.0)
  • Get fs2mod-py into PyPi. Installing will then be as simple as "pip install fs2mod-py".
  • Add an auto-update feature (?)
  • Check and improve OS X support

The auto-update feature is going to be a PITA. There will be ~5 ways a user can install fs2mod-py: Use git, download the Windows EXE, use pip, download an archive from github, use a Linux bundle.
The update method probably will be different for every single case.

P.S.: @m!m: Sorry, missed this post earlier.
EDIT 11/09/2014 : Progress ! Most of the mod and branch storage logic is done. I still have to refactor my first draft of the build storage logic and integrate it with the rest, polish the forms a bit, and install a third party account management bundle. After these steps we'll be ready for a first test. Expect this to be in something like 10 days.
:yes: That's great news!

I have been looking at the installation process a bit and I noticed that there is no way of specifying the folder of the package file that should be extracted. New mods should have all files in the root folder of the zip file but to support mods that were created before that we need some way of specifying the folder name (for example "MediaVPs_2014" would be the value used for the MediaVPs 2014 package).
Isn't that the purpose of the "dest" property? If an archive contains the MediaVPs_2014 folder, then you simply set dest to "". If it contains the VP files, you set it to "MediaVPs_2014". Or did you mean something else?

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
fs2mod-py would complain with "Mod Windows SSE wasn't found!" if you tried to install a mod that depends on it on any OS other than Windows (or on a processor that doesn't support SSE instructions). The message is only logged, right now, but I'll add a message box for the user.
If a mod depends on a package that isn't available on your platform, you won't be able to install it since you can't expect it to run without that dependency.
Ok, I'll just remove the packages that don't match when I initially parse the repo information.

P.S.: @m!m: Sorry, missed this post earlier.
EDIT 11/09/2014 : Progress ! Most of the mod and branch storage logic is done. I still have to refactor my first draft of the build storage logic and integrate it with the rest, polish the forms a bit, and install a third party account management bundle. After these steps we'll be ready for a first test. Expect this to be in something like 10 days.
:yes: That's great news!

I have been looking at the installation process a bit and I noticed that there is no way of specifying the folder of the package file that should be extracted. New mods should have all files in the root folder of the zip file but to support mods that were created before that we need some way of specifying the folder name (for example "MediaVPs_2014" would be the value used for the MediaVPs 2014 package).
Isn't that the purpose of the "dest" property? If an archive contains the MediaVPs_2014 folder, then you simply set dest to "". If it contains the VP files, you set it to "MediaVPs_2014". Or did you mean something else?
I meant that there should be some way of specifying which folder inside the archive contains the files. Currently the mediavp archived all have their files in the MediaVPs_2014 folder. My implementation of the installation doesn't install the files into your FreeSpace directory but instead into a directory "mod/<id>/<version>". This makes sure that there are now conflicts between two versions of the same mod. That is currently not possible as some archives don't have the files in the root directory of the archive. You could upload a new version of the archive but uploading multiple GB just for moving files in an archive doesn't seem like a good idea.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
I see two solutions to this problem:

Add a file map which allows you to specify paths inside the archive and where their contents should be placed.
Code: [Select]
{
    "filename": "<filename>", // required
    "is_archive": true, // optional, default: true
    "dest": "<destination path>", // optional, default: ""
    "urls": ["<url1>", "<url2>"], // required
    "file_map": { // optional
        "Media_VPs": ""
    }
}

Add a "move" action:
Code: [Select]
"actions": [
    {
        "type": "move",
        "files": ["Media_VPs/*"],
        "dest": ".",
        "glob": true
    },
    {
        "type": "delete",
        "files": ["Media_VPs"]
    }
]

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
I like adding the move action. That feature can also be used for other things so it's not something that can only be used in specific cases.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Updated the schema once again. I've added the move action and documented the delete action.

The move action is also useful if you want to replace a single file. You could just add the patched file to the downloads and add something like this:
Code: [Select]
"actions": {
    "type": "move",
    "paths": ["example.vp.patched"],
    "dest": "example.vp"
}