[Lykurgos88]

Because it's not possible to use the same folder to work with both MVPs 3.6.12 and MVPs 2014.  If you use 2014 you must include the file; if you use 3.6.12 you must not include the file.  So we use two folders to deal with the two mod configurations.

As for why we didn't just include the file and be done with it, well, we're all about compatibility here at the FSPort.  Most FSPort campaigns still have mod.ini files which point to 3.6.12; those would all be broken if we forced the upgrade, and then we'd have to deal with the inevitable technical support requests.   Additionally, there might be situations where you would want to run one FSPort-based mod using 3.6.12 and one using 2014 (one example I can think of is troubleshooting a mod error).  Having two folders retains that possibility.

I understand why there was the segregation with the normal mediaVPs (original, 3612 and 2014). It was of course for compatibility reasons. Especially the mediaVPs_3612 seems to be uncooperative with many mods from the year 2009. The different versions also have vastly different content and table file structures. The overall size has also grown by a landslide.

But here, the "new" "fsport-mediavps_2014" is almost identical to the older "fsport-mediavps". One single table file adjustment for ships hasn't broken ANY compatibility with FSPort campaign mods. The campaigns I previously listed work 100% fine after a modest mod.ini change. For me, it would be a HUGE waste of space to make 2 different fsport-mediaVPs folders with 99,9% identical content. My FS2 folder is already almost 40 gigs so this waste of space is just getting silly at this point.

But yeah, I guess there will always be idiots that can't change a single line of mod.ini with notepad. 

A steam-like content synchronization hub tool would be neat for FSO (I realize there is some effort to do this, but I also realize it's time consuming and difficult). It could automatically update mod.ini files when there is no risk of breaking things (like in this case).

It looks like you've removed the primarylist line and merged it into secondarylist.  That can lead to unintended results.  For example you might find that low-poly models show up in unexpected places.

  I thought only the loading order mattered. Specifically: "fsport-mediaVPs" must be loaded into the game before "fsport" and "mediaVPs_2014" should be last thing to load. At least I didn't end up having low-poly (FS1 era) models with using "secondarylist" only.

[Macfie]

To use the mediavps-2014 with other FSPort mods you have to change the mod.ini files anyway.

[Goober5000]

But here, the "new" "fsport-mediavps_2014" is almost identical to the older "fsport-mediavps". One single table file adjustment for ships hasn't broken ANY compatibility with FSPort campaign mods. The campaigns I previously listed work 100% fine after a modest mod.ini change. For me, it would be a HUGE waste of space to make 2 different fsport-mediaVPs folders with 99,9% identical content. My FS2 folder is already almost 40 gigs so this waste of space is just getting silly at this point.

But yeah, I guess there will always be idiots that can't change a single line of mod.ini with notepad. 

The problem is that the mod.ini AND the presence/absence of the patch must be changed together.  I wanted to do everything possible to minimize the number of problems people could cause for themselves.  Go read the Steam forums if you want to discover a taste of that.

This method requires no editing of files, only renaming them.  And regardless of the file that is renamed, the mod will still work.  Even if people rename the mod.ini incorrectly, it won't break, it will just run a different configuration.

You're right that it takes up a large amount of space though.  I thought that was the best trade-off, but it now occurs to me that it may be possible to minimize the space by using both folders together.  This will require an even more complicated command line though...

It looks like you've removed the primarylist line and merged it into secondarylist.  That can lead to unintended results.  For example you might find that low-poly models show up in unexpected places.

  I thought only the loading order mattered. Specifically: "fsport-mediaVPs" must be loaded into the game before "fsport" and "mediaVPs_2014" should be last thing to load. At least I didn't end up having low-poly (FS1 era) models with using "secondarylist" only.

The loading order does matter, and you changed it by removing primarylist, which causes a mod to be loaded before the active mod.  If you look at the command line in the launcher, you will see the difference between what was there...

Code: [Select]

-mod fsport-mediavps,fsport-str,fsport,MediaVPs_2014
...and what you changed it to...

Code: [Select]

-mod fsport-str,fsport-mediavps,fsport,MediaVPs_2014
Using ST:R as an example, your change will cause the Vasudan ship hulks in mission 14, "Chasing Shadows", to be low-poly instead of high-poly.

[jr2]

Goober, would it be possible to use something like QuickPAR (see MultiPar, below) to minimize the download sizes?

QuickPAR uses par files sort of like RAID stripes -- I don't know how, but you can change the original file anywhere and do anything to it -- the only caveat is the par file needs to be generated so as to be large enough to cover the changed data.

I don't know how you would make it work, but QuickPAR would basically allow you to only download the amount of file size that was changed between the original and modified versions.

You'd have to figure out a way (if possible!) to make QuickPAR either patch the original files to new versions, AND/OR create new files from the old ones.

e.g.:

Updating to MVPs 2014 from MVPs 2012 while keeping 2012, you'd only need to (on the dev side) generate a par file for the 2014 MVPs, download that par file (on the user end), then either have QuickPAR copy the new files to the 2014 directory (alternatively, just use a copy command to copy 2012 to the new 2014 directory, then QuickPAR the 2014 directory up to date).

I think this could be automated with the Installer.  Not sure about the amount of time needed to be invested in a project such as this, but I imagine the bandwidth savings would be immense.  Especially for updates that are small (1-100MB in size) but would normally require re-downloading the entire mod (1-5GB).


EDIT:  parchive par2cmdline (DOS, Win32, OSX, Linux, source etc etc).

EDIT2: Plans are also to introduce PAR3 (support for directory structures among other things.)

EDIT3: I'm behind the power curve. QuickPAR has been superseded by MultPar, which has support for PAR3

[Goober5000]

It's a nice idea, but it would probably require a large amount of work to integrate with the Installer.  Just the fact that it's written in C and not Java introduces a great deal of complexity.

[jr2]

I was thinking more along the lines of calling the command line app externally... maybe using a library -- not sure how that all would work, but it should be similar to using 7-zip I would think.  (Wish I knew more about the process, but that is what I read between the lines of what I don't understand.. am I basically correct?)

[ngld]

The installer uses Java bindings for 7zip which is easier than calling a command line app (which also would have to be extracted from the jar file).
I'm not sure how the java bindings work but if you're using a command line app, you'd have to either bundle at least three version (Windows, Linux, Mac) or build three versions of the installer.

After looking at MultiPar's website, it seems it only works on Windows. There's also xdelta which should work on Windows, Linux and Mac. It's a small and simple program which can generate deltas (diffs) for binary files.

[jr2]

They have the source available, and supposedly it runs on WINE, but yeah, not sure. 

Just an idea.  I mean, I or anyone else could generate par files or diff files, but the end user would have to install and manually work out how to patch.. not good, recipe for a mess.  I wonder if there's any software specifically for patching large data files in a standalone way (like the 1.2 patcher used for FS2, for example - or did that simply replace the files patched with newer versions?).

[Goober5000]

After looking at MultiPar's website, it seems it only works on Windows. There's also xdelta which should work on Windows, Linux and Mac. It's a small and simple program which can generate deltas (diffs) for binary files.

That's interesting; I hadn't seen xdelta before.  I did some searching around myself and ran across zsync, which is basically rsync over HTTP.  Even better, there is a Java implementation called jazsync.  I checked out the code to keep it on hand for later tinkering.

However, I later realized that it will not be possible to use any file diff algorithm with the current implementation of 7Zip-JBinding.  This is because it is designed to extract files one entry at a time.  You have to extract an integer number of entries; it is not possible to extract part of an entry.  This means that file diffing will only be possible if the entire file resides on the server, and I doubt people will want to upload zip files and the files contained in the zip.

As a trivia note, FSPort did provide patches several years ago, from version 2.3 through 3.0.4.  (Here is one.)  But creating the patches was tricky and people seemed happy to just download the entire files, so we dropped it.

They have the source available, and supposedly it runs on WINE, but yeah, not sure.

It seems to have escaped you that the source is in C and the Installer is in Java...

[jr2]

That were more me thinking along the lines of perhaps it (MultiPar) could be gotten to be cross-platform. 
Can Java call something else not written in it's own language and pass arguments to it? 
Then you have cross-platform Java call the (now cross-platform) xyzMerge/diff app and pass the arguments along that you want, perhaps after already copying the files to be modified to a new directory (assuming xyzMerge/diff doesn't support using one file + par/diff file as source for a new file in a new location).

[The E]

Can Java call something else not written in it's own language and pass arguments to it? 
Then you have cross-platform Java call the (now cross-platform) xyzMerge/diff app and pass the arguments along that you want, perhaps after already copying the files to be modified to a new directory (assuming xyzMerge/diff doesn't support using one file + par/diff file as source for a new file in a new location).

Sure, you can do that in Java. But as ngld pointed out, this would mean that the installer effectively loses some of its multi-platformness, as it would necessitate bundling the compiled executables for whatever external tools you want to call with it, not to mention that this introduces a bunch more complexity in terms of testing on various configurations.

[ngld]

After looking at MultiPar's website, it seems it only works on Windows. There's also xdelta which should work on Windows, Linux and Mac. It's a small and simple program which can generate deltas (diffs) for binary files.

That's interesting; I hadn't seen xdelta before.  I did some searching around myself and ran across zsync, which is basically rsync over HTTP.  Even better, there is a Java implementation called jazsync.  I checked out the code to keep it on hand for later tinkering.

That... looks very interesting. The only downside is that it only works with .tar.gz files. Most archives I've seen around here are either .rar or .7z.

However, I later realized that it will not be possible to use any file diff algorithm with the current implementation of 7Zip-JBinding.  This is because it is designed to extract files one entry at a time.  You have to extract an integer number of entries; it is not possible to extract part of an entry.  This means that file diffing will only be possible if the entire file resides on the server, and I doubt people will want to upload zip files and the files contained in the zip.

Well, it is possible if you generate the delta on the server or beforehand. I could implement this on the Nebula server. Whenever a new mod is added to the repository, the server already downloads the archives to generate checksums. Since it's also archiving old versions, it has access to the old and new files. It could then build an archive which contains new files, deltas of changed files and a list of delted files.
If you're using zsync, then it's even easier: Just zsyncmake on the .tar.gz archive and upload the generated .zsync file.

A steam-like content synchronization hub tool would be neat for FSO (I realize there is some effort to do this, but I also realize it's time consuming and difficult). It could automatically update mod.ini files when there is no risk of breaking things (like in this case).

Did you take a look at this? </shameless plug>

BTW, if you want to save space couldn't you put only the changed table in the fsport-mediavps_2014 folder and "fsport-mediavps_2014,fsport-mediavps" in the primarylist of the mod.ini file?

[Lykurgos88]

I wanted to do everything possible to minimize the number of problems people could cause for themselves.  Go read the Steam forums if you want to discover a taste of that.

Yeah, I had the unfortunate pleasure of experiencing the FS2 (steam edition) "registry error" forum rage last summer. It was sickening to see how you FSO guys somehow got the blame for the broken game when you were just trying to offer a better alternative. I too made a "FSO for dummies" thread in Steam hub, but I don't know how many read it.

The loading order does matter, and you changed it by removing primarylist, which causes a mod to be loaded before the active mod.

Ok, this is good to know. Thanks Goober! 

Did you take a look at this? </shameless plug>

I was aware of this alternative installer, but I didn't know it was aiming for these new features. Truly excellent work, well done 

BTW, if you want to save space couldn't you put only the changed table in the fsport-mediavps_2014 folder and "fsport-mediavps_2014,fsport-mediavps" in the primarylist of the mod.ini file?

Well actually I just kept the old name "fsport-mediaVPs" even after I put the new table file patch in it. This is because I already used the "unofficial 2014 patch" for fsport before, so I have already manually edited the mod.ini file of about 10 fsport campaign mods to refer to "mediaVPs_2014" in the secondarylist instead of "mediaVPs_3612". I could have re-edited the mod.inis again, but I felt lazy so I just kept the old fsport-mediaVPs name.

The reason I mentioned the whole "hard drive space" issue was because I thought it was a very clumsy solution to make 2 almost identical "mediaVPs" mods, when one is installing via the FSOpen java installer. But I get Goober's reasoning for this now. As the whole existence of this offtopic thread shows, FSO does really have a major challenger with content update synchronization (meaning that it's mostly manual instead of being automated or easily controlled).

[Goober5000]

Well, now you have me wanting to test the alternate streamlined mod arrangement.   I will see whether it's workable or not.

Also, I may be able to implement a patching utility for the Installer as well.  Stay tuned.

[jr2]

I knew if I poked the wizards enough they could find an awesome solution.  y'know... that, or turn me into a toad.   



Thanks for all the magic you do, guys.