Author Topic: BETA development of the FreeSpace Open Installer  (Read 44695 times)

0 Members and 1 Guest are viewing this topic.

Offline Hellzed

  • 28
Re: RELEASE: FreeSpace Open Installer, v2.0 BETA
A standard for packaging mods could be a good idea (it would be about : putting everything at the root of the archives, or putting everything in a folder named like the mod inside the archive ; packaging everything inside VPs and avoiding subfolders ; choosing a single archive format that doesn't suck, hint : RAR is terrible, 7Z is good but not installed by default on linux distros, ZIP is inefficient, BZ2 and LZMA are awesome...).

I ran into this issue while making my installer. Bulletproofing against non standard packaging was harder than bulletproofing against user input.
As I had to rely on Linux unpacking command line tool, i can tell you that "unzip" is perfectly fine, "7z" is not that great, and "unrar" is terrible.

 

Offline mjn.mixael

  • Cutscene Master
  • 212
  • Chopped liver
    • Steam
    • Twitter
Re: RELEASE: FreeSpace Open Installer, v2.0 BETA
Post-testing response.

Now we're getting somewhere! This version seems to be much more consistent and reliable.  :yes:

It seems like if the first hash check fails, it refuses to check the other hashes. This is isn't highly necessary, but it could be much more helpful. I had one MD5 wrong for the BtA download, which I fixed, only to suddenly find out the second was wrong too. It'd be handy if it did them all and gave all the warnings at once. (I'm thinking of a scenario where someone has to ask the mod creator to fix the config file once, twice, three times because of untested MD5s.)

Could you do the same scrolling fix to the other sections of the installer (installing... and install review)?

Still not ending the process properly for on my computer.
Cutscene Upgrade Project - Mainhall Remakes - Between the Ashes
Youtube Channel - P3D Model Box
Between the Ashes is looking for committed testers, PM me for details.
Freespace Upgrade Project See what's happening.

 

Offline MatthTheGeek

  • Captain Obvious
  • 212
  • Frenchie McFrenchface
Re: RELEASE: FreeSpace Open Installer, v2.0 BETA
Dependencies between parent and child mods should now be fixed!  The GUI should no longer get stuck if a child mod fails to install.  (Though the download bar still bounces back and forth... I'll still need to fix that.)
Confirming that, however now you get this:


I don't suppose "null" is expected behaviour (this is supposed to refer to all the MVPs children).

Didn't notice any improvement regarding the reliability of DLs yet.

logs here
People are stupid, therefore anything popular is at best suspicious.

Mod management tools     -     Wiki stuff!     -     Help us help you

666maslo666: Releasing a finished product is not a good thing! It is a modern fad.

SpardaSon21: it seems like you exist in a permanent state of half-joking misanthropy

Axem: when you put it like that, i sound like an insane person

bigchunk1: it's not retarded it's american!
bigchunk1: ...

batwota: steele's maneuvering for the coup de gras
MatthTheGeek: you mispelled grâce
Awaesaar: grace
batwota: oh right :P
Darius: ah!
Darius: yes, i like that
MatthTheGeek: the way you just spelled it it means fat
Awaesaar: +accent I forgot how to keyboard
MatthTheGeek: or grease
Darius: the killing fat!
Axem: jabba does the coup de gras
MatthTheGeek: XD
Axem: bring me solo and a cookie

 

Offline zookeeper

  • *knock knock* Who's there? Poe. Poe who?
  • 210
Re: RELEASE: FreeSpace Open Installer, v2.0 BETA
Looks great to me, assuming that the downloads work and barring some issues with the downloads themselves and their descriptions.

I can only think of a few little things to improve in the installer itself:

1. What's a MediaVP? I'd suggest using the term "graphical upgrades" or "audiovisual upgrades" in the description for the "basic" and "complete" mode of installation options. Certainly any newcomer who doesn't know what MediaVPs means will likely end up downloading them anyway because they're part of the basic option, but explaining it would save them a moment of confusion.

2. The splash screen says "Freespace SCP", then the title bar says "FreeSpace Open Installer", the header image says "Freespace Installer" and the text talks about "FreeSpace 2 Open". If I'm a complete newcomer who's just been pointed to the installer as a way to an upgraded FreeSpace experience, I'll be somewhat confused as to whether this is a FreeSpace 2 thing only or whether it also has something to do with FS1 (and it does, because FSPort). Sure it requires a FreeSpace 2 install, but even so the 2 appears after "FreeSpace" in one place but not in another. I think "FreeSpace 2 Open" should be used everywhere to prevent confusion as to what this is about.

3. The window is not resizable. It's annoying when I'm downloading and can't make the window taller so I could keep an eye on all bars at once.

 

Offline Goober5000

  • HLP Loremaster
  • 214
    • Goober5000 Productions
Re: RELEASE: FreeSpace Open Installer, v2.0 BETA
EDIT: The 'NOTE' section doesn't seem to be shown to the user anywhere.. is that just a legacy 'table flag', bug, or future feature?
Originally, it was a message displayed to the user after installation of that mod had completed.  The NOTE entry for the SCP config file tells the user to install OpenAL.

Right now it isn't displayed anywhere, but I think I'll add it to the log of installation results that shows up on the final page.  If you want to add a "be aware of this before you install the mod" message, like the 3.6.12 MVP constraint, you could probably put that in the the mod description.

It should be possible to add support for bold, italic, hyperlink, etc., in the More Info box, but I would want to defer that until after the more critical stuff is fixed.


A standard for packaging mods could be a good idea (it would be about : putting everything at the root of the archives, or putting everything in a folder named like the mod inside the archive ; packaging everything inside VPs and avoiding subfolders ; choosing a single archive format that doesn't suck, hint : RAR is terrible, 7Z is good but not installed by default on linux distros, ZIP is inefficient, BZ2 and LZMA are awesome...).

I ran into this issue while making my installer. Bulletproofing against non standard packaging was harder than bulletproofing against user input.
As I had to rely on Linux unpacking command line tool, i can tell you that "unzip" is perfectly fine, "7z" is not that great, and "unrar" is terrible.

The installer uses a wrapper around 7zip, so anything 7zip can extract, so can the installer -- regardless of whether 7zip is installed on the machine itself.  So people can feel free to standardize on .7z if they like.


Now we're getting somewhere! This version seems to be much more consistent and reliable.  :yes:
Great! :)

Quote
It seems like if the first hash check fails, it refuses to check the other hashes. This is isn't highly necessary, but it could be much more helpful. I had one MD5 wrong for the BtA download, which I fixed, only to suddenly find out the second was wrong too. It'd be handy if it did them all and gave all the warnings at once. (I'm thinking of a scenario where someone has to ask the mod creator to fix the config file once, twice, three times because of untested MD5s.)
Good idea.  I'll change this.

Quote
Could you do the same scrolling fix to the other sections of the installer (installing... and install review)?
Uh... I could, but that's more tricky, because the Installing section has panels of variable height.  (Especially if you expand/collapse a few of them.)  It's possible, but it would take some custom implementation.

Quote
Still not ending the process properly for on my computer.
Drat.  I'll keep looking into this.


I don't suppose "null" is expected behaviour (this is supposed to refer to all the MVPs children).
It's not getting the name for some reason.  That's probably an easy fix; I bet it's the same bug that directed certain mod-specific logging into the main logger instead of the mod logger.


1. What's a MediaVP? I'd suggest using the term "graphical upgrades" or "audiovisual upgrades" in the description for the "basic" and "complete" mode of installation options. Certainly any newcomer who doesn't know what MediaVPs means will likely end up downloading them anyway because they're part of the basic option, but explaining it would save them a moment of confusion.
This is probably something that mod authors can clarify using the More Info box.  Keep in mind that most config files are maintained by the mod authors themselves.  I was only able to change a few of them (SCP, FSPort, Sync/Transcend/Windmills) because I had access to those sites' FTP space.

Quote
2. The splash screen says "Freespace SCP", then the title bar says "FreeSpace Open Installer", the header image says "Freespace Installer" and the text talks about "FreeSpace 2 Open". If I'm a complete newcomer who's just been pointed to the installer as a way to an upgraded FreeSpace experience, I'll be somewhat confused as to whether this is a FreeSpace 2 thing only or whether it also has something to do with FS1 (and it does, because FSPort). Sure it requires a FreeSpace 2 install, but even so the 2 appears after "FreeSpace" in one place but not in another. I think "FreeSpace 2 Open" should be used everywhere to prevent confusion as to what this is about.
I suppose I could standardize on this.

Quote
3. The window is not resizable. It's annoying when I'm downloading and can't make the window taller so I could keep an eye on all bars at once.
You're not the first person to say so, but I'm going to defer that for now.  Making it resizeable has the potential to introduce layout problems.

 

Offline mjn.mixael

  • Cutscene Master
  • 212
  • Chopped liver
    • Steam
    • Twitter
Re: RELEASE: FreeSpace Open Installer, v2.0 BETA
The scrolling is mostly because it's suuuuper slow, but that can be lived with or deferred until more pressing issues are worked out.

You mentioned renaming should now work. Is there a special install.txt command for that? (I want to be thorough with the install.txt instructions).
Cutscene Upgrade Project - Mainhall Remakes - Between the Ashes
Youtube Channel - P3D Model Box
Between the Ashes is looking for committed testers, PM me for details.
Freespace Upgrade Project See what's happening.

 

Offline Goober5000

  • HLP Loremaster
  • 214
    • Goober5000 Productions
Re: RELEASE: FreeSpace Open Installer, v2.0 BETA
It's this:

Code: [Select]
RENAME
[from]
[to]

And with any luck, I'll have a new version out this evening with a bunch more fixes.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: RELEASE: FreeSpace Open Installer, v2.0 BETA
I guess you should limit the number of concurrent downloads...

Most .zip files failed with the known bug "java.util.zip.ZipException: invalid entry CRC" and the fs2_open_3.7.0.tar.bz2 file is failing with a random hash.
Code: [Select]
ERROR com.fsoinstaller.wizard.InstallItem  - Computed hash value of 77954e11f8a20db448a859c66440ab3e does not match required hash value of 4191abf91a4ce7cbbf1e53026713b581 for file 'fs2_open_3.7.0.tar.bz2'!The computed hash changes with every try but if I download the file myself, the hash is correct.

The installer got stuck when the 7zip bindings failed. Maybe you should display some kind of error message?
The exception was only in the logs:
Code: [Select]
java.lang.RuntimeException: SevenZipJBinding wasn't initialized successfully last time.
Here are the logs for my first and second try.

 

Offline mjn.mixael

  • Cutscene Master
  • 212
  • Chopped liver
    • Steam
    • Twitter
Re: RELEASE: FreeSpace Open Installer, v2.0 BETA
It's this:

Code: [Select]
RENAME
[from]
[to]

And with any luck, I'll have a new version out this evening with a bunch more fixes.

Works, and can even be used to move files around successfully. Any other commands I should know about? You are, literally, the only reference for the install.txt anymore it seems.

I have NAME, DESC, FOLDER, DELETE, RENAME, URL, HASH (with sub commands MD5, SHA-1, or SHA-256.. and I'm assuming the latter 2 are listed exactly like so.), NOTE, and VERSION.
Cutscene Upgrade Project - Mainhall Remakes - Between the Ashes
Youtube Channel - P3D Model Box
Between the Ashes is looking for committed testers, PM me for details.
Freespace Upgrade Project See what's happening.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: RELEASE: FreeSpace Open Installer, v2.0 BETA
It seems there's also MULTIURL:
Code: [Select]
MULTIURL
[url 1]
[url 2]
[url x]
ENDMULTI
It works like URL but the installer will try all listed URLs until one works.
DESC, MULTIURL, and NOTE need to be terminated with an END... token.

The list of tokens is in src/com/fsoinstaller/common/InstallerNodeToken.java and the parsing is done in src/com/fsoinstaller/common/InstallerNodeFactory.java, it seems.

 

Offline Goober5000

  • HLP Loremaster
  • 214
    • Goober5000 Productions
Re: RELEASE: FreeSpace Open Installer, v2.0 BETA
Someone else is diving into the code! :D

The full list of tokens is this:
Code: [Select]
NAME
DESC
ENDDESC
FOLDER
DELETE
RENAME
URL
MULTIURL
ENDMULTI
HASH
VERSION
NOTE
ENDNOTE
END

 
Re: RELEASE: FreeSpace Open Installer, v2.0 BETA
this launcher gives only failed
none done

 

Offline Goober5000

  • HLP Loremaster
  • 214
    • Goober5000 Productions
Re: RELEASE: FreeSpace Open Installer, v2.0 BETA
this launcher gives only failed
none done

I will fix it. :nod:

 
Re: RELEASE: FreeSpace Open Installer, v2.0 BETA
please put 2014 media vps
in installer
thanks

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: RELEASE: FreeSpace Open Installer, v2.0 BETA
Um, Goober? The dmtools.txt file uses a "SUBFOLDER" token.
It seems nice but isn't implemented. Do you plan to add it to the installer?

 

Offline Goober5000

  • HLP Loremaster
  • 214
    • Goober5000 Productions
Re: RELEASE: FreeSpace Open Installer, v2.0 BETA
I plan to fix the dmtools mod file; whether I add SUBFOLDER or do something else has not been decided yet.  First, I need to finish fixing the cancellation mechanism and then fix the SevenZipException.

 

Offline mjn.mixael

  • Cutscene Master
  • 212
  • Chopped liver
    • Steam
    • Twitter
Re: RELEASE: FreeSpace Open Installer, v2.0 BETA
just a note... plz2update when installer is ready. http://www.fsoinstaller.com/
Cutscene Upgrade Project - Mainhall Remakes - Between the Ashes
Youtube Channel - P3D Model Box
Between the Ashes is looking for committed testers, PM me for details.
Freespace Upgrade Project See what's happening.

 

Offline niffiwan

  • 211
  • Eluder Class
Re: RELEASE: FreeSpace Open Installer, v2.0 BETA
I've been running more tests; and unfortunately it doesn't seem to me that the newer versions are any more reliable that the original, I'm getting failures all over the place, seemingly at random :(

One issue I'm fairly confident is repeatable - if a config file (e.g. cutscene.txt) specifies a nested path like data/movies and the base dir (data in this case) doesn't exist then the install fails. Is this a situation where the installer should create the base dir, or should the config file specify it?  (And of course, this is somewhat artificial since a Freespace2 install should have /data created anyway, I only noticed this because I'm using a mostly empty dir for testing, not my main FSO dir).

Back to the CRC errors / hash failures, I've noticed that when a zip CRC failure occurs it's during the download.  However (after commenting out the code that deletes files on failure) there's files some from the zip successfully extracted on disk. I'm kinda spit-balling here, but is it possible that the unzip is starting prior to the download being completed, and it's running out of downloaded data to process?

edit: clarification
« Last Edit: January 28, 2014, 06:32:46 am by niffiwan »
Creating a fs2_open.log | Red Alert Bug = Hex Edit | MediaVPs 2014: Bigger HUD gauges | 32bit libs for 64bit Ubuntu
----
Debian Packages (testing/unstable): Freespace2 | wxLauncher
----
m|m: I think I'm suffering from Stockholm syndrome. Bmpman is starting to make sense and it's actually written reasonably well...

 

Offline m!m

  • 211
Re: RELEASE: FreeSpace Open Installer, v2.0 BETA
I just looked through the code of the Downloader class on svn an noticed something suspicious. There is a static downloadBuffer field which is used to buffer the downloaded data. You mentioned that downloading is done multithreaded which means that if that class is still being used then the buffer data is getting corrupted if there are multiple files being downloaded at the same time.
I haven't actually run the code to be sure that this is an issue, just wanting to point this out now before I do some testing later today.

 

Offline Goober5000

  • HLP Loremaster
  • 214
    • Goober5000 Productions
Re: RELEASE: FreeSpace Open Installer, v2.0 BETA
I've been running more tests; and unfortunately it doesn't seem to me that the newer versions are any more reliable that the original, I'm getting failures all over the place, seemingly at random :(
I haven't addressed the download exceptions yet.  I'm fixing the auxiliary bugs first, and I have a new set of changes that I haven't created a build for yet.

Quote
One issue I'm fairly confident is repeatable - if a config file (e.g. cutscene.txt) specifies a nested path like data/movies and the base dir (data in this case) doesn't exist then the install fails. Is this a situation where the installer should create the base dir, or should the config file specify it?  (And of course, this is somewhat artificial since a Freespace2 install should have /data created anyway, I only noticed this because I'm using a mostly empty dir for testing, not my main FSO dir).
That's correct.  I didn't think I would need this behavior, but I guess I do.  This has been fixed.

Quote
Back to the CRC errors / hash failures, I've noticed that when a zip CRC failure occurs it's during the download.  However (after commenting out the code that deletes files on failure) there's files some from the zip successfully extracted on disk. I'm kinda spit-balling here, but is it possible that the unzip is starting prior to the download being completed, and it's running out of downloaded data to process?
No, the download pipes directly into the extractor, so there's no missing data.  But I think m!m solved the problem...


I just looked through the code of the Downloader class on svn an noticed something suspicious. There is a static downloadBuffer field which is used to buffer the downloaded data. You mentioned that downloading is done multithreaded which means that if that class is still being used then the buffer data is getting corrupted if there are multiple files being downloaded at the same time.
I haven't actually run the code to be sure that this is an issue, just wanting to point this out now before I do some testing later today.
This has got to be it. :yes:  I can't imagine how that static keyword snuck in, but it shouldn't be there.  I've fixed this, and I'll upload a new build tonight with my other changes.