[jg18]

After looking at Hellzed's recent work on installer scripts, it occurred to me that the most difficult part of the FSO install process is putting together the MediaVPs, as I discovered when setting someone up back in January. A script that produces the MediaVPs_3612 folder (or MediaVPs_2014 once that's ready), given a folder with the downloaded .zip files, could well be worth its weight in gold, at least until we have a working installer.

As a stretch goal, it could take a provided list of MD5 sums and would tell you if any of your downloads were bad, although it'd require you to download an external tool to compute the MD5s.

If a batch file proves too difficult, maybe we could come up with something extremely simple using one of the Windows installer creator packages.

I suppose this could be done in coordination with Hellzed's work, although the intent is to keep the task simple enough that this could be written and tested very quickly and easily.

I'll see what I can come up with this weekend, although someone else might come up with something first. Watch this space.

EDIT: Slight clarification.

[z64555]

*watches the space*

[niffiwan]

sound good

FYI, the Debian FSO packages seem to have an installer script in them; it can download/install the GOG/CDROM files, the mediavps & fsport.  Only issue would be that it's tied to the Debian packaging system, i.e. makefile based, etc and thus may not be easily transportable to something standalone.

Anyway, if you want to have a look, the makefile is in:
freespace2_3.7.0+repack-1_amd64.deb
  data.tar.xz
    /./usr/share/games/freespace2-open/packages/freespace2-mod-mediavps/debian/rules

[jr2]

Can the Debian installer use cygwin?

[Hellzed]

My guess is that it is much too dependent on the debian packaging tools to be portable (even to distros outside debian derivatives).

[jg18]

So RL happened this weekend and I couldn't get to it. I'll try for tomorrow. Really, I'm hoping it could be as simple as

Code: [Select]

unzip *.zip
or maybe something a bit more sophisticated (bash syntax):

Code: [Select]

for i in `ls *.zip`; do unzip $i; done

My guess is that it is much too dependent on the debian packaging tools to be portable (even to distros outside debian derivatives).

Agreed.

EDIT: Corrections.

[Hellzed]

Actually, it's a bit more complicated : some VPs are in the root folder of the archive, while others are in subfolders. So you have to do either : a mod specific install script with hardcoded paths to VP files inside the archive ; or perform a search inside the zip file to find VP files, which would be compatible with more mods and easier to maintain when a mod packaging changes.

I know you are aiming at a Media VP installer script, but the root/subfolder issuer is something i ran into.

[jg18]

The idea was to keep this extremely simple and limit it to MediaVPs only, since it's the only mod that's "required" or at least strongly recommended to install as part of the FSO install process. For any other mod, people would need to install it themselves. There could be a script for the 3.6.12 MediaVPs and a script for the new 2014 MVPs once it's ready. I thought that with the MediaVPs, all files were in a subfolder in the zip files, but I haven't looked at it in a year.

I guess the script could be adjusted to something like (3.6.12, bash syntax)

Code: [Select]

for i in `ls *.zip`; do unzip $i; mv *.vp MediaVPs_3612; done

I guess the script could be adjusted to fit both 3.6.12 and 2014 once it's ready, maybe even 3.6.10, but again, the idea is to keep it very simple.

Note that I literally just woke up and am not capable of thinking anything particularly complex right now. I just noticed that last night my pseudo-scripts were trying to unzip vp files instead of zip files. Ugh.

EDIT: Corrections.

[Hellzed]

Currently it looks like this on my computer :
(this checks the integrity of the installed VP files, checks if you have the .zip files of the mod, and builds a list of what to reinstall/redownload, avoiding to re dl stuff twice ; the redownload reinstall process is not in a working state yet so i didn't include it)
(i'm not a real coder, so i'm doing what i can...)

To try this, you must create a "dl_cache" folder in your FS2 root directory, modify the script so FS2_ROOT_DIR points to your FS2 root directory.
the files "vp" and "download" should be in the same directory as the main sh script.
Right now, all it will do is check stuff and tell you if your Media VPs are in a working state.

Code: [Select]

#!/bin/bash
#Récupérer à nouveau le root directory
#récupérer les index download et vp sur mon serveur
#ajout d'un mini fichier update dans les unified mod installer, mettre à jour et remplacer les versions des mods. Automatique ou demander à l'utilisateur ?

#ajouter les checks permissions et check inputs

FS2_ROOT_DIR="/home/loris/FreeSpace 2 beta"
MOD_NAME="mediavps_3612"
CACHE_NAME="dl_cache"

vp=vp
download=download

MOD_DIR="$FS2_ROOT_DIR/$MOD_NAME"
CACHE_DIR="$FS2_ROOT_DIR/$CACHE_NAME/$MOD_NAME"

INTERACTIVE_MODE="false"
x=1
y=1


#boucles while pour faciliter les retours à l'étape précédente. Traiter tout le script avec cette méthode (+propre)

#tester la connexion à Hard Light Productions. +mon dépôt de fichiers de mods

while read line
do
reinstall=false
redownload=false

vp_name=`echo $line | cut -d\  -f2`
vp_zip=`echo $line | cut -d\  -f3`

if [ -f "$MOD_DIR/$vp_name" ]
then
echo "$vp_name : Installed file found"
vp_checksum=`echo $line | cut -d\  -f1`
current_checksum=`md5sum "$MOD_DIR/$vp_name" | cut -d\  -f1`
if [ $current_checksum = $vp_checksum ]
then
echo "$vp_name : File verified"
else
echo "$vp_name : File not verified"
reinstall=true
fi
else
echo "$vp_name : File not found"
reinstall=true
fi



if [ $reinstall = true ]
then
reinstall_list[$x]="$vp_name"
(( x++ ))

download_name=`cat $download | grep $vp_zip | cut -d\  -f2`
if [ -f "$CACHE_DIR/$download_name" ]
then
echo "$vp_name in $download_name : Cached file found"
download_checksum=`cat $download | grep $vp_zip | cut -d\  -f1`
current_cache_checksum=`md5sum "$CACHE_DIR/$download_name" | cut -d\  -f1`
if [ $current_cache_checksum = $download_checksum ]
then
echo "$vp_name in $download_name : Cached file verified"
else
echo "$vp_name in $download_name : Cached file not verified"
redownload=true
fi
else
echo "$vp_name in $download_name : Cached file not found"
redownload=true
fi
fi

if [ $redownload = true ]
then
    if echo ${redownload_list[@]} | fgrep --word-regexp "$download_name" 1>/dev/null
    then
    echo "$download_name already in download list. Skipping."
else
redownload_list[y]="$download_name"
echo "Will download $download_name ."
(( y++ ))
fi
fi

#3 modes d'action
# auto : si fichier trouvé, on ne fait rien, si fichier non trouvé mais en cache, on installe, si fichier pas en cache, dl + install
# interactif : on pose les questions (re télécharger, écraser les fichiers etc)
# force-all : on re télécharge tout et on re décompresse tout)

#gestion wget : sed pour garder l'avancement uniquement (barre + % + MBps)



echo -e ""

done < "$vp"

for item in ${!redownload_list[@]}
do
download_url=`cat $download | grep ${redownload_list[$item]} | cut -d\  -f3`
echo -n "Downloading ${redownload_list[$item]} :   0%"
wget --progress=dot:force -c $download_url -P "$CACHE_DIR" 2>&1 | grep --line-buffered "%" |  sed -u -e "s|\.||g" -e "s|,||g" | awk '{printf("\b\b\b\b%4s", $2)}'
    echo -ne "\b\b\b\b"
    echo " DONE"
done

for item in ${!reinstall_list[@]}
do
archive=`cat $vp | grep ${reinstall_list[$item]} | cut -d\  -f3`
case $archive in
*.zip|*.ZIP)
unzip -ju "$CACHE_DIR/$archive" "**${reinstall_list[$item]}" -d "$MOD_DIR"
;;
*.7z|*.7Z)
7z e "$CACHE_DIR/$archive" "${reinstall_list[$item]}" -r -o"$MOD_DIR"
;;
esac
done

#echo ${reinstall_list[@]}

file : vp

Code: [Select]

a22db283154c07e5cfb5951c410e3600 FSU-MVP.bmp MediaVPs_3612.zip
ef9c1864ee8d90bd661ebfdc624dc76a mod.ini MediaVPs_3612.zip
0a555881c327a60b2ea951fc65f3cbe5 MV_Advanced.vp MV_Advanced_3612.zip
4b2ebaba4b78b96dcd40aef1075c2e79 MV_AnimGlows.vp MV_AnimGlows_3612.zip
b25e9831a7b7a168f4e77591f27ed48e MV_Assets.3612.vp MV_Assets_Update.zip
b96fd80dd26c1597a42b2c550dd799cb MV_Assets.vp MV_Assets_3612.zip
9c30cbe8f137c2b688dd31f62043dadd MV_CB_ANI_1.vp MV_CB_ANI_1.zip
1532f8dd71184c0ae3473274a9b3eba8 MV_CB_ANI_2.vp MV_CB_ANI_2.zip
7b3502189dde3de5fe185b02191ef2d7 MV_Effects.3612.vp MV_Effects_Update.zip
e272eecaf368ab501f5d8ff667aba472 MV_Effects.vp MV_Effects_3612.zip
af17fe7c440f9fe97aa272ee4dfe9ba3 MV_Music.vp MV_Music_3612.zip
82905009785c30b505f441d2446a6286 MV_RadarIcons.vp MV_RadarIcons_3612.zip
f68c3b2e338295bffa7a9650777a8b50 MV_Root.3612.vp MV_Root_Update.zip
907e522ba79167fa13cdf033dc5c8a0f MV_Root.vp MV_Root_3612.zip
e8d78f5c99e753a3e2416d584ae74840 readme.txt MediaVPs_3612.zip
file : download

Code: [Select]

c019c57629f352ff68b00c0a5478dca5 MediaVPs_3612.zip http://mvp.fsmods.net/3612/MediaVPs_3612.zip
37d110c741e55141f8d71c64f989f850 MV_Advanced_3612.zip http://mvp.fsmods.net/3612/MV_Advanced_3612.zip
270ebae69c867905c5b3ba389309ae67 MV_AnimGlows_3612.zip http://mvp.fsmods.net/3612/MV_AnimGlows_3612.zip
d6eff4d6a3801ec64b8795407735defd MV_Assets_3612.zip http://mvp.fsmods.net/3612/MV_Assets_3612.zip
76f322d13e1701da885174dbf85104fb MV_Assets_Update.zip http://mvp.fsmods.net/3612/MV_Assets_Update.zip
f8e6011c0f26ad9eae9665c0456849cb MV_CB_ANI_1.zip http://mvp.fsmods.net/3612/MV_CB_ANI_1.zip
8e32344458d53ff7be48c91b7d4cf158 MV_CB_ANI_2.zip http://mvp.fsmods.net/3612/MV_CB_ANI_2.zip
9a8d3f08da8c207bc73ae0271a62f8d1 MV_Effects_3612.zip http://mvp.fsmods.net/3612/MV_Effects_3612.zip
c0de5bea8050a09c21d0375824eddf65 MV_Effects_Update.zip http://mvp.fsmods.net/3612/MV_Effects_Update.zip
eb5b11004e6823571bb6ef5a937f8eb2 MV_Music_3612.zip http://mvp.fsmods.net/3612/MV_Music_3612.zip
bdf3c34e608a4f03d2a8da2ea076ce67 MV_RadarIcons_3612.zip http://mvp.fsmods.net/3612/MV_RadarIcons_3612.zip
a2ed156ffdb055a9466c206771cc4854 MV_Root_3612.zip http://mvp.fsmods.net/3612/MV_Root_3612.zip
0ef08f10dcf8cbbb3640329de64bea17 MV_Root_Update.zip http://mvp.fsmods.net/3612/MV_Root_Update.zip


What i'm planning to do next :
1) the download screen (show what's downloading, the download speed and basic info using wget, being able to resume a previous download)
2) "vp" and "download" will be in the same zip file, so users will only have to downoad this file, launch it with the installer, and the install process should be automated from there
3) this will integrate nicely with the GOG extractor script.

The end process will be :
- get the fs2 bash installer
- tell it where your GOG installer is, wait for a bit
- tell it if you want a stable or a nightly buid of fs2_open, this step should be quick
- tell it where your MediaVPs net installer is, wait for a bit longer
- play !

EDIT : I just found out how to perform a search inside a compressed file in bash, so we don't have to know the exact structure of an archive to find only what we need : the VP file. This will play nicely with any mod using zip , tar.bz or 7zip .

And as you can see it's fairly easy to create a mod descriptor usable with this script. You just have to run the md5sum on the zip files folder, then on the VP files folder, append the corresponding archive name every VP file name (this way we can handle archives that have multiple VPs inside), append the download urls to the zip list... Done !

[Bullhorn]

This is just great.

Trés bon!

This was a glaring gap in the whole FSO installation process.

Bh

[Hellzed]

I've just updated the script, now it handles downloading files, creating a cache, and unpacking the whole stuff.

Now it needs a lot of polishing, including checking if directories exist, if we can write to these...
Then i'll add user input : an "interactive mode" to leave the choice of re-downloading/installing to the user, or force everything.

2 questions now :
- Do you think the cache should stay in the FS2 root folder, or be stored in a hidden folder in the $HOME folder ? (i think the second one would be the best way, so it wouldn't appear wxlauncher mod management as a mod directory)

- Do you think it should re-check integrity of downloaded files ?/newly installed VPs ? (and retry the process if somethink is broken ?)

Now i'm seriously thinking about completely reorganising the code of the full installer script so it could be used as a command line tool (no pseudo GUI in a console), and make 2 utilities : a console installer which would use the command line, and a simple GUI for the same command line.
I'm wondering how this script could be ported to Windows too... Could anyone try it on Mac OS X ?