Author Topic: Possibly breaking filesystem change  (Read 5508 times)

0 Members and 1 Guest are viewing this topic.

Offline m!m

  • 211
Possibly breaking filesystem change
A recent change has introduced a new filesystem check that validates if the case of the name of the data directories is correct as defined by the engine. This does only affect development files and not VP files so the number of affected mods should be relatively small (I know that the development version of Diaspora has this issue).

The reason this check has been introduced is that while such a configuration will work on Windows, it will not work with Linux or Mac (or any OS which has a case sensitive file system). To fix it, you simply have to rename all the data directories to use the lower case version of the name (you might need to rename it twice since earlier version of Windows doesn't like renaming files where the old and new name only have case differences).

 

Offline Spoon

  • 212
  • ヾ(´︶`♡)ノ
Re: Possibly breaking filesystem change
First off, thanks for making a thread. Not being informed about changes has been one of my biggest irks with the scp.

To clarify, so I got this straight. Because Unix is a OS for dirty peasants, the data folder can't start with a delicious capital D, but it has to be lower case? What about every other folder inside the data folder? All my folders start with caps, because my ocd demands it.

TBH I don't quite agree with the notion that breaking mods for this is 'justified'. I don't develop with unix, and if the game doesn't care either way when the files are in a VP, then why force this change upon us windows using devs? I don't really mean to make a big issue out of this or anything, I just find it kind of annoying.
Urutorahappī!!

[02:42] <@Axem> spoon somethings wrong
[02:42] <@Axem> critically wrong
[02:42] <@Axem> im happy with these missions now
[02:44] <@Axem> well
[02:44] <@Axem> with 2 of them

 
Re: Possibly breaking filesystem change
Unix users still unpack VPs for development sometimes, and it's terrible user experience if that makes everything break.
The good Christian should beware of mathematicians, and all those who make empty prophecies. The danger already exists that the mathematicians have made a covenant with the devil to darken the spirit and to confine man in the bonds of Hell.

 

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: Possibly breaking filesystem change
I understand it is a pain, but the pain on your side is rename some files, the pain on a case-sensitive filesystem user's end is not being able to run the content when not in a VP file.  Some mods release without VPs, etc.  Telling people that this shouldn't be done because some users will have breakage seems like the best solution to me, in this case.  From a developer standpoint, we have to try to avoid existing within our own bubble and realized this is a project used by users, some in a minority, of many different platforms.  Maintaining good compatibility with those platforms is often a big draw of talent to the development pool, as there are many Linux and Mac users out there that look for projects that cater to more than the PC Master Race :)
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline Spoon

  • 212
  • ヾ(´︶`♡)ノ
Re: Possibly breaking filesystem change
I understand it is a pain, but the pain on your side is rename some files, the pain on a case-sensitive filesystem user's end is not being able to run the content when not in a VP file.
Whats stopping the case-sensitive filesystem user from renaming the content when its not in a vp file?
Actually, renaming these thing is going to be a really annoying pain in the ass, because it breaks the svn folder structure. 
I really wish we could like, not do this. This has been a known issue for years now, why change this now all of the sudden? Or like, why not just have this dead stop error message on just linux and mac builds? I don't see why it would need to affect windows builds too.

Also, still waiting on that clarification if its just the data folder or every folder.
« Last Edit: February 07, 2017, 02:05:23 pm by Spoon »
Urutorahappī!!

[02:42] <@Axem> spoon somethings wrong
[02:42] <@Axem> critically wrong
[02:42] <@Axem> im happy with these missions now
[02:44] <@Axem> well
[02:44] <@Axem> with 2 of them

 

Offline The E

  • He's Ebeneezer Goode
  • Moderator
  • 213
  • Nothing personal, just tech support.
    • Steam
    • Twitter
Re: Possibly breaking filesystem change
I think Spoon has a good point here: an Error might be overkill in this case. For the purposes of this, a one-time warning whenever a given directory is accessed (or even just a log print) should be sufficient.
If I'm just aching this can't go on
I came from chasing dreams to feel alone
There must be changes, miss to feel strong
I really need lifе to touch me
--Evergrey, Where August Mourns

 
Re: Possibly breaking filesystem change
It seems like the only reason for doing that would be so that users can indefinitely delay fixing their development files.

Spoon: why is renaming directories in SVN such a hassle?
The good Christian should beware of mathematicians, and all those who make empty prophecies. The danger already exists that the mathematicians have made a covenant with the devil to darken the spirit and to confine man in the bonds of Hell.

 

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: Possibly breaking filesystem change
Well, not all mods have packaged their files, is one reason an error might be preferred to a warning.  If all mods ever were distributed in VP files and end users never had to mess with it, we could just log a warning.  But that's not the case.  Expecting users of a mod to go through the hassle of renaming files that they wanted to download and play with (every user that runs into the issue) vs the developer of a mod just doing a one time fix, seems like putting the burden of work in the wrong place doesn't it Spoon?  I don't understand the fix in question enough to know if it's all folders or just data, sorry.
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline Spoon

  • 212
  • ヾ(´︶`♡)ノ
Re: Possibly breaking filesystem change
I think Spoon has a good point here: an Error might be overkill in this case. For the purposes of this, a one-time warning whenever a given directory is accessed (or even just a log print) should be sufficient.
Yes, thank you. Just a log print should be enough.
Don't throw unnecessary warning messages or literally stop me from developing with an error message because I'm not developing in a unix compatitible way, which is literally irrelevant to me as the dev, and barely relevant to the end user because the end user is unaffected either way because of the VP's. And if they want to go ahead and unpack for whatever reason and do development things, then they can rename the folders themselves, surely?
This is really kind of a dumb thing to force this way, guys.

PH: Because the svn is case-sensitive, and since all my folders have caps, it means I have to delete all the old folders and files, then do a whole new checkout on every single file , and upload several gigs at the amazing speed of 80kb/s, because thats as fast as the svn allows me to upload, ever.
No thanks.

Well, not all mods have packaged their files, is one reason an error might be preferred to a warning.  If all mods ever were distributed in VP files and end users never had to mess with it, we could just log a warning.  But that's not the case.  Expecting users of a mod to go through the hassle of renaming files that they wanted to download and play with (every user that runs into the issue) vs the developer of a mod just doing a one time fix, seems like putting the burden of work in the wrong place doesn't it Spoon?  I don't understand the fix in question enough to know if it's all folders or just data, sorry.
Effectively all major mods ever have been distributed in VP files. The only one I can think of that has not done that, is Dimensional Eclipse, and DE shipped its files with lower case so its still not relevant. Making this into error (or a reoccuring warning) only serves to **** me over. Because please, feel free to point me to any recent relevant case in which an end user on a mac or linux system has ran into the problem of a mod they tried to run having a Data folder, and thus being unable to play it.
I bet you can't, because its not a common occurence at all. Because as I just said, effectively all mods, that are campaigns, get distributed in vp's.

This has effectively never been an issue of end users getting ****ed by a capitalized Data folder. This is just a thing that is randomly decided to be forced because it came up as an issue with the Diaspora dev team on github recently. And now Linux modders go "Yeah! We should totally force in the most heavy handed way possible because reasons!"
No, don't. This is dumb.
« Last Edit: February 07, 2017, 02:31:17 pm by Spoon »
Urutorahappī!!

[02:42] <@Axem> spoon somethings wrong
[02:42] <@Axem> critically wrong
[02:42] <@Axem> im happy with these missions now
[02:44] <@Axem> well
[02:44] <@Axem> with 2 of them

 

Offline m!m

  • 211
Re: Possibly breaking filesystem change
What about every other folder inside the data folder? All my folders start with caps, because my ocd demands it.
Every folder has to use the case that the engine expects. Currently, the engine expects all directories to use the lower case names.

I think Spoon has a good point here: an Error might be overkill in this case. For the purposes of this, a one-time warning whenever a given directory is accessed (or even just a log print) should be sufficient.
Yes, thank you. Just a log print should be enough.
Don't throw unnecessary warning messages or literally stop me from developing with an error message because I'm not developing in a unix compatitible way, which is literally irrelevant to me as the dev, and barely relevant to the end user because the end user is unaffected either way because of the VP's. And if they want to go ahead and unpack for whatever reason and do development things, then they can rename the folders themselves, surely?
This is really kind of a dumb thing to force this way, guys.
The reason I chose to make this an Error is that on Unix a misnamed data directory breaks the game. FSO can't be run with those directory names on those platforms so we have to make it an Error there. Since we generally try to make all Warnings and Errors  consistent across all platforms I decided that Windows should also use an Error. If it's actually an issue to just rename the directories then we could make it a Warning or a log print on Windows.

However, you shouldn't ignore that Warning or Error since it would make it impossible for someone on Linux to run your development files. If, for example, you found an engine bug in your mod and wanted me to fix it using your development files, I wouldn't bother looking at the issue simply because I can't run your mod. It's in your own best interest to use the folder names that have the best compatibility across platforms but if it's a huge deal for mod developers we could lower the severity on Windows to a Warning. A log message would probably also work but most people just ignore that so it's less than ideal but considering that you would probably encounter ~20 Warnings if all your directories are misnamed it may be our only solution here.

This is just a thing that is randomly decided to be forced because it came up as an issue with the Diaspora dev team on github recently. And now Linux modders go "Yeah! We should totally force in the most heavy handed way possible because reasons!"
No, don't. This is dumb.
No one "forced" this change. I submitted this because I identified an issue and thought that it would not affect anyone. That's exactly why we have nightly builds so that we can quickly determine that a recent change broke the game for someone.

 

Offline AdmiralRalwood

  • 211
  • The Cthulhu programmer himself!
    • Skype
    • Steam
    • Twitter
Re: Possibly breaking filesystem change
I would personally prefer that FSO on case-sensitive filesystems just be willing to read from any capitalization of the folder structure. Don't we already do this for mod folders because of the MediaVPs?
Ph'nglui mglw'nafh Codethulhu GitHub wgah'nagl fhtagn.

schrödinbug (noun) - a bug that manifests itself in running software after a programmer notices that the code should never have worked in the first place.

When you gaze long into BMPMAN, BMPMAN also gazes into you.

"I am one of the best FREDders on Earth" -General Battuta

<Aesaar> literary criticism is vladimir putin

<MageKing17> "There's probably a reason the code is the way it is" is a very dangerous line of thought. :P
<MageKing17> Because the "reason" often turns out to be "nobody noticed it was wrong".
(the very next day)
<MageKing17> this ****ing code did it to me again
<MageKing17> "That doesn't really make sense to me, but I'll assume it was being done for a reason."
<MageKing17> **** ME
<MageKing17> THE REASON IS PEOPLE ARE STUPID
<MageKing17> ESPECIALLY ME

<MageKing17> God damn, I do not understand how this is breaking.
<MageKing17> Everything points to "this should work fine", and yet it's clearly not working.
<MjnMixael> 2 hours later... "God damn, how did this ever work at all?!"
(...)
<MageKing17> so
<MageKing17> more than two hours
<MageKing17> but once again we have reached the inevitable conclusion
<MageKing17> How did this code ever work in the first place!?

<@The_E> Welcome to OpenGL, where standards compliance is optional, and error reporting inconsistent

<MageKing17> It was all working perfectly until I actually tried it on an actual mission.

<IronWorks> I am useful for FSO stuff again. This is a red-letter day!
* z64555 erases "Thursday" and rewrites it in red ink

<MageKing17> TIL the entire homing code is held up by shoestrings and duct tape, basically.

 

Offline m!m

  • 211
Re: Possibly breaking filesystem change
I already implemented that but I was told that we don't actually support upper case folder names so instead I built the current solution. I am not sure if the solution I presented in that PR is actually working since I needed to make some changes to the core CFile data structures.

Mods are different because they are handled by the command line system before CFile is initialized.

 

Offline Spoon

  • 212
  • ヾ(´︶`♡)ノ
Re: Possibly breaking filesystem change
The reason I chose to make this an Error is that on Unix a misnamed data directory breaks the game. FSO can't be run with those directory names on those platforms so we have to make it an Error there. Since we generally try to make all Warnings and Errors  consistent across all platforms I decided that Windows should also use an Error. If it's actually an issue to just rename the directories then we could make it a Warning or a log print on Windows.
Yes, I am aware that it breaks the game for Unix. What I can't understand the logic of, is that, this has always been the case, so why suddenly break the game for windows? This literally accomplishes nothing. It just breaks the game for one additional platform now.

However, you shouldn't ignore that Warning or Error since it would make it impossible for someone on Linux to run your development files. If, for example, you found an engine bug in your mod and wanted me to fix it using your development files, I wouldn't bother looking at the issue simply because I can't run your mod. It's in your own best interest to use the folder names that have the best compatibility across platforms but if it's a huge deal for mod developers we could lower the severity on Windows to a Warning. A log message would probably also work but most people just ignore that so it's less than ideal but considering that you would probably encounter ~20 Warnings if all your directories are misnamed it may be our only solution here.
Yes, I'll keep this in mind for the first time it will happen that a Unix coders will go through the effort to try and fix a WoD bug. Usually case-sensitivity has not been the roadblock here.
Since in all my years developing WoD, the count of coders looking through WoD files has been pretty abysmally low. I usually just resign myself in having to create some test case in retail. But hey, I'll keep it in mind, who knows if this scenario will ever happen. Hell, I couldn't even get a mac build for WoD when I asked for it, repeatedly.
Either way, a log warning or just have fso not be picky about case-sensitivity would be very preferable over breaking the windows build.

No one "forced" this change. I submitted this because I identified an issue and thought that it would not affect anyone. That's exactly why we have nightly builds so that we can quickly determine that a recent change broke the game for someone.
I meant forced as in, this thing being forced upon me.
Urutorahappī!!

[02:42] <@Axem> spoon somethings wrong
[02:42] <@Axem> critically wrong
[02:42] <@Axem> im happy with these missions now
[02:44] <@Axem> well
[02:44] <@Axem> with 2 of them

 

Offline AdmiralRalwood

  • 211
  • The Cthulhu programmer himself!
    • Skype
    • Steam
    • Twitter
Re: Possibly breaking filesystem change
I was told that we don't actually support upper case folder names
Except FSO handles it inside VP files just fine, so clearly we do support it; we could just extend that support outside of a VP archive.
Ph'nglui mglw'nafh Codethulhu GitHub wgah'nagl fhtagn.

schrödinbug (noun) - a bug that manifests itself in running software after a programmer notices that the code should never have worked in the first place.

When you gaze long into BMPMAN, BMPMAN also gazes into you.

"I am one of the best FREDders on Earth" -General Battuta

<Aesaar> literary criticism is vladimir putin

<MageKing17> "There's probably a reason the code is the way it is" is a very dangerous line of thought. :P
<MageKing17> Because the "reason" often turns out to be "nobody noticed it was wrong".
(the very next day)
<MageKing17> this ****ing code did it to me again
<MageKing17> "That doesn't really make sense to me, but I'll assume it was being done for a reason."
<MageKing17> **** ME
<MageKing17> THE REASON IS PEOPLE ARE STUPID
<MageKing17> ESPECIALLY ME

<MageKing17> God damn, I do not understand how this is breaking.
<MageKing17> Everything points to "this should work fine", and yet it's clearly not working.
<MjnMixael> 2 hours later... "God damn, how did this ever work at all?!"
(...)
<MageKing17> so
<MageKing17> more than two hours
<MageKing17> but once again we have reached the inevitable conclusion
<MageKing17> How did this code ever work in the first place!?

<@The_E> Welcome to OpenGL, where standards compliance is optional, and error reporting inconsistent

<MageKing17> It was all working perfectly until I actually tried it on an actual mission.

<IronWorks> I am useful for FSO stuff again. This is a red-letter day!
* z64555 erases "Thursday" and rewrites it in red ink

<MageKing17> TIL the entire homing code is held up by shoestrings and duct tape, basically.

 

Offline m!m

  • 211
Re: Possibly breaking filesystem change
Except FSO handles it inside VP files just fine, so clearly we do support it; we could just extend that support outside of a VP archive.
I have written the code to support different cases on Linux so with a bit of testing we could use that if the consensus is that we want to support that case.

 

Offline Spoon

  • 212
  • ヾ(´︶`♡)ノ
Re: Possibly breaking filesystem change
I just don't get why the first line of thought was: "We've got 2 platforms this is bugged on and 1 platform on which this is not. Let's make sure it's not going to work on all 3 platforms."
Rather than: "We've got 2 platforms this is bugged on and 1 platform on which this is not. Let's make sure it's going to work on all 3 platforms."  :confused:
Urutorahappī!!

[02:42] <@Axem> spoon somethings wrong
[02:42] <@Axem> critically wrong
[02:42] <@Axem> im happy with these missions now
[02:44] <@Axem> well
[02:44] <@Axem> with 2 of them

 
Re: Possibly breaking filesystem change
Trying to overlay a case-insensitive filesystem over a case-sensitive one is a bit of a kludge and liable to lead to interesting edge cases, I guess. It sounds like we already have a pretty good implementation of it though?
The good Christian should beware of mathematicians, and all those who make empty prophecies. The danger already exists that the mathematicians have made a covenant with the devil to darken the spirit and to confine man in the bonds of Hell.

 

Offline karajorma

  • King Louie - Jungle VIP
  • Administrator
  • 214
    • Karajorma's Freespace FAQ
Re: Possibly breaking filesystem change
I'm going to have to agree with Spoon on this one. Having a warning pop up on debug builds is one thing but forcing every single developer who uses SVN to rename their folders and then go through the massive hassle of checking in / out all the changes on SVN is probably the wrong way to go about this.
Karajorma's Freespace FAQ. It's almost like asking me yourself.

[ Diaspora ] - [ Seeds Of Rebellion ] - [ Mind Games ]

 

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: Possibly breaking filesystem change
Just out of curiosity, is the svn headache really all that bad?  As I understand it, a few 'svn mv' commands are all that would be required, issued on the directories themselves, not individual files.   According to http://stackoverflow.com/questions/3941291/a-sane-way-to-rename-a-directory-in-subversion-working-copy anyway, it makes it look pretty painless.
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline mjn.mixael

  • Cutscene Master
  • 212
  • Chopped liver
    • Steam
    • Twitter
Re: Possibly breaking filesystem change
Just out of curiosity, is the svn headache really all that bad?  As I understand it, a few 'svn mv' commands are all that would be required, issued on the directories themselves, not individual files.   According to http://stackoverflow.com/questions/3941291/a-sane-way-to-rename-a-directory-in-subversion-working-copy anyway, it makes it look pretty painless.

I tried that one time to hopefully mitigate a significant file restructure... it still re-uploaded all 2GB of data in the folder I was working on.
Cutscene Upgrade Project - Mainhall Remakes - Between the Ashes
Youtube Channel - P3D Model Box
Between the Ashes is looking for committed testers, PM me for details.
Freespace Upgrade Project See what's happening.