Hard Light Productions Forums

Modding, Mission Design, and Coding => FS2 Open Coding - The Source Code Project (SCP) => Test Builds => Topic started by: The E on October 31, 2015, 04:44:57 am

Title: Large-Address-Aware builds
Post by: The E on October 31, 2015, 04:44:57 am
One of the problems we have on Windows is that FSO on Win is a 32-bit application. This restricts the available memory space to 2GB, and chances are that there are scenarios where our memory usage goes beyond that. Making the engine fully 64-bit-compatible takes some doing, but it is possible to set a flag called "Large Address Aware" that allows us to address up to 4GB of memory.

Please test these builds especially if you frequently encounter "malloc failed" errors.

https://www.dropbox.com/s/yab58duapqyku8j/FS2_Open_3_7_3_Large_Address_Aware.7z?dl=0
Title: Re: Large-Address-Aware builds
Post by: General Battuta on October 31, 2015, 07:32:21 am
This is all Aesaar's fault.
Title: Re: Large-Address-Aware builds
Post by: Dragon on October 31, 2015, 07:37:55 am
Well, Steve-O and Nighteyes had their part in it, too. :) I'll try it out.
Title: Re: Large-Address-Aware builds
Post by: chief1983 on October 31, 2015, 10:18:30 am
Hey, I had never heard that 32bit apps were limited to 2GB until doing some investigation of my own.  Everyone was all, FSO usually peaks around 2GB so it must not be hitting limits or anything, and then I was like, hmm, apparently 32bit apps _are_ limited to 2GB by default...so no we're seeing what it takes to ensure 32bit builds can reach the 4GB boundary (or ~3GB or so on 32bit Windows, with a tweak to the Windows settings).
Title: Re: Large-Address-Aware builds
Post by: snake60 on October 31, 2015, 06:09:23 pm
If you're talking about the /3GB switch on winXP 32bits, be careful with it.
In the past (10 years ago I would say) I had a laptop with this kind of configuration at work (xp 32bits + 4gb ram + /3GB switch) which I used for Solidworks and I had a lot of problems with the nvidia drivers crashing randomly or simply not starting when the /3GB was used.
For what I can remember, it was linked to the fact that those drivers were loaded in a very specific adress in the memory and were causing all kind of strange and random behaviors, like 800x600 16bit color desktop at windows start or when applications were trying to access those adresses to use more than 2GB.
Sorry not to be able to be more precise than that (it's oooold), but it was really a pain in the butt at the time, so just so you know !
Title: Re: Large-Address-Aware builds
Post by: Dragon on October 31, 2015, 06:46:01 pm
Nope, that's another thing entirely. This is about the program being made aware of the fact that there can be more than 2GB available. Yes, if you're running XP with 3GB switch, it'd make FSO capable of actually using the extra memory, but it won't mess with memory settings of the system itself. Those are separate (if tangentially related) issues. It should have no impact on drivers or anything like that.
Title: Re: Large-Address-Aware builds
Post by: snake60 on October 31, 2015, 09:26:52 pm
Yeah, I know ! It should have been exactly the same logic on my old laptop, the /3GB switch was just here for solidworks to be able to use more than 2GB. But the nvidia drivers were crashing randomly or not even starting at all as soon as the /3GB switch was used. I can't explain why to you, I'm just telling you this so you don't oversee this potential issue for xp32 users who would like to use this possibility after it had been integrated.

Edit :
I've found a related article on the microsoft support site : https://support.microsoft.com/en-us/kb/319043 (https://support.microsoft.com/en-us/kb/319043)
It's explaining the problem, and the use of an additionnal switch /USERVA which can define the limit more precisely to avoid it.

Edit2 : a guy explaining exactly the same problem : http://arstechnica.com/civis/viewtopic.php?f=17&t=369534 (http://arstechnica.com/civis/viewtopic.php?f=17&t=369534)
Title: Re: Large-Address-Aware builds
Post by: chief1983 on November 01, 2015, 07:58:35 am
Dragon, he's right in that if we were to make LAA default on our builds, 32bit systems would still need the 3GB switch to allow FSO more than 2GB, which is the tweak I was referring to in parentheses in my previous post.  But it is true that setting can have unintended conequences, and running FSO with more than 2GB memory may only be reliable on 64bit platforms.
Title: Re: Large-Address-Aware builds
Post by: Phantom Hoover on November 01, 2015, 08:23:45 am
If you're running on a 32-bit system I suppose the drivers might not themselves be large address aware (or indeed might have needed the extra memory), whereas on a 64-bit system they'd just use the entire address space. LAA seems like a hack for 32-bit applications on 64-bit systems, I'm not surprised it's risky elsewhere.
Title: Re: Large-Address-Aware builds
Post by: chief1983 on November 01, 2015, 09:08:57 am
A carefully audited 32bit system might have been able to get away with it, perhaps in a server role, but yeah a daily user's system just might not work well with it.
Title: Re: Large-Address-Aware builds
Post by: jr2 on November 01, 2015, 03:29:25 pm
Would it be possible to easily detect the system configuration and, if x86 and proper settings aren't in place, return an error: "Warning, you don't have <insert necessary Windows settings hacks>, please post on the forums for assistance"?
Title: Re: Large-Address-Aware builds
Post by: Fury on November 02, 2015, 12:29:19 am
Those hacks should not be forced on people with 32-bit OS because they are not guaranteed to work reliably. Those people should instead stay away from high-res texture/effects packs to keep memory usage under 2GB.

Or you know, keep up with the times and upgrade to 64-bit OS.
Title: Re: Large-Address-Aware builds
Post by: The E on November 02, 2015, 12:44:02 am
That's my thinking too. Sooner or later, it is inevitable that mods will be made that require more than 2GB RAM. Given that the LAA flag can only cause problems in a scenario that is becoming more unlikely every day, I see no reason not to include it by default.
Title: Re: Large-Address-Aware builds
Post by: chief1983 on November 03, 2015, 10:47:33 am
It can definitely cause problems if our code isn't actually Large Address Aware.  Running this through the tests I posted earlier, by reversing the Windows memory management model and seeing if we run into issues, is the minimum that should be done before this were to become the default.
Title: Re: Large-Address-Aware builds
Post by: The E on November 03, 2015, 11:11:28 am
Based on the guides you posted, we should not be in any danger.

What do you mean by "reversing the Windows memory management model"?
Title: Re: Large-Address-Aware builds
Post by: chief1983 on November 03, 2015, 11:26:55 am
One of the SO links (http://stackoverflow.com/questions/2288728/drawbacks-of-using-largeaddressaware-for-32-bit-windows-executables) I posted in the other thread where we discussed this, mentioned forcing Windows to use a Top Down memory model via a registry setting.  Doing this would apparently ensure that hidden issues with LAA would surface 'almost immediately' instead of 'eventually'.
Title: Re: Large-Address-Aware builds
Post by: Phantom Hoover on November 09, 2015, 02:33:10 pm
I was just chatting about this on #scp, apparently FSO can be compiled and run as a 64-bit executable on Windows, it just hasn't been tested much. Seems like you'd be much, much better off focusing efforts on that than on LAA builds.
Title: Re: Large-Address-Aware builds
Post by: Bryan See on November 10, 2015, 10:35:10 am
What about testing Large-Address-Aware builds on Windows 10 (future updates included)? EDIT: What about higher RAM memory (16 GB)?
Title: Re: Large-Address-Aware builds
Post by: chief1983 on November 10, 2015, 11:50:55 am
FSO would only be using ~4GB (maybe only 3.5GB?) on a 64bit OS with an LAA build, no matter how much more memory is available.  However, it seemed like testing that LAA expands us from 2 to 4GB isn't all that hard to do, and any issue locations that crop up probably should be addressed, as it's likely where a size_t should be used in place of an int, etc.
Title: Re: Large-Address-Aware builds
Post by: Phantom Hoover on November 10, 2015, 02:07:47 pm
We already know that FSO doesn't have any word size issues outside of Windows platform code, because people have been using 64-bit Linux builds for years, and AdmiralRalwood is now distributing functional 64-bit Windows builds. Testing those and getting a futureproof solution seems a much better use of everyone's time than testing LAA builds and getting one extra gigabyte.
Title: Re: Large-Address-Aware builds
Post by: The E on November 10, 2015, 02:09:16 pm
Quite so.
Title: Re: Large-Address-Aware builds
Post by: Bryan See on September 24, 2016, 03:02:27 am
Have all the large-address-aware features committed to master branch already? I haven't yet to hear from them regarding this.
Title: Re: Large-Address-Aware builds
Post by: DahBlount on September 24, 2016, 03:15:00 am
afaik, LAA is largely irrelevant since 64-bit is being provided in the nightlies these days.
Title: Re: Large-Address-Aware builds
Post by: m!m on September 24, 2016, 03:56:39 am
afaik, LAA is largely irrelevant since 64-bit is being provided in the nightlies these days.
Exactly, 64-bit builds are to proper solution for dealing with out-of-memory errors.
Title: Re: Large-Address-Aware builds
Post by: Trivial Psychic on September 24, 2016, 07:57:41 am
That, and switch to water after you've had two beers.  ;)
Title: Re: Large-Address-Aware builds
Post by: AdmiralRalwood on September 24, 2016, 06:53:50 pm
As far as I am aware, 32-bit builds made of current master remain non-LAA, but it's irrelevant because anyone who could benefit from LAA can just use 64-bit instead.
Title: Re: Large-Address-Aware builds
Post by: jr2 on September 25, 2016, 07:24:09 am
Yeah, there's precious little reason to not use 64-bit OSes (and therefore benefit from 64-bit builds) other than: a) you're somehow still on Windows XP and don't have good XP 64-bit drivers or b) you're somehow stuck on an ancient 32-bit processor  or c) you're somehow stuck with less than ~3GB of RAM.

So, the reasons for not using 64-bit builds can be summed up with: your system is outdated.
Title: Re: Large-Address-Aware builds
Post by: chief1983 on September 25, 2016, 10:36:41 am
That's not necessarily true.  Many windows 7 machines that could run fso were sold with 32 bit windows, usually at or less than 4gb ram, I guess do get rid of old licenses or something.  They have the hardware but not the OS to run it.  I don't know if that's still going on with windows 10 but those users may have still upgraded and therefore be running otherwise stable up to date platforms.
Title: Re: Large-Address-Aware builds
Post by: jr2 on September 25, 2016, 12:05:24 pm
That's not necessarily true.  Many windows 7 machines that could run fso were sold with 32 bit windows, usually at or less than 4gb ram, I guess do get rid of old licenses or something.  They have the hardware but not the OS to run it.  I don't know if that's still going on with windows 10 but those users may have still upgraded and therefore be running otherwise stable up to date platforms.

Quote from: http://answers.microsoft.com/en-us/windows/forum/windows_7-windows_install/is-a-windows-7-license-key-valid-for-both-32-bit/70d546cd-b6e3-44d8-a6c8-fd7feb7d1915
Quote
I received a NFR copy of Windows 7 Ultimate at the Microsoft launch today.  It has a valid (I hope!) license key but the DVD is for 32-bit.  Is the same license key valid for 64-bit installations as well?  In other words, if I get a copy of the 64-bit DVD, can I use the license key I just received?

This will be a fresh install onto a system that currently has 32-bit Vista.

Thanks,
   .../Ed

Quote
Yes, it is valid for either 32 or 64 bit, its up to you which architecture you want to use. Please note 1 key, 1 license. You can't install 32 bit one computer and install 64 bit on another computer using the same key.

Quote
Quote
Now that I know Win 7 64 bit works great with my hardware.  Can I legally upgrade my Windows Vista 32 bit to Windows 7 32 bit?  And keep both versions of Windows 7 on the same machine.
If you have one Windows 7 license you can only have one install of Windows 7.
To install Windows 7 32bit you will first need to format/delete the 64 bit install.
 
You will need to buy a second license/key to install Windows 7 twice at the same time. There is no discount on a second license if you already have a license.

http://store.microsoft.com/microsoft/Windows-Windows-7/category/102

Quote
Yes, it is valid for either 32 or 64 bit, its up to you which architecture you want to use. Please note 1 key, 1 license. You can't install 32 bit one computer and install 64 bit on another computer using the same key.


TL;DR: You are licensed for both architectures, but not both simultaneously.  User would need to dump their 32-bit install and use the same key to install 64-bit.
Title: Re: Large-Address-Aware builds
Post by: chief1983 on September 25, 2016, 01:02:50 pm
We can't just expect our user base to reformat to play FS2.  It shouldn't be necessary as 32bit windows machines are still on the market.
Title: Re: Large-Address-Aware builds
Post by: m!m on September 25, 2016, 03:07:16 pm
We are not requiring anyone to reinstall another version of Windows since the 32-bit builds work fine in most cases. However, if people want to use more advanced features that require more memory then 64-bit builds are the only viable solution. Furthermore, enabling LAA could introduce bugs that are not present in the current 32-bit builds and given that most people can use the 64-bit builds I see no good reason to enable LAA.
Title: Re: Large-Address-Aware builds
Post by: chief1983 on September 25, 2016, 03:10:34 pm
Really, since it can be set after the fact, we might just want some instructions on how to do that and be done with it.
Title: Re: Large-Address-Aware builds
Post by: m!m on September 26, 2016, 03:40:42 am
Well, if it's that easy then we could just enable it in the nightlies and let people test it. However, I still think that we don't need to do that since the issues that should be reduced by this are already fixed by 64-bit builds.
Title: Re: Large-Address-Aware builds
Post by: The E on September 26, 2016, 04:58:32 am
I would agree with m!m here.
Title: Re: Large-Address-Aware builds
Post by: jr2 on September 26, 2016, 08:10:19 am
You could just point them to the utility over here (https://www.techpowerup.com/forums/threads/large-address-aware.112556/) if they wanted to enable LAA.  As has been said before, though, that's probably opening a can of worms, and that fact should be mentioned in big, red, flaming letters, along with the disclaimer that we don't support LAA-enabled x86 builds.  Those stuck on x86 and not willing to reformat might want to try it for the ability to use the advanced features without reformatting.
Title: Re: Large-Address-Aware builds
Post by: The E on September 26, 2016, 08:22:38 am
We do not support LAA. We will also not point people towards utilities to set that flag.

End of discussion.