Author Topic: Large-Address-Aware builds  (Read 2789 times)

0 Members and 1 Guest are viewing this topic.

Offline The E

  • He's Ebeneezer Goode
  • Global Moderator
  • 213
  • Nothing personal, just tech support.
    • Skype
    • Steam
    • Twitter
Large-Address-Aware builds
One of the problems we have on Windows is that FSO on Win is a 32-bit application. This restricts the available memory space to 2GB, and chances are that there are scenarios where our memory usage goes beyond that. Making the engine fully 64-bit-compatible takes some doing, but it is possible to set a flag called "Large Address Aware" that allows us to address up to 4GB of memory.

Please test these builds especially if you frequently encounter "malloc failed" errors.

https://www.dropbox.com/s/yab58duapqyku8j/FS2_Open_3_7_3_Large_Address_Aware.7z?dl=0
**** every cause that ends in murder and children crying. ― Iain Banks
Join the fun at the HLP IRC channel. Get the latest spam and gossip as long as it's fresh!

 

Offline General Battuta

  • Poe's Law In Action
  • 214
  • i wonder when my postcount will exceed my iq
Re: Large-Address-Aware builds
This is all Aesaar's fault.

 

Offline Dragon

  • Citation needed
  • 212
  • The sky is the limit.
Re: Large-Address-Aware builds
Well, Steve-O and Nighteyes had their part in it, too. :) I'll try it out.

 

Online chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️À➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: Large-Address-Aware builds
Hey, I had never heard that 32bit apps were limited to 2GB until doing some investigation of my own.  Everyone was all, FSO usually peaks around 2GB so it must not be hitting limits or anything, and then I was like, hmm, apparently 32bit apps _are_ limited to 2GB by default...so no we're seeing what it takes to ensure 32bit builds can reach the 4GB boundary (or ~3GB or so on 32bit Windows, with a tweak to the Windows settings).
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline snake60

  • 25
Re: Large-Address-Aware builds
If you're talking about the /3GB switch on winXP 32bits, be careful with it.
In the past (10 years ago I would say) I had a laptop with this kind of configuration at work (xp 32bits + 4gb ram + /3GB switch) which I used for Solidworks and I had a lot of problems with the nvidia drivers crashing randomly or simply not starting when the /3GB was used.
For what I can remember, it was linked to the fact that those drivers were loaded in a very specific adress in the memory and were causing all kind of strange and random behaviors, like 800x600 16bit color desktop at windows start or when applications were trying to access those adresses to use more than 2GB.
Sorry not to be able to be more precise than that (it's oooold), but it was really a pain in the butt at the time, so just so you know !

 

Offline Dragon

  • Citation needed
  • 212
  • The sky is the limit.
Re: Large-Address-Aware builds
Nope, that's another thing entirely. This is about the program being made aware of the fact that there can be more than 2GB available. Yes, if you're running XP with 3GB switch, it'd make FSO capable of actually using the extra memory, but it won't mess with memory settings of the system itself. Those are separate (if tangentially related) issues. It should have no impact on drivers or anything like that.

 

Offline snake60

  • 25
Re: Large-Address-Aware builds
Yeah, I know ! It should have been exactly the same logic on my old laptop, the /3GB switch was just here for solidworks to be able to use more than 2GB. But the nvidia drivers were crashing randomly or not even starting at all as soon as the /3GB switch was used. I can't explain why to you, I'm just telling you this so you don't oversee this potential issue for xp32 users who would like to use this possibility after it had been integrated.

Edit :
I've found a related article on the microsoft support site : https://support.microsoft.com/en-us/kb/319043
It's explaining the problem, and the use of an additionnal switch /USERVA which can define the limit more precisely to avoid it.

Edit2 : a guy explaining exactly the same problem : http://arstechnica.com/civis/viewtopic.php?f=17&t=369534
« Last Edit: October 31, 2015, 10:11:35 pm by snake60 »

 

Online chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️À➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: Large-Address-Aware builds
Dragon, he's right in that if we were to make LAA default on our builds, 32bit systems would still need the 3GB switch to allow FSO more than 2GB, which is the tweak I was referring to in parentheses in my previous post.  But it is true that setting can have unintended conequences, and running FSO with more than 2GB memory may only be reliable on 64bit platforms.
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 
Re: Large-Address-Aware builds
If you're running on a 32-bit system I suppose the drivers might not themselves be large address aware (or indeed might have needed the extra memory), whereas on a 64-bit system they'd just use the entire address space. LAA seems like a hack for 32-bit applications on 64-bit systems, I'm not surprised it's risky elsewhere.
The good Christian should beware of mathematicians, and all those who make empty prophecies. The danger already exists that the mathematicians have made a covenant with the devil to darken the spirit and to confine man in the bonds of Hell.

 

Online chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️À➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: Large-Address-Aware builds
A carefully audited 32bit system might have been able to get away with it, perhaps in a server role, but yeah a daily user's system just might not work well with it.
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline jr2

  • The Mail Man
  • 212
  • It's prounounced jayartoo 0x6A7232
    • Steam
Re: Large-Address-Aware builds
Would it be possible to easily detect the system configuration and, if x86 and proper settings aren't in place, return an error: "Warning, you don't have <insert necessary Windows settings hacks>, please post on the forums for assistance"?

 

Offline Fury

  • The Curmudgeon
  • 213
Re: Large-Address-Aware builds
Those hacks should not be forced on people with 32-bit OS because they are not guaranteed to work reliably. Those people should instead stay away from high-res texture/effects packs to keep memory usage under 2GB.

Or you know, keep up with the times and upgrade to 64-bit OS.

 

Offline The E

  • He's Ebeneezer Goode
  • Global Moderator
  • 213
  • Nothing personal, just tech support.
    • Skype
    • Steam
    • Twitter
Re: Large-Address-Aware builds
That's my thinking too. Sooner or later, it is inevitable that mods will be made that require more than 2GB RAM. Given that the LAA flag can only cause problems in a scenario that is becoming more unlikely every day, I see no reason not to include it by default.
**** every cause that ends in murder and children crying. ― Iain Banks
Join the fun at the HLP IRC channel. Get the latest spam and gossip as long as it's fresh!

 

Online chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️À➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: Large-Address-Aware builds
It can definitely cause problems if our code isn't actually Large Address Aware.  Running this through the tests I posted earlier, by reversing the Windows memory management model and seeing if we run into issues, is the minimum that should be done before this were to become the default.
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline The E

  • He's Ebeneezer Goode
  • Global Moderator
  • 213
  • Nothing personal, just tech support.
    • Skype
    • Steam
    • Twitter
Re: Large-Address-Aware builds
Based on the guides you posted, we should not be in any danger.

What do you mean by "reversing the Windows memory management model"?
**** every cause that ends in murder and children crying. ― Iain Banks
Join the fun at the HLP IRC channel. Get the latest spam and gossip as long as it's fresh!

 

Online chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️À➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: Large-Address-Aware builds
One of the SO links I posted in the other thread where we discussed this, mentioned forcing Windows to use a Top Down memory model via a registry setting.  Doing this would apparently ensure that hidden issues with LAA would surface 'almost immediately' instead of 'eventually'.
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 
Re: Large-Address-Aware builds
I was just chatting about this on #scp, apparently FSO can be compiled and run as a 64-bit executable on Windows, it just hasn't been tested much. Seems like you'd be much, much better off focusing efforts on that than on LAA builds.
The good Christian should beware of mathematicians, and all those who make empty prophecies. The danger already exists that the mathematicians have made a covenant with the devil to darken the spirit and to confine man in the bonds of Hell.

 

Offline Bryan See

  • Has anyone really been far as decided to use even go want to do look more like?
  • 29
  • Visionary modder and coder
    • Skype
    • Steam
    • Twitter
Re: Large-Address-Aware builds
What about testing Large-Address-Aware builds on Windows 10 (future updates included)? EDIT: What about higher RAM memory (16 GB)?
« Last Edit: November 10, 2015, 11:52:08 am by Bryan See »
Bryan See - My FreeSpace Wiki User Page (Talk, Contributions)

Campaigns:
FreeSpace: Reunited - Shattered Stars

Ships:
GTS Hygeia, GTT Argo, SC Raguel

 

Online chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️À➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: Large-Address-Aware builds
FSO would only be using ~4GB (maybe only 3.5GB?) on a 64bit OS with an LAA build, no matter how much more memory is available.  However, it seemed like testing that LAA expands us from 2 to 4GB isn't all that hard to do, and any issue locations that crop up probably should be addressed, as it's likely where a size_t should be used in place of an int, etc.
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 
Re: Large-Address-Aware builds
We already know that FSO doesn't have any word size issues outside of Windows platform code, because people have been using 64-bit Linux builds for years, and AdmiralRalwood is now distributing functional 64-bit Windows builds. Testing those and getting a futureproof solution seems a much better use of everyone's time than testing LAA builds and getting one extra gigabyte.
The good Christian should beware of mathematicians, and all those who make empty prophecies. The danger already exists that the mathematicians have made a covenant with the devil to darken the spirit and to confine man in the bonds of Hell.