Author Topic: [3.7.2-ARM] Unaligned Memory Access at modeloctant.cpp:28 (title edited)  (Read 1677 times)

0 Members and 3 Guests are viewing this topic.

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: [3.7.2-ARM] Unaligned Memory Access at modeloctant.cpp:28 (title edited)
Well the good news is the performance gain alone may be enough incentive for a mass effort to fix all the old models floating around.
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline ShivanSpS

  • 210
Re: [3.7.2-ARM] Unaligned Memory Access at modeloctant.cpp:28 (title edited)
That is on ARM, i dont think x86 will have any performance gain im afraid. But i will check just to be sure.

At any rate, ARM is becoming more and more powerfull and common.

 

Offline ShivanSpS

  • 210
Re: [3.7.2-ARM] Unaligned Memory Access at modeloctant.cpp:28 (title edited)
I was just looking at FSO code and i found this on modelread on the SLDC case.

Code: [Select]
void swap_sldc_data(ubyte *buffer)
{
#if BYTE_ORDER == BIG_ENDIAN
        char *type_p = (char *)(buffer);
int *size_p = (int *)(buffer+1);
*size_p = INTEL_INT(*size_p);

// split and polygons
        vec3d *minbox_p = (vec3d*)(buffer+5);
vec3d *maxbox_p = (vec3d*)(buffer+17);

minbox_p->xyz.x = INTEL_FLOAT(&minbox_p->xyz.x);
minbox_p->xyz.y = INTEL_FLOAT(&minbox_p->xyz.y);
minbox_p->xyz.z = INTEL_FLOAT(&minbox_p->xyz.z);

maxbox_p->xyz.x = INTEL_FLOAT(&maxbox_p->xyz.x);
maxbox_p->xyz.y = INTEL_FLOAT(&maxbox_p->xyz.y);
maxbox_p->xyz.z = INTEL_FLOAT(&maxbox_p->xyz.z);


// split
        unsigned int *front_offset_p = (unsigned int*)(buffer+29);
unsigned int *back_offset_p = (unsigned int*)(buffer+33);

// polygons
unsigned int *num_polygons_p = (unsigned int*)(buffer+29);

unsigned int *shld_polys = (unsigned int*)(buffer+33);

if (*type_p == 0) // SPLIT
{
*front_offset_p = INTEL_INT(*front_offset_p);
*back_offset_p = INTEL_INT(*back_offset_p);
}
else
{
*num_polygons_p = INTEL_INT(*num_polygons_p);
for (unsigned int i = 0; i < *num_polygons_p; i++)
{
shld_polys[i] = INTEL_INT(shld_polys[i]);
}
}
#endif
}

PLEASE dont tell me that there is a size 1 char on the begining of the ubyte* making all pointers unaligned :(
memcpy can definately help here but that itself is considered a unaligned access, but one that, with a little luck the kernel can handle. Maybe is better to just disable SLDC on ARM with a command line switch.

Not sure what INTEL_INT and INTEL_FLOAT does here, and they are also used at boundbox that may be causing issues on release build, my guess is just coying data like memcpy would do but they do provide the correct size for the arch.
In that case a memcpy with a sizeof(float/int) should work the same way.

 

Online taylor

  • Super SCP/Linux Guru
  • Moderator
  • 212
    • http://www.icculus.org/~taylor
Re: [3.7.2-ARM] Unaligned Memory Access at modeloctant.cpp:28 (title edited)
ARM is little endian so that code is unused in this case. But yeah, that 1 byte at the start is :sigh:. Even if that swapping code is unused, that 1 byte means the shield collision data is going to be unaligned.

INTEL_INT/INTEL_FLOAT are for byte swapping.

 

Offline ShivanSpS

  • 210
Re: [3.7.2-ARM] Unaligned Memory Access at modeloctant.cpp:28 (title edited)
Then disabling SLDC data loading on archs that are not x86 is the way to go, because the workaround i came out with in fvi.cpp with all those memcpy that are done at every shild hit is in no way faster and fixing SLDC for non x86 archs would mean having to do it again, chaging that char for an int, making models incompatible.

Maybe there is some other way, but no other comes to mind right now. Dealing with it would need to do an unaligned access at some point or the other, that you can never be sure that the kernel is going to be able to fix or not.

Now if i can only figure out what bit of data is causing the release build to crash at model load on bsp_collide_tree...

 

Offline Goober5000

  • HLP Loremaster
  • Administrator
  • 214
    • Goober5000 Productions
Re: [3.7.2-ARM] Unaligned Memory Access at modeloctant.cpp:28 (title edited)
Why would the model alignment affect FPS?  Aren't models completely loaded into memory before the mission starts?  So during the mission they should be referenced from their FSO data structure, not from the POF file.

 

Offline ShivanSpS

  • 210
Re: [3.7.2-ARM] Unaligned Memory Access at modeloctant.cpp:28 (title edited)
Ok now i got the release build into WCS game, with, shields, models and collision working, but now it gets a

Code: [Select]
Thread 1 "fs2_open_3.7.2" received signal SIGSEGV, Segmentation fault.
0x00448d24 in l_Shipclass_Model_f(lua_State*) ()

When a fighter is destroyed :( this never ends, again the debug build works perfectly. Worst thing is im not even sure were that function is.
There has to be some unaligned data still. Retail game data works perfectly.

Why would the model alignment affect FPS?  Aren't models completely loaded into memory before the mission starts?  So during the mission they should be referenced from their FSO data structure, not from the POF file.

My best guess is that some data is still being access with a reference to BSP_Data, bsp_data is stored as it is in memory for the model and submodels and i think some stuff is just using a reference, not copying that data somewhere else. Thats is the best i can think off, i have two folders, one with WCS with default models and other with aligned models, while being in the hangar waiting for launch i get 17-18fps with unaligned models, with aligned models 19-21, using the same client that has been modified to work with unaligned models.

Its either that or a side effect of something else.
« Last Edit: December 02, 2019, 07:41:44 pm by ShivanSpS »

  

Offline Col. Fishguts

  • voodoo doll
  • 211
Re: [3.7.2-ARM] Unaligned Memory Access at modeloctant.cpp:28 (title edited)
Maybe there is some other way, but no other comes to mind right now. Dealing with it would need to do an unaligned access at some point or the other, that you can never be sure that the kernel is going to be able to fix or not.

This thread triggers PTSD flashbacks from when I was trying to get a Doom sourceport working on a Sparc workstation (sparc64 doesn't take a performance hit with unaligned memory access, it just goes lolnope SIGBUS).

Unaligned POF data needs to be fixed externally, but the unaligned structs in FSO code you should be able to fix by telling gcc to enforce alignment during compile time:

https://gcc.gnu.org/onlinedocs/gcc-3.1/gcc/Type-Attributes.html

Sprinkle in "__attribute__ ((aligned (4)))"
(or 8) where needed.

I know other compilers (like the Sun CC) have command line flags to enforce that globally during compilation, but gcc seems to be missing such a feature?
"I don't think that people accept the fact that life doesn't make sense. I think it makes people terribly uncomfortable. It seems like religion and myth were invented against that, trying to make sense out of it." - D. Lynch

Visit The Babylon Project, now also with HTL flavour  ¦ GTB Rhea

 

Offline ShivanSpS

  • 210
Re: [3.7.2-ARM] Unaligned Memory Access at modeloctant.cpp:28 (title edited)
Im really not worried about SLDC data, its optional, i can comment the part were is loaded and thats it, thats good enough for now.

After looking at the code whats needs to be done about SLDC is to increase pof version, change those chars for an int, make the code process that data taking that in consideration and the new data must be saved in a new pof chunk type so it does not break compatibility with older FSO versions. I can also make my aligment tool to update the SLDC data to the new format and chunk type.

None of that is hard to do, it just need time and some testing. But its just not my priority right now.

Release now runs into a segfault when a non retail ship is destroyed, that most likely means a remaining issue with BSP_Data. IT also runs into another segfault at VC4 driver when entering the techroom, even trought i can get in-game.

 

Offline ShivanSpS

  • 210
Re: [3.7.2-ARM] Unaligned Memory Access at modeloctant.cpp:28 (title edited)
OK, i given up in trying to workaround the release 3.7.2 build to work with unaligned models, a shame because i was really close. If someone wants to use unaligned models on ARM he needs has to use the 3.7.2 Debug build without SLDC(commenting out SLDC loading on modelread, altrought im going to add a new command line to disable SLDC data loading) and the "hack" memcpy method on modeloctant point_in_octant i mentioned on page 2. That and removing the "forced on" statics display on game screen on debug build(when i find were that is) is the way that in going to go when i upload the first version of the modified 3.7.2. Im feel like im failed here but well it just not possible to workaround properly the unaligned models. Is a miracle that it works on debug build for wharever the reason.

On that, i dont think it is possible to fully fix the game either, if i disable kernel aligment fixup, the game (with retail game data) it loads and you can get ingame, it will crash with a bus error in, for example, hitting a ship on scripting.cpp, so playing with retail game data still yields aligment errors somewhere in the code. And this is 3.7.2, no idea what new suprises could be hidding on 3.8.0+

My objective right now is making my aligment tool good enoght so models can load with kernel fixup disabled like retail ones do, and check if that fixes the crashes on release build.
« Last Edit: December 03, 2019, 08:25:37 pm by ShivanSpS »

 

Offline ShivanSpS

  • 210
Re: [3.7.2-ARM] Unaligned Memory Access at modeloctant.cpp:28 (title edited)
Ok, today i did a mayor step forward as models are now aligned enoght to load in-game like the retail ones do on a unmodified 3.7.2 (other than ignoring SLDC data) whiout a bus error with kernel fixup disabled, with everything working, collision, shields, etc. unaligned defpoints vertex offset was causing a lot of troubles, that is now fixed.

But once again i ran into the same segfault as before when a ship is destroyed.

Code: [Select]
Thread 1 "fs2_open_3.7.2" received signal SIGSEGV, Segmentation fault.
0x00448d24 in l_Shipclass_Model_f(lua_State*) ()

I still dont know what that is, maybe a script? is debug builds disabling scripts or something?

The thing is... im running out of things to align in the pofs. BSP_Data is now properly aligned. so is the OBJ2 chunk, the other chunks are soft-aligned (this means, size offset is corrected and the chain adjusted for this, but strings inside do not).
FUEL,DOCK,SPCL,PATH and TXTR all have strings inside that could cause issues i need to take a look and fix them, but thats all that remains to be done. And not sure if any of that means anything to a ship breaking up thats causing the segfault. Maybe special points(SPCL)?
« Last Edit: December 04, 2019, 07:17:58 pm by ShivanSpS »

 

Offline Goober5000

  • HLP Loremaster
  • Administrator
  • 214
    • Goober5000 Productions
Re: [3.7.2-ARM] Unaligned Memory Access at modeloctant.cpp:28 (title edited)
This is part of the scripting system.  There have been several fixes to that over the years; it's entirely possible that this is related.  See these PRs for example:
https://github.com/scp-fs2open/fs2open.github.com/pull/2119
https://github.com/scp-fs2open/fs2open.github.com/pull/1960

You may be stuck with coding up an alignment-fixer-loader, unless you want to be restricted to only a subset of mods.  There are literally thousands of POFs that have been released for FreeSpace -- perhaps tens of thousands -- and it would be impractical to fix them all.

 

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: [3.7.2-ARM] Unaligned Memory Access at modeloctant.cpp:28 (title edited)
Perhaps, but starting with ones released on Knossos would be a big win and a much smaller subset.
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline ShivanSpS

  • 210
Re: [3.7.2-ARM] Unaligned Memory Access at modeloctant.cpp:28 (title edited)
My idea was to also integrate the aligner to FSO, verify the .pofs the game can see on startup, update the ones as necesary and place them somewhere that has higher priority than the .vps. Or align each chunk as it being loaded that is easier. But before any of this the tool must be in perferct working condition and do not cause a bus error itselft, i havent tryied run it on ARM directly yet.

Well it looks like ill have to fix every chunk that has a string, every chunk that has a string inside is problematic, for example the TXTR acording to wiki it is:
'TXTR'
int num_textures
for each texture,i
 string tex_filename    // texture filename

But is really

'TXTR'
int num_textures
for each texture,i
 int filename_length
 string tex_filename    // texture filename

This means it is really easy to fall into unaligned positions as you need the length of the current texture to move the pointer and parse the next one. SPCL is much, much worse has it has vector data after two strings. So there is no way around it.
« Last Edit: December 05, 2019, 07:39:47 am by ShivanSpS »

 

Offline ShivanSpS

  • 210
Re: [3.7.2-ARM] Unaligned Memory Access at modeloctant.cpp:28 (title edited)
OK, TXTR and SPCL done, only FUEL, DOCK and PATH remains now.

Also ive made an interesting discovery on the segfault, it seem indeed to be a bug with FSO 3.7.2 and not related not anything else. The strange thing is this only happens on 3.7.2 LINUX Release config, dosent happens on debug or in Windows. Wharever it is, it dosent happens on 3.7.4 but 3.7.4 cannot be used because it is definately running on software 3D renderer. If this is true this also means i was successfull in quickly workaround FSO to work with unaligned models whiout actually having to go and align it, this is something that could be easily switched on/off with a command line argument.

Other explanation could be that it dosent happens on 3.7.4 due to running in software 3d and/or low fps.

Goober, i did check those issues on git but they are already based on 3.8.0, meaning there are files there that dosent exist on 3.7.2, im going to compare 3.7.2 and 3.7.4 scripting system files now.

EDIT: there has been some extensive changes there from 3.7.2 to 3.7.4, no way that i can just drop in the files, its not gona work, so finding exactly was wrong and how to fix it is going to take some trial and error.
« Last Edit: December 05, 2019, 09:29:47 pm by ShivanSpS »

 

Offline ShivanSpS

  • 210
Re: [3.7.2-ARM] Unaligned Memory Access at modeloctant.cpp:28 (title edited)
Alignment tool released here https://www.hard-light.net/forums/index.php?topic=96108.msg1890470#msg1890470

Keep in mind i did basic testing, there could be bad data being producced by this process.

its simple to use the alignment logic on FS2_Code on modelread.cpp as long it does not couse a segbus itseft... i did my best to exploit memcpys that i know it works on FSO to the max. Thats the next move after fixing the script segfaults.

In the end, the models are aligned as the will ever be, but the script error on release config still happening on 3.7.2, and 3.7.4 is fine just runing on software 3d renderer. Im comparing script files from 3.7.4 to 3.7.2 but there are so many changes that im not sure where to start.

 

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: [3.7.2-ARM] Unaligned Memory Access at modeloctant.cpp:28 (title edited)
I wonder if we could integrate this conversion into knossos/nebula if it ever gets stable enough...
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline ShivanSpS

  • 210
Re: [3.7.2-ARM] Unaligned Memory Access at modeloctant.cpp:28 (title edited)
I dont see why not, once is fully tested and the vp writing code is working (im been smashing my head on the wall the whole day), is posible to either integrate the code or use it as a external tool, picking up the vp with the models and reeplacing it with the new version. Thats why i went for direct vp read/write support.

 

Offline ShivanSpS

  • 210
Re: [3.7.2-ARM] Unaligned Memory Access at modeloctant.cpp:28 (title edited)
OK, the very SLOW performance on 3.7.4 is caused by shaders, by using "Disable GLSL Support" the game is fully playable. But it is a little bit slower than 3.7.2. So yeah, 3.7.4 works fine, no crashes due to the scripting system like 3.7.2. Good to know that those crashes are unrelated to alignment, since the models are now aligned and i dont see anything wrong with them.
Now i dont know what to do, try to fix 3.7.2 scripting system or move to recomend 3.7.4 with disabled glsl for RPI3/4...
« Last Edit: December 09, 2019, 05:51:34 pm by ShivanSpS »

 

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: [3.7.2-ARM] Unaligned Memory Access at modeloctant.cpp:28 (title edited)
Just compile it default disabled on ARM maybe.  Just like SLDC.  Solved!
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays