Hard Light Productions Forums
Modding, Mission Design, and Coding => FS2 Open Coding - The Source Code Project (SCP) => Cross-Platform Development => Topic started by: Elyus on November 24, 2020, 12:13:03 pm
-
I'm doing testing with an M1 Mac mini, and FS2 runs fine through Apple's Rosetta 2 translation layer, but I wanted to try a native build as well. I've seen discussion on these forums around ARM support for various other devices, and I'm happy to report that the code is 100% ready-to-go with Apple's arm64 implementation! There's just a couple caveats to keep in mind for anyone building an Apple binary right now:
- Libraries: the configure process still detects macOS and downloads prebuilt libraries into lib/prebuilt, but these are x86_64 versions
- SDL2.framework can be built natively directly from source (https://www.libsdl.org).
- ffmpeg can also be built natively from source, although I followed this recommended simple configure (http://www.ffmpeg-archive.org/FFmpeg-on-Apple-Silicon-Success-td4693516.html) as not all dependencies may be ported.
- Unlike sdl and ffmpeg, freetype can already be installed through native homebrew. However, whether by error in the brew formula or Apple's headers, it complains about some missing symbols in the bzip2 dependency. That's easy to fix with a new build from bzip2's code (https://www.sourceware.org/bzip2/downloads.html).
- During the build process, a few small issues and some Mac idiosyncrasies pop up:
- As mentioned above, freetype may try to link against the wrong libbz2 unless you've built a new one and added it to the linker flags.
- Also, libpng's NEON optimization is not ready for arm, so you'll need to disable it by defining PNG_ARM_NEON_OPT=0, per this discussion (https://lists.freedesktop.org/archives/libreoffice/2019-September/083539.html).
- Lastly, Apple signing... macOS has removed the option to run code from "Anywhere" and adhoc signing is no longer reliable in Big Sur. It's silly that I have to worry about signing an executable I just built, but I guess Apple wants all users to become very familiar with the `codesign` tool. Anyways, during `make`, the `embedfile` tool will fail to run until the adhoc signature is replaced, and once the final fs2_open app is built, you'll need to `codesign` the executable and all embedded libraries before it will run.
As you can see, it's just handling libraries and Apple signing. I expect everything will become simpler when homebrew supports more arm64 packages. In the meantime, I thought I'd share this in case anyone was curious about Freespace making the jump to Apple's new systems.
Now, the bad news. Unsurprisingly, the M1's integrated GPU isn't all that impressive in FS2. Benchmarks have indicated performance on par with some average laptop GPUs and some games are running quite well, so I would expect a little better, but I'm only getting 20-30fps in Freespace. Interestingly, Rosetta is only a little less than native, so Apple's graphics translation is quite fast. At any rate, while I'm satisfied with this performance on their very low-end chip, Apple is going to have to unveil some very compelling graphics technology if they truly want to drop discrete GPUs.
-
Were you running a mod with modern graphics or retail data? I'm curious if it was even running in hardware acceleration at all? Any chance you could run a fast debug build and provide a debug log?
Honestly adding native support to our build ecosystem for macOS is now just one more thing on top of the list of needs. It seems like every time one thing for mac is fixed, two more things manage to break before anyone can reap the benefits. I believe we no longer have an automated build pipeline at all for Macs now, thanks to losing TravisCI support. I still love my work Macbook but the ecosystem just keeps making things difficult.
-
First of all, great work on getting FSO to run on the M1!
However, Apple is not making our life easier, that is certain. GitHub actions do have Mac support but that uses a later version of MacOS that already requires some of those code sign changes so I was unable to get that to work.
Even after all that, the deprecation of OpenGL basically means that at some point we will likely not have a graphics API to work with anymore. Judging from our current graphics development manpoower (which is basically only me) we are also not going to add Metal support.
All in all, I do not see the situation of FSO improving on Mac :sigh:
-
Actually that last one might not be a huge problem to overcome. https://moltengl.com/moltenvk/ seems to be a Vulkan to Metal translation layer, so if we do get Vulkan support, we could be translating one modern API to another one, and hopefully that remains a supported library.
-
Well, MoltenVk requires us to first port to Vulkan though :P
And even then, we will need to find a Mac graphics developer to do the actual conversion since MoltenVk is not a complete reimplementation of Vulkan as far as I know so some changes will be necessary.
Finally, I would like to note that I am very reluctant to invest valuable time into supporting an OS that makes every effort to make the developers job even harder so at least for me Mac is very low on the priority list.
-
Yeah I know it's not ideal, but I also know of mod and SCP contributors who are still primarily Mac users so I'm wary of giving up on it entirely.
-
@chief1983, that's a good point about graphics acceleration. I'm not sure, but I did create a FASTDBG build and flew around the first mission a bit. The fs2_open.log is attached; it's using the current mediavps and some custom flag presets from an old version of the launcher.
It does appear there's an error acquiring an OpenGL context. It's true as @m!m points out, Apple's deprecation of OpenGL is problematic in that they're letting the supplied version languish while pushing developers to their Metal framework. That said, the fact that they bothered porting OpenGL indicates they'll leave it around for some time, although undoubtedly it will receive little support and no updates.
I completely understand the concerns with handling macOS builds, but I am hoping things will begin to stabilize past this transition. Apple's recent push to abandon legacy technologies has been focused on preparing for this transition. Plus, with Apple, anything could happen. They've been down the road with proprietary graphics frameworks before: QuickDraw3D was the big thing before ditching it when they went to OpenGL; maybe they'll consider supporting Vulkan some day? :)
[attachment deleted by admin]
-
At least the error I could find are "expected" on Mac since we try to create higher versions of OpenGL first starting with 4.5 and at 4.1 we finally get one.
Regarding their support, the main issue is that at some point we are going to use features exclusive to OpenGL 4.2 or higher at which point Mac becomes impossible to support. We had that situation before with fixed-function OpenGL and it was not pretty. If Vulkan or MoltenVk works at that point it will not be an issue but that is a big "if".
-
@Elyus: nice work. :yes:
That said, things aren't looking rosy for future Mac support....
Finally, I would like to note that I am very reluctant to invest valuable time into supporting an OS that makes every effort to make the developers job even harder so at least for me Mac is very low on the priority list.
Unfortunately, I'm inclined to agree. Although I've recently done some (https://github.com/scp-fs2open/fs2open.github.com/pull/2863) work (https://github.com/ngld/knossos/pull/219) to improve Mac support, it looks to me like Apple is intentionally moving away from cross-platform compatibility. Even if we can find a coder willing to dedicate themselves to Mac support, there's a limit to how much the SCP can bend over backwards to support an OS whose developer has shown no interest in making cross-platform support easy.
It's ironic, in that Microsoft is moving towards cross-platform with things like Hyper-V and the new .NET. Ah, well.
Yeah I know it's not ideal, but I also know of mod and SCP contributors who are still primarily Mac users so I'm wary of giving up on it entirely.
Yeah, I was a Mac user when I joined SCP :) and I doubt we'll give up on it entirely. But Mac support is bound to get worse over time, and even with good intentions, the SCP can't do a whole lot about it.
-
Actually, the performance is not too bad for the M1. Specially if you are talking about 1080p, AMD Vega 11 is not that much faster really.
What i do find interesting is that you were able to run mediavps on the M1 on arm64 with the unaligned models and shields system not causing crashes, i really had to solve quitte of a few issues to do that on the RPI, it looks like Apple handles that a lot better.
-
Actually, the performance is not too bad for the M1. Specially if you are talking about 1080p, AMD Vega 11 is not that much faster really.
What i do find interesting is that you were able to run mediavps on the M1 on arm64 with the unaligned models and shields system not causing crashes, i really had to solve quitte of a few issues to do that on the RPI, it looks like Apple handles that a lot better.
The M1 does include some special bits for memory access in order to speed up x86 emulation, I think that may be the reason why?
-
Yeah, i guess they really had to ensure the CPU could properly handle that. It could also be their kernel, the linux kernel actually does a good job in fixing some of the unaligned memory access, because if i disable the unaligned access fixing in linux, FSO will still crash in... i think it was the scripting area.
-
For anyone else stumbling on this thread and trying to self-compile, i confirm that the current latest stable release (23.0.0) can be built on Apple Silicon Macs by following these steps:
- ensure you have all dependencies (freetype, ffmpeg, sdl2) installed w/ homebrew
- edit lib/freetype.cmake and lib/libpng/CMakeLists.txt as outlined in PR4337 (https://github.com/scp-fs2open/fs2open.github.com/pull/4337/files) to disable NEON and use the system freetype library
- follow Linux build instructions (https://github.com/scp-fs2open/fs2open.github.com/wiki/Building-on-Linux):
mkdir build && cd build
cmake -DFFMPEG_USE_PRECOMPILED=OFF -DFSO_FREESPACE_PATH=/path/to/Freespace2 ..
make -j<N> # Note we don't use the "install" target because it's broken
- application bundle should be in build/bin
I can submit a PR with the NEON & freetype fixes, along with updating relevant "Build on Mac" docs/wikis. How does that sound?
I know this has less value than actually fixing the production of redistributable Mac builds in CI, but that's the best I can offer right now considering my current knowledge of cmake and available time.
-
Is the only thing needed to make a distributable build a new set of the libs compiled for Silicon? Also I tend to use xcodebuild on Mac, and generate an xcode project file with CMake, any reason that wouldn't still work? I just got a new Silicon Macbook at work again (had one back in February but changed jobs then) and would be interested in helping out if there are solvable problems to get this going again.
-
There is also the requirement of signing/notarization. It sounds complicated but can be actually pretty straightforward in my experience.
E.g. our build script at work for a native Mac app looks like this:
# Build, archive, notarize
xcodebuild \
-scheme "$env" \
-configuration Release \
-archivePath "$PWD/Release/App/$env.xcarchive" \
archive
xcodebuild \
-exportArchive \
-allowProvisioningUpdates \
-archivePath "$PWD/Release/App/$env.xcarchive" \
-exportOptionsPlist $PWD/ExportOptions.plist
until xcodebuild \
-exportNotarizedApp \
-archivePath "$PWD/Release/App/$env.xcarchive" \
-exportPath $PWD/Release/App; \
do \
echo wait 10s...; \
sleep 10; \
done
# Zip exported app
ditto -c -k --sequesterRsrc --keepParent "$PWD/Release/App/$env.app" "$PWD/Release/App/$1.zip"
Xcode will need an Apple developper account. I understand they're free for open-source projects (https://developer.apple.com/support/membership-fee-waiver/). Do we have one already?
If you're familiar w/ generating xcode projects w/ CMake, that could be useful since it might be an easier path than getting notarization up & running without Xcode (example (https://github.com/Monetra/libmonetra/blob/master/CMakeModules/Notarize.cmake))
-
My macOS compilation notes are unfortunately a bit thin compared to my other distros, as I never made a note of how we built the pre-packaged libs the last time, etc. This is all I've managed to make note of in it so far:
macOS
Extracting library dSYM files
find . -name '*.dylib' | xargs -I {} dsymutil `basename {}` -o "{}.dSYM"
Compiling
Ensure latest CMake app is installed after updating Xcode.
cmake -DCMAKE_BUILD_TYPE=Debug -GXcode -DCMAKE_OSX_ARCHITECTURES=x86_64 -DCMAKE_XCODE_ATTRIBUTE_CODE_SIGNING_ALLOWED="NO" ../
xcodebuild -configuration Debug
PCS2
brew install wxmac devil [email protected] # latest boost 1.66 causes warnings about being too new
# Install SDL.framework to ~/Library/Frameworks/
cmake -GXcode -DBOOST_ROOT=/usr/local/opt/[email protected] -DCMAKE_CXX_STANDARD=11 -DCMAKE_OSX_DEPLOYMENT_TARGET=10.13 ../
As you can see it could use some updating.
-
Yeah this is incomplete but still pretty valuable since it shows an easy path to using xcodebuild for notarization. I just tested your script w/ an arm64 target instead of x86-64 and the tweaks mentioned earlier (https://www.hard-light.net/forums/index.php?topic=97138.msg1917944#msg1917944) and it builds fine on my machine. This is not redistributable yet however since we don't have prebuilt ffmpeg/freetype UB2 libraries to bundle. Right now, this relies on dynamically linking to shared, homebrew-provided ffmpeg & freetype libraries.
I never made a note of how we built the pre-packaged libs the last time
The last known compilation instructions I have seen are in https://github.com/scp-fs2open/scp-prebuilt/tree/master/doc
They're pretty much re-useable as is, since — if I remember correctly — bundled frameworks and libraries don't need to be signed/notarized individually. The application bundle is signed, notarized and stapled as a whole when running xcodebuild -exportNotarizedApp or xcrun altool --notarize-app
I just got a new Silicon Macbook at work again (had one back in February but changed jobs then) and would be interested in helping out if there are solvable problems to get this going again.
I'm also interested in helping.
AFAICT, this is what needs to be done to have redistributable Mac builds again:
prebuilt UB2 ffmpeg lib done PR12 (https://github.com/scp-fs2open/scp-prebuilt/pull/12)- prebuilt UB2 freetype lib PR open PR10 (https://github.com/scp-fs2open/scp-prebuilt/pull/10)
prebuilt UB2 SDL2 lib done PR8 (https://github.com/scp-fs2open/scp-prebuilt/pull/8)- creating an Apple Developper account
- application bundle notarization
- upgrade github action workflow PR open PR4337 (https://github.com/scp-fs2open/fs2open.github.com/pull/4337)
It would be probably good to have an issue on GH to track progress.
It would also be nice to have the inputs of @virusman and @notimaginative — these are Github handles, anyone know their handles here? — since they seem to have put quite a bit of effort in the past on Mac support.
Edit: updated the TODO list since PR12 was just approved
-
We should still have a shared SCP Gmail account although it may have been forced into 2FA with an individual's phone, so not sure how useful using it still is. I'll double check though. Not sure what other options there might be for creating a shared SCP apple dev account.
-
Thanks
If 2FA gets in the way, a Google Group could be a good alternative. They provide a public email (e.g. [email protected]) and it’s easy to add/remove members.
-
meego are you on Discord? Most SCP devs are more active on discord, including myself.
EDIT: Here is a link to our discord https://discord.com/invite/QFdueKEYrN
EDIT2: Ok, I just saw that you have virusman's and notimaginative's discord handles. I pinged you on discord because that's where we can have the easiest communication. We don't do a lot of SCP-prebuilt PR's, so I'm fuzzy on the proceedure. I'll put some effort into making sure they get merged.
-
Google group is probably a good idea so we can have an email to tie to shared accounts, if we didn't already do something like this for github. Does Apple do org accounts with multiple members for open source projects I wonder?
-
Google group is probably a good idea so we can have an email to tie to shared accounts, if we didn't already do something like this for github. Does Apple do org accounts with multiple members for open source projects I wonder?
I would think so. "Normal" Apple developper accounts typically have an "owner" user (i.e. "Team agent" in Apple parlance), and lower privileged users can be created as needed