Hard Light Productions Forums
Modding, Mission Design, and Coding => FS2 Open Coding - The Source Code Project (SCP) => Topic started by: Quanto on April 05, 2014, 11:27:50 pm
-
The Preamble:
In the #wod channel we had a conversation.
About model conversion, and how much of a convoluted process it is, the fact that .cob sucks, .pof is older than dirt, and dae is not as reliable as we wish it could be.
I went on to mention that the SpringRTS project had long ago replaced the ancient .3do format with one of their own. One that was modern and supported a number of features the old one was never able to do. From there they created a tool similar to PCS2 called "Upspring" and export plugins for 3DsMax and Blender.
The Proposal:
To do what has long need to be done and create a replacement for .cob, dae. and .pof all at the same time.
We call it ".pomf"
or "Parallax Object Motion File"
The new Short-term goal is to create native opencollada support within freespace.
Github Link: https://github.com/elQuanto/OpenCOLLADA (https://github.com/elQuanto/OpenCOLLADA)
The key feature here being native support for keyframed animations. We want to start slow with things such as translation, rotation, and scaling animations.
I would love to see us create bone based animations as well, but in the distant future.
Other features include updating PCS2 to support the format, and to create 3D Studio Max and Blender plugins to import and export models.
The Team:
Quanto
DahBlount
Vaktezraj
-
Do I hear big massive cannons recoiling? Oh yes I would support this.
-
I would jump on on this ship if I knew the first thing about code at this time. You have my support in this though. It is long overdue.
-
The main situation here after some discussion on #scp appears to be mainly making an extension of the .dae format, with better plugin support for Max/Blender and aim to make this more of an "Animated DAE" project than a new file format entirely.
DAE would really be augmented by hierarchy improvements and tbh, most of the work for the plugins are already done, as such I suggest we go with the FCollada library as our basis as it is one of the most advanced libraries AND is open-source.
Unfortunately FCollada is dead so we'll have to go with OpenCOLLADA. OpenCollada already has plugins for Max/Maya/Blender and they appear to be new and improved. Most of the work I see here would be implementing into FSO and PCS2.
Another part would be gaining popular support among the modders/authors. We need to listen to what they want and need.
I for one think we should augment PCS2 with this instead of creating an entirely new conversion interface.
-
Switching over to OpenCollada native support is now our short-term goal.
Here is a link to our github repository for any who wish to offer assistance.
https://github.com/elQuanto/OpenCOLLADA (https://github.com/elQuanto/OpenCOLLADA)
-
Well, I already use OpenCOLLADA to get models from Max -> PCS2.
So are we looking to get the engine to read OpenCollada DAEs? or just piggybacking/extending it to bring into a program that will make our "new" format that the engine will use (that carries less of the generalist baggage and is more optimized for FSO?)
-
We're going to work on getting FSO able to use OpenCollada .daes first and THEN we will extend and optimize them for FSO.
-
Just so you know, there is already a library out there that can handle parsing of quite a few 3D formats, including DAE: https://github.com/assimp/assimp
-
Here's my perspective on this. I generally like the idea of teaching FSO how to read other model formats. However, what I am not all that keen on are derivatives of standard formats that need special exporters and converters; I'd rather want all the FSO-specific bits in a separate table entry. Reason being that each piece of custom infrastructure adds another item on the list of things we need to support and develop; I do not want to create a large ecosystem of barely understood pieces of software that will collapse in a few years.
-
A sub section in the ship the entry? Or a similar small file accompanying each mesh?
-
I don't think those specifics matter at this stage.
-
If this happens, PCS needs to be able to generate this sort of table data. Manually fine-tuning those things is very convenient, especially when there's a lot of data (some stuff I made for BP has well over 50 glowpoints!). Manually writing in such a huge amount of model points is a real chore (that's why I never got around to adding RCS thrusters to ships I wanted to have them on).
-
i like standard model format (dae for example) + pof chunk data, but in a table. it gives you a fast new way to add features to the model format (stuff normally generated with pcs2) without having to recompile pcs2 to support them. engine supports new model feature? add it to the table, better/faster than reconverting all your models. you can get other cool things like keyframe animations which i think dae supports.
-
Just so you know, there is already a library out there that can handle parsing of quite a few 3D formats, including DAE: https://github.com/assimp/assimp
We didn't know that. We'll look into it.
When we get FSO able to parse DAE we'll most likely add some optional bits to the tables. Keyframe animations are another good point worth using DAE for as well.
-
Maybe PCS2 could be a better place to start? It does handle both pof and DAE and the source is available too...
http://sourceforge.net/p/alliance/pcs2/ci/master/tree/ (http://sourceforge.net/p/alliance/pcs2/ci/master/tree/)
-
As one of the SCP coders, can I strongly reiterate that you should aim to use an open standard format, without breaking modification (if any modification at all).
1. It means you straight away get a bunch of infrastructure and tooling that "just works" e.g. Blender plugins / converters
2. Doesn't leave the SCP project with technical debt maintaining an ecosystem of slightly modified versions of upstream tools, per The_E.
Good luck to you!
-
Just so you know, there is already a library out there that can handle parsing of quite a few 3D formats, including DAE: https://github.com/assimp/assimp
i like how this library supports every quake format ever. even quake3 maps. quake3 maps, in my freespace.
-
OK, since Quanto and Vaktezraj might not be helping you, DahBlount. I'm interested in working this as well. I know about C++ and Object-Oriented enough, and I am ready to take up the challenge.
-
Alright I'll get you hooked up with Git access.
-
Found this https://github.com/sbarthelemy/collada-dom
The COLLADA Document Object Model (DOM) is an application programming interface (API) that provides a C++ object representation of a COLLADA XML instance document.
-
I would love if FSO was able to natively read Collada, it would make model creation way easier, 90% of my personal model bugs come from buggy conversions from DAE to pof... specially with hierarchies and sub object positions.
-
It will be more convenient for the modder but not much more convenient for the user unfortunately. DAE is an XML file type meaning it takes some time to parse however DAE can be stored as a C++ Object (PCS2 uses this, known as Collada DOM).
The question is, do we sacrifice faster parse times for more convenient import and extra goodies?
-
We already build binary cachefiles for POFs. Why would we not do that for a DAE model?
-
Never said we wouldn't do that. The problem itself lies in the fact that XML files take longer to parse.
The main reason we're adding DAE to FSO is the animations and other user data bits that can be stored.
-
Okay, but how does a one time increase in the parse time, which is felt by the modder because the cachefiles are mostly portable, going to inconvenience the users?
The animations and other data would be stored in the cache as well, otherwise what would be the point of the caching?
-
I thought FSO parsed the .pof file still, albeit a more confined portion of it, alongside the cache file?
-
I thought FSO parsed the .pof file still, albeit a more confined portion of it, alongside the cache file?
It does. I believe the only thing the cache files contain are the BSP trees (which are information that the model file does not contain) and the model file as a whole always gets read and parsed by the game.
-
index buffers
-
I thought FSO parsed the .pof file still, albeit a more confined portion of it, alongside the cache file?
It does. I believe the only thing the cache files contain are the BSP trees (which are information that the model file does not contain) and the model file as a whole always gets read and parsed by the game.
Actually, it's the other way around. The cache files are the geometry for rendering purposes. And I don't know how you come to the conclusion that the model files do not contain BSP data when the spec explicitly says it does :P
-
What's the sketch at the moment?
What are we leaning towards?
I look forward to seeing another ten years down the line when the engine is pushing out Advent Children quality animation. :yes:
-
It does. I believe the only thing the cache files contain are the BSP trees (which are information that the model file does not contain) and the model file as a whole always gets read and parsed by the game.
Actually, it's the other way around. The cache files are the geometry for rendering purposes. And I don't know how you come to the conclusion that the model files do not contain BSP data when the spec explicitly says it does :P
Yeah, you're right, I got things mixed up; as far as I can tell, the game generates some BSP data on-the-fly on modelload (model_collide_parse_bsp()), but of course the model file itself contains some, too.
So, yes, disregard what I said. :sigh:
-
Okay. Even under the current system would DAE not take longer to parse even with the cache file?
-
So... surprisingly, no one has asked yet.
What kind of difference are we talking in parse length? Like 10% increase... 25%... 50%....? Cause, hell, if it takes a little longer to parse but we get all the goodies that come with it, would that not be worth it?
Seriously, how much of a time difference are we actually debating here?
-
I am not sure that that discussion is actually relevant. Worst case, there'd be a lengthier cache generation step, i.e. a one-time increase in load times.
In any case, getting this feature to work right has to be the first priority.
-
Which was pretty much my point. :)
-
so it's not so much replacing POF with dae/collada, as it is coming up with a dialect of dae/collada that can hold all pof information and be converted to pof without any pcs rigging and giving the game engine conversion capability. Given how many mods distribute the index buffer cache files currently what you would basically be doing is circulating your source files along side the files the game is actually using, inflating the download size in the process.
I'm honestly surprized the IB cache is till being used, and it hasn't been either integrated into the pof or a generation algorithm using a hash map hasn't been made resulting in dynamic creation fast enough not to need a cache.
-
Would the SCP be opposed to using the pre-compiled VC++ 8/9/10 assimp.lib and dependencies to integrate Assimp or should I try to integrate the code itself? The lib works with MSVC 05, 08, and 10 but we would need to compile versions for MSVC 11 and 13 as well as the environments for other OSes.
-
Since we have to compile it on OSX and Linux anyway, I would recommend distributing the source files.
-
In 2014, Quanto, DahBlount and Vaktezraj mulled the effort to replace the POF format with a native OpenCOLLADA support. Their efforts have not materialize. So I decided to step in, and based on the latest version (2018). My GitHub is: https://github.com/bryansee/OpenCOLLADA (https://github.com/bryansee/OpenCOLLADA). I've started coding it on my repository: https://github.com/bryansee/fs2open.github.com (https://github.com/bryansee/fs2open.github.com).
My original total-conversion project, as well as Wings of Dawn, demands such. So, the question is: where do I need to go from here?
-
My original total-conversion project, as well as Wings of Dawn, demands such. So, the question is: where do I need to go from here?
Bryan, do not EVER presume to speak for other developers. If spoon needs OpenCollada support in FSO, he knows how to tell us (But since he's spoon and thus awesome and aware of the issues involved, knows that this isn't likely to happen soon).
Secondly, if you are serious about trying to implement this yourself, you need to do the following.
Step 1: Learn how pof reading works.
Step 2: Implement a collada loader.
Step 3: Find a way to transform the collada geometry into the internal formats the engine is using.
Step 4: Find a way to embed the metadata the pof format provides, either by embedding it in the .dae or by providing supplementary files.
This, by the way, will only get you to feature parity with the pof reading code, and that alone is, in my estimation, a multi-week project (as in, several weeks of concentrated coding effort). For a hobby project such as this, it's easily a multi-month endeavour, and that's assuming that the coder in question is familiar with the FSO code and actually knows how to write C++. This is NOT a beginner project, and despite your proclamation of being a "visionary modder and coder", your past attempts at making contributions to FSO do not fill me with confidence that you actually know what you're doing. I'd be happy to be proven wrong about this, of course.
EDIT:
Also, Bryan, you've been posting about your projects on this site for almost ten years now. In that time, you have completed almost none of them. Why should we have any confidence in your ability to finish this in a reasonable timeframe?
-
Ah well ****e, I wasn't aware of this demand at all. Is WoD going to be okay without it? (https://i.somethingawful.com/forumsystem/emoticons/emot-ohdear.png)
Bryan, do not EVER presume to speak for other developers. If spoon needs OpenCollada support in FSO, he knows how to tell us (But since he's spoon and thus awesome and aware of the issues involved, knows that this isn't likely to happen soon).
(https://i.somethingawful.com/forumsystem/emoticons/emot-allears.gif)
-
Ah well ****e, I wasn't aware of this demand at all. Is WoD going to be okay without it? (https://i.somethingawful.com/forumsystem/emoticons/emot-ohdear.png)
Admit it, the lack of a visionary model format is the reason you haven't released WoD yet. :P
-
Admit it, the lack of a visionary model format is the reason you haven't released WoD yet. :P
I don't want to admit it, but it might be true (https://i.somethingawful.com/forumsystem/emoticons/emot-ohdear.png)
-
Also, if anyone were tackling this, besides the FSO POF loading code, the PCS2 POF loading code might be good to have alongside as a reference.
-
As per The E, I'm learning on how pof reading works. I know what I am doing.
-
Good luck with that then...............
-
Good luck with that then...............
Thanks a lot, Nightmare...
EDIT:
Also, Bryan, you've been posting about your projects on this site for almost ten years now. In that time, you have completed almost none of them. Why should we have any confidence in your ability to finish this in a reasonable timeframe?
Because I'm busy with my academic things from Tarc and UPM. Nowadays, my parents have now become somewhat authoritarian "monster parents" since the start of the Masters. They want to exclude anything but the Masters work. I've finished College and Degree work.
-
Why should we have any confidence in your ability to finish this in a reasonable timeframe?
Because I'm busy with my academic things
...Well, that's a reason not to have confidence in your ability to finish anything else in a reasonable timeframe.
-
Considering that it's been a long time, and, deciding that OpenCOLLADA would be a mess, I've decided to embark on replacing the POF format with glTF. See this new thread (https://www.hard-light.net/forums/index.php?topic=97430.0) for more.
-
:nervous:
-
:nervous:
You should head over to the gltf thread. gltf is much more advantageous to OpenCOLLADA because the former is a mess, while the latter is much more manageable. Equal to POF as The E said at the least.