Plan for Sam Contapay
by Sam Contapay · 03/28/2002 (10:33 am) · 7 comments
Well here is another update on the Maya to DTS exporter. Hope no one is getting mad about all these plans I just feel that the more I write, hopefully, something will stick out in my head on another path to try. Went home last night and began work on the DTS Reader. Let me step back a bit. The work on the DTS Reader and Exporter is all based on the ms2dts. Since that program allegedly works I decided just to extend the class and use the DTS SDK. But I have come across some interesting things that I should make the GG community aware of especially people planning on writing an exporter. So hopefully, others writing an exporter do not have to make the same mistakes I am making ;)
The DTS Reader I decide to extend the DTS::Shape class. Since it already has a read function why reinvent the wheel. I wrote a new class inhereited from DTS::Shape and was up and running, put up a new window dialog around it and decided to start stepping through code. I try to import one of the DTS files I have created from a MAYA file I get an error, was already expecting that. But here is the odd thing that happened. I tried reading in a known working DTS file from /torque/examples and get the same error on the same line?!?! I was totally shocked. Why would an error be generated on a known working DTS file? It happend in the read(istream& in) method of DTS::Shape. The line I believe was the Stream.Set(Get)Checkpoint(0) line.
Odd this makes me start thinking wildly. Is the DTS SDK something I can base an exporter on? Or is it out of date? Or, is it only specific to Milk Shape files? I have also emailed this same inquiry, more detailed though, to Lightwave Dave who has stepped up and offered his assistance, I will let you know how that works out. I'm thinking of now just basing the exporter on the TSShape class since we know that class works, its used by Torque to actually read in the file.
Here are some other oddities I came across in the DTS SDK. Stepping throught the read portion of the code I came across numGroundAnimations which is read in, however, I did not see this information being written out to a DTS file from the save() method. Also checkpoints, what do checkpoints do? I need to analyze this further as I do not know, or even know if they are necessary. I don't see the use of checkpoints in the max2dts_4 exporter so wonder if they are necessary, or something based on MilkShape only files.
I'm creating a website just to put all my findings and source files. I'm going to start uploading source now so that people can start perusing it and learning what a total *spaz* coder I am ;) Don't laugh I try to comment the source so people can at least follow. The source will be written in VC6 SP5.
As usual thoughts and opinions on the matter would be greatly appreciated.
The DTS Reader I decide to extend the DTS::Shape class. Since it already has a read function why reinvent the wheel. I wrote a new class inhereited from DTS::Shape and was up and running, put up a new window dialog around it and decided to start stepping through code. I try to import one of the DTS files I have created from a MAYA file I get an error, was already expecting that. But here is the odd thing that happened. I tried reading in a known working DTS file from /torque/examples and get the same error on the same line?!?! I was totally shocked. Why would an error be generated on a known working DTS file? It happend in the read(istream& in) method of DTS::Shape. The line I believe was the Stream.Set(Get)Checkpoint(0) line.
Odd this makes me start thinking wildly. Is the DTS SDK something I can base an exporter on? Or is it out of date? Or, is it only specific to Milk Shape files? I have also emailed this same inquiry, more detailed though, to Lightwave Dave who has stepped up and offered his assistance, I will let you know how that works out. I'm thinking of now just basing the exporter on the TSShape class since we know that class works, its used by Torque to actually read in the file.
Here are some other oddities I came across in the DTS SDK. Stepping throught the read portion of the code I came across numGroundAnimations which is read in, however, I did not see this information being written out to a DTS file from the save() method. Also checkpoints, what do checkpoints do? I need to analyze this further as I do not know, or even know if they are necessary. I don't see the use of checkpoints in the max2dts_4 exporter so wonder if they are necessary, or something based on MilkShape only files.
I'm creating a website just to put all my findings and source files. I'm going to start uploading source now so that people can start perusing it and learning what a total *spaz* coder I am ;) Don't laugh I try to comment the source so people can at least follow. The source will be written in VC6 SP5.
As usual thoughts and opinions on the matter would be greatly appreciated.
About the author
#2
Yup, I got your email Sam, and have sent one right back at you :o)
Phil, the MilkShape code does sort of have the functionality that you've asked for. Unfortunately, like you said, the DTS format is really horrid, so it really isn't always obvious as to what needs to be done.
Once I've got my LightWave plug-in up to the same level as the MilkShape one (ie: get animation working), I'll see about documenting what needs to be done to use the base code. If necessary, I'll also see what can be done to simplify the original base code to make it even easier to write an exporter.
Thinking out loud here: I'm wondering if what is needed is to have an intermediary format that is simple enough for everyone to write against. Then have a translator to convert to the final DTS format. Kind of like what is done to build DIF's ie: WorldCraft to common Map format; Map to DIF. And, to be trendy, this intermediate format could be in XML, which would remove any endian issues. Hmm, I may give this some more thought...
- LightWave Dave
03/28/2002 (12:03 pm)
Greetings!Yup, I got your email Sam, and have sent one right back at you :o)
Phil, the MilkShape code does sort of have the functionality that you've asked for. Unfortunately, like you said, the DTS format is really horrid, so it really isn't always obvious as to what needs to be done.
Once I've got my LightWave plug-in up to the same level as the MilkShape one (ie: get animation working), I'll see about documenting what needs to be done to use the base code. If necessary, I'll also see what can be done to simplify the original base code to make it even easier to write an exporter.
Thinking out loud here: I'm wondering if what is needed is to have an intermediary format that is simple enough for everyone to write against. Then have a translator to convert to the final DTS format. Kind of like what is done to build DIF's ie: WorldCraft to common Map format; Map to DIF. And, to be trendy, this intermediate format could be in XML, which would remove any endian issues. Hmm, I may give this some more thought...
- LightWave Dave
#3
Also, David, has been a great help and I am currently working with him on solving the MAYA issue. David, I would be happy, and more than willing to work with you, on coming up with an intermidatery solution as you propose. I think it would be great for the community.
The resource I will write will of course give David much props, since anyone who can get this to work is genious :D
But rest assured Phil, and the rest of the GG community, that a easier solution to save information is in the works and hopefully David will accept my invitation of helping out with a "more" simplistic solution.
Plus I have been working with XML at work so using that knowledge would be great in getting a sort of intermidatery format.
David if you want to work on the documentation together and a base class I would be happy and more than willing to participate as I see this as great step in the right direction.
03/28/2002 (12:13 pm)
Phil the current plan is to get this up and running and write a resource on the file format, as well as a class to use the DTS format more simplistically ;)Also, David, has been a great help and I am currently working with him on solving the MAYA issue. David, I would be happy, and more than willing to work with you, on coming up with an intermidatery solution as you propose. I think it would be great for the community.
The resource I will write will of course give David much props, since anyone who can get this to work is genious :D
But rest assured Phil, and the rest of the GG community, that a easier solution to save information is in the works and hopefully David will accept my invitation of helping out with a "more" simplistic solution.
Plus I have been working with XML at work so using that knowledge would be great in getting a sort of intermidatery format.
David if you want to work on the documentation together and a base class I would be happy and more than willing to participate as I see this as great step in the right direction.
#4
03/28/2002 (1:06 pm)
Funny, I was thinking about XML as a format for this mythical Torque 3D app *grin* Actually I guess it would be XGL, wouldn't it? http://www.xglspec.org/ No point reinventing the wheel.. unless the wheel is too complicated.
#5
But once more David thanks and let us see how it goes.
03/28/2002 (2:34 pm)
Just got some good advice from David. I'm going to give it a shot when I get home. He made some points to my existing code, that I didn't think about before, or knew about because it is sort of like reading Chinese =)But once more David thanks and let us see how it goes.
#6
I've heard you mention your dislike for the dts format several times. I hear you, I sympathize, I appreciate it...but you also have to understand what it does to appreciate that it is actually rather elegant. The code:
1) reads the dts file in one block, which was very important for the console titles that were being worked on at the time
2)parses that block depending on a number of preference settings, something that was requested by the Tribes 2 team but never actually used in the game,
3)converts the poly lists to either a single triangle list, many triangle lists, or a single list of triangles -- which is faster depends on your video card and what platform you are using. This is something else the T2 team simply did not have time to act on, so in T2 the first option is always used. On the PS/2, we were finding the 3rd was better.
4)skips the first N detail levels depending on which setting you have chosen. The idea here is that low-end machines don't even need to load the highest detail level. This is another feature that we thought we wanted but never ended up using. Note that this process is very tricky because different details of the same mesh share vertices, so if one mesh is dropped, we have to make sure the vertices still make it into the shape.
5) reads versions of the shape that are 2 years old. Really, the GG community should strip this stuff out -- it's one of the most confusing parts of the routine.
6) the helper macros, which help make the code hard to read, and which are admittedly not commented and should be (my bad for sure), handle endian issues (which is why there are versions of them for U8's, U16's, U32's, etc) and debug checking to make sure the shape is not corrupt.
Note that our mindset at Dynamix at the time was that the way you write an exporter is that you build the shape in memory and call the write method on the shape.
I understand that this doesn't necessarily work for the GG community because you want to release the source code to the public without worrying about the SDK. (I'm sure you don't want to deal with Torque linking issues when building exporters either).
I think that your idea and others of building an intermediate format for shapes is an excellent one. For the app that converts the intermediate format into the binary format, though, why not use the Dynamix strategy of building the shape in memory and then calling the write function. Do that, and no one has to figure out the file format. :)
And now an offer: I know I'm not the only busy person in the world. But I am busy. As many of you know, we at BraveTree have our paws in many things, and my time is committed to BraveTree. If my interests weren't elsewhere, I'd be thrilled to be involved in the thriving GG community more. Instead, I've purposely kept myself on the outside. So, I can't offer to write said tool for you. But, I am willing to correspond with you guys if you want answers to some questions.
ps. Re; TCP: you guys have been focusing on what you want the engine to do. I think you should first focus on documenting what the engine does and how it does it. It's related to that old parable: "if you don't know where you're going then any road will do". In this case it's "if you don't know where you are then you can't pick a road" (you get the idea :). My comments above are very much related to this issue.
Regards,
clark
03/29/2002 (10:40 am)
Phil -I've heard you mention your dislike for the dts format several times. I hear you, I sympathize, I appreciate it...but you also have to understand what it does to appreciate that it is actually rather elegant. The code:
1) reads the dts file in one block, which was very important for the console titles that were being worked on at the time
2)parses that block depending on a number of preference settings, something that was requested by the Tribes 2 team but never actually used in the game,
3)converts the poly lists to either a single triangle list, many triangle lists, or a single list of triangles -- which is faster depends on your video card and what platform you are using. This is something else the T2 team simply did not have time to act on, so in T2 the first option is always used. On the PS/2, we were finding the 3rd was better.
4)skips the first N detail levels depending on which setting you have chosen. The idea here is that low-end machines don't even need to load the highest detail level. This is another feature that we thought we wanted but never ended up using. Note that this process is very tricky because different details of the same mesh share vertices, so if one mesh is dropped, we have to make sure the vertices still make it into the shape.
5) reads versions of the shape that are 2 years old. Really, the GG community should strip this stuff out -- it's one of the most confusing parts of the routine.
6) the helper macros, which help make the code hard to read, and which are admittedly not commented and should be (my bad for sure), handle endian issues (which is why there are versions of them for U8's, U16's, U32's, etc) and debug checking to make sure the shape is not corrupt.
Note that our mindset at Dynamix at the time was that the way you write an exporter is that you build the shape in memory and call the write method on the shape.
I understand that this doesn't necessarily work for the GG community because you want to release the source code to the public without worrying about the SDK. (I'm sure you don't want to deal with Torque linking issues when building exporters either).
I think that your idea and others of building an intermediate format for shapes is an excellent one. For the app that converts the intermediate format into the binary format, though, why not use the Dynamix strategy of building the shape in memory and then calling the write function. Do that, and no one has to figure out the file format. :)
And now an offer: I know I'm not the only busy person in the world. But I am busy. As many of you know, we at BraveTree have our paws in many things, and my time is committed to BraveTree. If my interests weren't elsewhere, I'd be thrilled to be involved in the thriving GG community more. Instead, I've purposely kept myself on the outside. So, I can't offer to write said tool for you. But, I am willing to correspond with you guys if you want answers to some questions.
ps. Re; TCP: you guys have been focusing on what you want the engine to do. I think you should first focus on documenting what the engine does and how it does it. It's related to that old parable: "if you don't know where you're going then any road will do". In this case it's "if you don't know where you are then you can't pick a road" (you get the idea :). My comments above are very much related to this issue.
Regards,
clark
#7
Thanks for your comment. I know for one I would be happy to take you up on this offer :) I'm hoping to start documenting the DTS file format soon, but I'm trying to figure out how to do this without violating any of the license agreements for the source code.
I forgot about the source license issue, since I was going to release source for my exporter so others can look at it, I'm going to have to rethink this since it does use code from the Torque source tree.
03/29/2002 (12:57 pm)
Clark,Thanks for your comment. I know for one I would be happy to take you up on this offer :) I'm hoping to start documenting the DTS file format soon, but I'm trying to figure out how to do this without violating any of the license agreements for the source code.
I forgot about the source license issue, since I was going to release source for my exporter so others can look at it, I'm going to have to rethink this since it does use code from the Torque source tree.
Torque 3D Owner Phil Carlisle
If you ever get an exporter working, could you please please please write (and even go so far as to write some docs) for a class that can essentially be used as a stream object that writes out DTS files from any program.
What I mean is this..
dtsmodel.addvertices(
dtsmodel.addtextures(
dtsmodel.addanimation(
dtsmodel.savedtsfile();
Just doc the interfaces to this model class and it can be used by anyone else. This is kind of like the class i saw in the milkshape. Only with documentation.
Once we have that, it should be relatively easy to make a model exporter for anything.
Phil.