Torque Dynamic Linking and Threading

#1 We've talked about making some parts of the engine into DLLs (such as render or device core, etc), but it's not setup for it now and would be a bit of re-working to fit that in now. I don't know of any current link issues with the SDK.

The VC6 library project is used by the the tools. Some of the tools use core engine functions, something that needs to be cleaned up a little. There are a lot of engine objects which can't be placed in a library though (because of they way we use the persistence stuff to construct objects), so we can't have a core library for both the main engine and the tools.

We don't use threaded code so we don't need to compile multi-thread safe :) In fact it would slow the engine down as the multi-thread safe standard library uses a lot of critical sections to protect functions (guess those really aren't all that slow, but it adds up).

#2 There is dynamic linking and then there is splitting the engine into seperate source components... the later would be much more work then the former... which is some grunt work of exporting symbols...

Not being able to use torque outside of a single dll or only inside the projects exe is a major hinderance to code reuse, seperation, and the building of components... I have run smack into this before, it's one of the reasons dynamic linking was invented...

This is illegal:

MyDll.dll linked to static Torque.lib

My.exe linked to MyDll.dll and static Torque.lib

This is what has bit me with static libraries in the past... the core functionality that you want to use in components of your engine can only be linked to one! Otherwise you will get conflicts.

Perhaps you could give me a case where the tools and engine can't share a library? This may point towards something...

I don't know if this is feasible: but I hereby offer to get Torque up and running inside a dll.. not a huge deal really but hours of mindless EXPORTS

*EDIT*

The offer is only good for windows boxes... I don't know about Macintosh/Linux... but hey... add a project called Torque DLL which defines TORQUE_BUILD_DLL that exports the classes... and woohoo you can use torque about anywhere you want...

*END EDIT*


-J

#3 Another take (BTW: my .plan has a very quick description of the situation):

Ok... I don't think the value of dynamic linking is in dispute... Perhaps there is some concern over work/code breakage? It's really just another way of compiling the existing code... some fodder:

In a sentence: Wherever Torque is in your project it has to ALL be there... you can't link to it... currently this most likely means the exe...

Dynamic Linkage promotes modularity, code reuse, and SHARING... in addition, it can be added to the system without breaking anything...

Aside from script, Torque is monolithic... this statement could be argued, but only due to source availiablity... indeed, everyone has source access to Torque, which I firmly believe in... THE PROBLEM: if you derive from an engine class or simply USE engine functionality: you need to COMPILE with that code in the SAME MODULE which in turn NEGATES that code's use in ANY OTHER MODULE... this is a significant limitation and as it turns out, an artifical one.

Think about this: (simply) changing the engine to be _DYNAMICALLY_LINKED_ REMEDIES THE SITUATION...

What are plugins in 3DSMAX or Milkshape? dll's... Why? Cause they can be dynamically loaded and unloaded... this is not a monolithic design, there is good code seperation, and people who DON'T HAVE ACCESS TO ENGINE SOURCE can make plugins... you certainly wouldn't expect to need the complete max source to make a plugin? If you DID need the source to derive from a sdk base class would a plugin even be possible? No...

I have been working with The Nebula Device from RadonLabs (www.radonlabs.de) ... it is an opensource 3d engine/game system which has a very nice plugin system, this of course is based on dll's .. it works... many users have contributed FULL SUBSYSTEMS to the engine... other OpenSource projects have been COMPLETELY rolled in, many of which were ALREADY dll's just add some glue...

When I make a Torque game application I have to take the entire engine source and plop it into my application? I start out swimming in a sea of source files? Anything I write which uses Torque MUST go here... If I want to use something published by another developer it goes here... CVS updates happen here... everything is here... again... EVERYTHING IS HERE...

When I make a Torque utility application I get the benefit of static linking to a lib file? It is my understanding that this isn't possible with a game application? I know some situations that don't work with static linking... I would venture the problem is one or more of these... Does this limitation carry over to the land of dynamic linkage? I doubt it... I've written some zany stuff over the years... dynamic linking is much like having all the source in one spot...

Modularity, Reuse, and SHARING: by simply adding dynamic linkage as an option to developers you facilitate this:

----- CASE 1
Torque.dll (the whole thing... no huge changes... just exporting the necessary stuff)

game.exe
-----

At the most basic level a developer's game application is free of clutter... there is very clear code seperation: THAT is torque THIS is me... most sdk's have this design... at the very least they are compiled into a library... actually Torque is the only one I remember seeing where one compiles the engine into the application... this is an undesireable situation

----- CASE 2
Torque.dll (the whole thing... no huge changes... just exporting the necessary stuff)

MySupportingDllOne.dll

MySupportingDllTwo.dll

game.exe
-----

Whoopee! I just further broke down my codebase and Torque didn't stop me from doing it!!!! I can reuse Torque code in any of my dll's or in my game exe... if I want to use any of the functionality I wrote in a tool I just create a new project and set my dependencies accordingly!

----- CASE 3

Torque.dll (the whole thing... no huge changes... just exporting the necessary stuff)

MySupportingDllOne.dll

MySupportingDllTwo.dll

SomebodyElsesSupportDllOne.dll

SomebodyElsesSupportDllTwo.dll

SomebodyElsesSupportDllThree.dll

game.exe
-----

Now this is where things get cooking... developers (including garagegames staff) can release addons to the engine as neatly contained packages... (possibly even value added ones)... not as zip files of sources you merge against your application (ick)... everyone can happily compile and link with Torque code... I can even publish MySupportingDllTwo.dll and NOT MySupportingDllOne.dll to the community! Maybe I used some commercial library stuff in MySupportingDllOne.dll that I cant release... or maybe I don't WANT to publish MySupportingDllOne.dll... whatever the case may be... The community wins!!! Hooray for us!!!!


BTW: What happens when an addon gets a bit of redesign and there are a few dead sources? What happens when there are conflicts between two addons? I don't see much in the way of addons, nor do I see people wrapping existing codebases in for use with Torque... maybe this has something to so with it? Let's say it doesn't... what happens when people ARE pumping out code?

And then there are the tricks... I wrote a system similar to Dungeon Siege's automatic script binding... you parse a dll's exports and unmangle the C++ declaration out of it... with this information, the function pointer, and some minor assembly you can automatically generate script glue...

I could take the engine source.. add the export declarations and then when I get new from cvs add them again for any changed file... this sucks, because other developer's don't get the functionality and addons to the engine remain difficult to create and maintain... it's also a pain in the ass

Any good document on dynamic linking will likely do a much better job of describing why this approach is significantly better... the Torque engine is very powerful and well designed... I want to use it wherever I can, dynamic linkage helps me to do this...

In closing, anyone left clinging to the monolithic.exe approach wouldn't have to change their code one bit... and if they wanted to take advantage of the things I've mentioned they only need LINK differently...

-J

#4 I have no idea what sort of project you are looking to use Torque in, but I can't really see why you would need to dynamically link in the engine in the future.

The current tools require the engine to be linked in, but once the text model/map/whatever format is worked out, that need should be gone. You might still want to link in some subset for manipulating the model format (as a matter of fact, that makes a lot of sense so that some file format changes can be made transparent to the tools/engine. An example would be some sort of binary compiler for the text format).

You seem pretty gung-ho on the whole dynamic linking thing, but the only examples you have been citing havn't really shown where dynamic linking is a win over static linking. The only real win you would get would be an easy upgrade to the tools without having to recompile them. Everything else you have mentioned wouldn't be any easier or harder with static linking.

Like I said, maybe it would be helpful if you explained what exactly you are trying to get from dynamic linking.

Your examples all have a Torque.dll and a game.exe. Why would it be a win to seperate out the torque stuff from the game stuff? Most developers here are building on top of torque, not along side of it. If you want to build your changes as dlls and link them in, nothing is stopping you. You can very easily add in some scripting glue to let the scripting language call into your dlls and you are set. Most people will need to change something in the Torque engine, so building it as a dll doesn't seem to get you anything.

Josh

#5 Modularity, code reuse, and sharing...

My examples do not only have a torque.dll and a game.exe look again.. there are three cases described... as to the comment of building along side Torque.. notice where Torque.dll lies, yup, on top... if anything, building with the Torque source in your .exe is building along side it...

Currently everything is linked differently and the engine source is tied into every individual projects application.exe ... with dynamic linking things are consistant.. and work in all cases... tool or game...

Dynamic linking isn't harder then static linking... the compiler does the work..

And yes I am GungHo because I know how important this is... :)

-J

#6 Those are some great buzzwords, but they don't actually apply here.

The engine is already fairly modular. If you want to add in a new subsystem, it isn't that hard to do. You are free to add your code as a dll or whatever you want.

Code reuse. I don't see where dynamic linking buys you this. How is linking Torque in as a dll going to make your code more reusable. If your code is properly object oriented and abstracted, you should be as engine independant as you can be either way.

The only advantage to dynamic linking is that you can change out the dll at runtime. Everything else is *exactly* the same as a statically linked library.

Now, as Tim said above, there is some potential value to splitting the code up into several dlls/libraries, as this would let one do things like preview a model's appearance in 3d studio MAX with just the rendering engine linked in.

It seems like your main problem is running into link errors where both torque and your project are pulling in a dll. The solution to that is to just have one of the two pull in the dll. You don't need to recompile torque as a dll to do that.

Josh

#7 Dynamic linking isn't exactly like static linking except for switch outs at runtime... this is only one of it's virtues... a somewhat "more important" one is the ability to use the SAME code in DIFFERENT modules... EVERYBODY using Torque working in the SAME module is a bit odd wouldn't you say? :)


Dynamic linking allows multiple people working on multiple projects to use Torque in a consistant manner ... and not as some mutation of Torque Demo... we can use Torque consistantly in our projects and subprojects... we are not limited in any way WHERE we can use Torque functionality... AND code can be more readily shared...

This is exactly why the engine should be compiled with Multithreaded DLL code generation, it works for all cases.. people threading or not...

All of this for just adding export declations which can be turned on or off with a define! So if anyone is convinced dynamic linking buys them nothing; they can go on with having the entire engine build down at the lowest level: the application.exe

-J

#8 I'm sorry, but that is not quite correct.

Static linking lets you still enforce a contract, via header files. The header files are actually the *exact* same header files you use when dynamically linking, except for __declspec(dllexport) before each function or class.

You can still have some people working on the engine and others working on whatever you are linking the engine with and have those codebases seperate.

Josh

#9 I am not here to argue the differences between dynamic and static linking... these are documented...

I've said enough on the subject... one last battle cry:

"Torque on top!"

-J

#10 I agree that a Torque DLL makes perfect sense. Allows for a more modular use of the engine rather than hacking the included demo program.

I think it would even make sense to pull Torque completely out of the demo program. Make Torque *only* a DLL, and the demo program just uses the DLL. Though that would make porting to other OS platforms a bit different. I don't program for the Mac or Linux, so I have no idea how DLLs transfer across.

A side benefit of a DLL is that it makes Torque available to other languages and development environments, not just VC, CW, and GCC.

My thoughts on the issue...

-David
Samu Games

#11 The DEMO resides in script. Delete the scripts and the DEMO no longer exists. The EXE is the core engine.

#12 Yup.. the core engine is the .exe

The DEMO is script AND native code.. see player.cc... remove either and it no longer exists...

Unless all Torque development is happening in script,I am not sure how your comment impacts the issues presented?

-J

#13 player.cc defines how part of the engine works.

remove player.cc and you can no longer use the player object. But you could still use the vehicle objects. It would NOT prevent you from creating a game if the player.cc was not compiled into the engine. The capabilities from player.cc just would not be there.

#14 Script is great, the engine is great, people can make a great game with the engine...

This doesn't mean that the engine isn't being worked on and improved... heck, today there was a TODO list posted...

I would dearly like to see this added to that list... I certainly think the work involved and the benefits gained warrant this...

"Torque on Top!"

-J

#15 I've gotta go with Joshua on this. I guess I would add this... Creating separate dlls for the different parts of the engine only seems to make sense for a number of reasons. Don't mean to step on any toes, but if I do accidentally, just take this early apology, and realize these are my HUMBLE opinions and experiences. ;)

1. Being able to link to prebuilt dlls would save memory, reduce swapping, and allow multiple processes to reuse the same code. Lets say you wanted to create Game.exe and GameEditor.exe, if you had the core in dlls, you wouldn't have to include massive piles of source code, and get all the directories setup in your project. You could just link to the dlls.

2. Updates. If anyone has ever worked on large projects should know, sharing code files is a major blunder. What happens when you have 5 or 6 different apps all sharing the same code base, and a bug is discovered? Now you've hopefully only linked to shared code, and not included it separately in all the projects, but I've seen it, and it's a pain to clean up. Instead of having to deal with the bug in one spot, it's potentially in all of them. Cut, Copy, and Paste is not what is meant by code reuse. If the code were in dlls, and a bug was found, the code could be fixed, dlls rebuilt, and shipped out, without having to modify the executables that were linking to it, as long as the interfaces and return values weren't changes. Take for example programs that have auto updating to fix bugs. How easy it would be to just download a new dll, sling it into the common directory for those 5 apps, and whack, all five are now patched. You could do that with the source files, and an exe, but yikes... when I want to update something, I'd be more likely to download a 100k dll, as opposed to a big phat 2 mb exe. Some ppl aren't blessed with broadband... dlls could make that process a little simpler.

3. Sharing. Lets say I'm working on a project, and the dude coding the network objects finds he's got some great new fix for something. Perhaps I don't really care to see the files, or have to look at them in my project. I just wanna link to the dll and go, knowing the dude fixed it. I for one don't wanna have to look at the massive amounts of code I'm not gonna touch, zlib, and lpng, and so forth. Let the guys that get off on that stuff give me a dll that works, and can be updated. I don't want the source, chances are if it was broken and I'm not up on that topic, I probably couldn't fix it anyways.

4. Multilanguage, Multiplatform. Isnt' that one of the ideas of going multi this multi that, is that if I wanna go to Spain and sell my latest FPS, I don't have to rebuild my exe? I just add a new set of Spanish resource dlls? What about Multiple platforms? Why do I want to see the linus code if I know I'm only going for Windows? Just give me the win32 dll for networking and such and let me run with it.

5. Plugins, I for one would love to see some sort of plugin system, besides the scripting. Dlls are really a great way to implement that. I don't know of all the scripting in Torque, so I won't go there, but it seems like dlls would offer a 3rd party developer a great chance to make modifications to someone's game. Look at UT and Half-Life? Yes they use scripting, but the opportunity is there to use dlls if you're bold enuf.

I don't want to start a fire on this thing, and I'm sure I already have stepped on toes, but it seems that there are two camps with the engine design right now. I think it would be awesome to let a few guys dig in and convert the engine over to using dlls. Network, 3d, png, jpeg, zlib, scripting, and so forth. It's basically the same code, just approached from a different angle, and with a few symbols added. If someone wanted to tackle a project like that I'd love to help, and I'd love to see it in CVS. Then for all thos developers out there that wanna use the entire source in their project, great, it's still there. But, for those that wanna go a bit further, and get some dlls and go, great too!

Anywhoo, that's my 2 cents and then some. Keep talking guys, this could be really fun, and help a lot of people coming up, learn a lot of basics. I would have killed to have been able to see a project like this converted to using some dlls and to see the necessary exports and tons of other concepts.
laterz

#16 I would like to see stuff broken up a bit too..._BUT_ the more immediate thing would be to take about the whole thing... and export about everything... this is a define to export the classes, not some sweeping change... from here it would be possible to break off renderers and stuff... if desired...

Frankly, in my world I don't plan on touching too much of anything in the CVS, except for productivity hindering bugs... I also don't really mind that it is all lumped into one spot... give me a 5 meg debug dll and a 2 meg release dll... as long as I (and others) can link to the engine...

"Torque on Top!"

-J

#17 To me, this would be one of those "nice to have" features that should be at the bottom of the to-do list, saved for that rainy day when the to-do list is more of a done list ... unless someone wants to take it on as their pet project. I'm sure GG would look at it really closely if it is presented as a fait accompli just needing sign-off.


but first:

1. fix bugs in existing feature set
2. add "missing" features (essentially the T2 ripped-out stuff. plus things like doors that fairly basic in terms of feature list)
3. add "state-of-the-art" core technology features (ie. vertex shaders)

#18 This wouldn't qualify as a pet project... there isn't enough work to do...

I stated above that I would do the work... but as it is a MINOR change to about every header file... there probably is a better candidate...

What this means to coders working with Torque transcends the "nice" category...

-Joshua "Torque on top!" Ritter

#19 Don't get me wrong, I know for a fact I couldn't have written this thing all alone. There is some truly amazing things in there. I just can't really figure out why it wasn't designed that way to begin with...? I mean for the love of god, how could a company that can come out with a game like Tribes, and figure out all this scripting, and gui controls, things that I feel are freakin awesome as anything... yet not see fit to break this thing down a bit... That just seems strange. I know if I was planning a huge project, something of this size, i'd be all over specifying the interfaces, and getting some guys doing each one, each aimed at some dlls. Just seems easier to manage, but then again that's just me. And yes... I could be crack head that knows nothing of design, or it could be that maybe some or most of it got pushed aside for all the other advanced stuff, I just can't for the life of me understand the reasons not to get some of it broken off...
I'm gonna go check with my therapist... lol

#20 The engine's design is very good... it's code seperation is perhaps a little lacking.. but hey...

Let's try and stay on the subject of Dynamic Linking... how the engine should/could/would be broken up is another matter...

"Torque on top!"

-J