Compiling Multiple Projects from the Same Source Tree

Updated:

The resource has been updated to reflect the awesome feedback from the community!

Introduction:

Inspired by Philip Gregory's resource: "A resource for musicians" I decided to make a simple tutorial for maintaining the engine code base. The idea behind this is that T3D in particular shares the code base between projects. So if you modify the source code from the engine you affect every project. This may not be what you want. In my case it was not. I found this out the hard way and had to reinstall the T3D source to get the original code.
Note: GG, if you are listening, please provide an option for installing pieces of each release and/or selective repair of the install from the install program. That way when reinstalling only to get a portion of the engine one does not have to wait 15 to 20 minutes for the install to happen. I did try unzipping the file, but there is a binary wad of data I could not make heads or tails of. Thanks in advance.

Structure:

It is all about structure. The layout of the Torque 3D 1.2 directory is as follows for the relevant source directories:

- Torque 3D
-- Engine
--- source
-- My Projects
--- "project name"
---- source

Now the 2 directories we care about are the "source" directories. These contain the engine source (Torque 3D->Engine->source) and the project source (Torque 3D->My Projects->"project name"->source). Knowing the location of these 2 directories is important for what I will cover next. Use this is a reference.

Usage:

How does this help? Well, Microsoft Visual C++ (2008 for the example) has a great way to take advantage of overriding files from a project. This is a good thing. The way you take advantage of this feature is simple.

Say you have to make a change to a file in the T3D source. Lets say that you need to make a SimObject have a virtual function so you can override it in a later SimObject based class. This will mean that if you make this change it will affect every project that uses this same source code base. Unless, you take advantage of overriding a source file from the project source directory. For this example to make this change we would have to modify the file SimObject.h.

So in MS Visual C++ we would open our project and look in the "Solution Explorer" on the left side. Now there is going to be a DLL project named <project name> DLL. Under this you can drill down to the SimObject.h file like this: Source Files->Engine->console->SimObject.h. Now, right click on the file and select Properties. This will pop up a dialog. Find the option "Exclude From Build". This won't actually affect .h (header files), but it will put a red mark on the file. This is a visual indicator that you intend to override this file and just helps you later when tracking down which source file is overridden somewhere else.

Now we need to find the SimObject.h file in Windows Explorer so we can make a COPY of that file from the Torque 3D->Engine->source location to the Torque 3D->My Projects->"project name"->source location. Do not move the file, COPY the file. The path in Explorer would be Torque 3D->Engine->source->console->SimObject.h. Then it would be copied to the Torque 3D->My Projects->"project name"->source->console directory. Make sure the directory structure under "source" matches the directory structure under "Engine". It would look like this:

- Torque 3D
-- Engine
--- source
---- console
-- My Projects
--- "project name"
---- source
----- console

It should now be compiling against the new SimObject.h. To make sure we know about the file in the project we should add it back to the project in the Solution Explorer. To do this go into the project explorer again and right click on the "source" directory and choose Add->Existing Item. This will just add a indicator that this is part of the project for .h files. Now the SimObject.h file is part of the project again. Any changes you make to this copy of the SimObject.h will not affect other projects you create later on unless you decide to merge this file back into the main source directories again.

Great, but I have a huge project and I don't know what I changed in the main T3D source!

That is where I was at. It was a big project (for me at least) and I did not know everything I had changed from the original source. The solution was to reinstall T3D in a different directory and copy the source back to the original location of the source from my current T3D installation. I backed up my original source directory first by changing the directory name. Then I downloaded a great tool: gnuwin32.sourceforge.net/packages/diffutils.htm It allowed me to do a diff on the original source file directory against my changed source file directory. I ran a command window in the Torque 3D->Engine directory (-r is for recursive):

diff -r "source" "mysource" > changed.txt

So if "source" is the default source code for T3D and "mysource" is my backed up source files I have changed, then changed.txt will produce a diff that will show what changed from source to mysource. This could be applied by a program called "patch" (I think) to change the source to match your changes. Anyway, the changed.txt file can be used to show which files changed like for my project:

diff -r source orig 1.2/app/mainLoop.cpp source/app/mainLoop.cpp
373c373,383
<    Platform::setCurrentDirectory( Platform::getMainDotCsDir() );
---
>    //if(strnicmp(Platform::getExecutableName(),"python",6) == 0){
>    if(dStrnicmp(Platform::getExecutableName(),"python",6) == 0){
>       // if being launched from Python
>       Platform::setMainDotCsDir( Platform::getCurrentDirectory() );
>       //Platform::setCurrentDirectory( Platform::getCurrentDirectory() );
> 	  Torque::FS::Unmount( "game" );
>       Torque::FS::Mount( "game", Platform::FS::createNativeFS( ( const char* ) Platform::getCurrentDirectory() ) );
>    }else{
>       // if launched from Torque Toolbox or the executable wrapper for the DLL
>       Platform::setCurrentDirectory( Platform::getMainDotCsDir() );
>    }
diff -r source orig 1.2/console/compiledEval.cpp source/console/compiledEval.cpp
1297c1297
<             if(curObject)
---
> 			if(curObject)
1726a1727,1737
>                         break;
>                      }
> 					 // pyT3D
> 					 case Namespace::Entry::extScriptCallbackType:
>                      {
> 						const char *ret = nsEntry->cb.mScriptStringCallbackFunc(gEvalState.thisObject, nsEntry->mNamespace, callArgc, callArgv);
>                         STR.popFrame();
>                         if(ret != STR.getStringValue())
>                            STR.setStringValue(ret);
>                         else
>                            STR.setLen(dStrlen(ret));
diff -r source orig 1.2/console/console.h source/console/console.h
17a18,21
> // pyT3D
> //#ifndef _CONSOLEINTERNAL_H_
> //#include "console/consoleInternal.h"
> //#endif
119a124,126
> 
> // pyT3D
> typedef const char * (*ScriptStringCallback)(SimObject *obj, Namespace *nsObj, S32 argc, const char *argv[]);
diff -r source orig 1.2/console/consoleInternal.cpp source/console/consoleInternal.cpp
1227a1228,1243
> // pyT3D
> void Namespace::addScriptCommand( StringTableEntry name, ScriptStringCallback cb, const char *usage, S32 minArgs, S32 maxArgs, bool isToolOnly, ConsoleFunctionHeader* header )
> {
>    Entry *ent = createLocalEntry(name);
>    trashCache();
> 
>    ent->mUsage = usage;
>    ent->mHeader = header;
>    ent->mMinArgs = minArgs;
>    ent->mMaxArgs = maxArgs;
>    ent->mToolOnly = isToolOnly;
> 
>    ent->mType = Entry::extScriptCallbackType;
>    ent->cb.mScriptStringCallbackFunc = cb;
> }
> 
1300a1317,1319
> 	  // pyT3D
>       case extScriptCallbackType:
> 		 return cb.mScriptStringCallbackFunc(state->thisObject, this->mNamespace, argc, argv);
diff -r source orig 1.2/console/consoleInternal.h source/console/consoleInternal.h
77c77,79
<             BoolCallbackType
---
>             BoolCallbackType,
> 			// pyT3D
> 			extScriptCallbackType
126a129,130
> 			// pyT3D
> 			ScriptStringCallback mScriptStringCallbackFunc;
171a176,178
> 
> 	  // pyT3D
> 	  void addScriptCommand( StringTableEntry name, ScriptStringCallback, const char *usage, S32 minArgs, S32 maxArgs, bool toolOnly = false, ConsoleFunctionHeader* header = NULL );
diff -r source orig 1.2/console/scriptObjects.h source/console/scriptObjects.h
23c23,24
<    bool onAdd();
---
>    // pyT3D : made virtual
>    virtual bool onAdd();
diff -r source orig 1.2/console/simObject.h source/console/simObject.h
435c435,436
<       const char *getDataField(StringTableEntry slotName, const char *array);
---
> 	  // pyT3D added virtual
>       virtual const char *getDataField(StringTableEntry slotName, const char *array);
445c446,447
<       void setDataField(StringTableEntry slotName, const char *array, const char *value);
---
> 	  // pyT3D added virtual
>       virtual void setDataField(StringTableEntry slotName, const char *array, const char *value);
diff -r source orig 1.2/main/main.cpp source/main/main.cpp
25c25,26
<    sprintf(gameLib, "%s.dll", filename);
---
>    //sprintf(gameLib, "%s.dll", filename);
>    sprintf(gameLib, "_pyT3D.pyd", filename);

I was actually pleasantly surprised that I had changed relatively few original engine source files. I ended up having to move only 7 files to my project source directory. To help you understand how to determine which files changed you can see my list of changed files:

compiledEval.cpp
console.h
consoleInternal.cpp
consoleInternal.h
mainLoop.cpp
scriptObjects.h
simObject.h

Note: I did have to make some functions virtual in simObject.h. Can you spot them?

Conclusion:

I hope this all made sense for the people who would take advantage of this feature of MS Visual C++. I really like the ability to add and remove files at will. This makes it much simpler to keep code bases cleaner. This is probably an obvious feature to experienced coders, but for newer people blindly applying code resources it may save them some pain.

Enjoy

Thanks for corrections and suggestions:

1. Guy Allard for being the guinea pig and testing what I thought was working and wasn't. Guy is the guy who determined that .h files would not play nice unless in a compatible directory structure on the disk.
2. Lukas Joergensen for suggesting using WinMerge. That should help those console challenged folks out there.
3. Kerry Enfinger and Dan Keller for suggesting the use of version control software. That is definitely a good point that this method is not intended for that purpose. So make sure you use version control in addition! This is only for ease of compilation multiple projects from the same source.