Convoluted Console
by Jacob Dankovchik · in Torque 3D Professional · 08/04/2013 (3:34 am) · 13 replies
So I'm working on a function idea for Torque that deals with the console. I write the basic functions for what I need and then time comes to tack it on in to the console itself. Nice and simple, whenever you hit enter it checks this system and if good, runs the operation. Simple, right?...
First trick was finding exactly where the moment of entering something in to the console happens. I finally got it all tracked down and that's where the truly shocking moment came for me, and why I'm posting about it. I'm curious if it's just me, or if the console system truly is such a mess as it appears to be to me.
So the function where the magic seems to happen is actually the largest function I've ever seen... 1613 lines. Ok, no biggy, a lot of stuff gets sorted out all at once for entering console commands so I suppose it works. Nothing totally evil about such a large function. As for why? An enormous switch case 89 conditions large. Bit more than expected... but ok..
So then the problem comes in with what I'm trying to do, by greatly changing functionality of variables. So since it all points here, time to find out where entering a variable in to the console leads. And this is where things got a bit shocking. So the trick was to find out which case I need for a variable. I had a feeling there would be a few, so first step is finding out how many times the switch is run for calling on an existing variable.
How many does that work out to? 230. For calling on a variable, the giant switch is run 230 times. 228 for setting an existing variable to a new value. (curiously, changing is less than repeating an existing) So I wondered how many times it runs just for opening and getting to the main menu. 381,105 times. Even more impressive? A single shot of your weapon: 20,403 cycles of this huge switch.
The significance in this to me isn't really about performance. Granted, I'm sure better performance could be found here and if this is any indication, many other places in the engine. The real concern to me is just how all over the place the console seems to be. In trying to do my little project, what I expected to be relatively simple has turned out to be amazingly messy. Simply checking for, and reporting back the value of an existing variable created 230 executions of this check. Does anyone really know why it has to be like this?
It particularly surprises me SO much because I have a habit to occasionally look back on what I first learned on, the Quake 1 engine. So I looked towards Q1's console system. Granted it's far more simple in what it can do, but nobody would argue it's plenty functional. And with that in mind, there's not a single instance of something I just don't understand for being too complex. In fact there are many parts where I don't understand how it's so simple! Seems like it shouldn't even function without more stuff going on. So a similar approach, entering an existing variable and having it report back? Didn't get around to trying to setup a proper debug situation to test exactly, but was able to trace the line of events easily. About 10 different functions, each around 50-80 lines. No cycles of switches, no huge repeats of anything, and only 1 For statement that checks for the known variables.
I guess my point here is why is Torque's console just so... frightening? It defies my understanding for why it's THIS bloated. Surely we can do things simpler? And if so, is it something others would be interested in? This small project of mine may just expand into something bigger at this rate. Because right now, I want to smash my head through my keyboard from sifting through this.
First trick was finding exactly where the moment of entering something in to the console happens. I finally got it all tracked down and that's where the truly shocking moment came for me, and why I'm posting about it. I'm curious if it's just me, or if the console system truly is such a mess as it appears to be to me.
So the function where the magic seems to happen is actually the largest function I've ever seen... 1613 lines. Ok, no biggy, a lot of stuff gets sorted out all at once for entering console commands so I suppose it works. Nothing totally evil about such a large function. As for why? An enormous switch case 89 conditions large. Bit more than expected... but ok..
So then the problem comes in with what I'm trying to do, by greatly changing functionality of variables. So since it all points here, time to find out where entering a variable in to the console leads. And this is where things got a bit shocking. So the trick was to find out which case I need for a variable. I had a feeling there would be a few, so first step is finding out how many times the switch is run for calling on an existing variable.
How many does that work out to? 230. For calling on a variable, the giant switch is run 230 times. 228 for setting an existing variable to a new value. (curiously, changing is less than repeating an existing) So I wondered how many times it runs just for opening and getting to the main menu. 381,105 times. Even more impressive? A single shot of your weapon: 20,403 cycles of this huge switch.
The significance in this to me isn't really about performance. Granted, I'm sure better performance could be found here and if this is any indication, many other places in the engine. The real concern to me is just how all over the place the console seems to be. In trying to do my little project, what I expected to be relatively simple has turned out to be amazingly messy. Simply checking for, and reporting back the value of an existing variable created 230 executions of this check. Does anyone really know why it has to be like this?
It particularly surprises me SO much because I have a habit to occasionally look back on what I first learned on, the Quake 1 engine. So I looked towards Q1's console system. Granted it's far more simple in what it can do, but nobody would argue it's plenty functional. And with that in mind, there's not a single instance of something I just don't understand for being too complex. In fact there are many parts where I don't understand how it's so simple! Seems like it shouldn't even function without more stuff going on. So a similar approach, entering an existing variable and having it report back? Didn't get around to trying to setup a proper debug situation to test exactly, but was able to trace the line of events easily. About 10 different functions, each around 50-80 lines. No cycles of switches, no huge repeats of anything, and only 1 For statement that checks for the known variables.
I guess my point here is why is Torque's console just so... frightening? It defies my understanding for why it's THIS bloated. Surely we can do things simpler? And if so, is it something others would be interested in? This small project of mine may just expand into something bigger at this rate. Because right now, I want to smash my head through my keyboard from sifting through this.
#2
08/04/2013 (4:54 am)
Still, those numbers seem excessive. But yes, it's not just the console, it's TorqueScript bytecode you're dealing with now.
#3
I've gotten my little thing plugged in and working at this point and as I go in more, it just seems more and more messy, to be honest. It's like one function was tacked on to another and so on forever, until it becomes a tangled pile instead of working together.
08/04/2013 (5:18 am)
Yeah, I know TS is tied in to it. What shocks me though is just how excessive every action seems to be. Again, the 230 iterations just to report the value of an existing variable.I've gotten my little thing plugged in and working at this point and as I go in more, it just seems more and more messy, to be honest. It's like one function was tacked on to another and so on forever, until it becomes a tangled pile instead of working together.
#4
So what it is I'm working on is getting a solid CVar system in Torque, much like Quake's. The idea here is getting key numbers in the game, like graphics, sound, and world settings, setup in a cvar. Makes changing these numbers super easy without having to create new engine functions for each one and opens access to it throughout the engine, just have to plug in the cvar name. This will allow for much easier control over all the little numbers that play an important role.
The next step from here is to clean up the code and functions a bit and get it polished out a bit right. After that, bring it properly in to script which will be the bigger task here. My end goal here is to eliminate most of the "$" global variables of script. There's just something about that system I never really cared for. Plus doing this will make tying in variables from script to engine extremely easy and straight-forward, and vice-versa.
Not to mention the changes it could do for what I was talking about earlier. Instead of 230 cycles of switches and if's for changing a variable, you get a very clean, short, and straight-forward chain of events.
08/04/2013 (11:48 am)
I ended up going to bed at 7 am, got on such a good roll with this. Got a couple hours of sleep and thought I'd show what's going on, maybe others will be interested.So what it is I'm working on is getting a solid CVar system in Torque, much like Quake's. The idea here is getting key numbers in the game, like graphics, sound, and world settings, setup in a cvar. Makes changing these numbers super easy without having to create new engine functions for each one and opens access to it throughout the engine, just have to plug in the cvar name. This will allow for much easier control over all the little numbers that play an important role.
The next step from here is to clean up the code and functions a bit and get it polished out a bit right. After that, bring it properly in to script which will be the bigger task here. My end goal here is to eliminate most of the "$" global variables of script. There's just something about that system I never really cared for. Plus doing this will make tying in variables from script to engine extremely easy and straight-forward, and vice-versa.
Not to mention the changes it could do for what I was talking about earlier. Instead of 230 cycles of switches and if's for changing a variable, you get a very clean, short, and straight-forward chain of events.
#5
www.garagegames.com/community/forums/viewthread/132794
Also, there are 90+ opcodes in the VM. That means there has to be 90+ cases in the switch statement. If you want to reduce the complexity of the VM you would need to work with the TS compiler potentially.
$ vars are the equivalent of a console variable in Torque. If you create a CVar system you will end up having to reference those vars in script. So either you need to add more opcodes to the compiler or tie it into the opcodes that lookup the $ vars. Or have some custom TS commands to manipulate CVars. That will add even more cycles through the VM.
It may not be the prettiest VM in the world, but it does function well. It might save you a lot of time and frustration to understand the VM first before you get mired into debugging foreign code. Trust me, I have been there.
08/04/2013 (12:16 pm)
You really should look at what James Urquhart has done with native type optimizations:www.garagegames.com/community/forums/viewthread/132794
Also, there are 90+ opcodes in the VM. That means there has to be 90+ cases in the switch statement. If you want to reduce the complexity of the VM you would need to work with the TS compiler potentially.
$ vars are the equivalent of a console variable in Torque. If you create a CVar system you will end up having to reference those vars in script. So either you need to add more opcodes to the compiler or tie it into the opcodes that lookup the $ vars. Or have some custom TS commands to manipulate CVars. That will add even more cycles through the VM.
It may not be the prettiest VM in the world, but it does function well. It might save you a lot of time and frustration to understand the VM first before you get mired into debugging foreign code. Trust me, I have been there.
#6
The thoughts within the OP were really just about an observation I made that I found rather shocking. Ultimately though, as I mentioned there, the performance difference of what's in place and something that's improved upon (which is surely possible) probably wouldn't even be worth the effort. Would make me feel better if I'm able to dodge it, but if not it doesn't matter anyhow.
08/04/2013 (12:22 pm)
Heh, I will admit mostly this is just about me playing around with a thought. But also, the original post here really isn't what it's about anyhow. If I still have to work within the system, oh well. It's really about getting a global variable system going that's a bit more intuitive and more easily tied together between script and engine.The thoughts within the OP were really just about an observation I made that I found rather shocking. Ultimately though, as I mentioned there, the performance difference of what's in place and something that's improved upon (which is surely possible) probably wouldn't even be worth the effort. Would make me feel better if I'm able to dodge it, but if not it doesn't matter anyhow.
#7
Out of curiosity I looked at LUAs VM and opcode list:
www.lua.org/source/5.2/
The LUA opcode list is 1/3 as long. The VM switch statement is about 350 lines of code. So it looks like to me that the size of the switch statement might be related to the complexity of the opcode list.
I do think there is a lot of gain to be had working with the Torque Script VM. A new set of eyes would be a good thing. One thing I have worked on is separating all the opcodes out to individual functions. The reason for this was to try and get the VM to take advantage of tail call optimization. I did not see a difference in execution speed, but I am not sure if the issue was string lookups actually dwarfing any small increase in speed the tail call would provide. I know James has seen a lot of speed increase. There are just some issues that need to be worked out.
08/04/2013 (2:01 pm)
Some of the biggest performance hits you get are string lookups. That is what James addressed in his work.Out of curiosity I looked at LUAs VM and opcode list:
www.lua.org/source/5.2/
The LUA opcode list is 1/3 as long. The VM switch statement is about 350 lines of code. So it looks like to me that the size of the switch statement might be related to the complexity of the opcode list.
I do think there is a lot of gain to be had working with the Torque Script VM. A new set of eyes would be a good thing. One thing I have worked on is separating all the opcodes out to individual functions. The reason for this was to try and get the VM to take advantage of tail call optimization. I did not see a difference in execution speed, but I am not sure if the issue was string lookups actually dwarfing any small increase in speed the tail call would provide. I know James has seen a lot of speed increase. There are just some issues that need to be worked out.
#8
If you want to have an indebt chat with someone about the evils of the console, chat w Vince Gee. One of the main things we have been working on is reducing the usage of the console, if not removing it completely. He knows that stuff inside and out.
08/04/2013 (5:32 pm)
Jacob:If you want to have an indebt chat with someone about the evils of the console, chat w Vince Gee. One of the main things we have been working on is reducing the usage of the console, if not removing it completely. He knows that stuff inside and out.
#9
08/04/2013 (10:08 pm)
I daresay that adding another system in parallel to the console system isn't necessarily going to reduce complexity for anyone ;P. But if it works for you, great, and it's good to get more pairs of eyes examining this part of the engine.
#10
not more complex, just different. it starts to adhere to a standard thats not GG/T3D based.
but you're right. Do what works for you :)
08/05/2013 (3:54 am)
@Daniel:not more complex, just different. it starts to adhere to a standard thats not GG/T3D based.
but you're right. Do what works for you :)
#11
My suggestion would be to be a bit more realistic with what you expect to get out of refactoring the system compared to how much time your going to spend on it. If you are not careful you can easily spend months trying to untangle what is a core mess in Torque3D.
08/05/2013 (10:42 am)
Torque's Console system is a full-blown script interpreter complete with namespacing and objects, logging, and persistence. Quake 1's console system on the other hand is a simple variable registry combined with a basic interpreter. The two systems are about as equivalent as a paper calendar is to an iPhone.My suggestion would be to be a bit more realistic with what you expect to get out of refactoring the system compared to how much time your going to spend on it. If you are not careful you can easily spend months trying to untangle what is a core mess in Torque3D.
#12
iPhone = fancy shmansy, wibbly wobbly, timey wimey machine
08/05/2013 (12:10 pm)
paper calender = old school appiPhone = fancy shmansy, wibbly wobbly, timey wimey machine
Torque Owner Demolishun
DemolishunConsulting Rocks!
Are you talking about the interpreter (in compiledEval.cpp)?
If this is what you are referring to it is the heart of Torque Script. The console would be just an interface to that interpreter. The engine is evaluating op codes for every script command that goes through. There is a lot going on in there.