Improving TorqueScript
by James Urquhart · in Torque 3D Professional · 11/05/2012 (11:52 am) · 50 replies
Hey everyone. A while ago as a side-project I decided to have a go at improving TorqueScript ( see http://www.garagegames.com/community/forums/viewthread/131792 ).
The previous "Torquescript has suboptimal performance" thread seems to have pretty much died down, so I figured it best to start from a fresher slate. I'm also interested in everyones perspective on this issue: How can TorqueScript be improved?
At the moment I'm not looking to completely replace the scripting engine, rather I'm focussing more on improving it. e.g. reducing its overuse of strings, adding useful features.
In my fork, besides fixing the "all function parameters are strings" issue (which I made a pull request for), I've revamped the type system so native value types can be used to represent the complex types as opposed to just strings (much like how complex types might be represented in something like Lua). I've also added support for value-based arrays so "object.field = {1,2,3};" and "%value = {1,2,3};" now actually works, though they don't work in expressions since it introduces too much ambiguity in the parser syntax which is a PITA to resolve.
Does anyone have any interest in these changes?
The previous "Torquescript has suboptimal performance" thread seems to have pretty much died down, so I figured it best to start from a fresher slate. I'm also interested in everyones perspective on this issue: How can TorqueScript be improved?
At the moment I'm not looking to completely replace the scripting engine, rather I'm focussing more on improving it. e.g. reducing its overuse of strings, adding useful features.
In my fork, besides fixing the "all function parameters are strings" issue (which I made a pull request for), I've revamped the type system so native value types can be used to represent the complex types as opposed to just strings (much like how complex types might be represented in something like Lua). I've also added support for value-based arrays so "object.field = {1,2,3};" and "%value = {1,2,3};" now actually works, though they don't work in expressions since it introduces too much ambiguity in the parser syntax which is a PITA to resolve.
Does anyone have any interest in these changes?
About the author
#22
"s1" @ "s2" = "s1s2".
So it would make sense for there also to be @=. I've often tried to use it as intuitively it should be there along with +=, -= *= etc.
11/08/2012 (12:30 am)
Well, @ is the torquescript string concatenation operator. "s1" @ "s2" = "s1s2".
So it would make sense for there also to be @=. I've often tried to use it as intuitively it should be there along with +=, -= *= etc.
#23
So I added @= in what seemed like the most obvious fashion to me. When I use it in the console, I get this error:
11/08/2012 (2:20 am)
Yes, I think it just makes sense from a completeness point of view, and it should only actually be a change to the grammar, not to the compiler or anything else. I might see if I can whip that up sometime. Also, if you're wondering why you'd use it over SPC or regular @, maybe consider this for loop to generate a random hex number:%hexDigits = ...;
%num = "0x";
for(%i = 0; %i < 8; %i++)
%num @= %hexDigits[getRandom(15)];I think the @= looks perfectly at home there!So I added @= in what seemed like the most obvious fashion to me. When I use it in the console, I get this error:
==>$str = "haha"; haha ==>$str; haha ==>$str @= "ha"; precompile size mismatch, precompile: 12 compile: 9 ==>$str; hahaNot enough blood in me right now to debug this, though I did find a comment that mentioned enabling an AST debugging define, which I will try if nobody can be immediately helpful. Figured I'd ask first since there are some TS VM gurus hanging around this thread!
#24
The reasoning is, lets say you have 2 variables, and they're digits. %a = 1 and %b = 1
In TS vanilla, %a += %b would yield %a == 2.
If += also can act as concatenation(not sure how robust James' work with natural types is) you may run into the problem of it opting to concatenate instead of adding, meaning %a == "1 1", which wouldn't be the desired result. Guy raises a good point about @ already being syntax for string concatenation, so I'd have to say we keep them separate to prevent any sort of mixup(unless James doesn't think this would happen with the types handling). keeping -= and += as math functions, and @= as a string function makes sense, and would keep things logically separate.
11/08/2012 (2:25 am)
I'm actually going to have to speak against Richard's suggest with using += to concatenate strings.The reasoning is, lets say you have 2 variables, and they're digits. %a = 1 and %b = 1
In TS vanilla, %a += %b would yield %a == 2.
If += also can act as concatenation(not sure how robust James' work with natural types is) you may run into the problem of it opting to concatenate instead of adding, meaning %a == "1 1", which wouldn't be the desired result. Guy raises a good point about @ already being syntax for string concatenation, so I'd have to say we keep them separate to prevent any sort of mixup(unless James doesn't think this would happen with the types handling). keeping -= and += as math functions, and @= as a string function makes sense, and would keep things logically separate.
#25
11/08/2012 (2:48 am)
For the record, I agree with Jeff - until/unless TS is properly type-aware, + and @ probably need to be separated. 
#26
No way, {1,2,3} is the C/C++ way of which is what TorqueScript/TribesScript is based of off, hence why .cs files stand for C Script. ;)
11/08/2012 (10:02 am)
Quote:In this case I just decided to extend the "obj.field = {1,2,3};" syntax which wasn't implemented correctly, so the {} just stuck. [] could certainly be used too.
No way, {1,2,3} is the C/C++ way of which is what TorqueScript/TribesScript is based of off, hence why .cs files stand for C Script. ;)
// C/C++ Example
int nums[] = { 1, 2, 3, 4, 5 };
// So this feels correct:
%nums = { 1, 2, 3, 4, 5 };
// Does not feel right:
%nums = [ 1, 2, 3, 4, 5]; // Declaring an array using the array subscript operator? WTF...
#27
Also, pet peeve: why .cs instead of .ts? It makes it a pain when everything assumes they're C# files.
11/08/2012 (2:11 pm)
Okay, fair point. Instead of associating TS with C-like languages, I tend to associate it with other 'scripting' languages - JavaScript and Python primarily, even though it unfortunately has very little in common with the latter. Most other modern languages I know of use the [] style for literal arrays, though Lua is a notable counter-example.Also, pet peeve: why .cs instead of .ts? It makes it a pain when everything assumes they're C# files.
#28
11/08/2012 (2:15 pm)
Quote:That goes back to Tribes when the .cs file type was meant to indicate c-like script. A few years later Microsoft comes along with C# which causes the current confusion for noobs who think .cs means C# scripting. We were here first ;)
Also, pet peeve: why .cs instead of .ts? It makes it a pain when everything assumes they're C# files.
#29
Will see if I can get @= working. Obv though it would be nicer if more generic concatenation could be done with operator overloading.
@jeff
A good point. Wouldn't be a good idea unless the entire thing was refactored to remove such ambiguities or at least make them less ambiguous.
@daneil
The compiler is a bit dumb in places when deducing types for the AST nodes, so sometimes it will miss adding an instruction, especially when you start changing the syntax.
As Michael said, the ".cs" thing goes back to Tribes. Nowadays I tend to think of the "c" as meaning "c-like" or "console".
11/09/2012 (12:54 am)
@joshuaWill see if I can get @= working. Obv though it would be nicer if more generic concatenation could be done with operator overloading.
@jeff
A good point. Wouldn't be a good idea unless the entire thing was refactored to remove such ambiguities or at least make them less ambiguous.
@daneil
The compiler is a bit dumb in places when deducing types for the AST nodes, so sometimes it will miss adding an instruction, especially when you start changing the syntax.
As Michael said, the ".cs" thing goes back to Tribes. Nowadays I tend to think of the "c" as meaning "c-like" or "console".
#30
ATM I seem to have broken mission loading again, but arrays themselves work. Basically they function as follows:
Currently there is no way of determining length, nor is there any way to add or remove values. However these things can be achieved by adding a relevant console function.
"$value[expression]" can still create variables. If the interpreter detects $value is an array it will index into the array, otherwise it will just make a variable as usual. ATM there are still a few issues. e.g. the field sanitization currently prevents the arrays from being passed into the console type functions.
Have fun for now!
11/09/2012 (1:35 pm)
Ok, i've pushed my current changes here: https://github.com/jamesu/Torque3D/tree/consoletyperefactorATM I seem to have broken mission loading again, but arrays themselves work. Basically they function as follows:
$value = {1,2,3};
echo($value); // "1 2 3"
echo($value[0]); // 1
echo($value[1]); // 2
echo($value[2]); // 3
$value[1] = "100"; // 100 ("1 100 3")
MainMenuAppLogo.position = {10,10};Currently there is no way of determining length, nor is there any way to add or remove values. However these things can be achieved by adding a relevant console function.
"$value[expression]" can still create variables. If the interpreter detects $value is an array it will index into the array, otherwise it will just make a variable as usual. ATM there are still a few issues. e.g. the field sanitization currently prevents the arrays from being passed into the console type functions.
Have fun for now!
#31
So one could do this:
Though, this would look a bit strange at first look to everyone.
11/09/2012 (2:24 pm)
If someone's adding @=, think we could also add SPC= ?So one could do this:
> %var = "Hello"; // no space! > %var SPC= "World!"; > echo(%var); Hello World
Though, this would look a bit strange at first look to everyone.
#32
So SPC is actually a macro or shortcut for @ " " @.
Same principal applies for NL and TAB.
11/10/2012 (9:21 am)
@Steven Saric: SPC isn't actually an operator as it's more like an internal macro:%str = "Hello" SPC "World!"; // translates to %str = "Hello" @ " " @ "World";
So SPC is actually a macro or shortcut for @ " " @.
Same principal applies for NL and TAB.
#33
11/11/2012 (9:18 am)
How about the switch statement - I get really annoyed by the way that it doesn't fall through to the next case.
#34
11/11/2012 (12:55 pm)
I'm a bit unhappy with that as well. Unfortunately, changing it would break all existing uses of the switch statement. Not sure if that's a problem or not. If it is, we could consider doing something like JavaScript's "use strict" declaration. 
#35
Just Curious
11/13/2012 (7:46 pm)
If your looking for speed, why not use DNT. CSharp is a compiled language and handles memory far better than TS will. Just Curious
#36
i think the answer is "learning" ( Reinventing the wheel).
TS is a complete new language.through it one can find out how to implement a new scripting language.
but for DNT we have to use one of the supported language(already made by someone else ).although for productivity and performance DNT is better for game development.
+ DNT provides only dll file.
not it's source. right?
11/13/2012 (8:12 pm)
"If your looking for speed, why not use DNT. CSharp is a compiled language and handles memory far better than TS will."i think the answer is "learning" ( Reinventing the wheel).
TS is a complete new language.through it one can find out how to implement a new scripting language.
but for DNT we have to use one of the supported language(already made by someone else ).although for productivity and performance DNT is better for game development.
+ DNT provides only dll file.
not it's source. right?
#37
11/13/2012 (8:19 pm)
correct, everything is a dll when your game is ready to ship. 
#38
That is one thing that came out of the discussion in the other thread that it was unique in that regard. You don't need a scripting language compiled for the platform. Python is weak in this regard as well. So there is value in improving TS as a very portable language.
11/13/2012 (11:57 pm)
TS has the unique ability to be usable on a mobile device. It is completely defined by C++ which makes it very portable to any platform. With the future of T3D being open source it makes sense to have a language that can be compiled for multiple platforms and not rely on external sources.That is one thing that came out of the discussion in the other thread that it was unique in that regard. You don't need a scripting language compiled for the platform. Python is weak in this regard as well. So there is value in improving TS as a very portable language.
#39
Using DNT would be nice, but it's not an option for everyone considering licensing and cross-platform issues.
Also unless I'm mistaken DNT makes use of the same bindings as TorqueScript, right? If so it would also be subject to "stringitis" overhead, i.e. all object fields and console functions parameters are exposed as strings. This is one of the things I am resolving with these changes.
Interesting you should mention memory management. By default the Torque memory manager is disabled, which inflicts a notable performance hit to TS depending on how bad the OS memory allocation is.
11/14/2012 (1:23 am)
@vince:Using DNT would be nice, but it's not an option for everyone considering licensing and cross-platform issues.
Also unless I'm mistaken DNT makes use of the same bindings as TorqueScript, right? If so it would also be subject to "stringitis" overhead, i.e. all object fields and console functions parameters are exposed as strings. This is one of the things I am resolving with these changes.
Interesting you should mention memory management. By default the Torque memory manager is disabled, which inflicts a notable performance hit to TS depending on how bad the OS memory allocation is.
#40
11/14/2012 (3:24 am)
No, DNT doesn't use the same bindings, it takes the code from each console function and rolls a pinvoke for it thus bypassing the console completely. When you call the pinvoke you are passing typed variables. 
Torque Owner Joshua Leeming
Default Studio Name
Seriously tho... double thumbs up for Richard's suggestion I take back my own James...it is a bandaid, even if it is cleaner than %mystring = %mystring SPC "World";