TorqueScript has suboptimal performance

#41 @Frank fair points.

1) I'll admit, I've been abusing string parsing to fabricate array-esque variables by white-space delimiting data I need to pass between functions, and other such 'odd' workarounds to an array object, which while it has it's benifits, a proper, fast array type would be very useful and welcome.

2) I'll admit, I don't use iterators too much, so I guess I can't speak here. Do you have a language that has iterator functionality that I can look at to see what you feel is a better implementation?

3) I may not been thinking of JIT in the right sense, but what advantages does that have over what TS does now? Speed?

4) What situations do you see this coming into play?

5) In all fairness, MMOs tend to be rather special beasts, but TS seems like it'd work for most all other non-super-network-heavy game types. Can you elaborate on the 'server integration' part, like, what would be the ideal here?

6) Well, I can see this being a negative if you came in learning a different language, but there are plenty of script languages that are only vaguely similar to others, so this is more an inconvenience. Going from a project with language X to one with language Y would likely face the same issues, no?

7) Sorry, can you give an example? I'm probably not thinking of what you mean with this here.

I think spending the time and effort maintaining a less featured language is fine if in the maintenance, such flaws are fixed or reduced.
I'm curious how much you consider would be needed to qualify the 'fully prototype a game' qualifier. My current game project's actual game code is likely...80% TS, with some necessary C++ legwork to set it up. I haven't stress tested it with 32 players yet, but as far as I've seen thus, it's plenty quick for a lot of stuff unless you start doing some heavy code that is called constantly(the Tactical AI kit runs into this with it's in-depth visibility calculations every half second)

In perspective, my project is technically a port of an old UT99 mod I was a part of 'back in the day', which obviously was written entirely in UScript. UScript was clearly fast enough for Unreal's purposes, and the mod. And I've more or less been direct-porting most of the code into TS, and it so far hasn't failed me performance-wise. That said, I've got enough experience to try and stay away from current weak points and failings, and many others may not have the knowledge to do so, but I think with some polish, cleanup and modernization of some parts, TS is still perfectly viable.
Heck, once I get to the point of stress testing this thing, if it hasn't exploded when I try it, i'll even toss a build your way so you can preen through the code to see what kind of gameplay legwork I've done.

I completely agree that if you're going to have a script language, it needs to be able to build a game, not just act as engine-code glue like it has in the past, but I think it's actually rather close to being able to do that.
(Wow this got rambly)

#42 @James: Yeah, I agree that scripts in iTorque (T2D at all) is one of the major bottlenecks. But that's a bit different topic.. We are talking about T3D atm, right?

@Frank:

Quote:
5. Server integration limited. TS is fine for FPS, but has few features for integration with MMO servers.

Disagree. Our servers (large-scale MMO) running very smooth using TS. Actually, I prefer to have TS comparing to any other. And it can be easily extended to do anything you can imagine/want.
Yes, for newcomers, TS syntax is not "easy-to-get-in", but once you get it, you can do a lot and very fast. And its performance is enough to make a prototype with most of game logic build in TS.
And I've seen games where everything was done only by scripting (using stock EXE files).

I'm not against optimizations, it should be done where possible. Actually, we are (here, at DL) already looking at what can be done on that.
We have done some steps a bit aside.. rewrote about 60% of the console parser/eval/etc stuff just to make it ready for 64-bit (as soon as we have it semi-functional - will push it to our GitHub repo).

#43 My vote is for replacing TS with Lua for a few reasons.

1) It's fast.
2) It's MIT licensed, so no issues with that.
3) It's widely used and has tons of library support.
4) Doesn't have horrible tabs for scoping like Python does.
5) Lots of people already know it, and there are plenty of resources for learning it.

That said I still like TorqueScript, but I think for both speed and ease of use and wide appeal Lua would be a better choice.

#44 Why not Fortran or Ada? ;)

shootout.alioth.debian.org/u32/which-programming-languages-are-fastest.php

#45 Why not JIT-C++ :P Bet scripting in C++ would be something new :)

#46 @Ross LuaJIT has problems with C++ interoperability on windows x86
luajit.org/extensions.html#exceptions
Look under the heading

Quote:C++ Exception Interoperability

Which is enough of a reason to just use lua itself. Though, we are looking to speed up TS, not really replacing it. I even need reminding of that one.

#47 2) JS and Python have pretty good iterator support:

#python
list = [1,2,3,4,5,9]
for i in list:
  print i
// JS
for each (var item in [1, 2, 3]) alert(item);

3)This would be icing, but if we had to choose a VM why not choose one with that. Python does NOT have a good JIT solution, but V8 does.
4)Middle ware solutions. This would require the ability to bind binary portions as well. I am also very used to just grabbing libraries and using them for Python. No integration other than import.
5)Yes, but they are becoming increasingly popular. MMOs are really game oriented social networks. There is a lot of potential there. What I was thinking here is that there are servers that could use T3D server as a module rather than its own app. If this is not really an issue as Fyodor pointed out then I retract this point. This could be solved with an external binding to a scripting language to PHP or Python. So in that sense this in not a valid criticism.
6)C++,C,Pascal,JS,Java,Python,Lua and more are well known and there is a talent pool that is pretty large for any one of those. Torque Script, umm, what the heck is that? BTW, I really hate the scripting language in Oblivion. The only thing that made that actually fun to code in was OBSE. I would much rather have had Lua in Oblivion. I think TS is worlds better than Oblivion script.
7)Maybe this is not a fair point, but I really like the abstraction of JQuery and metaprogramming of Python. Especially in Python I can control just about everything including attribute access, init, allocation, array subscripting, operator overloading, pretty much anything. This allows for doing some behind the scenes work in an object. I can even access a function as an object and assign variables. I actually do this in my callbacks for the ScriptT3D interface to solve a TS simobject referencing issue between TS and Python.

Edit:
I am not against improving TS, but lets get the bindings to a point to where people can add support for other languages easier than they can now. I was mainly answering the question as to why one would want to replace TS wholesale. There are benefits to both.

#48 Frank: I definitely agree, there should be a way to plug in other languages if people are more comfortable with that. I just like to debate ;)

Actually, I've made an occasional comment about how I was writing an in-engine(in the editor suite) script editor that was going to be fully featured.

Originally, it started out as a joke, but me and my friends were talking about having a top-level interpreter for the editor, so you could have a language.config file, and build a syntax/language interpretation config. The script editor would read that, and then handle the language you're writing. At execution of the game, it would reference the config, translate to TS, and run the TS like usual in the background.
We'd started talking on it as a joke about having our game scripts as LOLSCRIPT or Whitespace for giggles, but I realized if it could be made robust enough, and the interpretation happens at the game run so there's no extra overhead for the interpretation step, it'd actually be stupidly useful for people wanting to use other languages.
Dislike TS, but like lua? load a lua.lconfig and the script editor will let you write lua code, etc.

Obviously, you'd want to optimize TS, because you wouldn't get the optimization benifits of the language, but it'd be a good transitional utility for people getting used to working with Torque and TS.

The definite end-goal idea would be the plugin functionality though.

Anywho, as for your points, I see what you mean about the arrays/iterators. I think this could be remedied by proper arrays + some helper console functions to iterate through them sensibly though.

As for executing binary middleware, is that something you'd really do in script, though? I mean, what kind of middleware are we talking about? If it's script that comes with a kit or something, you can just exec() and bam, you've got all the stuff in there. Or am I totally missing what you're meaning?

With 6, the interpreter thing I was talking above would be a bandaid in the short term, and a proper plugin system for deep integration would fix that handily.

For the abstraction, I can kind see what you mean with that, and it could be beneficial, though I haven't used Python extensively, so I can't say too much on that.
(Arg, I talk to much)

#49 Honestly bindings with T3D could be as simple as knowing what objects and what functions do what and where they are located. As long as your language of choice can bind via libraries / dll for c, c++ there shouldn't be a big problem. The only challenge to that is knowing what is where and what it does as well as operating with the objects and memory as they should be without mucking it up.

I've experimented with that via python, php and linking up erlang. Only had a problem when the language mucked things up with the objects. Then with TS In the engine, there is the interop there you have to deal with but that was never an issue. PHP has it's own issues, as far as the implementation is concerned and interop between it and anything is painful.

EDIT: Yes, it's a simplified view, but nothing stops anyone from compiling the engine as a dll and linking it in.

#50 @smally,
That is the binding interface I am talking about. consoleXMLExport provides function names and number of parameters, but it does not address parameter types. the engineXMLExport does not seem to return anything. This needs to be addressed for external bindings.

#51 I know I'm a bit late but I'd like to contribute to the 'why hate TS' discussion. I see a lot of people making the argument that if TS is too slow, you can just convert logic to C++. What I see happening in the games industry, and computer science in general (and mind you, I'm not widely experienced, this is mostly a student's perspective) is a shift towards programmer productivity and flexibility rather than code speed. Iteration is the key word - make something quickly, test it to see if it's fun, modify, test again. And if you're doing that, you don't want to be doing it in C++. At least, I don't think many people do. I wouldn't mind, but that's because I was raised on a crunchy diet of C++, and I'm considered something of a weirdo among my classmates. Obviously if you iterate down to a design you really feel works, and you run into performance problems, you can do some C++-ifying. But during development and testing I doubt you'd want to be doing that - it locks you in. Aside from having to recompile all the time.

So what I'm getting at is that TS (or whatever language replaces it) needs to perform adequately to handle a large burden of gameplay-related code. It also needs to be a high-productivity language, by which I mean you shouldn't have to do things like

Quote:abusing string parsing to fabricate array-esque variables by white-space delimiting data I need to pass between functions

for such basic programming tasks as passing arrays around. And it's not even something you should consider requiring new programmers to figure out.

Built-in functionality is also important. One of the main reasons I use Python is because of the massive library ecosystem. Almost everything you could want to do is included out of the box. I exaggerate, but for common programming tasks it's all but true. XML/HTML parsing, Sqlite interaction, set operations, regular expressions - they're just some of the libraries I've used recently.

EDIT: Sorry if that came out a bit ranty. To summarise: TS is fine, but why settle for fine? Programmer productivity is a big issue.

#52 @dB,
I have never been as productive as I am in Python. Although for a TS replacement I think something like V8 would be more appropriate maintainable. Considering that Python does not have JIT and also exhibits issues with multiple cores if you don't know what you are doing with multithreading. Python is also more geared to importing a codebase, while JS is designed to be embedded from the start.

Another VM that would be interesting would be the Neko VM. It has a Haxe programming language (similar to Actionscript) and has tools for adding more languages to the suite of languages. It is currently used on webservers an is 20x faster than the PHP VM. Haxe when compiled for flash actually is faster than Adobe's solutions.

#53 Actually, I find I can iterate stupid fast in TS. Again, partially because I'm used to it, but that could arguably be said about any language. TS is apart from most regular languages, so there's a higher starting curve, but once you get used to it, you can iterate just as fast as another option.

While my example of the string parsing 'array' was indeed a case of 'we should just have this built in', the fact I had the ability to just build a workaround and move on with what I was attempting to do attests to the flexibility TS brings and the potential productivity it offers. If we could optimize the backend, and buffet the current functions with ones we feel are missing or could be better, then I'd imagine that as a iteration/prototyping language, TS would rank very highly indeed.

That said, if people are bent on pulling a replacement for TS, and optimization and iteration-friendly is key, what does everyone think of rigging Torque to support recompilation-while-running functionality, like what UE4 is doing?

A lot of super-heavy code would end out as C++ anyway, regardless of script language, and C++ is one of the most widely used languages ever, and is fast(you'd be changing at the engine level, so it's as fast as it gets).
The only limitation of doing all C++ is the compile times murdering iteration.
The realtime recompile would alleviate that almost entirely.
If you haven't seen what I mean, go check out the UE4 demonstration, they show it off while pimping their editor.
We could go that route, no "visible" scripting language, you could have a topmost script of sorts, like how UE4 is having their visual scripting kismet for quick idea iteration and artist/designer friendliness, and everything else is handled in the C++, which can be iterated by the programmers while the program runs.
That's probably the most efficient workflow you'll get, overall.

Some people have guessed they used something like this to accomplish it:
http://runtimecompiledcplusplus.blogspot.be/

#54 Strike the NekoVM. It is LGPL and it would be a pain to integrate without ending up infecting the license of the rest of the codebase.

@Jeff,

Now, if people really want to keep TS then lets do this:

1. V8 has a BSD license. So we can take what we want.
2. TS has issues that need fixed.
3. V8 has probably solved most if not all those issues.
4. Lets take what we need out of V8 or even re-target TS to V8 in a way that we can upgrade V8 as the technology gets better.
5. Make sure V8 can still run Javascript as an optional language (no need to throw that away needlessly).

The end result is we leverage the thousands of man hours and experience, we keep TS and make it blazing fast, we tell people we used the V8 tech to improve the Torque Script tech. Give kudos to Google. Perhaps Google might even lend a hand? Summer of code thing?

Next we target browser integration and replace Flash as a tech for hi-speed internet plugins. Since Torque is a MIT license we get net neutrality buy in and support from the OSS crowd. The net folks also like the fact that we are using a Javascript engine. We bring the much needed 3D tech HTML5 is looking for.

If we retarget Actionscript (Haxe?), perhaps even run flash files directly, we would get buy in from all the flash developers.

I dunno, stranger things have happened.

Edit:
Oh holy heck! Look at this!
Read the section that says: "Can compile to JavaScript by generating a single .js file"

If we had the right supporting function calls to match Flash targets we could use V8 with JS and get all the Haxe/Actionscript developers!

#55 Hm, as long as TS acts the same with the V8 backbone, I think that could be a pretty valid way of handling things. Like you said, it'd definitely give us the speed boost, but it'd let people that are able to ninja TS code keep at that, while opening V8 developer interests. Get a proper plugin layer in there for others to shove whatever language they want in at their preference, and we'd probably have something pretty solid.

#56 Haha! Eric Preisz said they had expertise in V8 right from the start.

What are people's thoughts on this direction?

Another question:
Can doxygen do global grouping of code lines in its documentation?
I want to start identifying to hook points for the console: functions, callbacks, variables, etc. If I can use doxygen to identify those into their own global grouping we can work to document the existing interface. This would give us a hit list for the console hook points whatever direction we decide to take.

#57 I like the idea of using the V8 engine and modifying it to both run TS and JS.
When the time comes I would like to know if it mixes with the DNT however, but lets take those worries for a later time.

#58 DNT is a completely external binding if I remember right. Maybe Vince can jump in here and explain the details.

However, I think we can keep everything the same including callbacks. That way all of those interfaces don't get broken. Maybe this is a naive thought so anyone with V8 experience might want to jump in here. I do think we need to move to fixed types on console commands though. That might break some external stuff.

#59 One reason I'm not a fan of V8 is because it's not a cross browser solution. Firefox doesn't use V8, neither does IE.. So why would be assume that we can target one js vm as a valid browser solution and then add fuel to that fire thinking we are going to retarget and invite flash into the party? Seriously, it's not a good path. Those browsers that Don't have v8 would either have to switch (highly unlikely) and if they don't then your game is now open to anyone with a browser that knows anything about JS. Can we say security issue? Not to mention MAJOR slowdown would occur because now v8 isn't running your scripts at least in the browser it wouldn't be, not unless they are running chrome.

EDIT: I'm not going to jump onto the HTML5 bandwagon for any game or anything you want to actually keep secure. Browsers shouldn't be the end all be all, and cook your dinner too application. It's a mistake that people are going to be sorely regretting if they allow the browser to be extended to the point of "it can do anything". They should just do what they are good at and get out of the way for the rest. Which they do, hence browser plugins. HTML wasn't designed for such things, nor should it be.

I guess the point is, if your going down the V8 path, your going to have to make a browser plugin run the entire game anyway (current situation anyway) to get the speed your looking for and you won't be able to rely on HTML5 to do so.

Why not tracemonkey instead of v8? If you want javscript and want it fast, then wouldn't tracemonkey be the more appropriate js vm since it is faster than v8?
brendaneich.com/2008/09/tracemonkey-update

#60 @smally,
We would have our own V8 embedded. It would not even use the JS engine in any of the browsers. It would be standalone. Flash is a plugin with its own VM. We would not be any different.

We could provide a completely open source plugin so that it would be a contender for being a component that is not reliant on a single vendor. The W3C is big on the tech for the web being vendor neutral. It would also give us control of OUR platform.

Quote:I guess the point is, if your going down the V8 path, your going to have to make a browser plugin run the entire game anyway (current situation anyway) to get the speed your looking for and you won't be able to rely on HTML5 to do so.

That was the intent from the beginning. I apologize for not being clear on that.

Why not any old VM? Licensing. SpiderMonkey of which TraceMonkey is a part of is MPL/GPL/LGPL tri-license. V8 is BSD so it is easy to know where you stand. It is the same reason I am not considering Neko, no matter how good it is.