ToqueScript Intellisense or Auto-Complete

#1 Sounds wicked. Would def. speed up dev time.

#2 As far as I know, Eclipse doesn't understand TorqueScript. Your best bet would be using TextPad/UltraEdit/TribalIDE/TorqueIDE. I use TextPad with a TorqueScript syntax highlighter file and am very content.

#3 I like syntax highlighting and use TorqueIDE for torquescript but man intellisense (drop down method/property/class) support would be killer. Not really that hard to implement as well if you have a nice definition of the code (xml etc). Maybe I will look into it.

Auto-complete/Intellisense really jumpstart the new programmers to the engine and really help you work well with modules built from other programmers.

#4 I maintain TIDE (TorqueIDE) and we always wanted to do autocomplete and "goto definition" type stuff for it... but I never really had the time to do it...
I was looking at some autocomplete stuff for Java and PHP for JEdit and Eclipse, but couldnt find anything that would have been easy to "hack" for Torque script support...
I was thinking that we could extend Torque to generate a full list of all the script functions etc. (like e.g. ConsoleDoc does) in XML or whatever format and then use that in TIDE ...
any takers to help me out on this project? ;)

#5 Sweet Stefan that would be killer. I have used it with the SunOne IDEs, VS.NET, VC6, and others such as Macromedia Dreamweaver (PHP, HTML, CSS autocomplete).

In the MS world it is ran by XML files that associate custom assemblies/classes.

If someone doesn't know what autocomplete is its after the developer clicks the "." such as "objData." the parse then follows that object up the tree to find the type then loads the list of avail methods right at the cursor position as a drop down list to select. You can also type a method then hit the "(" key and the same thing happens with the method parameters, etc.

The methods source or XML data of the methods would need to include params values and maybe a description right there as the methods are selected. This lets the developer see all methods avail and descriptions. This speeds your coding and sometimes makes you see things you may not see reading documentation. Its definitely good for the "play" aspect of coding, sometimes this leads to the best innovation if an environment is setup for fun.

I think a hack could be implemented pretty easily with some work. It would be cool to even do it for your own classes and script.

In .NET you use documentation comments above each method (name, desc, params, return type, etc) above each method, prop, class, etc and then compile or run the /doc arg to create the xml file from you comments in code. This could be easily done with an external app to gen this metadata file.

Once the xml file is written its not much to append a list at cursor point then the only real hard part is making sure you are getting thr correct object to look up the method, prop, etc metadata correctly.

I may take a stab at this possibly in a stripped down text editor to get the functionality but it would take a while to get in there. I think it will be beneficial for the entire development community with Tourque. It would be great in TIDE.

#6 *DAMNIT copy the text in this box of your reply before hitting any buttons, because now i have to retype everything from memory when I checked the 'notify' me box it just blanked out this one with my first response... arrgh!*

I use a tool similiar to this called Visual Assist .NET from Whole Tomato, http://www.wholetomato.com. I found that it has increased my efficiency and has helped me out a jam a lot! I am just getting back into playing with Torque after a "long" layoff and this would be a great project to work on.

If you want someone to help out with this metadata generation let me know. I have written a parser, Logzer web log analyzer http://logzer.sourceforge.net, so I am keen on developing an application that will go through the Torque source and extract the information to place into this metadata that you define in what every format. XML would be nice just define the schema for me that you would like to integrate into your application.

I am in between jobs so I can focus on this application, just let me know what form you want the metadata to be in as well as the XML schema and I can get to work on it, that is if you want this to be part of TIDE or any other application. Since I am between jobs at the moment I can focus on this exclusively and maybe get something up in a week or so.

#7 Well, I'd appreciate any help :)
dRawk - Ryan C: could you post an example XML structure that MS uses? Maybe we can get some ideas what we would need there... adapted to Torque script of course...
and maybe Sam could help us with parsing it...
question is: is xml really suitable? if we do a realtime search in a xml text file containing all the script classes, vars, etc. it will get really slow I fear... parsing huge xml files isnt exactly fast usually and nobody wants to wait like 20 secs. for a completion popup ;)
If we go the xml route we could use some standard java xml parser from Apache e.g., I have used those a lot for web stuff...
what do you guys think?

#8 Hm, I had a quick look at ConsoleDoc and this is how it works...
it scans all .cc and .h files for these strings/declarations:

void  Con::addCommand (const char *nsName, const char *name, StringCallback cb, const char *usage, S32 minArgs, S32 maxArgs) 
void  Con::addCommand (const char *nsName, const char *name, VoidCallback cb, const char *usage, S32 minArgs, S32 maxArgs) 
void  Con::addCommand (const char *nsName, const char *name, IntCallback cb, const char *usage, S32 minArgs, S32 maxArgs) 
void  Con::addCommand (const char *nsName, const char *name, FloatCallback cb, const char *usage, S32 minArgs, S32 maxArgs) 
void  Con::addCommand (const char *nsName, const char *name, BoolCallback cb, const char *usage, S32 minArgs, S32 maxArgs) 
void  Con::addCommand (const char *name, StringCallback cb, const char *usage, S32 minArgs, S32 maxArgs) 
void  Con::addCommand (const char *name, VoidCallback cb, const char *usage, S32 minArgs, S32 maxArgs) 
void  Con::addCommand (const char *name, IntCallback cb, const char *usage, S32 minArgs, S32 maxArgs) 
void  Con::addCommand (const char *name, FloatCallback cb, const char *usage, S32 minArgs, S32 maxArgs) 
void  Con::addCommand (const char *name, BoolCallback cb, const char *usage, S32 minArgs, S32 maxArgs) 
bool  Con::addVariable (const char *name, S32 t, void *dp) 
void addField(fieldname, type, Offset(internal_variable_name, class_name))
static void addField(const char*  Fieldname, const U32 fieldType, const U32 fieldOffset, const U32 elementCount = 1, EnumTable *table = NULL);
(uses: const U32 Offset(variablename, classname) to get fieldOffset)
ConsoleMethod(className, name, returnType, minArgs, maxArgs, usage )
ConsoleFunction(name, returnType, minArgs, maxArgs, usage)
IMPLEMENT_CONOBJECT(className)
IMPLEMENT_CO_CLIENTEVENT_V1(className)
IMPLEMENT_CO_DATABLOCK_V1(className)
IMPLEMENT_CO_NETEVENT_V1(className)
IMPLEMENT_CO_NETOBJECT_V1(className)
IMPLEMENT_CO_SERVEREVENT_V1(className)
IMPLEMENT_GETDATATYPE(className)
IMPLEMENT_SETDATATYPE(className)
void Con::registerType(S32  type, S32  size, GetDataFunction  gdf, SetDataFunction  sdf)

then it collects all those infos and writes various HTML files and / or one text file (only 46 kb large for a recent HEAD) ...
the text file looks like this:

ConsoleFunction:
  Name: OpenALShutdownDriver
  Return Type: void
  Usage: "OpenALShutdownDriver()"
  Min Arguments: 1
  Max Arguments: 1
ConsoleFunction:
  Name: OpenALRegisterExtensions
  Return Type: void
  Usage: "OpenALRegisterExtensions()"
  Min Arguments: 1
  Max Arguments: 1
ConsoleFunction:
  Name: alGetString
  Return Type: constchar*
  Usage: "(stringitem)\n\n"
  Min Arguments: 2
  Max Arguments: 2
...

it would be easy as pie to make it generate xml instead and use a Java Parser to use this info in JEdit...
BUT... ;)
I'd like to have every script function, object, variable etc. available, even those defined in the (global and custom) scripts...
so we'd have to scan all the cs and gui files, too... but scan for *what* exactly? "function" keywords? "new", "datablock", "%" and "$", etc. etc.? Anything else?
And this could easily blow up our little xml file I guess ...

Another question: does anybody know a way to get all this info (every script object, function, etc., even those defined in script) directly from the engine without parsing every script file?

Just some thoughts... gotta think about everything some more... ;)

#9 Stefan I'll take a look at some of the code as well tonight. You are right there maybe some overhead scanning a file and putting the information into an XML file. There is a huge performance hit with Logzer, as I scanned the log file and placed the information into a SQLite database. I don't think blowing up the XML file would be that much of a problem. Maybe before we start the IDE, whichever one we decide, we load the entire file into a Memory File that way access, search, and seek times would be an order of magnitude larger. The only delay would be in the beginning loading the file into memory. Yes this may take a lot of memory but chances are people playing games based on the Torque engine or developing has enough memory to handle this.

I would be happy to help with the parsing portion. Just let me know via email or post on this thread as I am now monitoring it.

#10 Sam: your help is greatly appreciated :)
Do you guys know/use Java btw.?
I think we could use DotComplete as a base and just replace the Java centric stuff (Parser etc.) with Torque Script stuff (I've done the same with TIDEBrowse, which is a stripped down version of JBrowser) - this way we get all the GUI setup etc.
Yeah, loading the file into memory on startup would be an option I guess...
Okay, so lets start to collect what info we'd need for intellisense...

Functions

"Complete Member" / "Signature Help"

- Function name
- Parameter count
- usage info
- return type
- path(s) to reference(s) -> "go to declaration" etc.
- ...

Objects / Classes

"Complete Class" / "Signature Help"

stuff like "new XYZ(){"


and what about
global / local variables
$myVar, %foobar, ...

Torque packages

etc.

Any additions, thoughts anyone?

#11 I'm not sure I'd worry too much about locar vars, but definitely $gobals. Locals would cause ya' some extra work - having to sense what local vars you have in the function you are currently editing, etc. I'm not sure it's worth it.

One o' these days I'll polish off the next version of TorqueIDE (er - we've got a definite 'name space' issue here - Two TorqueIDE's, one abreviated, one not! :-) and release it - I had to face similar issues as you Stefan. I've got mostly working intelletype stuff - but it's far from as complete as it should be. My solution to the problem was twofold:

1) I made use of the existing JavaDoc / dOxygen setup used on the C++ side for the scripting stuff. That helped pick up all the documenation for a function / datablock / file / element. Threw it in a big ol' file, along with all existing console functions (though I'm not making use of consoleDoc at the moment, but I need to.) This also meant that stuff you weren't interested in documenting when it came to files / elements didn't get picked up by the 'generate function list' command

2) I also parsed through and grabbed all datablocks and functions based on the datablock and function keywords. While you might not document one, I figured unlike $globals you are gonna want to know it at least exists.

One problem with $globals - there's no declaration for 'em. I decided (personal decision here, not something particularly logical) to ignore any /// undocumented $global, since I didn't have a real tense to work from (IE - the first occurance of a $global didn't really tell me jack about it, and since I could watch for a declaration or anything, putting stuff about undeclared globals just didn't make sense to me.) However, if you just compiled a list of $globals as you encountered 'em, it would be pretty easy for intellesense type stuff.

Between functions, datablocks, and $globals, that covers most of the big stuff.

BTW: If we both worked from the same concept (/// JavaDoc + dOxygen) it might be a cool thing, that way documentation works the same for both of the IDE's. I can send you an updated version of TorqueIDE plus some of the documenation I've done on it if you want to look at my implementation - I haven't implemented the full suite of JavaDoc stuff, but, it supports documenting funcitons, datablocks, files, and elements, along with the @param, @see, @note, @todo, etc. atributes.

#12 Hey Davis, thanks a lot for those tips!
Yeah, we should really keep script comments used for documentation conform to some standard, so I'm very open to use the Javadoc style, too - would be great if you could send me the latest TorqueIDE (hehe, yeah, we really need to do something with the names... hmmmm)
so that I can check it out and see what I can incorporate into TIDE :)
Also very true what you said about globals, guess we should stick to that, too...

#13 I know *some* JAVA as I am mostly a C++, and don't hate me for it, but it pays the bills, Visual Basic programmer. Knowing that a variable is a global may help, I don't know how TORQUE scripts handle collision of a local variable defined with the same name as a global.

The one thing others may want to know about globals is what functions call and use them, and where they are defined. This would help tracking down any problems with unknown changes to the global variable.

@Davis: that sounds like a good idea with having a common intellisense type approach, this is a good thing to know right off the bat. When I started creating Logzer I used mostly all the MFC Collection classes, someone was interested in porting it to *nix but then said it would be too long for something that already existed on *nix, Analog. So before development starts knowing if it will be cross platform is a good thing, I would move collections and everything off to STL or LOKI library if we are basing it on C++.

@Stefan: Like I said I know some "java" the problem for me is remembering all those class names and methods because the last IDE I used didn't have intellisense like capabilities for JAVA so I was always referencing a class diagram or book. If there is an IDE that has this function please let me know so I can download and re-evaluate my JAVA capabilities.

Sorry I can't look through any docs now on the source or figure out a way to make it work at the moment. I am at a friend's house on his computer, babysitting his child. I should be home around 3-4 PM PST to discuss this further. And, I will check in from time to time depending on how well behaved his child is :)

#14 @Sam: I got no beef with VB - I used to do it too ;-)

At the moment, most of the work for pulling data outta the scripts is already done in umm... MyTorqueIDE (hehehe - to identify it somehow, since our namespace collision is occuring in the same thread for once ;-). Nothing fancy - I need to dump it to something prettier like XML at some point, 'cause right now it's just a flat text file with very little formatting. It's all done in TorqueScript, so, it's completely cross platform. Give than StefanTorqueIDE is done in Java, it's gonna be the same case there too I'm sure.

Keeping track of where a function call is definined is a very interesting issue. What I did was in my ugly little function file, I kept track of what filename and linenumber the function was found. Why? Nothing prevents you from creating the same function twice in TorqueScript. It's actually very convenient for some interesting hacks too - just load up, say function MyCoolFunction() in main.cs, then later when you exec() game.cs, it's version of MyCoolFunction() overwrites it in memory (same goes for datablocks, etc.). However, for tracking of functions, it does make life a little more... interesting :-)

For the console functions, it's a completely different story :-)

As for cross-reference stuff - well, that's what searches are for most of the time. Rarely do you need that on paper as much as you need it while you are workin' on a problem - in which case, you just pull it all out dynamically (even though it's written in TorqueScript, it doesn't take long for MyTorqueIDE to parse through all .cs files to find occurances of a $global.) But, as I mentioned before, $globals don't have a declaration (nor do %local vars) in TorqueScript. Sometimes I wish they did ;-) I've thought about adding a quick little bit to MyTorqueIDE's 'standard' for documenation for %local variable elements, but I haven't gotten around to supporting a couple of pieces of the JavaDoc + dOxygen standard needed to do the job properly (I need to have sections available for that.) Supporting the whole standard is a big job :-)

Stefan: Ok. The version I'll send you is one rev outta date (I broke the IDE again while putting in some more of my "must have" functionality, and then this week it's been attack of the Business Meetings and Clients From Hell, so I'm in development limbo for a couple more days.) I'll have to check, but, I think that rev has the first concept version of intellisense type stuff in it. I'll have to check to make sure. The current version in the resource does have all the JavaDoc stuff in it. If the previous unreleased rev doesn't have the inital version of intellisense stuff in it... well, it's gonna have to wait until I get the chance to finish it up, which will be next week :-/

Cool - now if all the IDE's (that handle TorqueScript) stick with the same documenation standards / format, maybe GG will update the starter packs with the doc style. That would make life REALLY easy for someone who is just interested in script hacking :-)

Hmmm... naming. This is an interesting 'minor' issue. I could always be egotistical about it and call mine MidnightRyderIDE ;-) But I think I'll pass :-) ('Specially since I didn't write portions of it - you and Harold did :-)

#15 @Davis: Sounds good, and thanks for now looking down on me for using VB. Most C++ coders, even JAVA coders all tell VB programmers to use a "real" language. I never got into the language wars, at this moment, I like the power and elegance of C++ a lot. I am also going to look up MyTorqueIDE and download whatever version is available at the moment.

When you and Stefan work out the XML schema you want to use please let me know, and if you need my help to parse your current file into an XML file. It doesn't sound that difficult and would actually give me something to contribute back to the community. Also what language we are going to use and if we are going for cross platform compatibility. MFC for Windows helps you a lot build a solution faster, but it also dumbs you down a great deal. For example, the other day I had to look up the parameter call for the standard C++ "main" function because I am so use to working with Win32 WinMain, and with MFC you don't even see WinMain.

Looking forward to starting work on this project.

*UPDATE* Do you have a download linke to MyTorqueIDE I just did a search for it here on garage games under forums, and resources, and Google and found no website. Is it released yet for the general public?

#16 I've done 6 games that utilized VB + DirectX, and 3 of them are about to start showing up in retail space. I'm not gonna look down on anyone for using the same tools. It's all a matter of preferences really - most of 'em you mentioned (VB, Java, C++) can do the same job to varying degrees (except that whole crossplatform issue).

As for a Schema - you might want to take a look at [url=http://www.stack.nl/~dimitri/doxygen/manual.html"]the dOxygen website[/url] and get an idea what's goin' on and what the needs of any XML schema might be - it's not a real intense set of problems to address, but, it would take a little work.

The MyTorqueIDE and StefanTorqueIDE is a bit of a joke - both projects are called TorqueIDE (but Stefan abreviates to his to TIDE most of the time), so for the sake of this discussion that involves both projects, I jokingly refered to one with My and the other with Stefan. The MyTorqueIDE can be found here as a resource. BUT - there's a minor problem. 98% of the work is done in scripting - no problems there. But about 2% (and it's a rather impotant portion for things like intellisense, syntax highlighting, etc.) is done in C++ - you'd have to be a Torque licensee to make use of it, since ya' can't do the changes without it. And, just in case you are wondering - StefanTorqueIDE is much more mature at the moment. MyTorqueIDE has a ways to go (but has been getting that at a nice clip, as long as I don't get held up by customers and holidays ;-)

I'll assume like, oh, 0% of either project will have it's schema work done in C++. StefanTorqueIDE(TIDE) uses Java, and MyTorqueIDE uses 98% TorqueScript to do the job (the do similar jobs, but have very different approaches to portions of it.) So MFC, etc. are pretty much non-issues here :-)

#17 Ah no problem thanks for the break down. I am a licensed user of the Torque source base. Ah so 2% of the code is in C++ I can help with that if you want me to. I was thinking we were going to parse the information and create an XML file that we, meaning you guys :), have defined that both programs can read. If it was just reading in the information and creating that XML file then I would, and could help in that area. If this was a second application, the one I was volunteering to create, would take the existing source parse and then generate an XML file, and you need my help I am here. But if you are planning to do it within the respective IDE engines then I would not have the source code available for the IDEs.

XML schema was just wondering if we are going to settle on some tag system that would be uniform and describe the functions and anything else needed. If you guys need my help let me know, I am ready willing and able to assist.

Better way to some everything up, on my end, is what are our goals and plans, and do you need my help? :) lol

#18 Sam: Whoops, yep you are indeed a licensee :-) I glanced at your profile (habit) and missed that line somehow! ;-)

Well, honestly, I don't know what the 'plan' is for any XML files involved. The discussion you see here is all that Stefan and I have chatted about so far when it comes to doin' the same thing on it. For MyTorqueIDE, XML seems like a good idea, but, wasn't something I planned to address immediately. I've got something that's workable for the moment, but, isn't nearly as pretty as it should be :-) I'm sure IF both IDE's end up spewing out XML, it's going to be directly based on dOxygen's format - it already exists and is documented, and spewing out XML for your entire project is then as simple as spewing out the engine's docs and the script's docs using tool of choice. Also makes the various toolsets play nicely together, or separately that way.

In code, functions using JavaDoc & dOxygen standards look like this:

///  Short Description of function, one line.
///
///  Longer description, possibly multiple paragraphs.
///
///  @param  %parameter1   parameter 1 description.  Can be mutiline.
///  @param  %parameter2   parameter 2 description.  Or even mutliple
///                        paragraphs.
///  @param  %parameter3   parameter 3 description
///
///  @todo:  Stuff yet to be done
///
///  @see:   Other documentation

You can look in some of the Torque Engine stuff to see it in action, including the stuff I don't have support for in my version of the IDE. You can look in the docs for my version of the IDE for information about what I've got support for and what I don't (in particular, pop open the function browser, and look in there for TorqueIDE - I'm slowly trying to migrate that to be built automagically from in script documenation too. That way I don't have to deal with it all twice ;-)

#19 Wow I haven't checked in for a day and lots has been said.

Here's what the comments look like in MS world in addition to the Java samples above.

..btw what the code markup syntax?


///

///
///

///
///
///
public int ExecuteNonQuery(string spName, params SqlParameter[] commandParameters)
{
//create a command and prepare it for execution
SqlCommand cmd = new SqlCommand();


The compiler then creates this XML format from these comments and using Reflection (for torque i imagine you may need to put the method name and params all in the comment since no reflection to grab that from -- or could change the parser to tear out themethods and properties from the code itself).



B.Component.Web

Which intellisense uses to read the object (it also uses reflection here but if there is an XML file it grabs from there).

I think that it would be cool to parse out the code no matter if its commented or not. This way all code is instantly autocomplete enabled as long as the parser can handle this.

If there are two methods of the same name then they are both in there but maybe the filename or class is grabbed from file as well to differentiate.

>>>

It sounde like there may be some stuff out there already that we can use to do this.

So is anyone putting it into something that will be posted back to cmmunity? I will get to work on this if not, mine may only be windows based though just for speed of development.

#20 DRawk: Um, that's how the comments are handled for the Torque Engine, thus it's probably best that they follow the exact same format, rather than have a second format. JavaDoc + dOxygen is what's used to create the HTML (and other - spew out whatever you like with dOxygen :-) version of the Torque SDK docs online. Plus, making it tied to a single compiler / IDE implementation that isn't a cross-platform standard isn't a hot idea IMHO either. (And I'm not pickin' on ya - Visual Studio 6.0 is my standard compiler environment until I move to Mac :-)

And of course, we're only talking about in the Torque Script stuff, unless Stefan is talking about this for both TS and C++ (and if so, well, I'm only interested in the TS stuff, since my version of TorqueIDE only touches scripts :-)

I'm not quite ambitious enough to create something that pulls out much more than function blocks and datablocks if they aren't commented. Sure, you could make a list of local variables that are used - but since TorqueScript doesn't use declarations, that doesn't help ya a great deal. You've got no context to work with. So, I stuck with only pulling elements, files, etc. when they are documented. Makes life a little easier to deal with. Remember - this is TorqueScript. No declarations for variables, or parameters. Anything can be passed as anything you like, so, it's gonna be up to you to put the right information in @param's :-)

Quote:It sounde like there may be some stuff out there already that we can use to do this.

Well, yeah - there's a spec for proper documentation. There's a spec for what the output should probably look like if using XML (dOxygen's output when you use the XML flag) - though I'm waiting to see if Stefan agrees with me on that idea before I implement anything in my IDE (hey, I've already written this part twice now. I'll wait until things look ironed out before I do it a third time ;-). There's dOxygen it's self that will rip the documentation out, and serve it up in any tasty format you like :-) (And it's cross-platform ta' boot.) So, if you document your code using the stuff I provided above, you're already golden when it comes to spewing it out, and viewing it inside anything besides your IDE (well... actually, you can use the function browser in my version of TorqueIDE. But it's UGLY, and only reads a format the programmer dreamed up, not nice XML ;-)

Both of these are community projects, so, jump right in - anything Stefan or I do on either project is gonna be released back to the community I'm sure. (I can't think of anything that I would add that I wouldn't give back, and Stefan is usin' SourceForge to handle his stuff, so it seems pretty much the same situation there too!)

Btw: To find out the syntax for code, quote, etc, click on the little "Click Here" button below the post a reply area - it's got all the markup stuff in it :-)