scan.l and gram.y

#1 IIRC, it looked like they used the MKS parsers. The command lines used are in comments at the top of scan.cc and gram.cc.

It also looks like they used a command-line option that incorporated some core files (yylex.c and yyparse.c) whole-hog, so you might be able to use some other parser/lexer combo and figure out a way to incoporate those files, and you'd end up with a working product.

Sounds sticky though. If you don't have access to the MKS software, and you'd like to re-generate the console parser using some other tools, it might be possible. I've added a parser to one of the TGE tools using Cygwin flex and bison, and everything has compiled, linked, and run correctly, so it's possible to do that... I just don't know whether or not there will be any "gotchas" in the console stuff specifically, if you don't use the MKS tools.

Anyway let me collect some notes about using flex/bison, and I'll post those here in a few minutes. They may or may not be helpful to you, since you're using slightly different tools and a "wizard" that presumably helps use them in VC++ builds, but who knows. And maybe someone else would need to know this stuff to.

#2 Since I've turned the previous contents of this post into a HowTo, here's a link to that: Incorporating a parser into your project.

#3 Note for anyone worried about using GNU tools: as of version 1.24, the output of bison can be used in non-free software. The output of flex has always been OK for non-free software.

#4 Joel, you rock dude.

I'm using the BisonFlex AppWizard, which set up VC++ for you, and also use Wilbur Streett's Flex and Bison, I don't exactly know what the compatibility level is with cygwin flex/bison.

But that gives me great confidence :) Thanks!

#5 Thanks, I hope it helps.

I'm thinking of bundling all that stuff into a HowTo resource, since I went to the bother of writing it, so if you come across any specific areas that are different when using the bison and flex that came with that wizard download, let me know and I probably should incorporate that info into the HowTo.

BTW there's one possible difference I know of already: I think that the version of bison you have may generate files with the suffixes _tab.c and _tab.h rather than .tab.c and .tab.h. So some of the compile commands I describe above may need to be altered to take that naming difference into account.

#6 here's what I did:

1) Downloaded and install the BisonFlex Visual C++ AppWizard Installer

2) Add the appropriate .y and .l files to your project.

3) For grammar ( .y ) files:
a) Right click the grammar file, and click settings
b) Click the "Custom Build Tab"
c) In the commands box, type "bison -d -v -o $(InputName).cpp $(InputName).y"
d) In the Outputs box, type "$(InputName).cpp"

3) For scanner ( .l ) files, follow a) and b) and d)above, but for c), do the following:
c) type "flex -o$(InputName).cpp $(InputName).l



as far as the output files go, the -d switch to bison generates $(InputName).cpp.h :\

seems to just tack an ".h" on the end of it, no biggie.
that's it. It seems to work fine for me, but the true test will be rebuilding the gram.cc and scan.cc files using this method, and then re-building the engine to see if the console's broken or not.

#7 Hmm... getting a bunch of build errors. Including the lack of a CMD_reset function.

Also, the version of bison/flex I have doesn't keep track of line #'s

#8 Probably many of the errors you're getting is because you didn't specify a name prefix (which apparently should be "CMD"). With the Cygwin tools, the appropriate args would be "-p CMD" for bison and "-PCMD" for flex.

#9 Yeah, I did that, but the errors were in the fact that MKS lex created a routine called yy_reset(), and my equivalent was yyrestart()

those translate into CMD_reset() and CMDrestart()

CMD_reset gets called from compiler.cc, so I had to do some trick work to get it to all link properly without an "unresolved external symbol" error.

testing it out now...

#10 doh

didn't work

=(

#11 Doh!

OK, I'll take a shot at recompiling the console parser from the .l and .y files, and see what happens.

#12 Yeah, looks like the MKS tools provide a hook to redefine the character-by-character input (which this parser uses), and flex doesn't have an exactly corresponding hook. Probably with a little tweaking you could provide a custom definition for YY_INPUT to get the same effect. Could be interesting, I'll probably come back to it later if you don't beat me to it. :-)

#13 Heh.. I got the "Some object isn't properly out of the bins" error when I tried running the .exe

I'm steppin thru the code right now

#14 Since I've turned the previous contents of this post into a HowTo, here's a link to that: Recompiling changes to the console parser grammar.

#15 Something I should mention though.

If you look at the CVS logs, scan.cc (the file previously generated from scan.l/CMD.l) has been directly modified (from version 1.2 to 1.3), rather than modifying its source file. This is, wuh, very bad. Assuming those changes would be a good thing to have, hopefully it wouldn't be too hard to "back-port" them from scan.cc into the appropriate spots in scan.l. EDIT: Yeah, it's pretty straightforward.

#16 ick.. that's nasty.

/me fires up winmerge :\

PS: Thanks a f**kin bunch for the help :)

#17 You betcha.

I wouldn't be surprised if this is something I would have had to tackle sooner or later anyway.

EDIT: Here's a download link with my new versions of CMD.l and CMD.y: www.neogeographica.com/misc/TGE_console_parser.zip

I was going to post that in the HowTo, but since the HowTo will be viewable to non-licensees (unlike this forum) that's a bad idea. Anyway, the .zip just has the new CMD.l and CMD.y (including the changes assimilated from the new version of scan.cc); it doesn't include the changed compiler.cc, or the necessary changes to the VC++ projects or the makefiles.

#18 OK.. WinMerge'd the 2 (scan.l, scan.cc) files, and it seems to be just a little bit of additional error checking/handling.

UPDATE:

got it to work, F**K YEAH.

once again, thanks for all the help

#19 One more note!

When I was setting up the flex options for CMD.l, I was just cutting-and-pasting from the flex options for my other parser, so I ended up making it case-insensitive (with the -i option). This is bad! :-) The console parser needs to be case-sensitive.

#20 Why's that? Cuz I know Torque scripts are case insensitive in regards to commands.

Is the case-sensitivity resolved through the lexer's rules?