Httpobject And Php
by Benjamin L. Grauer · in Torque Game Builder · 02/05/2007 (1:27 pm) · 17 replies
edit : (for readers, this topic is somehow the continuation of this one : http://www.garagegames.com/mg/forums/result.thread.php?qt=57158 )
Now that I got into HTTP online things, I kind of like the idea of using an online db through php scripts (for things like global score ladder, etc.).
I've got some help from a PHP guy who set up the script and the bdd. And, through HTTPobject.post, I was finally able to add entries do a MySQL DB. But... not quite right.
I can just type an url like this one
Now, when I'm trying to do the same with script like this :
I don't quite understand what's going on, but it seems that the "&" doesn't get right. Why is this happening with Torquescript ?
Now that I got into HTTP online things, I kind of like the idea of using an online db through php scripts (for things like global score ladder, etc.).
I've got some help from a PHP guy who set up the script and the bdd. And, through HTTPobject.post, I was finally able to add entries do a MySQL DB. But... not quite right.
I can just type an url like this one
http://mygamesite.net/netstuff/profile_create.php?name=charles&date=2006-10-12and it create successfully the entry with "charles" as name and "2006-10-12" as date.
Now, when I'm trying to do the same with script like this :
$myHTTPObject.post("mygamesite.net:80", "/netstuff/profile_create.php", "name=charles&date=2006-10-12", "");
// ps : I dont know what mean the last field for post, since there is no documentation for it.it create the entry with "charles&date=2006-10-12" as name and "" nothing as date.I don't quite understand what's going on, but it seems that the "&" doesn't get right. Why is this happening with Torquescript ?
#2
02/05/2007 (1:36 pm)
When I do so, it creates an entry with nothing in it.
#3
Out of curiosity, why use the 'post' function instead of 'get'? When you type the URL into a browser (the case that works in your example) that's a 'get' call. Maybe the PHP code isn't pulling variables properly for 'post' type submissions. Try changing to $myHTTPObject.get and delete the fourth parameter.
Speaking of which, I briefly scanned the engine source and it looks like that fourth parameter in the .post() function isn't used at all, it sets a global variable but it never gets used.
02/05/2007 (3:37 pm)
From you description it sounds like the query string is getting encoded at some point along the line. This should be fairly easy to debug in PHP by dumping the variables to the log.Out of curiosity, why use the 'post' function instead of 'get'? When you type the URL into a browser (the case that works in your example) that's a 'get' call. Maybe the PHP code isn't pulling variables properly for 'post' type submissions. Try changing to $myHTTPObject.get and delete the fourth parameter.
Speaking of which, I briefly scanned the engine source and it looks like that fourth parameter in the .post() function isn't used at all, it sets a global variable but it never gets used.
#4
@Ben, i've not run into this issue before, as I've said, since I use the TCPObject and build the 'GET /file/path/script.php?query=string&etc=etc' header data manually --
It would appear, from a brief glance and no debugging performed, that the HTTPObject.get() only recognizes the existance of a 'single' query parameter, and converts all everything into html encoded hex equivalents -- here's a quick review:
This method here, is the cause of your issues --
I would have thought that perhaps maybe it expected params as "param1=value param2=value" but there is no code to convert spaces to "&", as all characters in the asciiEscapeTable are converted into there hex equivalents.
@Ben, looks like you either a) have to fix this method for your code to work, or b) use the TCPObject and do it the way I've been doing it, "manually" (ie; the hard-way)
The only real noticeable difference between HTTPObject and TCPObject is that HTTPObject knows how to build your HTTP data-streams -- basically, what it does internally is the following:
Aside from this, the HTTPObject -IS- a TCPObject -- and doing the bits that HTTPObject does for you, with a TCPObject just takes a few minutes of reading to learn the HTTP Protocol.
02/05/2007 (5:52 pm)
A quick review of the httpobjecs source show that in the onConnected, it calls "expandPath", which actually converts the & into the HTML encoded version, this seems to be a fairly 'design' of the httpobjects consideration of querystring parameters -- @Ben, i've not run into this issue before, as I've said, since I use the TCPObject and build the 'GET /file/path/script.php?query=string&etc=etc' header data manually --
It would appear, from a brief glance and no debugging performed, that the HTTPObject.get() only recognizes the existance of a 'single' query parameter, and converts all everything into html encoded hex equivalents -- here's a quick review:
void HTTPObject::expandPath(char *dest, const char *path, U32 destSize)
{
static bool asciiEscapeTableBuilt = false;
static bool asciiEscapeTable[256];
if(!asciiEscapeTableBuilt)
{
asciiEscapeTableBuilt = true;
U32 i;
for(i = 0; i <= ' '; i++)
asciiEscapeTable[i] = true;
for(;i <= 0x7F; i++)
asciiEscapeTable[i] = false;
for(;i <= 0xFF; i++)
asciiEscapeTable[i] = true;
asciiEscapeTable['\"'] = true;
asciiEscapeTable['_'] = true;
asciiEscapeTable['\''] = true;
asciiEscapeTable['#'] = true;
asciiEscapeTable['$'] = true;
asciiEscapeTable['%'] = true;
asciiEscapeTable['&'] = true;
asciiEscapeTable['+'] = true;
asciiEscapeTable['-'] = true;
asciiEscapeTable['~'] = true;
}
U32 destIndex = 0;
U32 srcIndex = 0;
while(path[srcIndex] && destIndex < destSize - 3)
{
char c = path[srcIndex++];
if(asciiEscapeTable[c])
{
dest[destIndex++] = '%';
dest[destIndex++] = getHex((c >> 4) & 0xF);
dest[destIndex++] = getHex(c & 0xF);
}
else
dest[destIndex++] = c;
}
dest[destIndex] = 0;
}This method here, is the cause of your issues --
I would have thought that perhaps maybe it expected params as "param1=value param2=value" but there is no code to convert spaces to "&", as all characters in the asciiEscapeTable are converted into there hex equivalents.
@Ben, looks like you either a) have to fix this method for your code to work, or b) use the TCPObject and do it the way I've been doing it, "manually" (ie; the hard-way)
The only real noticeable difference between HTTPObject and TCPObject is that HTTPObject knows how to build your HTTP data-streams -- basically, what it does internally is the following:
%tcp = new TCPObject() { };
%tcp.connect("www.hostname.com", 80);
function TCPObject::onConnected(%this)
{
%obj.send("GET .... \r\n\r\n");
}
function TCPObject::onLine(%this, %line)
{
// check if line is header, if so, discard
// else, call TorqueScript "onLine" and return data
}Aside from this, the HTTPObject -IS- a TCPObject -- and doing the bits that HTTPObject does for you, with a TCPObject just takes a few minutes of reading to learn the HTTP Protocol.
#5
02/05/2007 (5:53 pm)
Also, I might note, I would not allow the game to send -any- dates to the server, as a client-machine may have an extremely skewed clock and the date could be anything -- I would set the date according to the server, and perhaps allow the client to set the GMT Offset (Time Zone) for display in-game and on the website.
#6
Simply commenting out the line
02/05/2007 (6:55 pm)
Another simple workaround would be to use that one query parameter to pass a custom name/value string that gets manually parsed in php, but this is a bona-fide bug in the engine and needs to be submitted and hopefully fixed. Simply commenting out the line
asciiEscapeTable['&'] = true;in httpObject.cc effectively fixes it, though it does put the burden on the client script to escape any ampersands in your query parameters before sending. The bug has been around awhile too. I've got it manually fixed with a comment in both a 2004 TGE source tree and the T2D 1.0.2 beta source.
#7
I believe a correct fix for this would be to include some additional consolemethod's exposed from HTTPObject, one that 'escapes' query string values, where &'s in the string being passed to the escape are converted for you -- or, possibly passing in a 'list' of parameters, such as perhaps a Name/Pair array (a SimSet could be utilized for this, then internally converted into a query-string)
The problem here, is that if your query-string parameter is a string, and it contains an "&", that & must be escaped, but the "name=value&name2=value2" must not -- the only "good" way to do this is to take lessons from existing technologies that do this, such as PHP and ASP -- ASP.NET for example, you "add" parameters to a Collection, with PHP, there's the "htmlencode" and "htmlentities" functions, for escaping string data for the purposes of displaying <> in pages, or passing & in a query-string string parameter.
Unfortunately, the "HTTPObject" is undocumented, and, from previous discussions, it's undocumented for a reason -- it's considered "internal" from my understanding, and is not really even used.
Which is why I opted to use the TCPObject for my code ... allowing for greater functionality, including the ability to actually make "post" requests to a server, all through torquescript, with no C++ modifications.
02/05/2007 (8:34 pm)
Luke, unfortunately, amperstands that need to be escaped, must be escaped by the client -- The HTTP RFC requires this, as the "&" is used to denote query-string parameters -- PHP and ASP both allow you access to the raw query-string data, but this is an absolute 'no no' -- I believe a correct fix for this would be to include some additional consolemethod's exposed from HTTPObject, one that 'escapes' query string values, where &'s in the string being passed to the escape are converted for you -- or, possibly passing in a 'list' of parameters, such as perhaps a Name/Pair array (a SimSet could be utilized for this, then internally converted into a query-string)
The problem here, is that if your query-string parameter is a string, and it contains an "&", that & must be escaped, but the "name=value&name2=value2" must not -- the only "good" way to do this is to take lessons from existing technologies that do this, such as PHP and ASP -- ASP.NET for example, you "add" parameters to a Collection, with PHP, there's the "htmlencode" and "htmlentities" functions, for escaping string data for the purposes of displaying <> in pages, or passing & in a query-string string parameter.
Unfortunately, the "HTTPObject" is undocumented, and, from previous discussions, it's undocumented for a reason -- it's considered "internal" from my understanding, and is not really even used.
Which is why I opted to use the TCPObject for my code ... allowing for greater functionality, including the ability to actually make "post" requests to a server, all through torquescript, with no C++ modifications.
#8
Regardless you wouldn't be handling the raw query string, you'd be treating the singular param on the query string. It just so happens that you chose to pack a name/value array into that variable, and there's absolutely nothing wrong with that.
Still the HTTPObject is nothing special and without the engine source, a bit cumbersome to work with. But what do you gain from using HTTP at all? If you're going to write a browser engine in script on top of TCPObject, why not just write a socket app on the server and do as you please with the connection. You can do away with nonsense like obfuscated document paths and pseudo-secret handshake routines, implement nice sturdy encryption, and tailor it to the needs of the app being developed.
02/05/2007 (8:59 pm)
We're not exactly doing anything so profound that requires adherance to industry standards, lets not get carried away. :) These are hack-arounds for a bug in the engine, and they'd work fine. Regardless you wouldn't be handling the raw query string, you'd be treating the singular param on the query string. It just so happens that you chose to pack a name/value array into that variable, and there's absolutely nothing wrong with that.
Still the HTTPObject is nothing special and without the engine source, a bit cumbersome to work with. But what do you gain from using HTTP at all? If you're going to write a browser engine in script on top of TCPObject, why not just write a socket app on the server and do as you please with the connection. You can do away with nonsense like obfuscated document paths and pseudo-secret handshake routines, implement nice sturdy encryption, and tailor it to the needs of the app being developed.
#9
Using HTTPObject was simple to do (simply sending variable, and reading the php's echos with onLine callback), but it has this annoying bug, I don't have the source to correct it, and anyway it seems less secure than TCPObject (using 'header' data with a custom client-type would be enough 'security' for me).
Alright... so, it seems that I'll have to use TCPObject then.
02/05/2007 (9:05 pm)
@DavidUsing HTTPObject was simple to do (simply sending variable, and reading the php's echos with onLine callback), but it has this annoying bug, I don't have the source to correct it, and anyway it seems less secure than TCPObject (using 'header' data with a custom client-type would be enough 'security' for me).
Alright... so, it seems that I'll have to use TCPObject then.
%tcp = new TCPObject() { };
%tcp.connect("www.hostname.com", 80);
function TCPObject::onConnected(%this)
{
%obj.send("GET .... \r\n\r\n");
}
function TCPObject::onLine(%this, %line)
{
// check if line is header, if so, discard
// else, call TorqueScript "onLine" and return data
}This quote seems to be on the right track for how to use it, but I think I may need some more precisions about %obj.send(...) and onLine callback with header and such ^^'
#10
I, for example, am utilizing TCPObject to send/retrieve data to a 'master server', that just happens to be a web-script -- which allows me to cheaply host a 'master server', that can be queried easily -- basically, the code I'm working on, replaces the internaly 'queryMasterServer' ConsoleMethods, with custom code that communicates with a web-site -- yeah, theres more overhead in the data transfer, but it's just a master server, it's not the actual 'game' server (which is hosted by the 'hosting client')
As far as adhering to standards, it's best to do things the 'right' way, whether then the 'wrong' way -- it prevents issues from occurring later in the future and makes the code reusable -- why write code that can be used over and over, in a fashion in which it's useless or more complicated to use over and over?
I've completedly removed the HTTPObject from my C++ code, and replaced it with a TorqueScript class, so I simply do the following: "%http = new TCPObject() { superClass = "HTTPObject"; class = "myCustomClass"; };"
This works quite well for me, and I intend to eventually port it over to C++, but for proto-typing needs, TorqueScript is working quite well.
As far as doing things as "hacks" to by-pass an engine "bug" or "limitation", I also find that quirky -- the engine code should be fixed up so that the developer doesn't have to come up with "tricky" hacks, regardless of how simple they may be to come up with, or even implement --
I personally find the existance of the HTTPObject within the engine source to be quite wrong, it should either be there and be useful, or not be there at all -- as it's merely a wrapper around TCPObject that automates the HTTP Protocol for you (to some degree), I find it rather useless --
02/05/2007 (9:07 pm)
@Luke, there are a number of very good reasons to use HTTP -- one, hosting for PHP is much cheaper then a shell account that allows unlimited connections to connect to a background process -- I, for example, am utilizing TCPObject to send/retrieve data to a 'master server', that just happens to be a web-script -- which allows me to cheaply host a 'master server', that can be queried easily -- basically, the code I'm working on, replaces the internaly 'queryMasterServer' ConsoleMethods, with custom code that communicates with a web-site -- yeah, theres more overhead in the data transfer, but it's just a master server, it's not the actual 'game' server (which is hosted by the 'hosting client')
As far as adhering to standards, it's best to do things the 'right' way, whether then the 'wrong' way -- it prevents issues from occurring later in the future and makes the code reusable -- why write code that can be used over and over, in a fashion in which it's useless or more complicated to use over and over?
I've completedly removed the HTTPObject from my C++ code, and replaced it with a TorqueScript class, so I simply do the following: "%http = new TCPObject() { superClass = "HTTPObject"; class = "myCustomClass"; };"
This works quite well for me, and I intend to eventually port it over to C++, but for proto-typing needs, TorqueScript is working quite well.
As far as doing things as "hacks" to by-pass an engine "bug" or "limitation", I also find that quirky -- the engine code should be fixed up so that the developer doesn't have to come up with "tricky" hacks, regardless of how simple they may be to come up with, or even implement --
I personally find the existance of the HTTPObject within the engine source to be quite wrong, it should either be there and be useful, or not be there at all -- as it's merely a wrapper around TCPObject that automates the HTTP Protocol for you (to some degree), I find it rather useless --
#11
The example code I'm posting is actually 'pre-alpha' code for my 'Torque Forum/Blog GUI' kit that I started working on months ago, and have since put on the shelf ...
02/05/2007 (9:09 pm)
@Ben, I can provide you some example code for working with HTTP streams in TorqueScript if you'd like -- just email me at higginsd at zoulcreations dot com and I'll post the code back to this thread (I only request the email because I can't do it tonight, and need a reminder -- I'm forgetful)The example code I'm posting is actually 'pre-alpha' code for my 'Torque Forum/Blog GUI' kit that I started working on months ago, and have since put on the shelf ...
#12
The day this (or virtually any other) game engine works to the degree that the occasional clever hack to workaround an oddity or bug isn't necessary will be pure bliss, but I'm not holding my breath. :)
Sounds like your HTTPObject replacement would make a great resource, hope to see it put up someday. Maybe even eventually replace the existing C version, that'd be fantastic. I'm sure a lot of people would find that incredibly useful!
02/05/2007 (9:58 pm)
@David, I was being facetious about the HTTP comment, though I do personally enjoy the freedom afforded by a leaner server implementation, opens a lot of possibilities not afforded by simple the web-service approach.The day this (or virtually any other) game engine works to the degree that the occasional clever hack to workaround an oddity or bug isn't necessary will be pure bliss, but I'm not holding my breath. :)
Sounds like your HTTPObject replacement would make a great resource, hope to see it put up someday. Maybe even eventually replace the existing C version, that'd be fantastic. I'm sure a lot of people would find that incredibly useful!
#13
I've got the ground work laid out, just have to do some of the engine gutting and remove the references to the master server ConsoleMethod's so that my torquescript implementations take over from there -- currently, the entire thing is wrapped into a nice little TCPObject sub-class, just have to create a global instance of it, and expose some generic global functions --
02/05/2007 (10:31 pm)
@Luke, just about all of my code gets posted at some point or another -- this code though is being used in one of my game projects, and so won't be available until much later -- it's also currently in a fairly 'untested' state -- I've got the ground work laid out, just have to do some of the engine gutting and remove the references to the master server ConsoleMethod's so that my torquescript implementations take over from there -- currently, the entire thing is wrapped into a nice little TCPObject sub-class, just have to create a global instance of it, and expose some generic global functions --
#14
I call the script "httpPage.cs" and instantiate a copy of it like so:
This is a fairly "simple" torquescript version of "httpObject", without the quirks of "httpObject" -- basically, you tell it what to connect to, by simply giving it the entire URL -- you can then perform what ever logic you want to on the results --
I personally have my PHP scripts return back "text/plain" pages, and spit out INI style results, sometimes I spit out TorqueScript and "eval" it, and other times, I return XML and parse it with my XML Resource (which was posted as a .plan about a month ago)
Hope this helps ...
02/06/2007 (6:10 pm)
Ok, here's the code, this is a quick snippet that shows how to connect to a web-page, then retrieve the results, in the same fashion as HTTPObject, but using TCPObject instead:$MAX_HTTP_QUERY_STRING = 255;
function httpPage::get(%this, %url)
{
%this.Buffer = "";
%this.doBuffer = false;
%host = "";
%port = 80;
%page = "";
if(strpos(%url, "http://") == 0)
{
echo("HOST1");
%host = getSubStr(%url, 7, strpos(%url, "/", 8) - 7);
%page = getSubStr(%url, strpos(%url, "/", 8), $MAX_HTTP_QUERY_STRING);
}
else
{
echo("HOST2");
%host = getSubStr(%url, 0, strpos(%url, "/", 8));
%page = getSubStr(%url, strpos(%url, "/"));
}
if(strpos(%host, ":") < 0)
{
%host = %host @ ":" @ "80";
}
echo("HOST: " @ %host);
echo("PAGE: " @ %page);
%this.Address = %host;
%this.Page = %page;
%this.connect(%this.Address);
}
function httpPage::onConnected(%this)
{
echo("CONNECTED: " @ %this.Address);
%query = "GET " @ %this.page @ " HTTP/1.0\nHost: " @ %this.Address @ "\n\n";
%this.send(%query);
}
function httpPage::onLine(%this, %line)
{
echo("LINE: " @ %line);
if(!%this.doBuffer && %line $= "")
{
%this.doBuffer = true;
echo("DO BUFFER: TRUE");
return;
}
if(%this.doBuffer)
{
echo("ADDING BUFFER ...");
if(%this.Buffer !$= "") %this.Buffer = %this.Buffer @ "\n";
%this.Buffer = %this.Buffer @ %line;
}
}
function httpPage::getResult(%this)
{
return %this.Buffer;
}
function httpPage::onDisconnect(%this)
{
}
function httpPage::onConnectFailed(%this)
{
error("Connection Failed: " @ %this.Host @ ":" @ %this.Port);
}I call the script "httpPage.cs" and instantiate a copy of it like so:
new TCPObject(httpPage) {};
httpPage.get("zoulcreations.com/index.php?var=val&var2=val2");
echo(httpPage.getResult());This is a fairly "simple" torquescript version of "httpObject", without the quirks of "httpObject" -- basically, you tell it what to connect to, by simply giving it the entire URL -- you can then perform what ever logic you want to on the results --
I personally have my PHP scripts return back "text/plain" pages, and spit out INI style results, sometimes I spit out TorqueScript and "eval" it, and other times, I return XML and parse it with my XML Resource (which was posted as a .plan about a month ago)
Hope this helps ...
#15
02/06/2007 (6:11 pm)
Btw -- the $MAX_HTTP_QUERY_STRING is used due to an issue I was having with getSubStr not allowing me to pass 2 parameters and just default the last param to "the remainder of the string", if your queries are much smaller, or much larger, this variable can be modified -- I believe 255 is a fairly decent size for general use, but if your trying to performance tune, I'd trim it down ...
#16
Great ! Just like the HTTPObject Get without any issue, you rock ^^
(I didn't yet applied it, but it seems to be already battle tested by you, so I have confidence it'll work as expected)
PS : my PHP guy is asking me to use POST for security, I don't quite understand what is the difference, but, well, maybe I'm asking too much ^^'
02/07/2007 (4:10 pm)
@DavidGreat ! Just like the HTTPObject Get without any issue, you rock ^^
(I didn't yet applied it, but it seems to be already battle tested by you, so I have confidence it'll work as expected)
PS : my PHP guy is asking me to use POST for security, I don't quite understand what is the difference, but, well, maybe I'm asking too much ^^'
#17
Modifying my httpPage class to do posts would be fairly simple, however, it requires a bit more knowledge of HTTP Protocols --
Basically, a POST request is sent as such:
Now, the tricky thing is the 'Content-Length', and I believe there's another header or two that identifies it as a 'post' and not a 'get' request -- but the basics are there ...
The content-length must be the length of the entire content, following the headers -- so, it'd probably be best to build the 'post' portion in a seperate string, then use the strlen() function to build the Content-Length
Do a google search for "HTTP RFC Post" and you should be able to read up on it a bit --
Unfortunately, I'm a bit busy at the moment, but if an update to the httpPage that has a "post" method is necessary, you can email me and I'll put it on my "todo" list :)
02/07/2007 (5:57 pm)
@Ben,Modifying my httpPage class to do posts would be fairly simple, however, it requires a bit more knowledge of HTTP Protocols --
Basically, a POST request is sent as such:
GET http://site.com/index.php HTTP/1.0 Host: site.com:80 Content-Length: 9 VAR1=Data
Now, the tricky thing is the 'Content-Length', and I believe there's another header or two that identifies it as a 'post' and not a 'get' request -- but the basics are there ...
The content-length must be the length of the entire content, following the headers -- so, it'd probably be best to build the 'post' portion in a seperate string, then use the strlen() function to build the Content-Length
Do a google search for "HTTP RFC Post" and you should be able to read up on it a bit --
Unfortunately, I'm a bit busy at the moment, but if an update to the httpPage that has a "post" method is necessary, you can email me and I'll put it on my "todo" list :)
Torque Owner Chip Lambert
Crusader Games