Game Development Community

Downloading missing conent from a web server

by Neil Marshall · in Torque Game Engine · 05/30/2006 (9:00 am) · 5 replies

I'm trying to change TGE so that if you're not connected to a multiplayer server and you have missing content it will instead go to a remote web server, download the files and then go.

Right now in ResManager::fileIsMissing() I'm trying to open a HTTP connection and download the file, then check again. But I can't find any up to date information on how to properly transfer files to the data folder. How do I do this? Is there any documentation on HTTPObject that I'm missing? Most of the stuff that I've found is CScript instead of C++.
NextNext Thread
PrevPrevious Thread
More from Torque Game Engine


Torque Owner Gerald Fishel
Development Ninja
#1
05/30/2006 (9:49 am)
Looking at the HTTPObject code, it looks to be really limited and will only handle textual data.

If you need to download binary data you'll need to modify HTTPObject to parse atleast the Content-Length header, and modify the onReceive method to handle binary data rather than processing on a line-by-line basis.

This is something that I will need as well, so I may tackle it in the next couple of days, I'll post a solution when I get it working.

Torque Owner Stefan Lundmark
#2
05/30/2006 (9:51 am)
There's a resource named TCPObject Binary Transfer, take a look at that.

Torque Owner Neil Marshall
#3
05/30/2006 (12:12 pm)
I've implemented the binary transfers resource, but I'm having trouble with it still as I'm in C++ and not CScript.

TCPObject tcpDownload;
tcpDownload.connect("www.buildingtogo.com:80");
if (tcpDownload.Connected) {
   tcpDownload.setBinary(true);
   U8 *myString = dStrdup("GET /img/indexPic.jpg HTTP/1.1\nHost: www.buildingtogo.com\nUser-Agent: Torque/1.0 \n\r\n");
   U8 myLen = 89;
   tcpDownload.send( myString, myLen);
   tcpDownload.saveBufferToFile("c:\temp.jpg");
   tcpDownload.disconnect();
}

My first problem is I can't define the string properly, but I have a feeling I'm going to run into a problem where the http header is tacked onto the beginning of the image once I do get it to transfer properly. Can anyone help?

Torque Owner Gerald Fishel
Development Ninja
#4
05/30/2006 (1:07 pm)
Hi,

First, you'll want to use \r\n instead of just \n after each header. i.e.:
U8 *myString = dStrdup("GET /img/indexPic.jpg HTTP/1.1\r\nHost: www.buildingtogo.com\r\nUser-Agent: Torque/1.0\r\n\r\n");


How you handle the response will depend on the web server. In most cases, you can process the response header on a line by line basis, and if it's a binary file, there will usually be a Content-Length header that tells you how large the binary data is after the header. Then when you receive a blank line, i.e. \r\n by itself, what follows will be the binary data.

Some web servers will compress the response body, and there will be a Content-Encoding header, i.e. Content-Encoding: gzip. In that case you will need to decompress the data once it is all received.

If you need to support arbitrary web servers, then you may want to think about using third party libraries. On Windows you can use WinINet to easily download web files. I'm not sure about Linux and Mac, but maybe you can shell execute wget or something. But there's enough variables, that supporting HTTP file downloads on arbitrary web servers can be a fairly complex process.

But if it's just for a specific web server, you can probably hack something together that will be good enough.

Torque Owner Neil Marshall
#5
05/30/2006 (1:18 pm)
Just a specific server for now. I don't think it will have gzip compression either.