Libcurl implementation
by Fyodor "bank" Osokin · in Torque Game Engine · 11/18/2007 (3:44 am) · 51 replies
this is my own libcurl implementation into TGE 1.5.2 "as is". Feel free to use it for your own games.
---------------------------
What implemented:
download works for http and ftp (without login/password - tested)
download works for http and ftp (with login/password - not tested)
you can set to resume previously interrupted download/upload (tested)
you can cancel download (tested)
upload works, but not tested heavily
---------------------------
See the available script functions.
If you need upload, don't forget to call .setUploadFlag(true); :)
---------------------------
Instructions:
Download "full" curl package (I've used 1.17.1).
Unpack it into /lib/ folder (so you have the /lib/curl-7.17.1/* )
For "Torque Demo" (or your own project):
Add ../lib/curl-7.17.1/include to the project's "Addition Include Directories".
Add ;CURL_STATICLIB to project's "Preprocessor Definitions".
Add ;"../lib/curl-7.17.1/lib/Release" to project's "Additional Library Directories".
Add curllib.lib to project's "Addition Dependencies".
Add libcmt.lib to project's "Ignore Specific Library" (without this it won't link the EXE).
Add lib/curl-7-17-1/lib/curllib.vcproj project file into your solution.
In curllib project's properties:
Add ";../../zlib" to the project's "Addition Include Directories".
Add ";HAVE_ZLIB_H;HAVE_ZLIB;HAVE_LIBZ;CURL_STATICLIB;CURL_DISABLE_LDAP" to the "Preprocessor Definitions".
engine changes:
game/main.cc:
at the top where all other includes are, add:
Rebuild the "curllib" project.
Rebuild your project (Torque Demo).
If it don't compile and giving errors like "templates can't be used when "C" linkage.." for wspiapi.h (see this page for explanation), you need to change the curl/curl.h file in includes folder (changes in BOLD):
See next post for code :)
---------------------------
What implemented:
download works for http and ftp (without login/password - tested)
download works for http and ftp (with login/password - not tested)
you can set to resume previously interrupted download/upload (tested)
you can cancel download (tested)
upload works, but not tested heavily
---------------------------
See the available script functions.
If you need upload, don't forget to call .setUploadFlag(true); :)
---------------------------
Instructions:
Download "full" curl package (I've used 1.17.1).
Unpack it into /lib/ folder (so you have the /lib/curl-7.17.1/* )
For "Torque Demo" (or your own project):
Add ../lib/curl-7.17.1/include to the project's "Addition Include Directories".
Add ;CURL_STATICLIB to project's "Preprocessor Definitions".
Add ;"../lib/curl-7.17.1/lib/Release" to project's "Additional Library Directories".
Add curllib.lib to project's "Addition Dependencies".
Add libcmt.lib to project's "Ignore Specific Library" (without this it won't link the EXE).
Add lib/curl-7-17-1/lib/curllib.vcproj project file into your solution.
In curllib project's properties:
Add ";../../zlib" to the project's "Addition Include Directories".
Add ";HAVE_ZLIB_H;HAVE_ZLIB;HAVE_LIBZ;CURL_STATICLIB;CURL_DISABLE_LDAP" to the "Preprocessor Definitions".
engine changes:
game/main.cc:
at the top where all other includes are, add:
#include "sim/simCurl.h"change the beginning of initLibraries function:
static bool initLibraries()
{
[b]if (!SimCurl::initialize())
{
Platform::AlertOK("libcurl Error", "Unable to initialize the libcurl... aborting.");
return false;
}
[/b]
if(!Net::init())
...change the end of shutdownLibraries function:Net::shutdown(); [b]SimCurl::deinitialize();[/b] }
Rebuild the "curllib" project.
Rebuild your project (Torque Demo).
If it don't compile and giving errors like "templates can't be used when "C" linkage.." for wspiapi.h (see this page for explanation), you need to change the curl/curl.h file in includes folder (changes in BOLD):
#if !(defined(_WINSOCKAPI_) || defined(_WINSOCK_H))
/* The check above prevents the winsock2 inclusion if winsock.h already was
included, since they can't co-exist without problems */
[b]#ifdef __cplusplus
}
#endif[/b]
#include <winsock2.h>
#include <ws2tcpip.h>
[b]#ifdef __cplusplus
extern "C" {
#endif[/b]
#endifSee next post for code :)
About the author
Dedicated game developer.
#42
08/12/2009 (9:52 am)
I think you will still have one issue, as the line after the if test will try to put a value to a NULL pointer... so maybe make an exit with something like thatconst U8 *SimCurl::readLine()
{
if(!storagedesc.mem.ptr)
{
static U8 retBuf[1] = "";
storagedesc.mem.ptr = NULL;
storagedesc.mem.size = 0;
storagedesc.mem.c = NULL;
storagedesc.mem.rsize = 0;
return &retBuf[0];
}
// Should not be here after NULL setting
storagedesc.mem.c[storagedesc.mem.rsize] = 0;
U8 *tokPos = storagedesc.mem.c;
...
#43
I'm terribly sorry for being so sloppy. That will teach me. ;)
08/12/2009 (10:08 am)
You're right.. I dumped my code, and added a pointer to yours instead. :)I'm terribly sorry for being so sloppy. That will teach me. ;)
#44
I can't get it to compile at all under Windows.
This line:
Throws errors if I'm using GCC or VC++, but only under Windows, not under OS X.
[edit]
Fixed by using the placenew macro, and making sure placenew() is defined in platform.h before new itself is redefined as a macro.
[edit]
Not fixed. It just tried recompiling everything. Silly me for thinking it was moving on to the next step. I still need help.
[edit]
Fixed. For real. Conditionally undef the new macro used by the memory manager. around the function using placement new.
09/10/2009 (5:05 am)
Hey guys, any word on what might be going wrong with the placement new being used in SimCurl::write_data?I can't get it to compile at all under Windows.
This line:
new(s) MemStream(nsz, ptr);
Throws errors if I'm using GCC or VC++, but only under Windows, not under OS X.
[edit]
Fixed by using the placenew macro, and making sure placenew() is defined in platform.h before new itself is redefined as a macro.
[edit]
Not fixed. It just tried recompiling everything. Silly me for thinking it was moving on to the next step. I still need help.
[edit]
Fixed. For real. Conditionally undef the new macro used by the memory manager. around the function using placement new.
#45
09/18/2009 (5:08 pm)
I wasnt going to go through 44 messages on this topic, but I did want to point something out. That as this option has upload can we not beable to setlimit on bandwith and instead of loading all missions ata time. set a trigger places around where the player goes into the next mis and just download files before he even reaches it? this would create a VERY faster load time, or would it cause a crash in the system when switching missions ?
#49
and your "instructions" part up top was very handy, thanks Bank !
fwiw, i used curl 7.19.7, and there were a couple differences:
* the library is "libcurl.lib", not "curllib.lib".
* i didn't have to add "libcmt.lib" to the ignored-libs list.
.. of course, that's just compiling and linking. i haven't actually used it so far! should be fine.
11/06/2009 (1:26 am)
i'm doing another integration of libCurl into T3D,and your "instructions" part up top was very handy, thanks Bank !
fwiw, i used curl 7.19.7, and there were a couple differences:
* the library is "libcurl.lib", not "curllib.lib".
* i didn't have to add "libcmt.lib" to the ignored-libs list.
.. of course, that's just compiling and linking. i haven't actually used it so far! should be fine.
#50
So if you are really willing to use extensively, you need to address all those issues.
I hope I will be able to show you some videos of my new code where the system download all models when required from the server inside the mission without halting the player and so on.
11/06/2009 (9:19 am)
One important comment as I did a lot of work with this in order to implement a fully data streaming system (for mission objects and so on) inside T3D, this implementation is not thread-safe at all; there are a lot of issue with multiple parts of T3D code and mainly everything around the Con::execute and FileStream...So if you are really willing to use extensively, you need to address all those issues.
I hope I will be able to show you some videos of my new code where the system download all models when required from the server inside the mission without halting the player and so on.
#51
aye, i'm aware of the thread-safety gotchas. this particular instance is not super general-purpose and doesn't have to worry about them so much, but they're definitely something to keep very-much in mind. ie, i'm not even exposing it to script, and blocking operations on curl's part are fine. but yr absolutely right: to implement a flexible curl system for general use within torque definitely requires some attention to detail w/r/t thread-safety. eg, don't jump into script and then back into C and then back into script all as part of a single call-chain.
also, i was mistaken: i did need to add libcmt to the ignore list.
11/06/2009 (11:55 am)
hey frank, thanks.aye, i'm aware of the thread-safety gotchas. this particular instance is not super general-purpose and doesn't have to worry about them so much, but they're definitely something to keep very-much in mind. ie, i'm not even exposing it to script, and blocking operations on curl's part are fine. but yr absolutely right: to implement a flexible curl system for general use within torque definitely requires some attention to detail w/r/t thread-safety. eg, don't jump into script and then back into C and then back into script all as part of a single call-chain.
also, i was mistaken: i did need to add libcmt to the ignore list.
Associate Konrad Kiss
Thanks again. I've corrected the code above.