Game Development Community

Libcurl implementation

by Fyodor "bank" Osokin · in Torque Game Engine · 11/17/2007 (7:44 pm) · 63 replies

this is my own libcurl implementation into TGE 1.5.2 "as is". Feel free to use it for your own games.
---------------------------
What implemented:
download works for http and ftp (without login/password - tested)
download works for http and ftp (with login/password - not tested)
you can set to resume previously interrupted download/upload (tested)
you can cancel download (tested)
upload works, but not tested heavily
---------------------------
See the available script functions.
If you need upload, don't forget to call .setUploadFlag(true); :)

---------------------------
Instructions:

Download "full" curl package (I've used 1.17.1).
Unpack it into /lib/ folder (so you have the /lib/curl-7.17.1/* )

For "Torque Demo" (or your own project):
Add ../lib/curl-7.17.1/include to the project's "Addition Include Directories".
Add ;CURL_STATICLIB to project's "Preprocessor Definitions".
Add ;"../lib/curl-7.17.1/lib/Release" to project's "Additional Library Directories".
Add curllib.lib to project's "Addition Dependencies".
Add libcmt.lib to project's "Ignore Specific Library" (without this it won't link the EXE).

Add lib/curl-7-17-1/lib/curllib.vcproj project file into your solution.

In curllib project's properties:
Add ";../../zlib" to the project's "Addition Include Directories".
Add ";HAVE_ZLIB_H;HAVE_ZLIB;HAVE_LIBZ;CURL_STATICLIB;CURL_DISABLE_LDAP" to the "Preprocessor Definitions".

engine changes:
game/main.cc:
at the top where all other includes are, add:
#include "sim/simCurl.h"
change the beginning of initLibraries function:
static bool initLibraries()
{
   [b]if (!SimCurl::initialize())
   {
      Platform::AlertOK("libcurl Error", "Unable to initialize the libcurl... aborting.");
      return false;
   }
[/b]
   if(!Net::init())
   ...
change the end of shutdownLibraries function:
Net::shutdown();
   [b]SimCurl::deinitialize();[/b]
}

Rebuild the "curllib" project.
Rebuild your project (Torque Demo).

If it don't compile and giving errors like "templates can't be used when "C" linkage.." for wspiapi.h (see this page for explanation), you need to change the curl/curl.h file in includes folder (changes in BOLD):
#if !(defined(_WINSOCKAPI_) || defined(_WINSOCK_H))
/* The check above prevents the winsock2 inclusion if winsock.h already was
   included, since they can't co-exist without problems */
  [b]#ifdef __cplusplus
    }
  #endif[/b]
#include <winsock2.h>
#include <ws2tcpip.h>
  [b]#ifdef __cplusplus
    extern "C" {
  #endif[/b]
#endif

See next post for code :)
NextNext Thread
PrevPrevious Thread
More from Torque Game Engine
Page«First 1 2 3 4 Last »

Associate Konrad Kiss
Bitgap Games
#21
03/31/2009 (10:08 am)
I've found out today that the return string (ie. in readLine) is not ending with a zero char. That can cause buffer overflows. A simple hack is to add that 0 ourselves:

const U8 *SimCurl::readLine()
{
	storagedesc.mem.c[storagedesc.mem.rsize] = 0;

	U8 *tokPos = storagedesc.mem.c;

Hacky, but does the job, and less prone to failure...

Torque 3D Owner Thomas "elfprince13" Dickerson
#22
06/03/2009 (2:15 pm)
thanks for that update Konrad. This resource is great.

Torque 3D Owner Thomas "elfprince13" Dickerson
#23
06/03/2009 (3:23 pm)
hmmm, I'm getting a couple of errors now (I'm trying to compile this with XCode, for the record).

CURLE_OBSOLETE4 was not declared in this scope.
if(!ResourceManager->openFileForWrite(s, storagedesc.filename, File::WriteAppend))
			   {
				   delete stream;
				   stream = NULL;
				   ret = CURLE_OBSOLETE4;
			   }

CURLE_OBSOLETE10 was not declared in this scope.
if(!ResourceManager->openFileForWrite(s, storagedesc.filename))
			{
				delete stream;
				stream = NULL;
				ret = CURLE_OBSOLETE10;
			}


[edit]
So....CURLE_OK is defined, as are most of the CURLE_* status codes. Unfortunately I can't find where they would be defined that CURLE_OBSOLETEn *isn't*.

[edit2]
evidently the CURLE_OBSOLETE error codes are actually aliased versions of older (obsolete) error codes. Not sure where exactly it happens, but CURL_NO_OLDIES seems to be related? Anyway, I just replaced CURLE_OBSOLETE10 and CURLE_OBSOLETE4 with the actual old names, and it works, but now I'm getting some symbol not defined errors, elsewhere in code. I suspect I didn't link the library correctly.

[edit3]
Built libcurl as a framework using the script here. http://curl.haxx.se/mail/lib-2009-02/0265.html

[edit4]
Everything seems to be working, but I'm a little unsure as to how to POST over http instead of GET. I guess I'll mess about a little and see. If anyone's gotten it figured out already, please let me know.

Torque 3D Owner Thomas "elfprince13" Dickerson
#24
06/13/2009 (7:37 pm)
Everything seems to build fine under OS X. Windows took a little tweaking (the Windows SDK version included with Torque seemed to be clashing with the default one on the system, a problem which only appeared after I included the curl headers), however I'm getting one last syntax error from the Windows build.


In this little block here:
if(ptr != simcurl->storagedesc.mem.ptr)
		{
			((MemStream*)(s))->~MemStream();
			new(s) MemStream(nsz, ptr);
			s->setPosition(p);
		}

I get the following error:
Quote:1>..\engine\sim\simCurl.cc(63) : error C2061: syntax error : identifier 's'

In particular, it seems to be the line which reads
new(s) MemStream(nsz, ptr)

that is being referenced in the error message. Has anyone else encountered this?

Torque Owner elvince
Neo Studio
#25
06/18/2009 (11:28 pm)
I have just 2 errors I don't how to solve them properly:
hi,

In T3D, the executef have change its signature, so now the code :
Con::executef(2, cbFinish, this->getIdString());
Need to be changed.
There are 3 call to this function in the source code.
I thing, I might move 2 of them to
Con::executef(this, cbFinish, this->getIdString());
or Con::executef("2", cbFinish, this->getIdString());?
but the one in SimCurl::curl_progress_callback, I don't know how to fix it.

More, I have 16 errors like:
error LNK2019: unresolved external symbol __imp__freeaddrinfo@4 referenced in function _Curl_getaddrinfo_ex libcurl.lib

error LNK2019: unresolved external symbol __imp__ldap_unbind_s referenced in function _Curl_ldap libcurl.lib

Any clue how to solve that? Did I miss a step?
FYI, The libcurl dll has compile successfully.

Associate Konrad Kiss
Bitgap Games
#26
06/19/2009 (5:41 am)
@elvince: Just get rid of that 2 completely and it should work. The thing is, in TGE, the function had to be told how many parameters it should be expecting, but not anymore.

So that
Con::executef(2, cbFinish, this->getIdString());
becomes
Con::executef(cbFinish, this->getIdString());


Torque Owner elvince
Neo Studio
#27
06/20/2009 (4:14 am)
Thanks for the tips.

Konrad, do you know how can I solve those 2 errors:

Error 2 error LNK2019: unresolved external symbol __imp__freeaddrinfo@4 referenced in function _Curl_getaddrinfo_ex libcurl.lib
Error 3 error LNK2019: unresolved external symbol __imp__getaddrinfo@16 referenced in function _Curl_getaddrinfo_ex libcurl.lib

During the compilation, I get those. I'm trying to understand what link I should add.

Thanks,

Torque Owner elvince
Neo Studio
#28
06/20/2009 (7:36 am)
I find, wz need to had 'ws2_32.lib' to linker!

Torque 3D Owner Huan Li
#29
06/26/2009 (2:02 am)
in Torque 3D beta3, The new Thread() can not start a new thread as it would in beta2, the following fix will keep it working.

in SimCurl.cpp, find 
_G.thread = new Thread((ThreadRunFunction)threadFunction, 0, true);

replace the above line with:

//[ fix start
// first, set the thrid arg to false
_G.thread = new Thread((ThreadRunFunction)threadFunction, 0, /*true*/ false);

// then, start the thread manually
_G.thread->start();
//] fix end

Torque 3D Owner Tony Richards
#30
06/30/2009 (10:59 am)
Quote:I also found a bug of some sort: it is possible that when (or soon after) deleting the object, perform() gets called. _G.head never evaluates to false even after the deletion, so that was causing me some crashes.

Try changing _G.head to volatile. In multi-CPU / multi-core situations, the mutex will not impose a full memory barrier on _G.head, which means that sometimes _G.head will be NULL in one core and not NULL in another.

Personally I'd suggest starting a single thread and using a semaphore, event or condition variable to wake it up when the queue is non-empty. This will eliminate the significant amount of overhead required for starting a new thread.

Associate Konrad Kiss
Bitgap Games
#31
06/30/2009 (12:34 pm)
Thanks Tony. Truth be told, I'm not fluent in multi-core, multi-process situations, so I'll need some time to really dive deep into this. Until then, I will force this to run in a single core - if I can. Thank you again!

Torque Owner Adam Beer
Ignition Games
#32
06/30/2009 (12:53 pm)
Im getting a bunch of errors here:

simCurl.cpp
1>..\..\..\..\..\Engine\source\sim\simCurl.cpp(288) : error C2065: 'ResourceManage' : undeclared identifier
1>..\..\..\..\..\Engine\source\sim\simCurl.cpp(288) : error C2227: left of '->openStream' must point to class/struct/union/generic type
1>        type is ''unknown-type''
1>..\..\..\..\..\Engine\source\sim\simCurl.cpp(296) : error C2819: type 'ResourceManager' does not have an overloaded member 'operator ->'
1>        d:\IgnitionGames\Mach 1\MachFinal\Engine\source\core/resourceManager.h(20) : see declaration of 'ResourceManager'
1>        did you intend to use '.' instead?
1>..\..\..\..\..\Engine\source\sim\simCurl.cpp(296) : error C2039: 'openFileForWrite' : is not a member of 'ResourceManager'
1>        d:\IgnitionGames\Mach 1\MachFinal\Engine\source\core/resourceManager.h(20) : see declaration of 'ResourceManager'
1>..\..\..\..\..\Engine\source\sim\simCurl.cpp(296) : error C2653: 'File' : is not a class or namespace name
1>..\..\..\..\..\Engine\source\sim\simCurl.cpp(296) : error C2065: 'WriteAppend' : undeclared identifier
1>..\..\..\..\..\Engine\source\sim\simCurl.cpp(306) : error C2819: type 'ResourceManager' does not have an overloaded member 'operator ->'
1>        d:\IgnitionGames\Mach 1\MachFinal\Engine\source\core/resourceManager.h(20) : see declaration of 'ResourceManager'
1>        did you intend to use '.' instead?
1>..\..\..\..\..\Engine\source\sim\simCurl.cpp(306) : error C2039: 'openFileForWrite' : is not a member of 'ResourceManager'
1>        d:\IgnitionGames\Mach 1\MachFinal\Engine\source\core/resourceManager.h(20) : see declaration of 'ResourceManager'

I have included core/resourceManager.h but that didnt do anything. Anyone know whats up?

Torque Owner Adam Beer
Ignition Games
#33
06/30/2009 (12:59 pm)
Never mind, I got it. All the 'ResourceManager' instances should be 'gResourceManager'. Also, just so I know my one change was correct, I changed File::WriteAppend to Torque::FS::File::Write. Would this be the correct change?

Associate Konrad Kiss
Bitgap Games
#34
06/30/2009 (1:58 pm)
Quote:
I changed File::WriteAppend to Torque::FS::File::Write. Would this be the correct change?

Yes, that's correct for TGEA 1.8.1 and Torque3D

Torque 3D Owner Davide Archetti
Default Studio Name
#35
07/03/2009 (1:29 pm)
Hi,
I've implemented this in Torque3D. But I whish to use it mainly in C++, so I've derived a new class from SimCurl, made the SimCurl::perform function virtual, so when I run my derived class its perform function is called, inside it I call Parent::perform() and then I do my tasks with the lines I get back from the net.
Everything seems to work fine, but sometimes I get this assert:
"Error, someone didn't reset the water mark on the frame allocator!"
I have investigated a bit, and I found that when you do a Con::executef, internally the console use a FrameTemp<char>, that uses the FrameAllocator function responsible to set and change the water mark.
What I think is that, sice SimCurl runs in its own thread, it changes the smWaterMark variable (that is a static variable) so that the main thread finds the watermark in a different position, and this could be very dangerous because it says where the allocated memory begins.
The solution to this, should be to run the Con::executef atomically, but I'm not an expert in threading, so I don't know where to start.
Does someone else have seen this problem?

Torque 3D Owner Davide Archetti
Default Studio Name
#36
07/04/2009 (12:34 pm)
Ok, I reply to myself.
The problem is that inside my perform function I used the Con::executef(OBJ, A a .... ) function, and I've learned it isn't thread safe, instead the Con::executef(A a...) is thread safe, indeed the error I got went away.
Hope this can help someones ;)

Torque 3D Owner Frank Bignone
Darkhand Studio
#37
07/08/2009 (9:05 am)
Just put a resource with a simple install to add to T3D project generator here: Easy libCurl integration for T3D

Associate Konrad Kiss
Bitgap Games
#38
07/08/2009 (9:34 am)
Quote:
I find, wz need to had 'ws2_32.lib' to linker!

Thanks for telling us elvince, that saved me some headaches!

Associate Konrad Kiss
Bitgap Games
#39
08/10/2009 (4:37 pm)
If you're targeting a page that doesn't output even one character (ie. due to some suppressed error), you're gonna be in trouble when you call readline, because storagedesc.mem is not initialized. Do something like this to circumvent an endless while loop (I included the other addition I posted above):

Edit: see Frank's code a few posts below..

Torque 3D Owner Frank Bignone
Darkhand Studio
#40
08/12/2009 (12:11 am)
Just one comment, why are you allocating so much space in your code dMalloc(16<<10)?
Page«First 1 2 3 4 Last »