Libcurl Integration ThreadSafe 2
by elvince · 10/04/2010 (7:08 am) · 7 comments
The implementation of this resource is the continuity of this thread
It's based on the current Bank's implementation, move to T3D & updated by Franck Bignone. Thanks for their work!
After many issue with the current implementation, I follow another way proposed by a community member and here we go!!
It resolved the main bug: You can't get more than 2 url in the queue.
and also have all the latest bug fixes of the previous version.
Download files
For the setup of the libcurl in the project,
From Bank's thread:
Download "full" curl package (I've used 1.17.1).
Unpack it into /lib/ folder (so you have the /lib/curl-7.17.1/* )
For "Torque Demo" (or your own project):
Add ../lib/curl-7.17.1/include to the project's "Addition Include Directories".
Add ;CURL_STATICLIB to project's "Preprocessor Definitions".
Add ;"../lib/curl-7.17.1/lib/Release" to project's "Additional Library Directories".
Add curllib.lib to project's "Addition Dependencies".
Add libcmt.lib to project's "Ignore Specific Library" (without this it won't link the EXE).
Add lib/curl-7-17-1/lib/curllib.vcproj project file into your solution.
In curllib project's properties:
Add ";../../zlib" to the project's "Addition Include Directories".
Add ";HAVE_ZLIB_H;HAVE_ZLIB;HAVE_LIBZ;CURL_STATICLIB;CURL_DISABLE_LDAP" to the "Preprocessor Definitions".
engine changes:
engine/app/mainloop.ccp:
at the top where all other includes are, add:
Scripts samples:
I mainly tested the url text fetcher more than the download part, so if you find any issue, you can post them here and I will try to solve them (or the community :D).
As someone raised, the Url is 256 characters long in this implementation.
It's based on the current Bank's implementation, move to T3D & updated by Franck Bignone. Thanks for their work!
After many issue with the current implementation, I follow another way proposed by a community member and here we go!!
It resolved the main bug: You can't get more than 2 url in the queue.
and also have all the latest bug fixes of the previous version.
Download files
For the setup of the libcurl in the project,
From Bank's thread:
Download "full" curl package (I've used 1.17.1).
Unpack it into /lib/ folder (so you have the /lib/curl-7.17.1/* )
For "Torque Demo" (or your own project):
Add ../lib/curl-7.17.1/include to the project's "Addition Include Directories".
Add ;CURL_STATICLIB to project's "Preprocessor Definitions".
Add ;"../lib/curl-7.17.1/lib/Release" to project's "Additional Library Directories".
Add curllib.lib to project's "Addition Dependencies".
Add libcmt.lib to project's "Ignore Specific Library" (without this it won't link the EXE).
Add lib/curl-7-17-1/lib/curllib.vcproj project file into your solution.
In curllib project's properties:
Add ";../../zlib" to the project's "Addition Include Directories".
Add ";HAVE_ZLIB_H;HAVE_ZLIB;HAVE_LIBZ;CURL_STATICLIB;CURL_DISABLE_LDAP" to the "Preprocessor Definitions".
engine changes:
engine/app/mainloop.ccp:
at the top where all other includes are, add:
#include "simCurlManager.h"change the beginning of initLibraries function:
static bool initLibraries()
{
if (!SimCurlManager::initialize())
{
AssertISV(false,"Unable to initialize the libcurl... aborting.");
}
if(!Net::init())
...change the end of shutdownLibraries function:Net::shutdown();
...
Net::shutdown();
SimCurlManager::deinitialize();
...
}Scripts samples:
if (!isObject(curler)) new ScriptGroup(curler);
function curler::addTextTask(%this, %url,%cbOK, %cbFailed, %cbProgress)
{
%id = getStringMD5(%url@%Action);
%sc = curler.findObjectByInternalName(%id);
if (!isObject(%sc))
{
%sc = new SimCurlItem() { internalName = %id; };
%this.add(%sc);
%sc.setURL(%url);
%sc.url = %url;
// Callbacks
if(%cbOK!$="")
%sc.setFinishCallback(%cbOK);
if(%cbFailed!$="")
%sc.setFailedCallback(%cbFailed);
// For the moment there is an issue in 1.1 Beta 3 for this function.
//I suffest not to use it. (not thread safe)
if(%cbProgress!$="")
%sc.setProgressCallback(%cbProgress);
%sc.ts = getSimTime();
AddSimCurlTask(%sc);
return %sc;
}
return false;
}I mainly tested the url text fetcher more than the download part, so if you find any issue, you can post them here and I will try to solve them (or the community :D).
As someone raised, the Url is 256 characters long in this implementation.
About the author
Recent Blogs
• T3D - Post Event Current time - Safer• Libcurl Integration ThreadSafe
• StackPanel T3D
#2
You can remove the engine api by living the console method and callback to the old style declaration. The rest of the code will work fine. .
10/06/2010 (8:05 pm)
Yes it s for beta version. You can remove the engine api by living the console method and callback to the old style declaration. The rest of the code will work fine. .
#3
10/07/2010 (4:49 am)
I can't get this to work. I got it to compile but it crashes when trying to make a new simcurlitem.
#4
Can you please check that you have done the initialize part?
10/07/2010 (6:51 am)
I add the engine changes in my resource implementation that where described in older implementation.Can you please check that you have done the initialize part?
#6
Here is the way I am starting the download:
Is there anything noticeably wrong with it that would make downloading not work?
03/16/2011 (10:36 am)
I have been trying to get downloading with this resource setup and it doesnt seem to be fully working for me. I am trying to use it for an auto-patch system and when the file starts downloading, it stops after 7 kb. Heres the download method I added:function curler::addDownloadTask(%this, %url, %file, %expectedSize, %md5, %cbOK, %cbFailed, %cbProgress)
{
%id = getStringMD5(%url);
%sc = curler.findObjectByInternalName(%id);
if (!isObject(%sc))
{
%sc = new SimCurlItem() { internalName = %id; };
%this.add(%sc);
%sc.setURL(%url);
%sc.url = %url;
if(%expectedSize<1) // we treat "", "0" or "-1" as "unknown"
%sc.fileSizeExp = -1;
else
%sc.fileSizeExp = %expectedSize;
%sc.setFileName(%file);
%sc.file = %file;
%sc.hash = %md5;
// Callbacks
%sc.callBack = %cbOK;
if(%cbOK!$="")
%sc.setFinishCallback(%cbOK);
%sc.callBackFail = %cbFailed;
if(%cbFailed!$="")
%sc.setFailedCallback(%cbFailed);
%sc.callBackProgress = %cbProgress;
if(%cbProgress!$="")
%sc.setProgressCallback(%cbProgress);
%sc.ts = getSimTime();
%sc.runCount = 0;
if (isFile(%file) && fileSize(%file)==0)
{
if($curl::debug) error("Removing zero-sized file!");
deleteFile(%file);
}
if (isFile(%file))
{
if($curl::debug) error("File exists!!!");
%fileSize = fileSize(%file);
if(%sc.fileSizeExp > 0 && fileSize(%file)==%sc.fileSizeExp)
{
if($curl::debug) error("We know the expected file size and file sizes are the same!");
if(%sc.hash!$="" && strlwr(%sc.hash)$=getFileMD5(%file))
{
if($curl::debug) error("File sizes is the same and hashes too! Nothing to do!");
schedule(10, 0, %sc.callBack, %sc);
return true;
}
else
{
if($curl::debug) error("File size is the same, but hash is different! Deleting file!");
fileDelete(%file);
}
}
if(%sc.fileSizeExp > 0 && fileSize(%file) > %sc.fileSizeExp)
{
if($curl::debug) error("Existing file is greater than we expecting! Deleting file!");
fileDelete(%file);
}
if(%sc.fileSizeExp > 0 && fileSize(%file)>0 && fileSize(%file) < %sc.fileSizeExp)
{
if($curl::debug) error("Setting resuming from:" SPC fileSize(%file));
%sc.resumingFrom = fileSize(%file);
%sc.setResumeFrom(%sc.resumingFrom);
}
}
AddSimCurlTask(%sc);
return %sc;
}
else
{
error("The download is already in progress... not starting a new one!!!");
return false;
}
}Here is the way I am starting the download:
curler.addDownloadTask("http://example.com/patches/patch1.exe", "patches/patch1.exe", -1, "", "getPatchOK", "cbVersionError", "getPatchProgress");Is there anything noticeably wrong with it that would make downloading not work?
#7
07/21/2011 (7:05 pm)
Just as a side note, I should mention that it appears you've included most of the necessary data structures for my cookie jar + POST resource (www.garagegames.com/community/blogs/view/18545), but not quite everything.
Torque 3D Owner Entr0py