Server->Client Resource downloading is in!
by Mark Frohnmayer · in Torque Game Engine · 04/03/2002 (5:22 pm) · 28 replies
Hey all,
Just finished up the checkin on version 2 of the resource downloading code, which now works with datablocks. So models, textures, interiors, terrains, sounds, etc... can all be downloaded from the server if they are missing on the client.
Details of the process:
When ghost always objects or datablocks are received, they are queued on a vector in the NetConnection. If they add properly, they are popped off the queue and processing continues. If they fail to add, or in the case of datablocks if they fail to preload, the list of resources that the object tried to load is sent to the server, which will download any missing ones to the client. The client then tries to reload the object. If it fails again, any new resources it requested are downloaded from the server.
If the server is unable to download any resources to the client and the client object still fails to add, then the connection terminates like before with the appropriate error message.
I also bumped the net protocol, since the new code is incompatible with the previous net version.
If anyone out there wants to hammer on this, go for it...
Just finished up the checkin on version 2 of the resource downloading code, which now works with datablocks. So models, textures, interiors, terrains, sounds, etc... can all be downloaded from the server if they are missing on the client.
Details of the process:
When ghost always objects or datablocks are received, they are queued on a vector in the NetConnection. If they add properly, they are popped off the queue and processing continues. If they fail to add, or in the case of datablocks if they fail to preload, the list of resources that the object tried to load is sent to the server, which will download any missing ones to the client. The client then tries to reload the object. If it fails again, any new resources it requested are downloaded from the server.
If the server is unable to download any resources to the client and the client object still fails to add, then the connection terminates like before with the appropriate error message.
I also bumped the net protocol, since the new code is incompatible with the previous net version.
If anyone out there wants to hammer on this, go for it...
About the author
#2
04/03/2002 (5:51 pm)
The code just uses the network system, so it has the same cap for downloading as any other client. It just uses the normal loading gui screen for displaying file download progress. 
#3
04/03/2002 (6:58 pm)
Thanks alot Mark good work!. I am updating now to check it out. I am curious about GUI's being transfered to the client also, How hard would it be to transfer custom server GUI files also so players could use new GUI's without having to go to a website or such to get them. Just an Idea that I thought would be useful. If this is already included ignore heh.. I guess I should have waited till it was done with the compile to post feedback. 
#4
Then I fired up my game puter with the same HEAD build (from 30 minutes ago) - query master goes fine, so I join.
The server does its thing, gets to the CADD: part for my client.
The client looks like this:
The joystick & mousestuff at the end came from me clicking in the window afterwards several times...
Anyways, it stuck there.
04/03/2002 (8:12 pm)
Mark, I set up a server, dedicated, waterworld mission.Then I fired up my game puter with the same HEAD build (from 30 minutes ago) - query master goes fine, so I join.
The server does its thing, gets to the CADD: part for my client.
The client looks like this:
The joystick & mousestuff at the end came from me clicking in the window afterwards several times...
Anyways, it stuck there.
#5
I have noticed something else, at least the release exe isn't releasing some resources, trying to delete it and it screams that it is in use.
Maybe something's not getting cleaned up?
04/03/2002 (8:51 pm)
Updating as we speak, see if I get the same probelm. However, you might want to leave and come back, the downlaod could be happening, check your client example dir and see if it's downloading stuff. I have noticed something else, at least the release exe isn't releasing some resources, trying to delete it and it screams that it is in use.
Maybe something's not getting cleaned up?
#6
04/03/2002 (9:49 pm)
Thx, Ed. I left it for 30 mins, still no go. But of course, since the server directories were exact copies of the client dirs, there should only have been the mission file to d/l anyway, no ?
#7
I changed the shortcut to read (blah blah) -missions fps/data/missions/waterWorld.mis (forward slashes) and away it went. It started the mission loading. Eventually the client said "sending request for takeme.wav" which was apparently satisfied. Then it sent a request for " .wav", (no file name, just the dot-extension). At this point the server crashed.
04/03/2002 (10:16 pm)
Update - I went back and scanned the server console again. I noticed that it said it couldn't find mission fps\data\mission\waterWorld.mis (note the back slashes). That's probably the problem right there.I changed the shortcut to read (blah blah) -missions fps/data/missions/waterWorld.mis (forward slashes) and away it went. It started the mission loading. Eventually the client said "sending request for takeme.wav" which was apparently satisfied. Then it sent a request for " .wav", (no file name, just the dot-extension). At this point the server crashed.
#8
04/03/2002 (10:45 pm)
Hrm... it crashed? Doh! Ken, if you can dig a bit further and give me some more info that'd be great... I wasn't able to crash my server at all by requesting files...
#9
----------------------------------------------
TORQUEDEMO caused an invalid page fault in
module TORQUEDEMO.EXE at 0177:0059a769.
Registers:
EAX=00b1f6c0 CS=0177 EIP=0059a769 EFLGS=00010216
EBX=018c55ec SS=017f ESP=00b1f6b8 EBP=018c5884
ECX=00000000 DS=017f ESI=0188054c FS=452f
EDX=616820b1 ES=017f EDI=0000003f GS=0000
Bytes at CS:EIP:
8b 11 56 50 57 ff 52 04 6a 7f 68 0c 27 68 00 6a
Stack dump:
01880554 018c55ec 00000030 0184d8ec 00562928 00562d67 018c5884 018804bc 018804bc 018c55ec 0056301f 018804bc 01880554 0188050c 00562f8c 0188050c
------------------------------------------------
I was working through with the debug build trying to catch it in the act. I fell asleep in my chair and just woke up 15 mins ago ... going to bed now.
04/03/2002 (11:49 pm)
This is hte crash dump from the crash mentioned above:----------------------------------------------
TORQUEDEMO caused an invalid page fault in
module TORQUEDEMO.EXE at 0177:0059a769.
Registers:
EAX=00b1f6c0 CS=0177 EIP=0059a769 EFLGS=00010216
EBX=018c55ec SS=017f ESP=00b1f6b8 EBP=018c5884
ECX=00000000 DS=017f ESI=0188054c FS=452f
EDX=616820b1 ES=017f EDI=0000003f GS=0000
Bytes at CS:EIP:
8b 11 56 50 57 ff 52 04 6a 7f 68 0c 27 68 00 6a
Stack dump:
01880554 018c55ec 00000030 0184d8ec 00562928 00562d67 018c5884 018804bc 018804bc 018c55ec 0056301f 018804bc 01880554 0188050c 00562f8c 0188050c
------------------------------------------------
I was working through with the debug build trying to catch it in the act. I fell asleep in my chair and just woke up 15 mins ago ... going to bed now.
#10
Takeme.wav is a placeholder for some audio profiles. Eitehr put a real wav in there, or comment out that profile.
I bet that will fix it.
That being said, there are lots of placeholders in the original scripts, might be a better thing to have a more graceful error catching ;)
04/04/2002 (8:22 am)
So it sounds like it is crashing on things that don't actually exist.Takeme.wav is a placeholder for some audio profiles. Eitehr put a real wav in there, or comment out that profile.
I bet that will fix it.
That being said, there are lots of placeholders in the original scripts, might be a better thing to have a more graceful error catching ;)
#11
04/04/2002 (10:22 am)
Ken, if you could run the server in the debugger and get a file/line of the crash that would be great. So far I've been unable to reproduce it.
#12
If it has a particle (or presumably another datablock) that is missing, the debug build says:
Invalid Connection ObjectID with a retry or cancel. retrying invalid page faults, cancle quits to desktop.
amd the console echos: Connection error: Missing particle texture /fps/blah/blah
This is a client local to the server, means I have a bad datablock in my scripts. Correcting that datablock and the crash goes away.
I suspect this is what the other crash may be?
04/04/2002 (2:43 pm)
I can get it to crash, head from last night.If it has a particle (or presumably another datablock) that is missing, the debug build says:
Invalid Connection ObjectID with a retry or cancel. retrying invalid page faults, cancle quits to desktop.
amd the console echos: Connection error: Missing particle texture /fps/blah/blah
This is a client local to the server, means I have a bad datablock in my scripts. Correcting that datablock and the crash goes away.
I suspect this is what the other crash may be?
#13
(My server doesn't have MS Studio on it, nor a vid card that will support the client...)
04/04/2002 (4:24 pm)
grrr... can't get it to crash when running locally. Have to load the client one of my kid's puters and try it.(My server doesn't have MS Studio on it, nor a vid card that will support the client...)
#14
I ONLY get the crash when the server is dedicated, and only in Release build. And of course, this is my standard Alpha test set up for my team & BETA test setup for the BETA testers... Debug build does not have a problem.
04/04/2002 (4:51 pm)
OK Mark, here's the scoop:I ONLY get the crash when the server is dedicated, and only in Release build. And of course, this is my standard Alpha test set up for my team & BETA test setup for the BETA testers... Debug build does not have a problem.
#15
Crash is 100% reproducible on the server when the server is in dedicated mode, release-build (only).
04/05/2002 (4:57 pm)
Bump for Mark.Crash is 100% reproducible on the server when the server is in dedicated mode, release-build (only).
#16
Trying to debug further.
d
04/05/2002 (9:11 pm)
FYI - new code looks like it breaks the Mac build completely at the moment. And that's in a single-player mission start...Trying to debug further.
d
#17
04/06/2002 (8:54 am)
Mark left yesterday afternoon for Portland, he probably look at this until Monday.
#18
Maybe the Win memory handling is doing memclears and the Mac isn't? I don't recall this, but anything is possible... ;)
d
04/06/2002 (12:59 pm)
I added a fix to netconnection to pre-initialize the download member vars to null/zero. That at least temp fixed the crash, though I'm getting notify callbacks when I would assume I wouldn't get any (since 'current net download file' ptr is always null).Maybe the Win memory handling is doing memclears and the Mac isn't? I don't recall this, but anything is possible... ;)
d
#19
Did you get the change David checked in? And does it still crash your dedicated server? I'll run an optimized build tomorrow at the office and see if I can repro it there.
04/09/2002 (5:21 pm)
Hey Ken,Did you get the change David checked in? And does it still crash your dedicated server? I'll run an optimized build tomorrow at the office and see if I can repro it there.
#20
04/09/2002 (5:55 pm)
Yes Mark, it now works in dedicated mode, release-build with David's new check-in.
Torque 3D Owner Robert Blanchet Jr.
My only real concern here is the load that this puts on the server. Will the client recieve the data at the same rate as it would normally if it had all the files?
I guess what I'm trying to say here is I don't want an individual who doesnt have some maps and textures to hog all the bandwidth from the server. Is there some kind of 'cap' in place to prevent this?
Also another question I have about this is, will there be a dialog of some sorts so the user knows they are downloading some stuff from the server that they do not have?
Thanks for the work! Greatly appreciated.