Sending Large Strings Via commandToClient/Server

#1 Use TCPObject? Torque's networking is geared towards sending lots of little bits of info, because that's what you need for a fast-action networked game. TCPObject gives you a stream you can write data cross without regard for length.

You can also write a modified version of command to client that breaks long strings up until multiple events and reassembles on the other side. There might be a resource like this already.

#2 Can you do a client to server TCPObject (I was thinking net services applications only)? I guess there shouldn't be a problem, I'll take a look at it.

As to a resource, I've searched and searched but haven't come up with anything. Of course, I may be asking the wrong question to the search engine. If you have a hunch as to what and/or where it is, I'd appreciate it.

Thanks for the help!

Edit: Uff! Just looked at TCPObject. Nothing there outside my scope, but there would be a lot of it for something simple. If you can't dig up that resource I might have to do that one myself to make commandToX work with long strings.

#3 I would suggest taking a step back to think about a way to avoid sending so much data from the server to the client. for example instead of sending down the long description and path to the thumbnail of the object, bake that stuff into the client and then just send the shortName of the object.

#4 Hey Orion,

I would if I could. My game makes heavy use of plugins and dynamic data, and when I get to multiplayer, there's just no way to avoid the need to send a lot of data to the client. Even if I managed to get the needed data down to a single byte, I'd still be limited to 254 objects on my map, which is not nearly enough. Because the positions and visibility of those objects can change at a moment's notice and the only definitive set of data is on the server because of that, I have to send a lot of data. For these functions, it's if they slow down the network a bit, because you can't mix the fast-action parts of the game and map viewing.

Thinking about it, I've come up with two options I'm going to look at:
1. I make a brand-new longCommandToClient/Server, and new functions into Bitstream for writeLongString and readLongString. Because of the way readString returns data, I can't just modify writeString to have a variant for long strings without MAJOR surgery to the engine.

2. Restructure the way I send the data, so instead of individual records, I send individual commandToClient's, each with a single record. This is almost in line with the avoiding sending so much data thought, IMHO.

#5 Heya, if you really want to use commandToClient to send all this data,
just go with approach #2: just use a bunch of individual commandToClients() with smaller amounts of data in each. the overhead for a commandToClient() is fairly minimal.

note it complicates the receiving end a bit, as you'll probably want to signal the client when it's gotten all the packets. also note that if the amount of data is truly large, you may experience a server-chug when the data is sent. TGE is single-threaded, so if you're spending a significant amount of time in script sending commands to one client, you aren't servicing other clients, ticking the simulation, etc. if you find that happening, you could do something like only sending so many C2Cs per second.

but really it sounds to me like you're trying to use commandToClient for something it wasn't meant for.

a couple ideas:

* data like long description and path to the asset is probably static with respect to the item name itself,
so even if you can't pre-bake it into the client, you probably don't need to send that down with every C2C.
eg, you could send down the dictionary of items during mission load, and then your C2Cs would only have the item name. another option is if you could get that data hosted by an HTTP server, you could use the HTTP Object to get it to the clients. HTTP is really better for transferring large amounts of data.
If you have a master server, perhaps the master server could help with this.

* you might be able to dynamically create datablocks on the server during server-side mission load to describe all this info, which would then be transmitted to each client when it connects. these would need to be custom datablocks you create, with a custom packData() function of course.

> For these functions, it's [okay] if they slow down the network a bit, because you can't mix the fast-action parts of the game and map viewing.

(i'm presuming you meant to include the word "okay").
if all the clients are either viewing the map or doing fast action, then that's true,
but if some clients may be viewing the map while some are doing fast action,
the fast action folks might experience some chugging, as i mentioned above.

#6 Can this data be sent at mission download time? If so I recommend looking at how the mission download works in torque and integrating your own code with it.

#7 The key thing here is that you can't send large buffers because Torque's networking breaks everything up into little pieces for greater responsiveness. Good fit for gameplay, bad fit for this situation. So cutting it up into small pieces is necessary regardless of when you send it.

Unless you send it over another transport, like TCP or HTTP. I think that would be the best solution TBH as it would require little additional C++ code (if any).

#8 Thanks guys. For what it's worth, I'm going to probably end up using a TCPObject as well as multiple commandToClients. I've already converted one instance of this over to the multiple command method, and it works just fine; wasn't that hard to do, even. I had to add a few different commands, one to reset the GUI, one to send an item, and one to signal that its good to go.

Orion, I am working a bit with tagged strings for things like the file path, the problem is that the same item on the map may have a different picture at any given time. Almost all of the text in the game is generated via a script that is eval'd during play, an artifact of the structure of the game versus the historic structure we are aping. The game covers 80 years of time, and the flavor text changes significantly during that time, as well as in response to in-game events and the attributes of the player; much of the text can and will actually change from moment to moment, which makes datablocks a bad fit for what we have going on. For stuff like individual map objects, the 255 limit is not a major problem, it would have to be quite the individual item in order to hit that.

But there are still cases where I'll need to send a single string that could be well in excess of 255 characters- dynamically generated flavor text, for instance, and for those cases I'm looking at a TCPObject, or a function that will break the text into 255-character chunks, stuff them into separate parameters of CommandTo and reconstruct the string from the params at the end, which could get a max string size of 20 * 255 = 5100 characters give or take. Ben, after sleeping on it, I think your first call was probably the best one, I just hate having to make a dedicated channel with a TCPObject just to send potentially long strings, but if that's what it takes that's what it takes. Poking around and looking at BitStream, it would be some major surgery to change the way it works at this point.

Oh, and Orion, yes, I did mean to include OK. Good catch! But you bring up a particularly good point about Multi. We haven't decided yet if the main player going into the map is going to do the same for the other players- our multi makes the other players quite secondary to the main/local player. So I'll have to definitely check the multiplayer responsiveness for a potential server chug and, as you said, use schedule to break up the commands over a second or so.

All in all, though, I'll have to put this on my list of resources to do after I get this game closer to completion- some convenient way to send large data strings without a datablock.

#9 If you're doing a resource for large string chunks, you should try including Zlib compression. Decompression is basically instant on most hardware.

#10 Note there's already Huffman compression for text strings in TGE. eg writeHuffBuffer(), readHuffBuffer().

#11 Does Huffman compression work for long strings, or is it an artifact of the 255 character limit?

#12 Good question.
huffman encoding in general is appropriate for arbitrary-length strings;
the particular implementation in TGE may be different.

looking at the code, it does seem to me that it's assuming length < 256, so it would need some retooling:

bool HuffmanProcessor::writeHuffBuffer(BitStream* pStream, const char* out_pBuffer, S32 maxLen)
{
   ...
   S32 len = out_pBuffer ? dStrlen(out_pBuffer) : 0;
   ...
      pStream->writeInt(len, 8);
   ...
}

edit: the code snippets above show that it's intended for strings < 256 because len = the length of the input string, and then len is being sent across the wire using only 8 bits, so the maximum transmissible length is 255.

#13 That was my read on it as well, thanks for the clarification. Huffman's a bit beyond my ken, so I'll probably look at Zlib when the time comes, but who knows when that will be?

#14 Ok, this is in the realm of "re-think your method" so take it or leave it. I'm guessing you have some sort of object model on your Server that tracks your game objects and makes decisions on behaviour and such. It looks like you have some server-side function (or set of functions) that looks at your game objects and decides what GUI elements to send to the client. You then construct your long string of info and send that to the client for GUI objects to be created.

Instead of that, what if you moved which data was actually being sent, and where the GUI "decision" was being made. Instead of sending the raw GUI information, why not send the actual game objects required for making those GUI decisions? Then, the client (upon receiving the updated game object info) uses that to decide what GUI elements to render and how to render them. This (in my mind) more fits in with the Torque model: send game info to the client, and the client decides how to display that info to the user.

This restructuring would require you to create an actual C++ engine class for all object types that are used in your game's object model, and then writing the BitStream network code to transmit them to the necessary Clients, as well as develop a scoping rule for those objects. However, this seems like more of a "right" way to do things, and is more in keeping with a Model-View-Controller style of thinking.

#15 Hmm... Would that help with the string transfers, though? It seems like that might indeed fit better with Torque's model and simplify other issues, but not this particular one. Then again hard to say without more details.

#16 Mark,

You're a thinking man, obviously. That is essentially what I'm doing- deciding what should be displayed on the client on the server and sending that down the line. I had considered doing exactly what you've just laid out when I first designed the game's architecture, but my thinking was that I don't really need that info on the Client if it's going to involve that much extra C++ coding, I can just use CommandTo and do what I'm doing now. The benefit of having it visible on the client didn't seem to outweigh the negative of having to write all of those NetObjects.

Unfortunately, that still wouldn't help the current predicament, which is getting a long string over the network, which is still necessary for dynamically generated text. Even with your model (give the client the info), I still would have object fields over 255 characters and be in exactly the same spot. The 255 character limit is part of BitStream::writeString, and it would still end up getting broken up and sent in little bits, and/or break the max packet size, etc. Realisitically, I'd need a TCP stream to get the info over to the client anyway, even if I'm doing it at the NetObject level inside the engine instead of with CommandTo at the console level. This also means that even a Datablock wouldn't be the answer for this, even if it was semi-static text (IE: created dyanmically but before the Datablock download phase).

#17 Ok, if you've considered it and decided against it, that's fine.

I have solved this same problem in the past by abstracting away the "direct" commandToClient calls into a higher function. I pass the entire string to this function and it handles sending it to the client. First, it gives an "initiation" command to the client preparing it to receive the compiled string, giving it a unique ID. Then it kicks into a loop where it breaks the larger string into smaller 255-character parts, sending each of those at a time, along with the "overall string" ID. One the client these pieces are received and stored in a variable using the overall string ID. Once all the "chunks" are sent, the server sends a final command telling the client the string is complete (again with the ID), and which point the Client does its thing.

This system should work even if you are sending multiple large strings to the same client simultaneously, since the string ID is being sent with it.

#18 Charlie, NetConnection::startSendingFile() etc might also make interesting reading.
that's the mechanism which sends whole files form server to client.

#19 Mark, Orion, thanks for the hints. Mark, I just might do that, depending on how well my TCPObject integration goes. By splitting up most of my code to use separate CommandTos, I've cut out the number of instances where I need to pass large strings, and for those I could do a wrapper as you've suggested instead of maintaining an entire separate TCP connection.

Orion, I will definately take a look at that one, I hadn't even heard of it before. It also solves a problem I have on the horizon, that of getting pilot data from A to B in multi.