Torque networking, applied to VOIP
by Anthony Lovell · in Torque Game Engine · 02/04/2004 (9:18 am) · 92 replies
I have made some solid progress at adding VOIP into Torque (I can sample the microphone and compress it using Speex... code should work on Mac and Windows, and I could explore Unix). I intend to eventually share my work with the community when I have it ready, but now I need to learn much more about Torque -- primarily, its networking gestalt.
For my first implementation, I would like to just use Torque's unreliable UDP platformNet to send streaming voice packets through the server to only those other clients who are near enough to hear the speaker (a simple distance limit). Then, on each of those clients, I'd like them to hear the sound emanate from the position of the speaking player.
I had been thinking that my means of coding this was to be a direct use of Net::sendto() but another idea occurs to me:
Is the proper way to do this to add the mouth of the Player as part of the data state that the network layer is supposed to disseminate to other clients, much like his orientation in the world? In such a model, a non-talking player would have no state to transmit to the net, but when my microphone layer hands me some compressed audio, I'd hand it to my local Player instance's mouth, and trust that the neat differencing logic could then realize that it needs to go out to the net. If this is the approach, would I have to take special measures to ensure you could hear people behind you (as they are out of camera scope -- I take it this is different than setting scopeAlways, as the distance is still pertinent)?
Also, on the receive side, would the key be to add an AudioEmitter (or a subclass of one) to the mouth of the speaker? Is there any existing high-level support for audio which is provided on a frame-by-frame basis, and not from a soundfile resource?
Thanks in advance for any counsel you can provide.
tone
For my first implementation, I would like to just use Torque's unreliable UDP platformNet to send streaming voice packets through the server to only those other clients who are near enough to hear the speaker (a simple distance limit). Then, on each of those clients, I'd like them to hear the sound emanate from the position of the speaking player.
I had been thinking that my means of coding this was to be a direct use of Net::sendto() but another idea occurs to me:
Is the proper way to do this to add the mouth of the Player as part of the data state that the network layer is supposed to disseminate to other clients, much like his orientation in the world? In such a model, a non-talking player would have no state to transmit to the net, but when my microphone layer hands me some compressed audio, I'd hand it to my local Player instance's mouth, and trust that the neat differencing logic could then realize that it needs to go out to the net. If this is the approach, would I have to take special measures to ensure you could hear people behind you (as they are out of camera scope -- I take it this is different than setting scopeAlways, as the distance is still pertinent)?
Also, on the receive side, would the key be to add an AudioEmitter (or a subclass of one) to the mouth of the speaker? Is there any existing high-level support for audio which is provided on a frame-by-frame basis, and not from a soundfile resource?
Thanks in advance for any counsel you can provide.
tone
#62
Sounds like a great project
10/30/2005 (10:39 pm)
Good to hear it is working and being worked on. I really wish I had the time to help out on this one as it sounds like a really interesting project.Sounds like a great project
#63
This is the kind of thing that I've been hoping for in Torque for quite a while. I wish had the skills to help out, but I'm just starting to learn the coding side of Torque. Anyway, thank you for everything you've done with this.
11/01/2005 (12:47 am)
@AnthonyThis is the kind of thing that I've been hoping for in Torque for quite a while. I wish had the skills to help out, but I'm just starting to learn the coding side of Torque. Anyway, thank you for everything you've done with this.
#64
I have Mac and PC working together nicely in "mouth to ear" mode wherein speaking avatars are heard by those others who are near them in 3D. I have a substantial groundwork laid for game-style channels (game, red team, blue team, etc), but I have not gotten that working just yet.
I'd be very surprised if the TGE community did not get a version of this made available (on some terms, and I'm not amenable to hiding this under a barrel or reaming people) in the early part of 2006.
Here is a rough overview (no promises on features, however... just an update)
A basic game's use of this would have the following script loaded on the client as the client initializes:
A few small changes to the options panel and its scripts places a new "Voice" tab which permits input device selection, a simple VU meter and a slider to adjust a software input gain on it (if someone could help me figure out how to drive hardware input levels for all TGE platforms, I'd be willing to listen!) A single key binding is added to support a "push to talk" button.
In the place in the client-side script where a client's connection is accepted, network event handlers are installed on his connection to the server so that the proper course can be taken with the VOIP data as it arrives. In this way, the code is well modularized (I think, anyhow):
That's basically it, aside from some boilerplate script the coders need not look at but simply hide away (it supports things like the VU meter and options dialog, suspends input handlers when the options dialog is open, etc).
The other, optional elements of scripting shake out like this:
The fun part (in my view) is that VOIP is simply the first feature I hope to make available. There is more to come having to do with voice in games, and I think the way I will present it will place TGE users on the fore-front for making games with the lowest possible barriers to fulfilling the unmet social promise of games. I look forward to getting people's perspectives on how to do that "the TGE way" as I go.
Have a good 2006!
tone
12/31/2005 (2:02 pm)
Well, despite your every reasonable expectation that this would just prove to be another initiative that you read about and then never hear of again, I have to report some solid success.I have Mac and PC working together nicely in "mouth to ear" mode wherein speaking avatars are heard by those others who are near them in 3D. I have a substantial groundwork laid for game-style channels (game, red team, blue team, etc), but I have not gotten that working just yet.
I'd be very surprised if the TGE community did not get a version of this made available (on some terms, and I'm not amenable to hiding this under a barrel or reaming people) in the early part of 2006.
Here is a rough overview (no promises on features, however... just an update)
A basic game's use of this would have the following script loaded on the client as the client initializes:
// create a global script object to control voice input and device selection, etc
$Vox = new Vox();
// Make sure the Vox audio input system initialized.
if ($Vox::systemFailure !$= "") {
MessageBoxOK("Vox Initialization Failed",
"The Vox system failed to initialize properly:" SPC $Vox::systemFailure @ ".");
}
// add voice-over-IP encoding processor to the speech input system
// this handles ONLY the "send" side of the system (take mike input, encode it, send it to server)
// it starts off disabled... we'll enable it when the player enters the game (or is given an avatar)
//
$VOIPin = new VOIPInputHandler();
$Vox.addInputHandler($VOIPin);A few small changes to the options panel and its scripts places a new "Voice" tab which permits input device selection, a simple VU meter and a slider to adjust a software input gain on it (if someone could help me figure out how to drive hardware input levels for all TGE platforms, I'd be willing to listen!) A single key binding is added to support a "push to talk" button.
In the place in the client-side script where a client's connection is accepted, network event handlers are installed on his connection to the server so that the proper course can be taken with the VOIP data as it arrives. In this way, the code is well modularized (I think, anyhow):
// add a handler to decode and play VOIP // data that arrives over the net $VOIPnet = new VOIPNetworkHandler(); $Vox.getConnection().addAudioNetEventHandler($VOIPnet);
That's basically it, aside from some boilerplate script the coders need not look at but simply hide away (it supports things like the VU meter and options dialog, suspends input handlers when the options dialog is open, etc).
The other, optional elements of scripting shake out like this:
bool Vox::startSampling(); // e.g.: push to talk
bool Vox::stopSampling(); // e.g.: key up on push to talk
S32 Vox::getNumInputDevices(); // e.g.: "Line in", "Direct Sound Input"
S32 Vox::getInputDevice(index);
F32 Vox::getInputGain(); // // 1.0 is norm -- a software multiplier at the moment
void Vox::setInputGain(F32);
S32 Vox::getNumInputHandlers(); // e.g.: VOIPInputHandler, others in future
VoxInputHandler Vox::getInputHandler(index); // VOIPInputHandler is an example
void Vox::enableInputHandlers(bool); // all installed ones, at once
void Vox::addInputHandler(VoxInputHandler);
void Vox::removeInputHandler(VoxInputHandler);
U32 Vox::getInputDecibels(); // audio propagation on server will be dependent on this. default is 70, max is 255
void Vox::setInputDecibels(U32);
bool VoxInputHandler::isEnabled(); // a superclass to VOIPInputHandler
void VoxInputHandler::enable();
void VOIPInputHandler::setOutboundPacketInterval(F32); // in seconds (in 20ms increments, minimum of 40 msecs)
F32 VOIPInputHandler::getOutboundPacketInterval();
U32 VOIPInputHandler::getEncodeQuality();
void VOIPInputHandler::setEncodeQuality(U32); // 0->10, default is 2 (7750 baud). Bitrate depends on this
// 0 == 3.95 kbps
// 1 == 5.75 kbps
// 2 == 7.75 kbps
// 4 == 12.80 kbps
// 6 == 20.60 kbps
//10 == 42.20 kbpsThe fun part (in my view) is that VOIP is simply the first feature I hope to make available. There is more to come having to do with voice in games, and I think the way I will present it will place TGE users on the fore-front for making games with the lowest possible barriers to fulfilling the unmet social promise of games. I look forward to getting people's perspectives on how to do that "the TGE way" as I go.
Have a good 2006!
tone
#66
Just to let you(or anyone else) know, OpenAL 1.1 now has a nice interface for capturing audio.
Essentially, it goes like this :
I hope that helps you :)
01/01/2006 (3:33 am)
Anthony,Just to let you(or anyone else) know, OpenAL 1.1 now has a nice interface for capturing audio.
Essentially, it goes like this :
// Initializing device
// (you can get the name(s) via enumerating the device list)
captureDevice = alcCaptureOpenDevice(name, frequency, format, buffersize);
if (alcGetError(captureDevice) != AL_NO_ERROR) {
Con::errorf("Unable to initialize capture device '%s'!", name);
return false;
}
// (later) Starting capture
alcCaptureStart(captureDevice);
if (alcGetError(captureDevice) == AL_NO_ERROR) {
capturing = true;
// Execute console commands, etc...
return true;
}
// (later) Checking if capture buffer full
ALint capturedSamples = 0;
talcGetIntegerv(captureDevice, ALC_CAPTURE_SAMPLES, 1, &capturedSamples);
if (capturedSamples >= max_captured_samples) {
// Full, so extract
U8 *buffer = new U8[capturedSamples * sample_size];
alcCaptureSamples(captureDevice, buffer, capturedSamples);
// Dispatch to net?
}
// (then) Stopping the capture
alcCaptureStop(captureDevice);
alcGetError(captureDevice);
// (finally) Closing the device
alcCaptureCloseDevice(captureDevice);I hope that helps you :)
#67
PortAudio is working semi-nicely, though I notice some defects (such as I cannot close down devices I open with it).
tone
01/02/2006 (7:23 am)
Oh, heck. I wonder if that is better than what I am using. No use rushing to switch over to it, but gee it would have been nice to know of it in advance.PortAudio is working semi-nicely, though I notice some defects (such as I cannot close down devices I open with it).
tone
#68
01/02/2006 (8:14 am)
What about software like 'Roger Wilco' or 'Teamspeak' in a game? Has anyone seen if the voice companies have SDKs for putting their voice chat software within the game itself?
#69
If I remember correctly, Anthony was the original creator of Roger Wilco. I think it's pretty cool that he's been working on voice for Torque. I'm guessing that getting VOIP in the OpenAL sound layer has been a challenge for him, though.
In addition to Anthony, it sounds like James and some other really bright folks are looking into this as well. Another guy (can't remember his name) is even trying to implement a better alternative to OpenAL. All of that is great news for the entire Torque community.
Aaron E.
01/02/2006 (8:43 am)
Jonathon,If I remember correctly, Anthony was the original creator of Roger Wilco. I think it's pretty cool that he's been working on voice for Torque. I'm guessing that getting VOIP in the OpenAL sound layer has been a challenge for him, though.
In addition to Anthony, it sounds like James and some other really bright folks are looking into this as well. Another guy (can't remember his name) is even trying to implement a better alternative to OpenAL. All of that is great news for the entire Torque community.
Aaron E.
#70
I'd cite these reasons
1. In the general case, outside tools are owned by some, but not all players. When it is built into the game, the external demands upon the players is reduced to the bare minimum: do you have a working microphone?
2. Outside tools rely on a hosting model separate from the game's. In the best case, everyone agrees on this host and has access to it and simply does more work to get voice going (they have to join two servers). In the worst case, people cannot agree on the server to be used and you get fractionation at the squads/clan level. but even in the best case, few relish the chance to enter the game and immediately be told to alt+tab out, start up the external voice app, and then manually enter some server and channel settings they struggle to remember accurately as they start this needless dance.
3. Building into the game permits game designers to offer the KIND of voice connectivity that seems right. It becomes INTIMATE rather than one size fits all. Examples include
a. the ability to model mouth-to-ear conversations where the tuning model is as natural as moving near those you want to speak to (this is important to my own game concepts)
b. having voice chat channels that automatically mimic affiliations in the game (e.g.: selecting a team in the game automatically puts you on that team's chat channel on voice)
c. ability to deliver realistic communication devices that are directly a part of the game, e.g.: radios subject to distance and atmospheric interference, telephones that must call each other, PA systems, etc. (this is also something very important to my game design ambitions)
I actually created Roger Wilco, and never really saw the external API used by any developers other than those who wanted to create a better interface than our simple application itself offered. Those games that bundled RW (e.g.: Baldur's Gate: Tales of the Sword Coast, or Flanker 2.0) never did any integration work despite the fact that the total effort required was approximately an hour of coding.
I hope that some fairly simple voice code in places like TGE will actually succeed in delivering better and more widespread voice in games than even successes like RW and TS have achieved. For the Indie developer, the advantages listed above might offer a means of leaping ahead of what "the big boys" offer in the way of voice.
tone
01/02/2006 (8:45 am)
Roger Wilco had a limited SDK and TS may have one, too, but this is not a substitute for a game having voice built in.I'd cite these reasons
1. In the general case, outside tools are owned by some, but not all players. When it is built into the game, the external demands upon the players is reduced to the bare minimum: do you have a working microphone?
2. Outside tools rely on a hosting model separate from the game's. In the best case, everyone agrees on this host and has access to it and simply does more work to get voice going (they have to join two servers). In the worst case, people cannot agree on the server to be used and you get fractionation at the squads/clan level. but even in the best case, few relish the chance to enter the game and immediately be told to alt+tab out, start up the external voice app, and then manually enter some server and channel settings they struggle to remember accurately as they start this needless dance.
3. Building into the game permits game designers to offer the KIND of voice connectivity that seems right. It becomes INTIMATE rather than one size fits all. Examples include
a. the ability to model mouth-to-ear conversations where the tuning model is as natural as moving near those you want to speak to (this is important to my own game concepts)
b. having voice chat channels that automatically mimic affiliations in the game (e.g.: selecting a team in the game automatically puts you on that team's chat channel on voice)
c. ability to deliver realistic communication devices that are directly a part of the game, e.g.: radios subject to distance and atmospheric interference, telephones that must call each other, PA systems, etc. (this is also something very important to my game design ambitions)
I actually created Roger Wilco, and never really saw the external API used by any developers other than those who wanted to create a better interface than our simple application itself offered. Those games that bundled RW (e.g.: Baldur's Gate: Tales of the Sword Coast, or Flanker 2.0) never did any integration work despite the fact that the total effort required was approximately an hour of coding.
I hope that some fairly simple voice code in places like TGE will actually succeed in delivering better and more widespread voice in games than even successes like RW and TS have achieved. For the Indie developer, the advantages listed above might offer a means of leaping ahead of what "the big boys" offer in the way of voice.
tone
#71
01/02/2006 (4:28 pm)
@Anthony: Wow, good job on RW! I didn't mean use RW/TS as an alternative, I meant actually code it into the game itself. I defenately would want all the extra benefits you describe above, just didn't know if you could implement a RW SDK into the actual game itself to where the integration was seemless, kind of like many games have done with gamespy integration. 
#72
Great work, I'll be watching!
Just a note on bandwidth.
I run a WISP, but I live too far away to get my own service (go figure).
I use VoIP (TS/Vent/etc) while using VNC and other remote desktop apps on a 28.8 modem (farm line) daily.
I couldn't live without it.
Compression can do wonderous things, but you know that. ;)
Ari
01/03/2006 (7:06 am)
@AnthonyGreat work, I'll be watching!
Just a note on bandwidth.
I run a WISP, but I live too far away to get my own service (go figure).
I use VoIP (TS/Vent/etc) while using VNC and other remote desktop apps on a 28.8 modem (farm line) daily.
I couldn't live without it.
Compression can do wonderous things, but you know that. ;)
Ari
#73
I would agree with most of what you say: having properly integrated voice is definitely the best option.
However, having integrated the teamspeak SDK with Torque myself, I wouldn;t agree that "In the best case, everyone agrees on this host and has access to it and simply does more work to get voice going (they have to join two servers)." -- the SDK allows TGE to launch the teamspeak client and point it to the correct URL at which the teamspeak server is running. Therefore it's no extra work for the players. As you pointed out, lip synching, 3D audio etc. aren't supported however...
01/03/2006 (7:17 am)
Anthony,I would agree with most of what you say: having properly integrated voice is definitely the best option.
However, having integrated the teamspeak SDK with Torque myself, I wouldn;t agree that "In the best case, everyone agrees on this host and has access to it and simply does more work to get voice going (they have to join two servers)." -- the SDK allows TGE to launch the teamspeak client and point it to the correct URL at which the teamspeak server is running. Therefore it's no extra work for the players. As you pointed out, lip synching, 3D audio etc. aren't supported however...
#74
tone
01/03/2006 (7:35 am)
You're right. I started riffing on what I disliked about the external tools themselves (when not integrated) as compared to a good and true integration.tone
#75
01/03/2006 (9:15 am)
@Sam: Did you or would you be willing to put up a tutorial on how to get TS SDK integrated into TGE?
#76
Reviewed the AudioEmitter class and am ready to continue making the one interface for VoIP and normal audio...
As far as PortAudio vs. OpenAL... the VoIP side should have an abstracted interface allowing it to be quickly used with any audio engine.
write me at sumdj@pcdj.com (not my hotmail) so we can coordinate coding
Cheers and happy new year!
:)
Sumner
01/04/2006 (5:47 am)
Greetings, Anthony...Reviewed the AudioEmitter class and am ready to continue making the one interface for VoIP and normal audio...
As far as PortAudio vs. OpenAL... the VoIP side should have an abstracted interface allowing it to be quickly used with any audio engine.
write me at sumdj@pcdj.com (not my hotmail) so we can coordinate coding
Cheers and happy new year!
:)
Sumner
#77
One of my TGE projects will use the VOIP feature, just wondering if you mind release a EA version now?
BTW I only need it on the windows platform.
Steven
03/12/2006 (9:51 am)
Anthony,One of my TGE projects will use the VOIP feature, just wondering if you mind release a EA version now?
BTW I only need it on the windows platform.
Steven
#78
one of my projects needs voip integrated into torque by the end of May, would you mind sharing some code please?
warvstar@gmail.com
05/05/2006 (6:13 pm)
Anthony,one of my projects needs voip integrated into torque by the end of May, would you mind sharing some code please?
warvstar@gmail.com
#79
07/05/2006 (8:13 am)
@Anthony - Any update on your progress with this?
#80
I'd hand this to someone who would promise to deliver it back to me in a usable form for TGE in return for a perpetual personal use license and a lot of credit, but not otherwise as I cannot spare any cycles.
I regret that I raised so many expectations.
tone
07/05/2006 (9:02 am)
I'm sorry to say I am no longer using TGE for a variety of reasons.I'd hand this to someone who would promise to deliver it back to me in a usable form for TGE in return for a perpetual personal use license and a lot of credit, but not otherwise as I cannot spare any cycles.
I regret that I raised so many expectations.
tone
Torque Owner Anthony Lovell
Ben has promised some help, but seems divided for his time (who woulda thought!). If anyone is committed to helping me nail this, contact me.
tone