Come ride the rapids

DISCLAIMER: This is a personal project undertaken by me, David Wyand. It does not represent work being performed by GarageGames, nor is it an indication of any future product releases.

Whether you believe in Web-two-point-oh or Web-it-blows, there's no denying that something is afoot. It seems that everyone and their dog, and their dog's fleas, have their own video stream. Ustream.tv, stickam.com, justin.tv, blogtv.ca are all infesting the Internet with images of people staring into space, sleeping, and performing faux stripteases. Video streaming for the rest of us. Well it's time to bring Torque into this space!

Row, row, row your boat...
What I've been playing with recently is integrating a DirectShow filter with TGE. This turns Torque's rendering pipeline into a DirectShow streaming source that may be fed into anything that accepts such a connection. And as you'll find out later, this allows you to do a lot of interesting things. This is accomplished through a custom built DirectShow library and a slightly modified TGE 1.5.



The DirectShow capture source filter I've built is presented to the operating system as any other video source. Other video source examples include a web camera or a TV tuner card. So in theory any application that accepts a web cam as a source would be able to connect to my filter's output pin. The filter currently supports the following formats:

* 32-bit RGB (RGBA with an opaque alpha channel)
* 24-bit RGB
* 16-bit YUY2 (4:2:2)
* 16-bit UYVY (4:2:2)

These should allow the filter to be accepted by nearly any DirectShow application. The only missing format -- 4:2:0 -- could be added if a need was found. The filter supports any push frame rate your computer may handle (regardless of Torque's frame rate) and any resolution. Although to access these features I'll need to create a GUI rather than modify a variable and recompile. :o)

To pass data between the filter and Torque, an externally accessible video buffer is provided. To transfer the data, the DirectShow filter is imported (it's a DLL) into Torque and the lockable video buffer is in shared memory suitable for a memory copy. More handshaking between the filter and Torque could be added in the future to allow for finer control of the filter from within Torque.

Life is but a dream
All of the work on the TGE side is performed by a custom canvas: GuiVideoCaptureCanvas. The neat thing about this is all you need to do is swap your regular canvas with this one (usually in a TorqueScript call) and you now have video streaming. Yes, really. The steps to streamdom are:

1. Drop in GuiVideoCaptureCanvas and recompile.
2. Modify statement in TorqueScript to use the new canvas.
3. Profit!

Now while we could stop there, where's the fun in that? We want more features! Features until you're mashing the keyboard with bloody, sinewy stumps!

Well the first feature the new canvas gives you is being able to build an overlay GUI that is only visible from the video stream.



In the above image I've added some text and a time stamp. This could be anything really as it makes full use of the Torque GUI. One idea I came up with is to route the chat window to a text GUI to make it readable from the video stream. You could also display team scores or the current leader. As I said... anything.

The second feature is you may build an overlay GUI that is only visible to the player.



As an example I've built a dialog that provides a preview on the video stream (another custom GUI control) and a couple of controls. This could be used to change the stream's settings, to pause it, etc. Here the Torque mouse pointer is automatically shown when the overlay is displayed if it wasn't previously, but it only renders for the player and not in the stream. If the mouse pointer is normally visible -- such as the game's main menu -- then it will also be visible within the stream.

There is some sneakiness required to make it all work. The following diagram provides a general outline of the canvas' rendering pipeline.




I'm making use of OpenGL to resize the rendered display to fit the video stream's resolution (top box in the diagram, right column). I've found it best to enable some form of antialiasing in the video card's driver to provide the best image possible. This could be done automatically from within Torque but that would require some more involved code changes.

Which stream will you take?
This is really cool and all, but what are its practical uses? Excellent question. Here are some of my thoughts. I suspect that others can come up with even better ideas.

Direct video capture
Using a program that can capture to disk a DirectShow stream, it's possible to directly record your Torque session. No more screen captures with FRAPS or Camtasia Studio. Just pure memory dumping goodness.

I've had great success with VirtualDub. Its a completely free, DirectShow capture and AVI editing program with some great stream capturing features. As an example the following video was captured using VirtualDub on my Inspiron 9300 laptop with its single core, non-hyperthreading processor. It was initially captured at 320x240 and 30fps using the RGB 24-bit format as an uncompressed AVI. I then used VirtualDub to trim the start and end, and saved it as a Divx AVI file with default settings.

'When Orcs Attack' Sample Video (12MB)

Play session streaming to the Internet
My DirectShow capture source filter may be used with the Adobe Flash player to stream your Torque play session over the Internet. As far as the Flash player is concerned it is just another web cam. Let the whole world see your Orc fragging skills. Me bash ork gud.

I've had great success with Ustream.tv and its more business oriented approach, rather than the sites devoted to social networking. They also allow you to record your streams and store them on their servers. As an example I've recorded the following live stream through the Flash player at 12fps and a quality of 80.

Recorded Live Orc Cam Stream Web Page

This could be used for live tournament play, live in-game guild presentations and training, live Machinima, etc. This is where being able to control a 3rd person camera in game can be quite handy. I'm sure others can come up with more novel ideas. :o)

Developer controlled Internet streaming
This is an area that I believe has gone untapped. Rather than having the players stream their sessions (or in addition to it), the game developer could have machines dedicated to the task. This would probably be most useful in a persistent game or at least one where the game developer has some control over the game's servers, but needn't be limited to it.

This technology could be used to market the game by showing what is happening right now at a particular location, for example. Or to highlight a current in-game event with a 'wish you were here' sentiment. You could have a number of camera locations that are cycled between, or have a number of individual video streams.

I could envision in a persistent world like vSide you could make the video streaming camera an actual object. Have the avatars share an in-game video camera and stream the video to your front page using Ustream's embedded player.


vSide streaming mock up for illustrative purposes

The original reason I started this project was to have a web cam into a virtual world that ran itself. It's something I've toyed with in one incarnation or another for quite some time. I still hope to one day.

My boat sprung a leak!
So what's left to do before I consider this code complete:

1. Add audio support. I haven't looked into this yet so it could end up being a deal breaker if it doesn't work out.

2. Set encoding parameters from a GUI and/or Torque. This includes resolution, frames per second, format, etc.

3. Enable/disable streaming from within Torque.

4. Display a user defined image when not streaming from Torque.

5. Look into OSX support under Quicktime.

Does this float your boat?
So does something like this interest people? What's your take on technology such as this? Would you be willing to purchase a code pack to put this into your game or simulation?

Thanks for reading!

- LightWave Dave