Game Development Community

Multitouch for Objects in iTGB v1.2

by Dave Calabrese · 05/13/2009 (7:12 pm) · 20 comments

EDIT: Fixed some missing / incorrect information.

You find yourself wishing to touch your iPhone or iPod Touch screen more than once - but try as you must, you simply cannot. A second touch will nullify the first. You feel alone - but you are not. You have officially entered....

THE TORQUE ZONE

*dun dun duuuun*


THE PROBLEM:
iTGB has multitouch capabilities built into it. These actions get passed through the mouse input managers - which is a good thing, as it allows them to make use of all the existing code for mouse callbacks. However, they do not work well. First off, true multitouch actions are getting passed back to the script level - which is okay, however there isn't much you can do with this. Secondly, if you touch a second location on the screen after touching a first, it will nullify your original touch.

So how to fix this? We're going to integrate a full 'ScreenTouch' input manager. With this it will give you the ability to add in the control and functionality you need. There are a lot of changes involved here, so you might want to get a barrel of coffee or tea or something. (Or, if you're a solid game programmer like myself - BEER! Or, Mead, which is my preference.)

Let's get started. First off, make sure you are running iTGB v1.2 - this fix is not guaranteed to work in versions before or after it.

Since there are a lot of files, I am going to list a file name, then the change needed rather than describing it all in detail. I am going to post the area of code it is in. Look for "//DMC - Added for Multitouch". I will also list the line number to add the code near. Here we go!!

game/demoGame.h
Around line 33
public:
   void textureKill();
   void textureResurrect();
   void refreshWindow();

   //virtual int  main(int argc, const char **argv);
   virtual bool mainInit(int argc, const char **argv);
   virtual void mainLoop();
   virtual void mainShutdown();
      
   void processPacketReceiveEvent(PacketReceiveEvent *event);
   void processMouseMoveEvent(MouseMoveEvent *event);
   void processInputEvent(InputEvent *event);
   void processScreenTouchEvent(ScreenTouchEvent *event); //DMC - Added for multitouch
   void processQuitEvent();
   void processTimeEvent(TimeEvent *event);
   void processConsoleEvent(ConsoleEvent *event);
   void processConnectedAcceptEvent(ConnectedAcceptEvent *event);
   void processConnectedReceiveEvent(ConnectedReceiveEvent *event);
   void processConnectedNotifyEvent(ConnectedNotifyEvent *event);
};

gui/core/guiCanvas.h
Around line 268
/// @param   event   Mouse move event to process
   virtual void processMouseMoveEvent(const MouseMoveEvent *event);
	
	virtual void processScreenTouchEvent(const ScreenTouchEvent *event); //DMC - Added for multitouch

   /// @}

gui/core/guiCanvas.h
Around line 312
virtual void rootMouseMove(const GuiEvent &event);
   virtual void rootMouseDragged(const GuiEvent &event);

   virtual void rootScreenTouchUp(const GuiEvent &event); //DMC - Added for Multitouch
   virtual void rootScreenTouchDown(const GuiEvent &event); //DMC - Added for Multitouch
	
   virtual void rootRightMouseDown(const GuiEvent &event);
   virtual void rootRightMouseUp(const GuiEvent &event);

game/main.cc
Around line 613
/// Process a mouse movement event, essentially pass to the canvas for handling
void DemoGame::processMouseMoveEvent(MouseMoveEvent * mEvent)
{
   if (Canvas)
      Canvas->processMouseMoveEvent(mEvent);
}

//DMC - Added for multitouch
void DemoGame::processScreenTouchEvent(ScreenTouchEvent * mEvent)
{
	if (Canvas)
		Canvas->processScreenTouchEvent(mEvent);
}
//DMC - End added for multitouch

/// Process an input event, pass to canvas for handling
void DemoGame::processInputEvent(InputEvent *event)
{
   PROFILE_START(ProcessInputEvent);

platform/event.h
Around line 50
enum EventTypes
{
   InputEventType,
   MouseMoveEventType,
   ScreenTouchEventType, //DMC - Added for multitouch
   PacketReceiveEventType,
   TimeEventType,
   QuitEventType,

platform/event.h
Around line 373
enum InputDeviceTypes
{
   UnknownDeviceType,
   MouseDeviceType,
   KeyboardDeviceType,
   JoystickDeviceType,
   ScreenTouchDeviceType //DMC - Added for Multitouch
};


platform/event.h
Around line 145
//DMC - Added for multitouch
/// Screen Touch Event
struct ScreenTouchEvent : public Event
{
	S32 xPos, yPos;
	U8    action;
	
	ScreenTouchEvent() { type = ScreenTouchEventType; size = sizeof(ScreenTouchEvent); }
};


platform/gameInterface.h
Around line 59
virtual void processEvent(Event *event);

   virtual void processPacketReceiveEvent(PacketReceiveEvent *event);
   virtual void processMouseMoveEvent(MouseMoveEvent *event); 
   virtual void processScreenTouchEvent(ScreenTouchEvent *event); //DMC - Added for multitouch
   virtual void processInputEvent(InputEvent *event);


platform/gameInterface.cc
Around line 138
//DMC - Added for multitouch
void GameInterface::processScreenTouchEvent(ScreenTouchEvent*)
{
	
}


platform/gameInterface.cc
Around line 93
case InputEventType:
         processInputEvent((InputEvent *) event);
         break;
	  case ScreenTouchEventType: //DMC - Added for Multitouch
		 processScreenTouchEvent((ScreenTouchEvent *) event); //DMC - Added for Multitouch
		 break; //DMC - Added for Multitouch
      case QuitEventType:
         processQuitEvent();
         break;



T2D/t2dSceneWindow.cc
Around line 2340
NOTE: This is an EPIC MEGAHACK to the EXTREME. I should probably rewrite this to use a boolean check in the GuiEvent....
}         
      }

	  #ifndef TORQUE_OS_IPHONE //DMC - Added for multitouch
      if( lastObject && lastObject->isProperlyAdded() && bFoundNew == true && lastObject->getUseMouseEvents() )
         lastObject->onMouseEvent( "onMouseLeave" , event, worldMousePoint ); 
	  #endif //DMC - Added for multitouch
   }

   // We have to dance a bit to get onMouseEnter to work
   for( U32 i = 0; i < pickVector.size(); i++ )


gui/core/guiCanvas.cc
Around line 424
return true;
		}
   }
   return false;
}

//DMC - Added for Multitouch
void GuiCanvas::processScreenTouchEvent(const ScreenTouchEvent *event)
{
		//copy the cursor point into the event
		mLastEvent.mousePoint.x = S32(event->xPos);
		mLastEvent.mousePoint.y = S32(event->yPos);
		
		//see if button was pressed
		if (event->action == SI_MAKE)
		{
			U32 curTime = Platform::getVirtualMilliseconds();
			mNextMouseTime = curTime + mInitialMouseDelay;
			
			mLastMouseDownTime = curTime;
			mLastEvent.mouseClickCount = mLastMouseClickCount;
			
			rootScreenTouchDown(mLastEvent);
		}
		//else button was released
		else
		{
			mNextMouseTime = 0xFFFFFFFF;
			rootScreenTouchUp(mLastEvent);
		}
}

void GuiCanvas::processMouseMoveEvent(const MouseMoveEvent *event)
{
   if( cursorON )
   {


gui/core/guiCanvas.cc
Around line 781
if (bool(mMouseControl))
      mMouseControlClicked = true;
}

//DMC - Added for Multitouch
void GuiCanvas::rootScreenTouchDown(const GuiEvent &event)
{
	mPrevMouseTime = Platform::getVirtualMilliseconds();
	mMouseButtonDown = true;

		iterator i;
		i = end();
		while (i != begin())
		{
			i--;
			GuiControl *ctrl = static_cast<GuiControl *>(*i);
			GuiControl *controlHit = ctrl->findHitControl(event.mousePoint);
			
			//see if the controlHit is a modeless dialog...
			if ((! controlHit->mActive) && (! controlHit->mProfile->mModal))
				continue;
			else
			{
				controlHit->onMouseDown(event);
				break;
			}
		}
	
	if (bool(mMouseControl))
		mMouseControlClicked = true;
}

void GuiCanvas::rootScreenTouchUp(const GuiEvent &event)
{
	mPrevMouseTime = Platform::getVirtualMilliseconds();
	mMouseButtonDown = false;

		iterator i;
		i = end();
		while (i != begin())
		{
			i--;
			GuiControl *ctrl = static_cast<GuiControl *>(*i);
			GuiControl *controlHit = ctrl->findHitControl(event.mousePoint);
			
			//see if the controlHit is a modeless dialog...
			if ((! controlHit->mActive) && (! controlHit->mProfile->mModal))
				continue;
			else
			{
				controlHit->onMouseUp(event);
				break;
			}
	}
}

void GuiCanvas::findMouseControl(const GuiEvent &event)
{
   if(size() == 0)
   {


platformiPhone/iPhoneInput.mm
Change the functions 'createMouseMoveEvent','createMouseDownEvent' and 'createMouseUpEvent' to read like the following:
bool createMouseMoveEvent( S32 touchNumber, S32 x, S32 y ) {
	
	ScreenTouchEvent event;
	event.xPos = x;
	event.yPos = y;
	event.action = SI_MAKE;
	
	TouchMoveEvents.push_back( touchEvent( touchNumber, x, y ) );	
	Game->postEvent(event);
	
	return true;//return false if we get bad values or something
}


bool createMouseDownEvent( S32 touchNumber, S32 x, S32 y ) {
	
	ScreenTouchEvent event;
	event.xPos = x;
	event.yPos = y;
	event.action = SI_MAKE;

	TouchDownEvents.push_back( touchEvent( touchNumber, x, y ) );
	Game->postEvent(event);
	
	return true;//return false if we get bad values or something
}

bool createMouseUpEvent( S32 touchNumber, S32 x, S32 y ) {
	
	ScreenTouchEvent event;
	event.xPos = x;
	event.yPos = y;
	event.action = SI_BREAK;

	TouchUpEvents.push_back( touchEvent( touchNumber, x, y ) );	
	Game->postEvent(event);
	
	return true;//return false if we get bad values or something
}


Phew! Okay... now, in your TorqueScript, all you need to do is setup your code like you would be interacting with things through mouse. So onMouseUp, onMouseDown, onMouseMove, onMouseEnter, onMouseLeave and so forth now all work - with multitouch on your iPhone / iPodTouch.

Huzzah!

#1
05/13/2009 (11:01 pm)
NOTE: I'm finding a lot of situations where this does not work. I'm working on writing a new input system for iTGB that will be used just for 'screenTouch' input. I'll update this resource with that when it's ready.


NOTE NOTE: Okay, this is fixed and functional. Use the resource and enjoy!
#2
05/14/2009 (8:58 am)
Excellent! Now somebody needs to make a pinch-zoom resource to go with it ;)
#3
05/14/2009 (10:34 am)
Cool stuff Dave :)
#4
05/14/2009 (1:22 pm)
Great work Dave, I also found multitouch lacking as-is, been meaning to get to it, but thankfully now I don't' have to :D.
#5
05/14/2009 (4:32 pm)
For anyone who has already tried implementing this, I had a wrong filename in one place and had left out some code. Props go out to Bret Patterson for pointing that out - thanks man! :)

@Ronny: After going through all this, I don't think that would be too difficult actually... unfortunately I'm now behind on 2 projects from spending 3 days banging through this (at least I needed this for them) so I'd better get back to work, lol. Once I have some free time if nobody has done it yet I might do a Pinch/Zoom resource myself.

@Afan and @Luke: Thanks!
#6
05/22/2009 (2:45 am)
This implementation works great for the game I'm designing, but it turned my touches into a dragging-like response. Because the touchscreen is so sensitive, it's as if I'm telling my game to press immediately in the next pixels over. I linked the touch to a sound and it's so rapid-fire that the sound is flanging. Does that make sense in my explanation?

I wanted to use one object as a button (a Shift or modifier key) and touch the screen to place another object, but my game thinks I'm trying to place 20 objects in one area.

Any idea how to fix this? Perhaps a onMouseUp somewhere?
#7
05/22/2009 (2:51 am)
I integrated a script-side fix into my own project for this. What I do is have a property called 'mouseDown' or 'buttonDown' on any object that you click down and up on. So the code looks kinda like this....

function myObject::onMouseDown(%this)
{
   if(!%this.buttonIsDown)
   {
      %this.buttonIsDown = true;
      <Do Stuff Here>
   }
}

function myObject::onMouseUp(%this)
{
   %this.buttonIsDown = false;
}


I designed it this way so that the simple 'onMouseDown' call will handle both down and dragging your finger (otherwise the implementation is MUCH more complex - this was an elegant way of handling everything).

Lemme know if that helps or if you need further help!
#8
05/22/2009 (3:02 pm)
Those lines of code saved the day and put my touchscreen interface back at ease. Thanks, Dave!
#9
05/25/2009 (1:45 am)
Dave,

I'm just starting to implement pinch to zoom and swipe to move (I'll put them up as resources when I'm done). After going through what you've done here (nice work!), I have a question: when you drag your finger across the screen, is that interpreted as a bunch of onMouseDowns? From my quick tests I think this is the case (I don't know how to write to a file from the simulator/device to confirm this or not). But this means that I'll need to implement the pinch zoom within only onMouseDown and onMouseUp?

Thinking out loud, I was thinking of implementing this within rootScreenTouchDown and have that tell a scene window where the mouse points are. The scene window would then do what it needs to do with the camera, whether that be move for just one mouse point or zoom for two. You obviously have more knowledge than me in this area, does this high level overview seem like a reasonable implementation?

And one more thing, does multitouch need to be enabled from the game builder options?

Edit:
After playing with it some more, where does it actually detect multiple clicks? I was expecting the iterator within rootScreenTouchDown to have a size greater than 2 (with the first iteration being a modeless dialog), but that doesn't look like it is ever the case.
#10
05/25/2009 (4:55 am)
@Justin: That is correct, it registers them as a bunch of mouse downs. This really seemed like the cleanest way to do it to be able to register multiple sliding 'cursors' at once (without totally rewriting how Torque handles input).

What the above code does to handle multiple clicks is simply allow you to send 'states' into objects. So onMouseDown and onMouseUp and so forth. I don't think you need to do anything in rootScreenTouchDown, actually... what I think you need to do is write a new kind of object, probably for the GUI side of things. Then this object can act as a transparent overlay to everything else (make sure it's always at the top of the GUI display stack). This control will be able to comprehend multiple mousedowns and mouseups and know how to count them, and be able to register small changes in the last 'mousedown' position to another. So if a mousedown is 3 pixels away from the previously recorded mouseDown1, then we can easily guesstimate that this MouseDown is both still MouseDown1 and has moved slightly - same logic for MouseUp. Then to register pinching, you would follow your stored MouseDown1 and MouseDown2 states for slight difference moving away from each other. To make sure data goes to objects beneath it, if the new pinch control did not register a pinch, then it sends data to the screen below it at the same points.

As I type this out, actually, I think this would in reality be really easy to write. Lemme know if you get stuck and I'll see if I can find some time this week to dig in there and write a "ScreenPinchControl".
#11
05/25/2009 (4:57 am)
@Justin: Oh, and don't worry about writing files from the simulator/device to check things. Just run the GDB from XCode while you have the game running after building for simulator or device. (The GDB button can be found just above the code window, but it's only there while you are debugging). This will bring up the debugger window which shows all console output... so you can add a bunch of echo/error/Con::printf statements to your TorqueScript/Engine code and see some results.
#12
05/25/2009 (5:14 pm)
Dave,

Thanks for the gdp tip, that helps out a lot.

I've been working on the pinch, and I think I almost have the base for it. I didn't like the idea of guessing which mouse point moved, after running some tests I found that I was able to move my finger by up to 100 pixels in one event change.

I have decided to remove rootScreenTouchDown and rootScreenTouchUp. Instead I am just using rootMouseDown and rootMouseUp. I did this because it makes the reading easier when implementing a second touch event. I am just using rootRightMouseDown and rootRightMouseUp to represent a second touch. I have also implemented a drag method, just because it was an easy conversion.

I have modified processScreenTouchEvent heavily, and have it almost working, except for one case. Is there a way within this method to determine how many touches there are? If the first finger has touched the screen, and another event was triggered, I need to know if there are one or two fingers on the screen to determine if this is a drag of the first event or a new event. In your comment to Bret, you said that all the touches are within a group. Where is that group? I think if I could access that group I could count the number of touches and then use that to determine if this event is a second touch or not.

For reference, my new processScreenTouchEvent is below. Line 29 and 30 are where I need to determine if there are one or two touches. The only downside of this approach is that you can only handle up to two touches, but it should be able to be modified if somebody needs more.

void GuiCanvas::processScreenTouchEvent(const ScreenTouchEvent *event)
{
   //copy the cursor point into the event
   mLastEvent.mousePoint.x = S32(event->xPos);
   mLastEvent.mousePoint.y = S32(event->yPos);
    
   //see if button was pressed
   if (event->action == SI_MAKE)
   {
      U32 curTime = Platform::getVirtualMilliseconds();
      mNextMouseTime = curTime + mInitialMouseDelay;
        
      mLastMouseDownTime = curTime;
      mLastEvent.mouseClickCount = mLastMouseClickCount;
      
       if(mMouseButtonDown){
           if(mMouseRightButtonDown){
               // To determine which mouses event this belongs to, compute the distance between
               F32 mouseDistance = sqrt(pow(mLeftMouseLastEvent.mousePoint.x - mLastEvent.mousePoint.x,2) +
                                        pow(mLeftMouseLastEvent.mousePoint.y - mLastEvent.mousePoint.y,2));
               F32 rightMouseDistance = sqrt(pow(mRightMouseLastEvent.mousePoint.x - mLastEvent.mousePoint.x,2) +
                                             pow(mRightMouseLastEvent.mousePoint.y - mLastEvent.mousePoint.y,2));
               if(mouseDistance < rightMouseDistance)
                   rootMouseDragged(mLastEvent);
               else
                   rootRightMouseDragged(mLastEvent);
           }else{
           // ----------------------------------->
           // If a second finger has touched the screen, call rootRightMouseDown.
           // Else, we are performing a rootMouseDragged
           // <-----------------------------------
           }
       }else
         rootMouseDown(mLastEvent);
   }
   //else button was released
   else
   {
       if(mMouseButtonDown && mRightMouseButtonDown){
           F32 mouseDistance = sqrt(pow(mLeftMouseLastEvent.mousePoint.x - mLastEvent.mousePoint.x,2) +
                                    pow(mLeftMouseLastEvent.mousePoint.y - mLastEvent.mousePoint.y,2));
           F32 rightMouseDistance = sqrt(pow(mRightMouseLastEvent.mousePoint.x - mLastEvent.mousePoint.x,2) +
                                         pow(mRightMouseLastEvent.mousePoint.y - mLastEvent.mousePoint.y,2));
           if(mouseDistance < rightMouseDistance)
               rootMouseUp(mLastEvent);
           else
               rootRightMouseUp(mLastEvent);
       }else if(mMouseButtonDown){
           rootMouseUp(mLastEvent);
       }else{
           rootRightMouseUp(mLastEvent);
       }
       if(!mMouseButtonDown && !mRightMouseButtonDown)
           mNextMouseTime = 0xFFFFFFFF;
   }
}
#13
05/25/2009 (10:03 pm)
@Justin:

Take a peak in iPhoneInput.mm, in the platformiPhone directory. Look for the function processMultipleTouches. Inside of there, we have our 3 touchEvent vectors which store all the input data from the touch interface. For each of the 3 input event types (up, down and move), the vector has a method of .size(). I THINK that is storing the number of touch events for that specific event type...
#14
05/26/2009 (12:01 am)
Dave,

I think I've got a better way to handle multiple touches. I'm still working on getting it to zoom, but I *think* the hard part is over. With the method below, I don't even have to guess or find the distance to figure out which finger is moving.. the iPhone can tell you (which I think is really cool, I would like to know how they are doing this.. the hardware has to be playing some role). But anyways, the touch events aren't handing the number of touches correctly. Currently, the touchNumber in iPhoneInput.mm pretty much means nothing. The three touch event vectors that you were talking about had hope at first, but processMultipleTouches screws with it and I didn't find them useful at all.

Within each touch call, the SDK provides you with a UITouch instance. This instance can not only tell you where your current touch is located, but also the previous touch location. I used this fact to come up with an active touches array that will help me keep track of which touch the new touch belongs to.

Edit:
My pinch/zoom and swipe/pan resource
#15
10/22/2009 (5:59 pm)
Awesome work Dave, managed all changes with strong coffee, early morning mead is not my thing!

Unfortunately i get no interaction with the simulator or my device once implementing this, (using 1.2) have you troubleshot (is that a word?) this before?

I compiled the Touch tutorial (supplied with iTGB) ok in iTGB but no action in Xcode??

Any help would be appreciated
Ash
#16
10/22/2009 (6:28 pm)
Ashley,

I have managed to get this to work with both 1.2 and 1.3 with multiple projects, so I'd say it's been pretty heavily tested and 'troubleshot' (dunno if it's a word, but I'm officially putting it into use!).

I would suggest first adding some output statements in the engine that read something like "Con::printf("@@@ Touch Input");" in the iPhoneInput.mm steps and see if the initial calls are getting activated. Then, keep adding debug statements through the code until you find exactly what is not getting called - then it should be easy to see what happened from there.
#17
10/23/2009 (4:57 pm)
Dave,
Great to have the multitouch expert onboard!
it was working afterall!
i didn't need to run any output statements like you detailed above (though helpful tip for the future)
i created another project that involved no mouse dragging and i can see multiple touches with behaviours that replace a sprite on touch and another that grows a sprite on touch, all worked as expected. very pleased!

the reason i attempted this is because i came across a thread that explains that onMousedragging was broken on release of 1.3.

i very much need this functionality to work (or something similar).
i would appreciate it if you could tell me how you are managing to move sprites by touch. if the above is true???

i can only do this in 1.2 (pre your amazing code changes) for single touch

Ash
#18
10/27/2009 (8:24 pm)
To be able to move a sprite, you best bet is to turn on physics for that object, then use the moveTo command.


When it's time for the object to move, first do this:
myObject.setUsesPhysics(true);

This is needed because in iTorque2D, moving is physics driven. Yes, physics are very expensive, so make sure to do .setUsesPhysics(0) when you don't need that object to move to save on processing cycles.


Then to actually move..
myObject->moveTo( t2dVector Position , F32 Speed , 1, 0, 1, 0.1f);

Now, notice I gave the C++ version of this. That's because if you are going to be updating the moveTo destination on every update pass from moving the finger, you're almost certainly going to want to do this from either a custom C++ object, or from a C++ component, as it will improve your performance noticeably.

If you just want to test this and see how it works in script, you can just do:
myObject.moveTo( X Y , Speed, 1,0,1,0.1);

Dump the function in the console to see what the rest of those arguments do. :)

Hope that helps!!


(NOTE: I should add that this works fine in 1.2 and 1.3).
#19
10/30/2009 (1:18 pm)
thanks for your reply Dave,
I am getting unknown commands in the console for both setUsesPhysics and moveTo. nor can I find details of these functions in the official documentation. I am sure I must be doing something wrong, can you point me in the right direction.
i much appreciate you taking the time to help.
Ashley
#20
11/20/2009 (7:16 pm)
setUsesPhysics is only available in the iTGB target - you'll get errors for this if you try and set it for the mac build (so you .isMethod(setUsesPhysics) to check first to avoid the error). As for moveTo... what kind of object are you attempting to apply this to?