Request for info from people who have tested games

#1 Well one day I was testing the newest build for Marble Blast when I looks out through the hole in the wall and sees this tank pull up and out gets one of Mark Frohnmayers' boys, so he comes in nice and friendly and says Mark wants to have a word with me, so he chains me to the back of the tank and takes me for a scrape round to his office and Mark is there in the conversation pit with Rick and Alex Swanson, the baby crusher, and two film producers and a man they called 'Kierkegaard', who just sat there biting the heads of whippets and he says 'I hear you've been a naughty boy Clement' and he splits me nostrils open and saws me leg off and pulls me liver out and I tell him my name's not Clement and then... he loses his temper and nails me head to the floor.

#2 Hey Brad,
I've tested quite a few games, most of them were hits, so obviously it was quite fun. I betaed Everquest, Planetside, Star Wars Galaxies, Shadowbane, Everquest 2, Ultima Online, Furcadia, Kyrne, Half Life, and Flashpoint. Although it is VERY exciting to get to play games before the rest of the world, it can also be a stressful operation. Crashing every few minutes, then when the thing you don't want to happen (crash/wrong output) you've gotta try to re-create the EXACT same conditions that you got the irregularities under, if you could recreate it, you wrote up a report about it, and if you couldn't, then you haven't done your job and have to keep going until you can. Most Beta testers don't bother doing a proper job and are just in it to get a pre-release version, but I like to think that if I can help solve even just one prob with the game then I've done my job, and might help both a player of said game, and a tech support guy of said game by not forcing them to spend hours troubleshooting.

#3 LOL @ Pat

Beta testing my own games proved to be extremely challenging. I had people along with me way back in the True-Vol days and really most couldn't take it. It was more like Alpha testing actually but getting a game to work how you want can be really tedious. It took me 10 months more than full time hours to create that first big game. Most of the time I had headaches from staring at the monitor so long and if I couldn't accomplish a certain goal it made it difficult to sleep. It was made tougher by the fact that it was a multiplayer game and I had to buy a laptop to test it myself. Laptop on 56K desktop on a cable modem ... I must have compiled the game 5,000 times bug testing thousands of gameplay scenarios.

Back then I was really scared of someone ripping off my work or just ripping into it so I only shared it with a few close friends etc... Of course now I realize people will only rip you off after they know what you're working on is successful ;) The last game I worked on wasn't too tough we devided the testing up amongst 10 or more people and it greatly reduced the beta testing pains ... of course it wasn't multiplayer. I've got to say that when developing multiplayer stuff it's hard to find anyone who'll stick with me through the bugs. It's just tedious.

#4

Quote:Anything anyone out there who has tested to show that game testing is anything but easy would be appreciated.

Not sure I follow you. If I were to say testing is easy would you even want to hear what I have to say, or are you specifically looking for ways to make it sound tough?

#5 Gonzo,

What I'm looking for is anything to get through to people that testing video games is more than playing a game and telling the developers whether the experience was fun or not. And thanks everyone for posting info. It is definitely appreciated.

BRAD

#6 Testing a game is HARD and rather often BRUTAL work. Pat Wilsons' description is not wholly unlike how it is to test a game!

In my time, I've been fortunate to beta-test a number of very stable games including World of Warcraft, Diablo II and its expansion, and a game on the Sega Dreamcast whose name currently escapes me. Of course there have also been the games that I have worked on or been asked to beta-test by 21-6 Productions.

If you pick up a game and play it, then report back to the company that "It was cool", you will almost never be asked to test another game. Many company's, in fact, actually require you to write a letter stating WHY you should test this game... check out www.betawatcher.com for proof of that!

The primary things about testing a game are checking for:

-Stability
-Doing every little stupid thing any user could think of and seeing if it breaks the game.
-Is it fun?
-Is it tedious?

Let me take Metal Gear Solid 2, for example. Have you played it? If you have, this is going to make a lot more sense. Had you been a tester for this awesome, fun game, one of your tasks would have been to press up to every part of every surface in the game and knock on it to see if it makes the right sound. This may be cool for the first HOUR, but after only an hour when you're not even down the first hall in the first area because you're busy knocking every damned object in site from every feasable direction, you're going to start realizing, this is a JOB. =)

The latest game I've been working on was TubeTwist. I tested the hell out of that game until it was almost impossible to break it. I had to click everywhere on every screen bang on key's while switching scenes, wildly move things around, and of course the usual trying to do stupid in-game tricks that the players MIGHT do to see if it causes issues.

In Warcraft 2, I remember hearing how one of the testers had so MUCH fun placing buildings on every single square of the map then attacking them with a peon... all to make sure it wouldn't break the game.

So, game testing can be fun, sure! I had a lot of fun finding ways to break games. The truth is that a company will almost never bring in testers until further into development, when they know it is stable enough to try. Maxis used what they called "Kleenex Testers" for The Sims, which was kinda neat. They would hire these people off the street and have them work for a week or two. They would play the game, try and break it, do everything they could think of, and in the end have to report about every odd bit and detail they found, especially including the interface. Maxis would then take the time to improve and modify and fix things based on what these testers said, then get a whole new batch of Kleenex Testers to try out the next variation on things, and they kept doing this until things were golden.

I shall give you one more example: Marathon 2. This is the funniest testing thing I have ever heard. Anyone who has made games for awhile can tell you that it is not uncommon at all for the developers to be complete masters at their game, because they have spent SO much time with it, and know ALL the tricks. Bungie knew this, and took into account the difficulty level of the game. Well, they kept tweaking the difficulty of things until the developers were able to - as a group - complete Marathon 2, on the most difficult setting, by themselves, using NOTHING but the KNIFE weapon. That's right. No ammo, no bullets, no missiles... just knives and a WHOLE FRIGGEN GAME to play through MANY MULTIPLE TIMES until things were balanced enough that they could beat it with just knives.

Now, ask your friends: Does all THAT sound like fun? And don't forget to take into account the massive reporting you are expected to do. "This is cool" will not even come close to cutting it. =)


-Dave C.

#7 Thanks Dave, that is the type of information that I was hoping for. Anyone else out there with beta testing stories that they would like to share?

#8 No, Gonzo, he was looking for accurate representations of testing.

If you were to say testing was easy, you'd just be wrong, or are looking for new ways to be condescending?

#9 Brad. I like to test, testing is a strength of mine because I always loved any type of puzzle I could get my hands on. Didn't matter how hard the puzzle(not jigsaw puzzles, mind puzzles, logic puzzles, math and word puzzles, etc...) I was addicted to them and they didn't last long at all. I've also been dealing with code for about 23 years now. Started when I was 12 with MS-DOS, then BASIC, MACHINE, COBOL, etc.... so for me to say it is easy is mearly a reflection of how I view it's difficult in relation to my skills. I get stumped sometimes, sometimes I try way to hard to find a problem that is practically jumping right out at me and I just dont see it till I just walk away and come back later, then it jumps out at me, lol. I've also developed all my own methods of testing and debugging through years of self teaching, and a lot of things I do I have never seen anyone else do so I tend to get things done pretty fast overall. That's the debugging aspect of it anyway.


Now, the game aspect is a tricky one to illustrate because you can have two equally experienced people that have drastic differences in skills not because one is smarter than the other or has a special gift that the other doesn't have but because the experience is reletive to environment and which environment they have their experience in will dictate the performance of the environment they are put into. To best explain what this means, I'll give you a theoretical me, and the real me. The theoretical me(TheoMe) works for a decent company, can be small or large, but the atmosphere is professional and straight forward. Guys in an office coding a game working on features of a milestone list piece by piece, a pretty generic place for the most part. Now the real me, doesn't work for any company at all, and didn't start out coding new programs from scratch. I started out as a modder in a game that was rampant with bugs and worst of all, cheaters. Cheaters everywhere. While the TheoMe was busy coding a game pretty much by the numbers and worrying about meeting timelines and feature requirements, the real me was in the trenches studying cheat techniques, scripts, tactics, and methods of exposing and exploiting cheats to better understand how to not only fix them when I leanr of it's existance, but more importantly how not to open a hole like that in the first place. Now, assuming all stays the same if you fast forward 5 years then what you end up with is a TheoMe that is laying down code at an advanced pace doing pretty much what he was always doing(just coding games) and then you got the real me who just happens to look over his shoulder one day at the code on his screen and says "Dude, you cant do that, within two weeks Cheaters will have a script that will open this function wide up because they'll interject a command via console that allows them to do this and they will gain instant control of your server and start admining themselves and kicking other players. They'll own your server. Doesn't even need to see it run to know it will be penetrated by someone eventually and the TheoMe had no clue what he was talking about untill it was explained to him.

#10 See the contrast here? Coder one being an in house code grinder is sitting there laying down code that is just waiting to be script hacked, and a "field" coder that has seen so many cheats and hacks that he can even recognize the type of bad code that is going to create a hack or penetration point. It's not the TheoMe's fault that he is writing bad code because the code is working and appears to be logical, and he's never actually even seen how someone would cheat, so he has no basis for reference of what not to do or what could happen. So there are definate things to look for when building a team. Me personally, if I had a choice of a guy that had worked for EA and RockStar, and Microsoft, all 3 and a guy that had modded games for himself and no pay for just as long as the other guy had worked with all those companies, I would take the modder in a heartbeat. IMO that "field" experience is huge and anyone that can specifically stop cheats is a huge value to me personally because I put a high value on it. I have seen cheating destroy great games and anything I work on I am very careful to make as tight as my skills will possibly allow


From there, you need someone that knows and has a good feel for exploitation that is not of the cheat type, but more of the breakdown type which is not so obvious at all. A theoretical example would be one of those crazy hacks that makes no sense and you always wonder "How in the hell did someone figure that shiznit out", just something whacke like, if you switch to weapon 2 and then simultaneously switch to weapon 1 and open your inventory menu while holding the fire button down and then hit your suicide button the instant your weapon changes to back to 1 and after you respwn you have 2,000,000 ammo and you are invisable, all they can see is your gun. As crazy as that sounds, much stranger stuff than that has been done and I have had to figure out not only how to fix it, but how in the heck someone ever figured out the sequence in the first place. Once that mystery is solve, I take my new knowledge and apply it to my other systems to see if this type of manuver or a variation of it can affect them also. This is where things pretty much demand experience and a lot of thinking and problem solving where sometimes the only fix is a very heavy rewrite of multiple systems. That is anything but easy in a lot of cases.


I'm speaking from a position of needing well honed debug and diagnosis skills that I have, and I only have two other people I know personally that can achieve these same types of results. I'm sure there are more people out there with these types of full spectrum skills, but I only know two people that I have actually worked with and have seen them and their skills in action so I end up doing at alot of this myself and use their help when I can get it. To tell you th truth, I wish more of these damn cheaters would turn from the dark side and learn to code their butts off so they could apply those skills they posses to stopping the very thing they are best at.

#11 I would hire an awesome cheater in a heartbeat to work for me because even though I have tons of experience in counter cheat production, countering cheats and studying cheats can never be my primary job duty or skill so a dedicated cheater is always going to have the advantage on me. The saving grace here is that I generally start out with code that is already very tight becuase of what I have learned over the years, but no matter how good you are there is still going to be stuff you miss because you cant think of everything. Sometimes it just flat out takes a different mind with a totally different way of thinking to expose a flaw. Luckily I won't be to far behind him when he exposes the cheat assuming the cheat can be detected through some means such as a log entry that doesn't make sense, somone having more points than you know they should have(player 1 has 20,000 points and the highest anyone has ever scored is 2,000 points), or just plain old see them do it while playing them, and I'll get it removed before it does any real damage, but that will mean I am always chasing and almost certainly never going to find the cheat before he does. So you have to stay on your toes.

Next stage is just a full blown test of everything. GEt on a server alone, and do everyhting you can do and make sure everything does indeed do what it is supposed to do. A final housecleanig if you will. This is pretty straight forward and systematic, and it's also pretty damn boring and time consuming. Say you were talking an FPR with 20 weapons and 3 different player types that can carry them. You'll need to load up in a character and equip every weapon you can and then basically you would run around and just fire the weapon while looking for things like, did the explosion work, did the kickback work, was the spent bullet subtracted from inventory, did the animation play correctly, when the weapon ran out of ammo di the count stop on zero and did the gun actually stop shooting, if you switch off the gun and come back does the ammo come back with it, etc.. ect.... and these testing procedures are completely dependant on the type of game, the operational aspects of it's assets, a skilled testor that not only knows how to run a full blown system test but also knows the game well enough to know that he actually did test every single aspect before he signs off on it.

Now, to me, this is where you start worrying about your fun. And in my case I could turn the game over to anyone that wanted to test it and they very well could just tell me what they like, and don't like about it and then I'll take corrective actions. But I wouldn't pay that person to play it and critique it because there is no shortage of people who will happily give you their opinion on something new like that. However, if some player was testing my game and he had feedback that was IMO very valuable I would definately reconsider a way to pay him some bucks to get him to stick around and help with new ideas and feedback. But this is the only way I would pay them. And I wouldn't even consider this person a testor because he's not realy testing anything but his opinions of it. So as you see, anyone who tests with me is definately going to put some serious effort and thought into the game and honestly they will not spend anywhere near as much time playing as you would suspect, and there would be plenty of times where they aren't actually playing at all.


So I'm in agreement with you on the work part for sure. A true testor in my book will actually spend a great deal of time testing and almost no time actually playing. Once all the stuff is actually ready to hand over and there are going to be reagular players just playing to see how it plays then the testors can go out and play with them and see the fruits of their labors as well as get a better feel of how they might be able to improve various aspects of the game.

#12 I dont know whats more sad. That you wrote a 20 page essay or that you think anyone cares enough about what you have to say to read it

JMO

#13 @ Pat and Charlie

Pat, last night Charlie let me know he was irritated with me for patronizing, to which I apologized to him about it and he shrug it off as no big deal because it was more like a series of posts that irritated him and offered to discuss it with via email etc.. etc... I didn't contact him. Instead I thought about what he said, and after a few hours, this thread come to mind. I remembered the "condescending joke"(supposed to be a joke) I had made to you and thought maybe that had been a part of what irritated him. I pretty much remembered what i had said to you but wanted to see again just to be sure. So I find this thread, scroll down to see my post again and before I got there, I saw your accusation(for lack of a better word) of me being condescending, and I was like, oh yeah, I remeber now! I explained my position to you and as a joke I finished with a lousy joke. Anyway, seeing it the second time around I realized the my first few comments were a little rougher than I thought they were originally and they pretty much made the rest of the post(including the joke) look more like a lecture than an explanation. So I just wanted you to know I had no intentions of lecturing you and the post looked like crap so I deleted it. If you didn't see it yet then don't worry about it, I just wanted you and Charlie both to know that I really didn't plan to come off that way.


Anyway, I realized none of it even mattered except for the your "condescending" inquiry. In all fairness, I really wasn't trying to be condescending to Brad when I asked him if he really wanted to hear what I had to say, it was just a simple inquiry to make sure I was giving him what he needed. But still, here was a second person who was suspect of me being condescending and I'm thinking Damn, Charlie was upset with me because of you, and you were upset with me about Brad, and since Brad couldn't have been the reason you were mad, then somewhere out there, I must have done it again to someone else. Sooooo I'm like ok. Now what? Have I done this to everybody? WTF?


Anyway, i'm not sure I wanna know if it goes way back. I might not be able to live with myself. :-( I just gotta watch what I say from now on. Again I just wanted you guys to know that aint me. Yeah, appearantly I have been screwing up at least 3 times and I had no intentions of doing any of it, I'm supposed to be a clown. Funny, make people laugh, but I guess lately my jokes have been off quite a bit. I have been through some major hell in the last 3 weeks so I guess I'm starting to snap at people.

#14 I just hate to hear anyone call testing anything but tedious and painful, but then again I'm in the middle of getting an Xbox game ready for certification. I have probably played, at this point, more Marble Blast than anyone else on the planet with the exception of Mark and the testing team that is working on Marble Blast. Months later bugs are still cropping up that would cause us to fail cert. Like, "What happens when you are playing online and you yank the network cable out on THIS screen." I am sure that any one of the Xbox cert companies are full of employees who would not let anyone utter the words "easy" and "testing" in the same sentance.

#15 I skip Gonzo's posts. Sorry Gonzo, I tried reading them.

Software testing is the HARDEST thing to get right. It's subjective, even experienced QA folks have a hard time determining if a bug is really a bug, or an intentional feature implementation.

Good QA folks are hard to find. It's a monotonous job, and the nuances in determining whether something is a "bug" or a poorly designed feature is tough, often a judgement call, especially in poorly documented designs. It's especially tough in games, as you have to add in the "is it fun" on top of all the OTHER stuff good QA should uncover.

Then there is the whole "prioritization" that needs to happen, and good QA people need to be able to assign SOME level of priority to the issues they find.

It's a very detail oriented job. For a bug report to be useful to me, I need to know exactly what you are doing when you encounter the issue.

http://scp.indiegames.us/e107_plugins/custompages/Mantis.php

That's the advice we give end users who want to contribute to the FS2 SCP and report bugs. Peruse that bug list, you'll quickly see the difference between a good bug and a bad bug report.

Good QA people help the team solve the problems they find by giving concise, reproduceable error reports.

So no, it's not easy.

#16 Wow. Lots of new posts. Thanks everyone. :)

BRAD

#17 Gonzo should write a book ... doh too late!

#18 I'm quite new here so a little background is in order. (Hi all by the way). I began work in the the games industry as a tester, jumped in on the bottom rung and tested a few games. Worked my way up to coder - did that for about 7 years and after the last company I worked for went under set up with a coder friend our own business. We develop games mostly, a small part of our business is selling online games, the major part is writing games for cabinets. Anyways...

It's not so much that testing is difficult - it's more mind numbing than anything. One game I had to spend about 2 weeks doing the same task over and over - start the game, drive round the entire map, into every collision point I can find and note down co-ordinates of any problems. The job wasn't tricky, it just took forever - once a map had fixes in it I had to do it all over again.

I didn't care too much whether it was an actual bug or not, if behaviour wasn't as expected then something was wrong and I'd let the producer decide if my bug shouldn't be there. For example, clicking Back and going to the wrong screen isn't technically a bug if that's exactly what the code says it should do but it is unexpected behaviour. As soon as you give too much thought to whether something is a bug or not it gets tricky. Safer to let the person who gets the reports just read it and put a line through it.

Once a bug was located then it's time to try and repeat that bug and see if you can get an insight on what causes it. 'If I do this does is still happen?', 'How about if I use that weapon instead?' 'what about if I go to this option screen first?'. Write all that down, type up the results, classify them, send them to the lead coder.

Edward Gardner makes very good points and is spot on. Good QA are hard to find, having been on the other side of the fence and worked with different standards of QA it can get quite painful if they're not up to scratch. As an example bugs are classified, we used a simple system...
A - show stopper, the game crashes, it hangs, you can't continue type thing. Have to be fixed before release.
B - severe, these really should be fixed before release. Things like, "clicking Back takes you to the wrong screen", etc.
C - minor, once the A and B are cleared off you start fixing these. This is stuff like "there's a stray pixel on the HUD graphic".
D - comments, just items like "that weapon would be better if it did this".

What did our test department decide to do midway through crunch time? Classify everything as 'A', why? Just so you'd read them and try to fix them. We had class A bugs like "it would be cool if you could fly a helicopter". Producer saw how many class A buger there were and nearly had a heart attack.

So, testing isn't difficult, but finding a good tester is. And testing doesn't stop when you're no longer a tester. Once a tester, always a tester.

On a final (phew!) note, here's a bug description from the not so good QA....

"When catamaran leaving the textures to walls then change".

This, unsurprisingly, earnt the tester in question (Dave) the nickname "Yoda-ve", complete with Photoshopped green head on Yoda's body.