Peer Review Program Proposal
by Shayne Guiliano · in General Discussion · 07/12/2002 (12:17 pm) · 35 replies
(Addition-Before you read this post, please note that the original intent of it has evolved somewhat and that the Peer Review Program I'm now thinking of is not well-represented by the first part of the post. Just a heads up.)
This post evolved out of a discussion over copyrights and team recruitment problems. An idea popped into my head reading Jeff's response that I wanted to propose to the community. I think, especially in a community like this, there are a lot of really creative people that have problems getting an idea to actually find support within the community because it's so hard to convince people that your idea is creative and unique and capable of igniting a fire in any developer willing to listen to it.
The problem is EVERYONE in this community believes that their game is the next great game, and that's just not the case, for whatever reasons. GG self-promotion ends up getting the "Boy Crying Wolf" kind of indifferent response that we've all gotten used to. I'm guilty of it too. Go to the Help Wanted Adds. How many "this will be the best game ever made" quotes will you find in the first hundred you read? The problem is, one of them might actually be telling the truth and be THE ONE.
In other words, this is a creative community without any kind of quality assurance infrastructure in place for evaluating the power, creativity, and thoroughness of ideas that deserve consideration. Undoubtedly, some ideas within the hundreds of active minds in this community, whether it be 1% or 50%, actually are the next big things. And if they are we have no means for knowing and this is a shame. I know there are lots of artists, programmers, and designers who would jump at the chance to work on a project if they could be convinced that it was truly worthwhile. Isn't that why you, we, are here, reading and writing on these boards. We all wanna make the next great game.
So, if this truly is a strong community, and I believe it is, then we should have some means for finding the great ideas in the community and supporting them for the benefit of the entire community. And I see some of this already going on so I know we want this to happen. And I believe the best answer to this problem could be to simply act like a community by creating a Peer Review Program.
Here's how I propose to do this.
-Invite the more experienced developers to agree to reviewing game designs that have met Peer Review entrance standards.
-Find some administrators to direct the program and filter entries so that the reviewers will not have to worry about their time being wasted. I volunteer to work on this and invite anyone to help. It will help the community and undoubtedly be educational.
-Create a set of standards that outline what is required for a design doc to be accepted for review. I think the GG leadership should be very involved in creating these standards. The entries should have length limitations to limit the time required for review and design requirements so that we can be sure all entries have been properly written and are fully ready for professional review.
-Create a set of standards for reviewers to report their opinions with and post these opinions online. The actual content of the design will not be published, just a professional analysis of the quality of it and recommendations for developers to work on it, maybe work on it or not work on it. Set the standards for review to be very high so that designers are forced to think at the highest level in their design efforts.
-Set up the entire system so that everyone involved is under confidentiality agreements. I believe that we should prob get some professional legal advice for setting this up. Granted, this is the most difficult aspect of the proposal because I know how much creative people fear NDAs, but I believe that we could come up with an agreement that makes it so everyone will feel comfortable. This protects everyone and would allow for creative designers to pitch their ideas to developers based on professional community recommendations. It also decreases the amount of effort needed for everyone to feel comfortable since only one NDA will have to be signed that covers everything that occurs within the Peer Review Program.
This process would do several things that could directly affect the success of the community. It will force designers to be more professional in their approach to design and this can only have a positive affect on the development of games, be they great ideas or average ones. Designers can be confident it will be at least seen by someone in the know without having to worry about it being stolen. I know it has been discourageing to me to be working hard on an idea not sure how I will be able to get some legit developers to see it. It will help the great community ideas get made because they will carry the quality assurance stamp of the community which will help to attract developers. It will also give the more experienced developers a chance to sift through the ocean of ideas so they can work on the games that deserve their expertise. It would be great if a designer could have his or her design doc reviewed and then have developers coming to her or him to work on the game.
Overall, I hope I've convinced you that this program is necessary and could have a tremendous affect on the quality of games that are developed within the community. I think it could set a model for other creative communities, such as indie film, to follow if we were able to make it a success. I also believe it will help to bring in more creative people into the community because we will be able to advertise to them "Just write down your idea in detail, and we'll check it out. If it's good then it prob has a good chance of being made. If it isn't then at least you will know."
If you have any interest in this program at all, be it as an entrant, administrator, or reviewer, please post something and I will contact you. We need to build some momentum for this thing so that the next great games can start being made.
(Addition)-After discussion with someone else on the boards, I am convinced that we should further discuss whether or not NDAs are necessary for this to work. I think it will encourage a atmosphere of professionalism that would be great to see in the community, though I would hate for it to limit the quality of reviewers to be involved. If you think you are a potential reviewer and this is the only thing holding you back, please tell us why so we can be more educated when thinking about what to do.
This post evolved out of a discussion over copyrights and team recruitment problems. An idea popped into my head reading Jeff's response that I wanted to propose to the community. I think, especially in a community like this, there are a lot of really creative people that have problems getting an idea to actually find support within the community because it's so hard to convince people that your idea is creative and unique and capable of igniting a fire in any developer willing to listen to it.
The problem is EVERYONE in this community believes that their game is the next great game, and that's just not the case, for whatever reasons. GG self-promotion ends up getting the "Boy Crying Wolf" kind of indifferent response that we've all gotten used to. I'm guilty of it too. Go to the Help Wanted Adds. How many "this will be the best game ever made" quotes will you find in the first hundred you read? The problem is, one of them might actually be telling the truth and be THE ONE.
In other words, this is a creative community without any kind of quality assurance infrastructure in place for evaluating the power, creativity, and thoroughness of ideas that deserve consideration. Undoubtedly, some ideas within the hundreds of active minds in this community, whether it be 1% or 50%, actually are the next big things. And if they are we have no means for knowing and this is a shame. I know there are lots of artists, programmers, and designers who would jump at the chance to work on a project if they could be convinced that it was truly worthwhile. Isn't that why you, we, are here, reading and writing on these boards. We all wanna make the next great game.
So, if this truly is a strong community, and I believe it is, then we should have some means for finding the great ideas in the community and supporting them for the benefit of the entire community. And I see some of this already going on so I know we want this to happen. And I believe the best answer to this problem could be to simply act like a community by creating a Peer Review Program.
Here's how I propose to do this.
-Invite the more experienced developers to agree to reviewing game designs that have met Peer Review entrance standards.
-Find some administrators to direct the program and filter entries so that the reviewers will not have to worry about their time being wasted. I volunteer to work on this and invite anyone to help. It will help the community and undoubtedly be educational.
-Create a set of standards that outline what is required for a design doc to be accepted for review. I think the GG leadership should be very involved in creating these standards. The entries should have length limitations to limit the time required for review and design requirements so that we can be sure all entries have been properly written and are fully ready for professional review.
-Create a set of standards for reviewers to report their opinions with and post these opinions online. The actual content of the design will not be published, just a professional analysis of the quality of it and recommendations for developers to work on it, maybe work on it or not work on it. Set the standards for review to be very high so that designers are forced to think at the highest level in their design efforts.
-Set up the entire system so that everyone involved is under confidentiality agreements. I believe that we should prob get some professional legal advice for setting this up. Granted, this is the most difficult aspect of the proposal because I know how much creative people fear NDAs, but I believe that we could come up with an agreement that makes it so everyone will feel comfortable. This protects everyone and would allow for creative designers to pitch their ideas to developers based on professional community recommendations. It also decreases the amount of effort needed for everyone to feel comfortable since only one NDA will have to be signed that covers everything that occurs within the Peer Review Program.
This process would do several things that could directly affect the success of the community. It will force designers to be more professional in their approach to design and this can only have a positive affect on the development of games, be they great ideas or average ones. Designers can be confident it will be at least seen by someone in the know without having to worry about it being stolen. I know it has been discourageing to me to be working hard on an idea not sure how I will be able to get some legit developers to see it. It will help the great community ideas get made because they will carry the quality assurance stamp of the community which will help to attract developers. It will also give the more experienced developers a chance to sift through the ocean of ideas so they can work on the games that deserve their expertise. It would be great if a designer could have his or her design doc reviewed and then have developers coming to her or him to work on the game.
Overall, I hope I've convinced you that this program is necessary and could have a tremendous affect on the quality of games that are developed within the community. I think it could set a model for other creative communities, such as indie film, to follow if we were able to make it a success. I also believe it will help to bring in more creative people into the community because we will be able to advertise to them "Just write down your idea in detail, and we'll check it out. If it's good then it prob has a good chance of being made. If it isn't then at least you will know."
If you have any interest in this program at all, be it as an entrant, administrator, or reviewer, please post something and I will contact you. We need to build some momentum for this thing so that the next great games can start being made.
(Addition)-After discussion with someone else on the boards, I am convinced that we should further discuss whether or not NDAs are necessary for this to work. I think it will encourage a atmosphere of professionalism that would be great to see in the community, though I would hate for it to limit the quality of reviewers to be involved. If you think you are a potential reviewer and this is the only thing holding you back, please tell us why so we can be more educated when thinking about what to do.
#22
07/17/2002 (9:09 am)
OK, he keeps editing the post above mine. So, my previous post has no more relevance to his last post.
#23
07/18/2002 (8:10 pm)
water-fall lifecycle model is dead for ages.
#24
10/15/2003 (12:07 am)
Randall why did you even bring back this old thread in the first place if you didn't want Shayne's created threads to show up in the active listings?
#25
LOL
He's passionate... gotta at give him credit for at least that!
-EricF
10/15/2003 (12:08 am)
So now it's the top SIX threads? :-PLOL
He's passionate... gotta at give him credit for at least that!
-EricF
#26
If you don't like reading things he posts, just don't read threads that have his name next to them.
His idea wasn't bad - it just wouldn't be appropriate for GG. One of the main benefits of GG is that people who don't have years of experience designing and programming games have a place to get started. Reviewing the designs would mess that up for some people.
If someone WANTS a review from their peers, they can just post a thread asking for one.
But him making this suggestion is no reason to get angry. It looks like you're mostly referring to some other thread I'm not aware of. But it sounds like whatever he posted wasn't so terrible that he needs to be banned from GG. People shouldn't be punished for expressing their opinions about game design in a game design forum.
I admit that a few of the things he said sounded vaguely insulting, but other people say similar things on GG.
10/15/2003 (7:44 am)
I disagree with Shayne's point of view in this thread too, but I think banning him is going way overboard.If you don't like reading things he posts, just don't read threads that have his name next to them.
His idea wasn't bad - it just wouldn't be appropriate for GG. One of the main benefits of GG is that people who don't have years of experience designing and programming games have a place to get started. Reviewing the designs would mess that up for some people.
If someone WANTS a review from their peers, they can just post a thread asking for one.
But him making this suggestion is no reason to get angry. It looks like you're mostly referring to some other thread I'm not aware of. But it sounds like whatever he posted wasn't so terrible that he needs to be banned from GG. People shouldn't be punished for expressing their opinions about game design in a game design forum.
I admit that a few of the things he said sounded vaguely insulting, but other people say similar things on GG.
#27
The more words something takes to describe, the less absolute that something is.
10/15/2003 (8:51 am)
Big J's Law of the Universe:The more words something takes to describe, the less absolute that something is.
#28
Indeed.
10/15/2003 (11:21 am)
Quote:The more words something takes to describe, the less absolute that something is.
Indeed.
#29
It would be a good idea if design was absolute.
If coming up with the uber design doc meant that would have have an
uber game that would be cool, but in the real world, game development is
mostly about implementation. You can have the best design doc in the world
but if you cant model and you cant program, your games gonna be crap no matter
how solid the idea/design is.
Also I dont see it necessary to send it to everyone.
I know when I hit a certain spot in development I will send
it to the GG guys for advice and direction, or input on what I have so far
and a few other people.
10/15/2003 (1:36 pm)
To me this idea is pretty flawed.It would be a good idea if design was absolute.
If coming up with the uber design doc meant that would have have an
uber game that would be cool, but in the real world, game development is
mostly about implementation. You can have the best design doc in the world
but if you cant model and you cant program, your games gonna be crap no matter
how solid the idea/design is.
Also I dont see it necessary to send it to everyone.
I know when I hit a certain spot in development I will send
it to the GG guys for advice and direction, or input on what I have so far
and a few other people.
#30
Number of pro-peer review posts after Jeff's post: 1 of 13.
No offense to Jeff and the greate GG gang meant by this; it seems much of the community is greatly influenced by what Jeff has to say. :)
10/16/2003 (7:00 am)
Number of pro-peer review posts before Jeff's post: 3 of 3.Number of pro-peer review posts after Jeff's post: 1 of 13.
No offense to Jeff and the greate GG gang meant by this; it seems much of the community is greatly influenced by what Jeff has to say. :)
#31
10/16/2003 (7:52 am)
This isn't surprising, most people are sheep. Not to invalidate any particular viewpoint.
#32
whatever... this thread is over a year old, some of the people who posted this time around may not have even been here then.... or they just never read this post then.
10/16/2003 (8:01 am)
Or the community is greatly affected by reaction of the person themself.
#33
10/16/2003 (8:14 am)
@Harold: Good insight... a bit of both.
#34
This is not some kind of cult by any stretch of the imagination, I agree that GarageGames can do what ever they want, it is their website, I think people disagree with Shayne because his idea(s) are not very well thought out, are flawed and unrealistic and his extremely poor presenation skills and the beligerent reactions to critisms. It has nothing to do with Jeff's ( or any GG staffs ) opinions or actions.
Shayne painted a huge target upon himself with the personal attacks ( not only the last big one aimed at Jeff ) there were many other personal insults aimed at many other people in other posts.
I am sure that just made some people curious if Shayne was just having a bad day or was that his personallity. Seems that he has had problems all along if you read thru this former posts and replies you can gather the same "tone" and "attitude" in every one of them.
10/16/2003 (10:44 am)
Quote:No offense to Jeff and the greate GG gang meant by this; it seems much of the community is greatly influenced by what Jeff has to say. :)
This is not some kind of cult by any stretch of the imagination, I agree that GarageGames can do what ever they want, it is their website, I think people disagree with Shayne because his idea(s) are not very well thought out, are flawed and unrealistic and his extremely poor presenation skills and the beligerent reactions to critisms. It has nothing to do with Jeff's ( or any GG staffs ) opinions or actions.
Shayne painted a huge target upon himself with the personal attacks ( not only the last big one aimed at Jeff ) there were many other personal insults aimed at many other people in other posts.
I am sure that just made some people curious if Shayne was just having a bad day or was that his personallity. Seems that he has had problems all along if you read thru this former posts and replies you can gather the same "tone" and "attitude" in every one of them.
#35
-J
10/16/2003 (11:03 am)
Yeah, I woke up in a foul mood... after a walk life is good again... people aren't sheep it would seem now :)-J
Torque Owner Shayne Guiliano
I believe that design is a conceptual discipline, and that methods exist that can increase creative freedom, and that a design should have at the least the following elements considered, written, reviewed, and reworked, and re-evaluated before the first one and zero hit the memory.
These are just my opinions so please respect them. I'm writing them so that everyone can have a clearer understanding of what I'm talking about when I talk about thoroughness of design and the use of methods to aid in design. Of course, once the program was started, we would work together on what the requirements for submission will be. These are the kinds of things that would be criticizable in a Peer Review Program.
If you don't like the ideas, that's fine. I'm just trying to be the best video game designer I can, and using this forum to as a means for expressing my ideas and getting feedback.
Also note: I do think all the pre-production needs to be done by one person. More heads are always better than one, and this is the best defense for the Peer Review Program. It'd be best to put together a pre-production deign team that can tackle all these elements, though I think it's important for there either to be a clear leader or for everyone to have an equivalent amount of passion and dedication.
Last note: Not all of these elements need to be considered for every type of game, though it can't hurt to consider them.
-Concept Art-All concept art should be finished and completed for all creature and level design. Storyboards laying out the entire game should be finished with proper perspective and emotion. If the designer doesn't know how to draw, he or she should take some classes. If this doesn't work, start looking for a partner to help you with the pre-production.
-Animation-Animation should be fully described in such a way that makies it possible for animation artists to come in and complete their work without being confused. If a designer has developed his or her characters properly, then there should be no problem in developing animation during production.
-A.I.-Yes, I believe artificial intelligence should be fully designed before the game starts and that this responsibility lies with the designer. A.I. has such a large impact on any game that I can't believe that most development teams just throw in their A.I. when it's 70% complete. If a designer can't do it on his or her own, then take the right classes, go buy A.I. Programming Wisdom, or get some help in designing it. This is probably the least developed part of our industry, or at least the worst implemented.
-Story Development-Who? What? Why? When? Where? Most of the time, story development is the most lacking aspect of a released games. The story should be emotionally engaging in a way that the intended audience can relate to. To think that most people gloss over this is unbelievable to me. Devil May Cry-Could have changed the industry if 5 fairly creative people had sat in a room for a week and developed a story that wasn't cliche and superficial. I must admit that video game storytelling doesn't have to be like traditional storytelling, but that doesn't mean it isn't important. There's a lot of ways to tell a good story. Pace, Impact, perspective, balance, etc. are the artistic elements that separate a well told story from a poorly told one. Storytelling is not typically used on sports games, though I believe it will be in the future.
-Character Development-Video games designers must develop characters in traditional and non-traditional ways. The best designers in the industry typically have the best character development. However, this is also overlooked in many released games. You should develop their personalities, what they look like, how they would act in certain situations. Control structure overlaps with character design. An example of this is Mario jumpin on people's heads. It is in his character to do this, even though it is part of the gameplay. Generally, you should develop them well enough so that anyone can almost predict what the characters will do in certain situations while maintaining a strong character dynamic. Kojima and Miyamoto are two of the masters at this. A simple good concept drawing with a well layed out control structure for an online-battle game would be an example of good character design.
-Level Design-This is something somewhat unique to video games and is inherently linked to gameplay because environments are typically interactive in one way or another, even if it's just a wall that blocks your way. Level design has taken some good steps in recent years, though everyone is still catching up to Miyamoto, who was the first to master modern interactive level design with the Mario and Zelda series. Level design is everything. In movies, every single thing on the screen should be there for a reason. In video games we too often just throw stuff in there without actually having a reason, though we should seek to have everything there for a reason.
-Camera-This is my biggest issue with the video game industry. Very rarely is it possible to say that the designers put any real work into the camera use. Why, I don't know. The camera is as important as anything in the game and can be very powerful for creating effect. Again, all the best games have good camera work. You should have thought of the persective that you think is best for the game before the game is finished.
-Control Structure- Though many great games have had tremendous control structures, many games lack depth of thought into the control structure. Granted, control structure is something that can only be perfected during actual implementation, but it can definitely be mentally experimented with before any code is written and must be in order to understand what is possible in the rest of the design. Have an initial control structure laid out before you begin production.
-Sound-Sound is so overlooked in the video game industry that it is also laughable. Devil May Cry, a multi-million dollar project, has some of the least professional mixing I've heard. This can be said for many games. Very few designers can claim to have taken sound to the next level and have actually used it to it's full potential. Sound is an amazing thing and I believe it's as powerful a tool for expression as video. Voiceovers are typically poorly directed and this should be the responsibility of the designers. Music is very effective in creating mood and should be used accordingly. Work hard to make the sound design up to par with the rest of it and it will pay off in the end. It's important to plan sound design because synchronization is so important.
-Vibration-Granted this is something that is only usually considered for console gaming, though it isn't usually very well-designed. Some of the best uses I've seen for vibration: Cosmetic-The entire Metal Gear Solid PS1 game. Functional-Using it to tell the pitcher where the strike zone is in All-Star baseball 2002-3. I think using vibration for effect is something that is still relatively unexplored in video games and it's a shame, because it could be such a powerful tool. It could also be used in the gameplay to give the player hints or additional functionality. Try to design vibration into the game or consider where it could be used to add to gameplay.
-Gameplay-This is something that has been defined so many different ways by VG designers that I lost count. I believe gameplay is everything from the moment the player turns on the game to the moment it's finished. The entire experience. Most people define gameplay as the dynamic interaction between the player and the computer where the player must provide input that affects the game. The universal artistic concepts of balance, pace, impact, etc. typically are what determines the difference between good gameplay and bad gameplay. I believe that this is the part of the design effort that could benefit most from researching past methods of design. It is what separates video games from all other media and probably makes video games so appealing. Gameplay design should be a pre-production process, though this typically benefits the most from in-production tweaking. A designer should strive to work out all the other elements of a game before production so that he or she can fully focus on gameplay during production. A director determining how the actors should act during a movie production is comparable to a designer directing the gameplay during production. Again, Miyomoto has never made a game with poor gameplay because he has such a great method. However, I do admit that, especially since gameplay always seems to be the main focus of production teams, most games released today do have pretty good gameplay. Very few games have gameplay that is as good as it could be, however. You must work hard to understand how all the gameplay elements work together to create a game and be able to communicate this before the production starts.
-Tool Development- Every single post-mortem I've read talks about how useful it would have been if they had gotten their tools finished before they had a group of 30 people waiting on them. Tools should be designed before any implementation occurs and should be the first thing that is implemented, before any artists are even hired. Tools should fall under pre-production because they should be mastered before being used for development. Your tools are so important because they are your means for bringing your design to life. You don't have to make your own tools, but you should evaluate everything you can.
-Tech Design-This is the hardest part of design, and should probably be considered at the same time everything else is being designed, though of course this can lead to problems if a designer doesn't have a realistic approach to their design and over-designs things that just can't be implemented. I guess my answer to that is be realistic. Have your stuff review by your Tech guy to see if it's possible.
-Visual Representation-What kinds of visual elements do you want in your game and how will you get there? What are the color themes? What kind of symbolism are you trying to inject into the images? What are the tech effects you will use to bring your game to life? Bump-mapping, per-pixel lighting, etc. Considering how visual video games are, I don't understand why so many games don't have more profound visuals. We should put in a lot of work into how we want the visuals to look in the end.
-Algorithm Design-Designers should be able to design agorithms and represent them visually or in written format. If you can't do it, then go take a class. A designer not knowing how to think about algorithms is like a movie director not knowing about acting. Being able to define and create algorithms will bring your game together and make you a powerful designer. Try drawing them out and debating them with other designers.
-R&D-The best production houses in the world live and die by R&D. Make demos of anything that's non-trivial before you begin production. These demos should be considered part of pre-production and should be reviewed by as many people as possible. This is when control structure and gameplay should first be tweaked with. You should have a very good feel for all the things you have demoed before you make any final decisions on design directions.
All these design elements require is a good mind, a pencil, and some paper. You don't need to have a big team together to finish the pre-production work. You may only need the help of a concept artist for drawing the pictures, a programmer for putting together R&D demos, and maybe a level designer or animator, though I think a lead designer, like a director for a movie, should be a pseudo expert in all these aspects of design.
In regards to the Peer Review Program, I think all of these issues will have to be addressed before a design can be submitted for full review, though I think it would be smart to set up an infrastructure that can also aid in the process. This will assure that the reviewers will not waste their time reading poorly developed designs and will also force designers to fully think out their design.
Note-You should be sure to leave room for artistic influence from your team when production starts.
Pseudo-Disclaimer-This is a theoretical argument that has come out of the last six months of me thinking about and researching video game design. I am applying these standards to everything I design and this helps to give me direction without limiting my creativity. I believe that all these concepts should be well understood by a designer if he or she really wants to unlock the full creative potential of video game design. My evidence for this is all the great games I've ever come across.
I believe the artistic side of video games comes from how well all these elements fit together when the game is done, much like the way all the molecules on a painting come together to make art. I've gotten these ideas from reading into the industry and reflecting on what I've read. I don't think any of them are revolutionary and they are even mostly obvious. However, I don't see many people sticking to them. If you think there's something I missed, please post it. That's what this is all about.
Maybe I'll post them on a new Post just so that we can get away from all the bickering above. Since this post's length has reached short book proportions, I'll try not to write any more long posts.