Follow Slashdot stories on Twitter

 


Forgot your password?
Close
typodupeerror
×
GameCube (Games) Entertainment Games

Quality Assurance In The Games Industry 75

Posted by simoniker from the make-it-better-somehow dept.
Thanks to NTSC-uk for their opinion piece discussing the perceived lack of 'quality assurance' in the videogame industry. Amid oft-repeated claims that "many games fall short of the mark" on overall quality, there are some more interesting arguments that QA testing "rarely promotes the criticism and fine-tuning of the most important aspect of design - gameplay." The author even goes on to suggest that hardware manufacturers should again get more involved in the quality of games on their machine: "Nintendo demonstrated during the 80s and early 90s how the power of the manufacturer can be used... to ensure that the design of new games, and particularly good gameplay, was top of the agenda - hence Nintendo's 'Seal of Quality'."
This discussion has been archived. No new comments can be posted.

Quality Assurance In The Games Industry More

Quality Assurance In The Games Industry

Comments Filter:

  • Double-edged sword (Score:3, Interesting)

    by sbszine ( 633428 ) on Monday October 20, 2003 @10:28PM (#7266512) Journal
    There are still a lot of console games released while suffering from major clipping and control issues, so some QA from console manufacturers would definitely be A Good Thing. Gameplay and content, on the other hand, are very subjective things, and things that console makers might be better off not dabbling in. Remember the santised RPG translations that appeared under the Seal of Quality in the NES days? I would hate to fire up FF:Crystal Chronicles and find Holy replaced with White, or bars replaced with cafes, as per the bad old days.
    • Driver 2 springs to mind in regards to console titles being released with huge problems. In Driver 2, at certain areas of the map, the framerate drops to unplayable levels and you just have to hope nothing goes wrong until you get past the area.

      Now if I recall, all games for the PSX had to be cleared by Sony to be approved for release on the console. Now while there was nothing like Nintendo's seal (which I personally take with a pinch of salt), this was a huge glitch in Driver 2 that was obviously ignored
    • There are still a lot of console games released while suffering from major clipping and control issues

      Clipping is mostly an accepted limitation of technology. Controll issues are often ruled as design issues and not "bugs" per se.
  • I was always under the impression that the Nintendo Seal of Quality was pure BS. Back when I was a kid, I suggested to one of my friends (an avid reader of Nintendo Power) that he buy Afterburner for his NES. (Yes, there was a NES Afterburner cartridge, as far as I can remember :). At any rate, he says to me something to the point of, "But it doesn't have the Nintendo Seal of Quality."

    Well of course it doesn't. Sega made it, and at the time, they were Nintendo's biggest competition. I had no doubts ab
    • I had no doubts about the quality of Afterburner as a game, at least in comparison to every other NES title out there at the time.

      That is not what the seal of quality assured, it was assurance of technical quality: The game does not crash or freeze, the walkthrough is possible, etc.

      It seems to me that the Seal of Quality was mainly just a way to keep unlicensed cartridges from selling.

      That too...
  • Quality assurance is an oxymoron. The fact is, with the suits saying "We gotta ship game X by this date or else there shareholders will bitch", there's no way quality assurance can be taken seriously.

    Games are consistently shipped with the "screw it, we can patch it later" mentality. Look back over the last few years. There have been games such as Pool of Radiance, Myth 2 etc... That shipped with fundamental bugs, not just in the game itself, but in the installer routine! The original versions of those gam
    • That's why I've always preferred the title 'quality assessment' when it comes to software QA. It rather honestly represents the situation - the testing team is there to tell you how bad things are and where they're broken, but they don't have the power to actually assure the quality of the product.

      In game related QA, also keep in mind that the salaries paid to game testers are severely sub-standard. The mentality typically is, "Hey, they get to play games all day, so we don't need to pay them much," and

  • QA in most non-game software developments involved creating automation to pound a particular piece of the software quickly and reliably.

    This would be kind of hard to implement in games.

    And if you've ever met a game tester, yeesh.

    • Re:Well (Score:2, Interesting)

      by Anonymous Coward
      AHHH but Nintendo DOES this (or at least they used to). I remember having a game in submission for final approval by Nintendo and they'd come back to us with a rejection for reasons such as "If you hit the reset button 30 times in a row the game locks up".

      Nintendo used to (and I assume still do) pretty rigorous hardware testing, code analysis (since there's certain things such as certain cpu registers you shouldn't access) etc.

      It's the bad gameplay that often gets ignored.
      • Nintendo used to (and I assume still do) pretty rigorous hardware testing, code analysis (since there's certain things such as certain cpu registers you shouldn't access) etc.

        Your assumption, as far as I've seen, is false. I've had three GBA games with crash bugs so far - Super Puzzle Fighter 2 Turbo, Megaman Battle Network 3, and Yu-Gi-Oh World Wide Edition, and the Puzzle Fighter and Yu-Gi-Oh bugs not only appeared in casual gameplay, but were easily replicated by both myself and many others. And accord
        • Haven't had any problems with MMBN3, but I've personally seen one on SSF2T-- getting to Akuma hardlocks the game in the US version. Yugioh WWD.. well, didn't see any crash problems in it, but I did notice a REALLY blatant case of cheating or two when I tried it. Absolutely horrible. SPF2T I haven't played enough to say. Your point, though, is quite valid-- when companies like Konami (Yugioh) and Capcom (SSF2T/SPF2T/MMBN3) can't get these games debugged, something's very wrong. Unfortunately, this is just a
          • Here are the bugs that I was talking about, to be more specific:

            Megaman Battle Network 3 Blue - A crash bug occurred only once or twice and they seemed to come out of absolutely nowhere, so I have no idea what set them off. It was a standard GBA crash bug, though. The screen became one single color (blue in this case) and game either went silent or made a horrible noise.

            Yu-Gi-Oh Worldwide Edition - The cards that require dice rolls cause the GBA to crash roughly one third or half the time they are used. T
  • ....is how the game companies often try to silence the critics of their games when they are released full of bugs or minus key features (such as gameplay). *cough*SWG*cough*. They were deleting posts left and right on the forums, locking out non-players, and many other things, simply because they had an unfinished game, and they didn't want people to know about it.

    And you know who we have to blame for all of this? UO.

    They started the trend of releasing a game that realistically was still in beta. Howeve

    • Blaiming UO for the problems of gaming being released while in beta stage is irrational. No one knew how popular MMO games would be at the time (ok there are exceptions, but those never really hit mass market) and UO was really the first one try it.
      • Just because nobody knew doesn't mean UO isn't to blame. Intentional or not, they set the standard for buggy releases, and incidentally, they became a success, so people modeled the behavior of what made them a success, and in this case, it happened to be a buggy release. I'm not saying that its their fault for being the first one to try an graphical MMORPG, but it is their fault for releasing it buggy as all hell. Regardless of whether it was the first game of not, they released a game that should have
    • This started happening a LONG time before UO. Games have always shipped with noticable bugs (anyone remember Epic on the Amiga? Six years before UO and bugged to hell and back) although the trend has been getting worse, and companies have censored their message boards since the first one went up. UO may have been the first vsible manifestation of it for you, but some places were pros at the game before then.
    • UO is a particularly bad example because that game was literally the first of it's kind, in terms of a huge MMORPG. Even MUDs don't come close to something on the scope of UO when it's released.

      I'm sure everyone at Origin learned from the mistakes they made with that particular release but in hindsight I really think that MMORPGs really do require a lot of player feedback in order to not only create a balanced setting but a fun (and working game). That said, Origin's QA/testing methodology is a far cry
  • Working for a some big corporations like consulting companies (like Accenture and IBM) and (when jobs were scarce) a gaming company (EA), I have found that while QA isn't perfect for video games; it is much worse for enterprise applications like banking software. Scary thought eh, that GTA is better tested than online bill pay...
    • That's so true in my experience. Games get tested by the programmers, the developers QA dept., the publishers QA dept. and the manufacturers QA dept.

      Business software gets tested by the programmers, and the end-users (and occasionally there is a QA tester at the developer).

    • The scary thing is that they let clowns like Accenture and IBM's consulantcy divison anywhere near financial software. I work for a bank and having dealt with both IBM and the company formerly known as Anderson Consulting I have come to the following conclusions:
      1. The only IBM staff that should be allowed on to a banks premises are those installing the servers. If they offer to look at your software or processes: shoot them.
      2. Shoot everyone at Accenture because they're all clueless motherf***ers. They make M
  • Nintendo Power. Those two words used to go together. Today in the US however, they do not. If Nintendo were to request major changes to a game they would be laughed at. They just don't have the market share to command the respect they used to. I would wager that if Nintendo did something to slow a third party game down, the third party would just cancel the Gamecube release. "Well it was buggy anyway so we don't want it." Riiiiiiiiight, it makes me laugh to see someone argue that fewer games is better
    • I would wager that if Nintendo did something to slow a third party game down, the third party would just cancel the Gamecube release. "Well it was buggy anyway so we don't want it." Riiiiiiiiight, it makes me laugh to see someone argue that fewer games is better.

      So that brings us to Sony and Microsoft. They can and do reject games because of quality. But they can't just go rejecting a game because they don't like it. If it functions according to their standards and as the publisher intends, what grounds do
      • Pain, we are often on different sides of the arguement, but we are on the same page here. Are you trying to convince me of something? Very little of my post was opinion. 95% was up to the minute fact.

        The only debate I could see is clipping, and I could see collision detection so poor that it could effect the end-user, but you would have to provide an example for me to evaluate. The best selling games (GTA and THPS) have some of the worst clipping out there, yet noone complains. In general, I stand b
    • Surprise, surprise, Sony DOES deny stuff arbitrarily. They've had a rather strict policy of no 2D stuff on PS2 since launch. The policy goes back to PSX days-- Capcom had a LOT of trouble getting some of the Rockman titles okayed for US release and in fact could NOT secure rights for Rockman Complete Works around 1998-1999.
    • Sony does and has rejected games cause they don't like it. I remember Sony wouldn't do a release of some Megaman game because they didn't want 2-D titles on the PS1 back in the day.
  • Getting a game out ASAP to coincide with a movie release or to make some quick money to make the books look better before the next quarter are the highest prioritize on most game companies from the way games look now. Nintendo seems to still be trying to control the quality of their games somewhat, but look at the libraries. Nintendo doesn't have nearly as many games out as the other guys, but nearly every game they have out is something of high quality. PS2 especially on the otherhand you have to dig th
    • You make this big rant about what you think happens within Nintendo vs other companies. Then you use the noun glitch as a verb. A PROGRAM CANNOT GLITCH. NOTHING CAN GLITCH.

      Please explain to me how to glitch. I want to go outside tonight and glitch. What does glitching look like?

      So what does your use of the word glitch say to me? It says you have never worked in the games industry and have no idea what you are talking about. That you are just spouting random, hateful things with nothing to back
      • Please explain to me how to glitch. I want to go outside tonight and glitch. What does glitching look like?

        This [video-fenky.com] is what glitching looks like. So go glitch to your heart's content tonight. Beware of mean ninja cuties.

      • Then you use the noun glitch as a verb. A PROGRAM CANNOT GLITCH. NOTHING CAN GLITCH.

        Considering that the English use of the word glitch as a noun is derived from a verb in German and Yiddish, I'd hardly say it's inappropriate to use glitch as a verb, especially since it's primary use in English only dates back to 1962 and always references a verb.

        Please explain to me how to glitch. I want to go outside tonight and glitch. What does glitching look like?

        The German and Yiddish verbs mean to slip, skid,
  • Superman Has the Nintendo quality seal, heck for that matter so does Aquaman, so does charlie's Angels for Game Cube and nfl blitz 2003 for gameboy advance.

    So Here's how much a nintendo quality seal Matters [gamerankings.com]
    • Oh whee, I got that link wrong. TO make that post work as I want it to, please pre-load gamerankings with the worst game sort routine, then click on that and it should work. Either that or you could just go yourself and click to the search for worst game.

      Right now it kinda backfires and makes it seem sarcastic, Sorry about that my friends!

  • Not really QA's fault. (Score:2, Interesting)

    by Anonymous Coward
    Console and PC development tracks are entirely different. Console manufacturers have stringent controls on what they allow to be published on their platforms. Obviously you can publish anything you want on the PC as you can publish it yourself.

    Fatal bugs (ie: crashes) are a lot less common on consoles than PCs. That's because they often must undergo a "burn in" process. For example, Xbox games have to stay running for 7 days - being played - without crashing before Microsoft will send the game to duplicati
    • Actually, the soak test / aging test is usually 72 hours, not during active game play. Usually this means going in and out of "attract mode". Often that means playing a movie over and over again, and never running the game itself.

      I suspect the real reson for the aging test is that MS avoid embarrassing public crashes on the in-store displays.
      • This is standard practice in the console world, not a Microsoft conspiracy. Sony et al have similar soak test requirements that must be passed before release.
      • The Morrowind crashes (screen freezes, loud buzzing from speakers, I seem to recall other games doing this once in a great while) after less than 7 days of play certainly seem to support your theory =) Say, that reminds me, I have to go sell my copy and buy the bug-fixed GOTY edition...

  • Just Focus On Bugs (Score:4, Interesting)

    by DarkZero ( 516460 ) on Tuesday October 21, 2003 @12:26AM (#7267466)
    For awhile there, console games were relatively bug free. On the Super Nintendo, Genesis, and to some degree the PlayStation, there were very few crippling bugs that every single person that played the game had to watch out for while playing. Lately, this has changed. Enter the Matrix was riddled tons of different bugs. Tomb Raider: The Angel of Darkness was filled with situations where Lara would get stuck in a jump and you would have to reset. Jak & Daxter had a bug that randomly occurred in the middle of the game that kept you from fully completing it. Knights of the Old Republic's bugs are just as infamous (though not as numerous) as Enter the Matrix's. Star Ocean: Til The End of Time, a best-selling Japanese console RPG, had crippling bugs. Yu-Gi-Oh! World Wide Edition is filled with bugs toward the end of the game, including a relentless crash bug that causes the game to crash during almost any battle with the final boss.

    These are some of the best-selling games in their respective countries and consoles, but they're riddled with software bugs and glitches that, in some cases, ruin really great gameplay ideas. And these are just the ones that are popular! Play any of the less popular GBA titles, such as Megaman Battle Network 3 or the GBA port of Super Puzzle Fighter 2 Turbo, and you'll find many more of those GBA crash bugs.

    We've actually gotten to the point where games made by veteran game developers like Capcom, Shiny, and Konami that have been certified by NINTENDO (of all companies!) are riddled with crash bugs, so I think gameplay is the least of our worries at this point. If you can't even play the damn game, then the gameplay doesn't really matter much.

    (And as a brief side note, some of the practices that the article mentions have already been standard at Sony for years. Sony Computer Entertainment America has wielded its broad monopoly in the United States to keep what it sees as "below average" Japanese PlayStation and PS2 games from entering the US. Some notable victims are The King of Fighters 2000, a Metal Slug title or two, a Persona game, and Goemon.)
    • Curiousity here, what buts in KOTOR? I haven't heard of them, and no one I know has run into them. Not doubting you, just curious what they are. I guess we were lucky to not run into them, Bioware has usually been good about QA.
      • Check the GameFAQs FAQ list [gamefaqs.com] for the game and go down to "Bug/Glitch FAQ" (GameFAQs doesn't allow direct linking to the FAQs). It explains the Stealth Glitch, which is the one that people started frantically spreading the day after KOTOR came out, but there are several smaller bugs in addition to that one. The bugs generally only occur if you're doing something weird, but that's still a far lower standard of quality control than most console games have had until this year.
    • (And as a brief side note, some of the practices that the article mentions have already been standard at Sony for years. Sony Computer Entertainment America has wielded its broad monopoly in the United States to keep what it sees as "below average" Japanese PlayStation and PS2 games from entering the US. Some notable victims are The King of Fighters 2000, a Metal Slug title or two, a Persona game, and Goemon.)

      Huh? What do bugs have to do with that? Those games aren't ported for many reasons, none of whic
      • I hardly call KOF 2000 and Metal Slug "CRAZY JAPANESE games" in fact the only one out of the list he provided would be Goemon.. which has been released twice on the SNES and N64 (Legend of the Mystical Ninja)
        • I know they're not, that's why I said "perceived". The general perception, warranted or not, is that a lot of those games simply wouldn't sell here. While I personally love Metal Slug, I'm hardly the mainstream gamer audience... and games like that, in this age of popular 3D violence and crappy gameplay, just don't sell that great.
      • Huh? What do bugs have to do with that? Those games aren't ported for many reasons, none of which include faulty software. Localization, and the perceived fact that some of those CRAZY JAPANESE games just wouldn't sell in this market are the main reasons.

        The article that was linked in the /. story was trying to encourage companies like Sony to wield their licensing power for reasons greater than preventing buggy software. All of those games are evidence that Sony is doing exactly that. None of them were r
    • Comment removed based on user account deletion
    • And what's even worse about titles like Enter the Matrix and Angel of Darkness selling well is that players (especially new players) come to expect their games to be buggy. ("This extremely popular game was buggy, why would others be any better?") So effectively, Atari and Eidos have lowered the bar for good games.

      --Jeremy

  • Seal of Quality (Score:2, Interesting)

    by H8X55 ( 650339 )
    The Nintendo Seal of Quality wasn't worth the paper it was printed on. It was nothing more than a Nintendo Fee, and promise to promote the game in Nintendo Power. Ever read a Nintendo Power review? Ever hear them trash a game? Were all of the games good? Nope. Nintendo Power was just propaganda. There were plenty of great games that came sans Seals.

    But feel free to step up now, i hear SCO is selling Seals of Quality for all the major Linux distros.
    • Ah more slashdot ignorance. Say it with enough conviction and it becomes true though right?

      There is a process to earn the Seal of Quality, though it really just means it was published with Nintendo's approval. Sony and Microsoft have similar processes as did Sega before them. So today it means no more or less than the other big players. Nintendo was the first to do this, and with it (and many other factors that would be OT here) they brought back video gaming and made it feasible as a business in the
      • Maniac Mansion was close to not getting the gracious Seal, however, they toned down some of their sophomoric humor in exchange for the proverbial pat on the head. Is this a quality issue? or more of a content issue? is this one of the ways nintendo saved american gaming?
        • Maniac Mansion as never "close to not getting the seal." If Jaleco refused to remove the nudity, crass humor and the hamster in the microwave the game simply would have not been released.

          Licensed NES publishers did not release ANY games without the seal of quality, which, as has been stated elsewhere, was really just a symbol for "this company is paying us for this bs seal and better distribution than unlicensed companies." Nintendo ruled with an iron fist back in those days, and unlicsensed companies coul
          • Remember Tengen? The published, released, and sold games without seals of approval. RBI Baseball was one of their better games, if you were into sports sims, and baseball.
            • Yes, they did publish games this way but they also broke the law to do it. They tricked the patent office into giving them the documentation for Nintendo's cartridge lock-out chip, which was proven in court, after they failed to reverse engineer it.

              Nintendo's Seal of Quality, at least in the old days, was a way of keeping companies from dumping massive amounts of bad software into the market. This had the negative affect of also keeping some good software from the market. They did it because they saw what

            • I was going to mention Tengen in my post, but figured it would just confuse people.. heh.

              Tengen was a licensed publisher at one point. All their games were sold in the same stores as all the other licensed publishers, the carts were the same as otherlicensed nes carts, etc.

              Then, the Tetris thing happened, and Tengen was no longer a licensed publisher.

              Nintendo no longer manufactured their carts (hence the black cartridges), and their games were no longer available in the same stores as licensed nes games.
    • The Nintendo Seal of Quality wasn't worth the paper it was printed on. It was nothing more than a Nintendo Fee, and promise to promote the game in Nintendo Power. [...] Were all of the games good?

      The games with the nintendo seal of quality are tested on TECHNICAL quality, not subjective quality.

      The nintendo seal of quality means that the game can be finished (!), that it runs correctly on nintendo hardware, that it has no major bugs (crashes), etc.

  • Blame the talking heads. (Score:4, Insightful)

    by MImeKillEr ( 445828 ) on Tuesday October 21, 2003 @08:00AM (#7269041) Homepage Journal
    As someone who works in software QA, all I can say is thus:

    If QA has the ability to block the release of something due to defects then this is an almost absolute way to ensure quality (other factors notwithstanding).

    If QA doesn't have to give their seal of approval before something goes out the door, then things will be released with defects (some known, others not).

    I was fortunate enough to work for a boss who stated to development and the project managers that he would not sign off on releases simply to meet deadlines. If the powers that be wanted something shoved out the door simply to meet customer expecations, they'd do so without QAs consent - and that we'd not take the blame.

  • Good QA is undervalued (Score:4, Funny)

    by codemonkey_uk ( 105775 ) on Tuesday October 21, 2003 @09:41AM (#7269904) Homepage
    IMHO high quality QA is hugely undervalued in the games industry as a whole. To many publishers just take on any old kids on their summer/winter vacation from collage, just for the final project phase, and do not see the hiring of quality staff in the QA department as a long term investment. Less than games half the companies I've encountered (ie worked for or interviewed at) take QA a seriously as I think they should.


    Anecdotal evidence doesn't count for much, but it can be entertaining, so here is a the "description" from a genuine bug report from a major US publisher, for a game I was working on a while back (not published):


    "-Music selection should very , as well as sound efects
    crashing ,sliding out , aswell as a annoncer threw the
    game making itmore real , funnny , and with the race
    atall times, when a accident acures, ect.    "

    This was submitted as a "class A", "In-Game GUI" bug.


    It's a cut and paste. No typos introduced in the retelling.

  • Heaven's yes, let's go back to the seal of quality. There was certainly nothing wrong with that. As everyone knows the seal of quallity was definitely synonymous with quality [google.com].
  • With pieces of buggy crap (see the Temple of Elemental Evil boards for a perfect example) basically ruining PC gaming, yeah...something has to be done. As it is now, you can't hold a company accountable because you can't RETURN buggy as hell software. It's quickly becoming impossible to do so because of the way the companies that MAKE the games tell stores they will do returns. It's a sad thing, because I can say that many good/quality games are tossed and never played because of the horrendous problems
  • from the article..

    <blockquote>And what about industry regulators? Organizations such as the IDSA and ELSPA fight piracy, and impose age ratings for new software, but nothing to raise awareness of the concerns caused by software glut and poor gameplay. To counteract this, trade bodies could establish their own independent departments employing experienced testers and gamers to evaluate the quality of new software. The outcome would enable developers to be given an independent assessment of their games

Slashdot Top Deals

Math is like love -- a simple idea but it can get complicated. -- R. Drabek

Close