Microsoft Shows Progress Toward Real-Time AI-Generated Game Worlds (arstechnica.com) 21
An anonymous reader quotes a report from Ars Technica: For a while now, many AI researchers have been working to integrate a so-called "world model" into their systems. Ideally, these models could infer a simulated understanding of how in-game objects and characters should behave based on video footage alone, then create fully interactive video that instantly simulates new playable worlds based on that understanding. Microsoft Research's new World and Human Action Model (WHAM), revealed today in a paper published in the journal Nature, shows how quickly those models have advanced in a short time. But it also shows how much further we have to go before the dream of AI crafting complete, playable gameplay footage from just some basic prompts and sample video footage becomes a reality.
Much like Google's Genie model before it, WHAM starts by training on "ground truth" gameplay video and input data provided by actual players. In this case, that data comes from Bleeding Edge, a four-on-four online brawler released in 2020 by Microsoft subsidiary Ninja Theory. By collecting actual player footage since launch (as allowed under the game's user agreement), Microsoft gathered the equivalent of seven player-years' worth of gameplay video paired with real player inputs. Early in that training process, Microsoft Research's Katja Hoffman said the model would get easily confused, generating inconsistent clips that would "deteriorate [into] these blocks of color." After 1 million training updates, though, the WHAM model started showing basic understanding of complex gameplay interactions, such as a power cell item exploding after three hits from the player or the movements of a specific character's flight abilities. The results continued to improve as the researchers threw more computing resources and larger models at the problem, according to the Nature paper.
To see just how well the WHAM model generated new gameplay sequences, Microsoft tested the model by giving it up to one second's worth of real gameplay footage and asking it to generate what subsequent frames would look like based on new simulated inputs. To test the model's consistency, Microsoft used actual human input strings to generate up to two minutes of new AI-generated footage, which was then compared to actual gameplay results using the Frechet Video Distance metric. Microsoft boasts that WHAM's outputs can stay broadly consistent for up to two minutes without falling apart, with simulated footage lining up well with actual footage even as items and environments come in and out of view. That's an improvement over even the "long horizon memory" of Google's Genie 2 model, which topped out at a minute of consistent footage. Microsoft also tested WHAM's ability to respond to a diverse set of randomized inputs not found in its training data. These tests showed broadly appropriate responses to many different input sequences based on human annotations of the resulting footage, even as the best models fell a bit short of the "human-to-human baseline."
The most interesting result of Microsoft's WHAM tests, though, might be in the persistence of in-game objects. Microsoft provided examples of developers inserting images of new in-game objects or characters into pre-existing gameplay footage. The WHAM model could then incorporate that new image into its subsequent generated frames, with appropriate responses to player input or camera movements. With just five edited frames, the new object "persisted" appropriately in subsequent frames anywhere from 85 to 98 percent of the time, according to the Nature paper.
Much like Google's Genie model before it, WHAM starts by training on "ground truth" gameplay video and input data provided by actual players. In this case, that data comes from Bleeding Edge, a four-on-four online brawler released in 2020 by Microsoft subsidiary Ninja Theory. By collecting actual player footage since launch (as allowed under the game's user agreement), Microsoft gathered the equivalent of seven player-years' worth of gameplay video paired with real player inputs. Early in that training process, Microsoft Research's Katja Hoffman said the model would get easily confused, generating inconsistent clips that would "deteriorate [into] these blocks of color." After 1 million training updates, though, the WHAM model started showing basic understanding of complex gameplay interactions, such as a power cell item exploding after three hits from the player or the movements of a specific character's flight abilities. The results continued to improve as the researchers threw more computing resources and larger models at the problem, according to the Nature paper.
To see just how well the WHAM model generated new gameplay sequences, Microsoft tested the model by giving it up to one second's worth of real gameplay footage and asking it to generate what subsequent frames would look like based on new simulated inputs. To test the model's consistency, Microsoft used actual human input strings to generate up to two minutes of new AI-generated footage, which was then compared to actual gameplay results using the Frechet Video Distance metric. Microsoft boasts that WHAM's outputs can stay broadly consistent for up to two minutes without falling apart, with simulated footage lining up well with actual footage even as items and environments come in and out of view. That's an improvement over even the "long horizon memory" of Google's Genie 2 model, which topped out at a minute of consistent footage. Microsoft also tested WHAM's ability to respond to a diverse set of randomized inputs not found in its training data. These tests showed broadly appropriate responses to many different input sequences based on human annotations of the resulting footage, even as the best models fell a bit short of the "human-to-human baseline."
The most interesting result of Microsoft's WHAM tests, though, might be in the persistence of in-game objects. Microsoft provided examples of developers inserting images of new in-game objects or characters into pre-existing gameplay footage. The WHAM model could then incorporate that new image into its subsequent generated frames, with appropriate responses to player input or camera movements. With just five edited frames, the new object "persisted" appropriately in subsequent frames anywhere from 85 to 98 percent of the time, according to the Nature paper.
Gill Bates said so (Score:4, Funny)
What's wrong with just pre-generating worlds? Is this a solution looking for a problem?
640 worlds should be enough for anyone!
Re: (Score:2)
What's wrong with just pre-generating worlds? Is this a solution looking for a problem?
640 worlds should be enough for anyone!
I was kinda wondering the same thing. I don't think the problems with modern games are going to be fixed by having the worlds generated by the computer. If the gameplay sucks, the mechanics suck, and the story and characters are incoherent, having an auto-generated world isn't solving anything except making the studios feel better because they don't have to worry about anybody getting the same game twice. Maybe that helps replayability, but does anybody care about whether a game is worth playing again if yo
Re: (Score:2)
Re: (Score:2)
Get creative. Googles Demo could generate full games. Adding generation like maps only helps to keep it consistent, as you can't rotate 360 in Googles Demos without the world changing, but the limitation is easy to lift by just adding a bit of normal storage that gets filled as the AI generates the world. You've seen an immovable object? Store that. You've picked up something to your inventar? Keep track of that instead of hoping the AI remembers what you picked up 15 minutes ago. Then condition the AI not
Re: (Score:2)
This things just generating video. Its not generating a UE5 project with properly mapped out levels sand shit, its just.... video
Re: (Score:2)
The interesting point is, that a good enough fake is the same as a game. They use the previous frames (video) and input to generate the next frame. The network interpolates a plausible game frame, allowing things like walking around in the game. Without long term memory you don't have a consistent map, but that's where you just need to add some memory. Change the input from (20 frames, input) to (20 frames, input, map seen so far) and then the next frame will more consistent with the map seen so far. It's a
Re: (Score:2)
640 worlds should be enough for anyone!
If we have to be fair, Microsoft's MS-DOS didn't have a 640Kb limitation. The IBM's PC-DOS did.
Some computers running MS-DOS could access 896Kb of conventional memory or even more.
Only a limited genres this is useful for... (Score:2)
This is only useful for a limited amount of genres. For example, dungeon crawls except in a MMO, for example, World of Warcraft having gates to random worlds which open up that have specific parameters. Maybe some survival horror games like something based in the Backrooms where one never knows what is on the next level that they have to deal with, be it running through snow in mountains, to going through tunnels made from gallium fast enough to not melt the tunnel around someone, etc.
There are limited ar
Re: (Score:1)
Such can be "faked" with traditional methods by generating random connection graphs and world parameters such that a world is never same since the chance of say 7 parameters being the same is effectively nil.
The parameters may be say average thickness and height of walls, number of warlocks per castle, friendliness of trolls, brick texture, avg. tree height, princess bra size*, magic spell duration, etc. (Diff world types maybe have diff parameters.)
* You know, things that matter to guys. Have a moob-mode
Re: (Score:1)
You win 2 neckbeard cards, and free Zoloft.
Rookie mistake. (Score:5, Insightful)
For games the goal usually isn't "endless levels", it is "fun levels".
Yes, there are some genres and styles where generating infinitely many levels is what players want. They're uncommon, and generally struggle to attract a following.
Most games focus on fun, they focus on "imperfectly balanced" design where each option has strengths against others yet also come with weaknesses, levels are thoughtfully considered with level goals, gameplay goals, overlapping objectives. A balance of positive and negative spaces, visual flow to drive players toward interesting areas, and story lines that fit with interest. If there is a story, it needs to integrate with the stories being told. Design is about player choices, aesthetics, and variety.
While AI may be able to tell stories and make virtual worlds, it's the difference between a machine generated novel versus a story written by a bestselling author. It might get to the point where it is able to create them, but I'd have no interest in playing them.
Re: (Score:3)
I think that's exactly what they'll be good at. Subjective is hard, objective is easy.
Creating a believable 'fun' world is a subjective problem but they're getting there.
Crunching the numbers to find the right balance of too easy/too hard etc is a much more objective calculation. It will be able to balance on the fly, no more coarse sliders, it will be able to see where you spend time, what you look at what you find interesting and what you don't and tailor to that. It's still a long way off but it seems
Re: (Score:2)
For games the goal usually isn't "endless levels", it is "fun levels".
Yes. That is the reason I stopped playing Halo: Too much boring repetition.
of course (Score:2)
By collecting actual player footage since launch (as allowed under the game's user agreement), Microsoft gathered the equivalent of seven player-years' worth of gameplay video paired with real player inputs.
Not a surprise, just a reminder that even when you pay to play a game, they still want to make more money off of you. They're nice like that!
Basically (Score:2)
Hello Games has beaten them by a decade or so, with the incredible fun and ginormous set of universes in No Man&s Sky. And most likely is still way bigger than the Microsoft progress. But hey, fret not: Microsoft can still learn a dozen things with Light No Fire, whenever it is out.
Point it at Second Life geometry (Score:2)
Yeah, it' still around.
SL has probably the largest unified collection of 3D geometry ever created.
Microsoft should point it at SL, use SL geometry as low-res hints, and use MS AI as the renderer.
Why decode video when the geometry has better metadata, especially on interiors?
It's so much more information richlooking at the data itself, instead of pure interpretation.
Second Life really doesn't have a point, other than being there, so it's a good fit for this.
And it doesn't have to reanimate avatars, just use
I checked the stats and (Score:2)
I'm not saying they're lying but...
They might be leaving out that the 7 years of real time player data was actually collected over 4 realtime years, and this game has an average player count of zero to two... with a release peak of almost 1.5k.
I suspect they don't have the variation in the dataset they're suggesting they do.
Dumb, mixes rendering problem into generation. (Score:2)
Congratulations, you invented Rogue-likes (Score:4, Insightful)
Such an innovation has never happened before.
The only reason you would need a "real-time AI-generated" world would be for gambling purposes to well and truly screw over the victims, err, I mean *players*. But we already have ways of doing that. I guess this gives plausible deniability because they can blame the AI.