Upscaling Retro 8-Bit Pixel Art To Vector Graphics 325
An anonymous reader writes "Two researchers — Johannes Kopf from Microsoft, and Dani Lischinski from The Hebrew University — have successfully created an algorithm that depixelizes and upscales low-resolution 8-bit 'pixel art' into lush vector graphics. The algorithm identifies pixel-level details (original paper — PDF) to accurately shade the new image — but more importantly, the algorithm can create smooth, curved contour lines from only-connected-on-the-diagonal single pixels. At long last, we might be able to play Super Mario Bros. on a big screen without stretching our beloved plumber's pixels to breaking point. You really must look at the sample images." Scroll down in the paper to see how their technique stacks up against some others, including Adobe's Live Trace.
Wonderful! (Score:4, Interesting)
Re: (Score:2)
An interesting concept, remastering old games. Sega started doing that a few years back with graphical updates of their classic games for the PS2, but I think they were only on sale in Japan. Monkey Island had this treatment recently too, and you can switch between original and updated graphics at any time in the game.
I think some games lend themselves to being updated because their graphics were hand painted and digitised. A lot of point-and-click adventure games were made that way. Other games' graphics w
Re: (Score:2)
This filter looks quite similar to the existing HQnX series of filters, but with even more aggressive smoothing (from the paper, they appear to use a similar quantisation matrix approach then vector-smooth the result). As the paper shows, the results are fine for a few cases of a single isolated sprite, but for a whole display it looks really nasty, even worse than HQ4X [imageshack.us].
My personal preference
Re:Yawn (Score:4, Informative)
Re: (Score:2)
The hq2x and hq4x filter are all specially made to handle pixel art shapes. Even the generic filters are selected for their ability to upscale pixel art well.
Re: (Score:2)
Have a look at the paper [nyud.net], if you haven't yet. Their algorithm is impressive, but even in the samples they chose, some things get better results (IMO) with hq4x (Fig. 10, Mario Bros. scene, perhaps?) or Photo Zoom 4 (Fig.9, That warrior with the sword).
Comment removed (Score:5, Informative)
Re: (Score:3, Insightful)
Re: (Score:3)
That said, I would like to compare the final results against a best of breed scalar like hq4x for the same final output resolution.
Re: (Score:2)
Re:Yawn (Score:4, Funny)
Too late, you already did.
-dZ.
Re: (Score:2)
The paper actually mentions them, compares how they work and describes what was improved.
Re: (Score:2)
Re: (Score:2)
Re: (Score:3)
Except that this algorithm vectorizes, instead of just upscaling the image with bigger pixels and smooth edges.
Except that this algorithm is intended to accentuate the information that is there. Their thesis is that, on such low-resolution images, each individual pixel may convey important info
Re: (Score:2)
Re: (Score:2)
What stop the emulator to have 2 display: the hidden one with original graphics where collision detection is done, and the HD one with the nice display?
Re: (Score:2)
You could vectorize the original graphics, edit the vector graphics and then finally export to a higher-res raster format.
Re:Wonderful! (Score:5, Informative)
It is. This has nothing to do with that, but is about giving a nicer representation to information that is there. All this does is removes jagged corners that result from up-scaling.
Sorry, but that's just plain not going to work.
Re: (Score:3)
I actually did RTFP and you are not quite as right as you blithely assert. The parent poster was correct.
This has nothing to do with that, but is about giving a nicer representation to information that is there.
Actually, the algorithm makes heuristic decisions which add information based on an understanding of the human visual system and assumptions about the problem space (it works well on hand-drawn cartoons, but not on down-sampled photos). For instance, they have heuristics for deciding which neighboring pixel fields should be attached to a pixel "island" by one of its four corners. In the original image
Re: (Score:3)
"Just create a version of this specialized in different subjects, e.g. faces."
Ok, you have done a pill that mitigates headhache.
Just create a version of this specialized in different subjects, e.g. cancer.
Re: (Score:3)
Reconstructing image details accurately is still impossible and always will be.
Why should that be always impossible? There is nothing all that hard in it, humans can do it pretty easily. The problem with these algorithms is that they are all pretty primitive and only really take local pixels into account, not the whole image, thus in the example in the article you have edges in the back and belly of dolphin, which are trivial to tell that they should be smooth, for a human at least. The highlights on th gradients are also wrong, as the algorithm apparently doesn't take them into prope
Re: (Score:3, Informative)
From TFA : "The other problem is that the Depixelizing Pixel Art approach always smooths images, even when an object shouldn't necessarily be smooth. For example, are Space Invaders really meant to be cute and round? Maybe, in the creator's eye, they had long, angular, razor-sharp mandibles and straight-out antennae! "
They a
Re: (Score:3)
How so? A video is just a series of fixed pictures, one after the other.
Yes, but it are pictures of the same object in motion. Super simplified example: Imagine you have a one-pixel camera. A single frame of video of that will tell you nothing by itself, as its just a pixel, but if you take the camera, move it around in the right pattern and add the images together with the correct offset you can get a full 2D picture of whatever you are photographing. Same can be done with a 2D camera, you have to reconstruct how the images are shifted relative to each other and then add them
PDF slashdotted (Score:2)
Re: (Score:2)
Yeah, the PDF is dead, as is their site where I was hoping to view some more examples [johanneskopf.de]. Maybe in a while...
Re:PDF slashdotted (Score:4, Informative)
http://johanneskopf.de.nyud.net/publications/pixelart/paper/pixel.pdf
Don't tell me people have never heard of the coral cache
Re: (Score:2)
Don't tell me people have never heard of linking. [nyud.net]
Not just the PDF, here's his website:
http://johanneskopf.de.nyud.net/publications/pixelart [nyud.net]
Re: (Score:2)
Re:PDF slashdotted (Score:5, Informative)
Re: (Score:2)
Hooray! Thanks.
Re: (Score:2)
Mirror: http://easy-iphone-unlocking.com/download/pixel.pdf [easy-iphon...ocking.com]
When mirroring something, try to do so on a site that won't get slashdotted even faster than the original.
I know the intent was good, but you only create more frustrated people, who now have had two links time out instead of just one.
Re: (Score:2)
Hosted via Google Docs:
here [google.com]
Re: (Score:2, Insightful)
Since I can't read the PDF either, I'm forced to infer from the dolphin picture that it's raining cheese and the dalai lama is coming over for lunch.
WTF original sketches? whose ass did you pull that from?
Re: (Score:2)
Re: (Score:2)
HTML version of the PDF via Google
http://webcache.googleusercontent.com/search?q=cache:15iKIgSAkDwJ:johanneskopf.de/publications/pixelart/paper/pixel.pdf [googleusercontent.com]
Re: (Score:2)
which doesn't have the images
Great. (Score:3)
Re: (Score:3)
Re: (Score:2)
Yes, a regular expression joke (Score:5, Funny)
Re: (Score:2)
What difference does it make? The poster is a soulless monster from the lowest depths of hell.
Re:Great. (Score:4, Funny)
Re: (Score:2)
One of the things holding me back from forking out the 100$ for the XBOX360 XNA license was my inability to make nice graphic images larger than your traditional ancient sprite types. (I am pretty good at making sprites, or at least I used to be. Been awhile since I have dabbled with it.)
I feel I could do some fantastic things with an algorithm like this.
Re:Great. (Score:5, Interesting)
Besides, do you think 1080 HD will be the resolution of choice 20-40 years from now? I imagine we'll be looking at wall-sized TVs at some point and 1080 pixels will look awfully blocky on a 10 foot wall. Sharp has already released a 7x680 x 4,320 pixel 85-inch screen, [mashable.com] how long before 4320 is the new 1080?
Re: (Score:2)
how long before 4320 is the new 1080?
Not too damned long. At a recent show, multiple panel manufacturers were showing off both 7" and 10" panels at 2560x1600. Nine of those 10" panels glued together gets you a very sweet 30" display at 7680x4800. Next year. I don't want to think about the price, though.
Re:Great. (Score:4, Informative)
The maximum theoretical human eye resolution (limited by aperture and focal length) is about 0.35mm per line pair at 1m. For most people with "normal" eyesight, the limit is closer to 0.7 mm at 1m. So for a TV that's 6 feet away and 2 feet tall, there's no point to a vertical resolution higher than ~1800 pixels, and 1080 is more than what most people can perceive. 4320 pixels at 6 feet makes no sense unless your TV is a minimum of 5 feet tall, and for most people, 15 feet ;) So yes, it really does take wall-sized TVs for that sort of resolution to even be useful. A 7" panel (~4" tall) with 1600 pixels vertical resolution? That's 0.06mm per pixel. The maximum theoretical distance in which that could be useful is under 7 inches, and more practically, closer to 3 inches away from the viewer. If you want that kind of resolution, you might as well make it into a planetarium ;)
Re: (Score:2)
You are assuming that the person stays perfectly still. When watching TV your head moves slightly all the time (breathing etc.) and this motion allows you to see a difference between resolutions above and below the thresholds you set out. It only works with high contrast edges though.
You can think of it a bit like a camera that takes two photos. For the second photo the camera is moved 0.5 pixels to the left, so it is now sampling between the pixels in the first image and doubling its horizontal resolution
Re: (Score:2)
And 15 years ago a video card capable of 1920x1080 didn't even exist to consumers. In fact in 1996 the first consumer 3D gaming solution was released with an amazing 4 megabytes of RAM! [wikipedia.org]
A good rule to follow is to never compare the hardware you're using now to what you'll be using 10 years from now. I have no doubt we'll
Re: (Score:2)
There were plenty of graphics cards capable of 1280x1024 and 1600x1200 back in those days. Heck, with 4 MiB of video RAM (4,194,304 bytes) you could use 12 bits per pixel and still have enough memory for 1920x1080 (2,073,600 pixels). Not exactly truecolor but it does allow you to use a palette of 4096 colors.
Then there are of course issues such as sync frequencies but I suspect such things could be worked around pretty easily (back then you pretty much wanted 72+ Hz vertical sync unless you enjoyed pain).
Re: (Score:2)
I can easily picture taking this one step further. With a couple additional steps, your sprite drawer should be able to recognize matching parts in other frames of the same sprite. You could then allow a customizeable time period -- say, 1/4 to 1/2 second -- where there's a linear interpolation between the two SVG graphics. Aka, when Mario jumps, his legs don't just instantly appear apart; they move there. A smart drawing algorithm could even intelligently determine the appropriate time period to have f
Re: (Score:2)
Re: (Score:2)
You're kidding, right? This is allowing us to play all those games from our youth without them looking like blocks at 1080 HD.
No, and no.
The first statement is false because most of "all those games" have sprites and other graphics with hard angles, which this algorithm can't handle; everything becomes a spline. I'm sure there are people who would appreciate it if games got transformed so it looks like the fighters hit each other with soft nerf bats instead of swords, or teeth stopped looking sharp and scary. I'm not one of them.
The latter statement is false because whether a bitmap image looks like blocks doesn't depend on the
Re: (Score:3, Insightful)
Kick ass! *I* do pixel art! (Score:2)
Being able to convert low resolution pixel art into nice vectors like that is kick ass. I wish they had released a utility to try it out on other materials. (like my own..)
I do some of my best work in a work envelope around 100x100 pixels.
Re: (Score:3)
Re: (Score:2)
...which is exactly what Inkscape uses for tracing, except it's got a nice UI and editing capabilities for the result...
Paper mirror (Score:5, Informative)
My verdict: Yeah, it looks all nice and smooth, but with all upscaling, it's basically interpolating data. The original just didn't have that much detail, so you can only get so much out of it. Sometimes it works, sometimes it doesn't.
(Oh, and it makes all text look pretty bad. Kinda Comic Sans-y, if I can say that without invoking instant hate.)
Impressive, interesting flaw with the keyboard pic (Score:5, Insightful)
EvilHom3r over on Reddit seems to have mirrored the paper (as images) here [imgur.com].
My verdict: Yeah, it looks all nice and smooth, but with all upscaling, it's basically interpolating data. The original just didn't have that much detail, so you can only get so much out of it. Sometimes it works, sometimes it doesn't.
(Oh, and it makes all text look pretty bad. Kinda Comic Sans-y, if I can say that without invoking instant hate.)
I know what you mean but it's pretty impressive nonetheless. The flaw I found interesting was in the keyboard image. I'm sure the intention here is square keys, but their algorithm made all the keys round. Some things in a pixelated image should not be smoothed but without human context that's a very hard call to make.
Re: (Score:3)
Also, the pins of the chip was a failure, becoming sawtooths instead of parallel pins.
The problem with this algorithm, like many others, is that it's subject to an extension of the Pareto principle. In 80% of cases, it may be an improvement, but that doesn't help if it makes the result worse in 20%. Cause it's the failures we notice, and judge quality by.
Algorithms like this also don't work well for animated sprites. Our imagination can fill in the gap between Mario holding the hammer above his head and
Re: (Score:2)
With the doom face example, their algorithm falls down slightly. Basically, it's almost as if they have to decide between a sharp edge/curve, or a gradiated colour section, like as if the decision is a dichotomy. I'm sure the math is complex, but they should always strive towards the most general all-encompassing technique, rather than try to build more arbitrary 'stages'. A sharp edge could be said to be a gradient, but a very sharp changing one, and vice versa...
Re:Paper mirror (Score:5, Informative)
Re: (Score:2)
Yes, but I can imagine many would try to solve the issue by adding a kludge creating more special cases, rather than approach the problem with the better goal of unifying the alleged difference between a sharp, or blurred edge. A matter of degree rather than type/quality.
Re: (Score:2)
Looking at the images, I don't think it'd be that hard to detect the level of antialiasing in the image. In fact, they're already kind of doing it when they decide whether to blend areas or to leave them distinct. To me, that leaves the obvious solution: rather than a binary between "sharp" and "blurred" being determined by the level of color difference, you use the level of color difference (with a particular focus on matching hue) to decide the maximum transition distance. So, for example, a stark colo
Re: (Score:2)
They point that out in the article indeed, the face is added as an example of where it goes wrong. Their algorithm simply doesn't work well on anti-aliased images (scaled-down photos), it's primarily designed to handle typical hand-made sprites with much sharper edges and colour differences.
Re: (Score:2)
You denigrated comic sans by the transitive property of bad. No hate from me.
PDF mirror (Score:2)
CORAL link to the PDF (Score:5, Informative)
It looked like it was on the verge of a Slashdotting, so I CORALized it.
http://johanneskopf.de.nyud.net/publications/pixelart/paper/pixel.pdf [nyud.net]
Very interesting, and quite effective.
Re: (Score:2)
And so the Coral link is now also slashdotted... :P
Flash image tracing (Score:2)
This looks a lot like what you might end up with if you use the image tracing function in Flash to convert bitmaps into...vectors.
The best I've come across (Score:2)
Re: (Score:2)
I wonder if it would be feasible to cache these results.... by definition these sprites don't change appearance very often. If you could detect that sprite A is being drawn at point X,Y, and just draw the cached and pre-generated high-res vector art in the right location, would that be any more practical for real-time rendering?
Re: (Score:2)
The Real Test (Score:2)
The real test for the Kopf-Lischinski algorithm will be how it can handle Nethack [bbspot.com].
It's easier with 8-bit (Score:2)
This looks good because it is specialized for 8-bit manually created graphics. 8-bit graphics with a broad palette has little shading, so it's usually unambiguous which pixel attaches to which adjacent pixel. Vectorizing 24-bit color images, or 8-bit images from photographs, requires more guesses about how wide things are and where the edges are supposed to be.
Nice if you really want to play 80's games in higher resolution.
But the whole point is... (Score:3, Interesting)
This project is technically interesting but as art it completely misses the point. Not every piece of human creation needs to be updated, upscaled, "improved" and redone.
I know the originals [imgur.com] make more sense to me.
On the next CSI: 8-bit (Score:3, Insightful)
Would be great for LCD upscaling in GPUs! (Score:3)
NVIDIA and AMD should implement this and related algorithms for their LCD upscaling!
The algorithm looks like it could be parallelized onto the GPU easily enough, and doing it transparently at the video card driver level would enable PC gamers to play classic games on large LCDs without the thumbnail sized pixels.
Just imagine what this scaling algorithm could do for classic low-res games like Diablo.
This is why closed-source sucks... I'd have done it myself by now if I had access to the AMD GPU Windows driver source code!
MAME support (Score:2)
Can someone link to the MAME build that supports this? ....
What do you mean they're not coding this directly into MAME source?!?!?!
Finally! (Score:3)
Commodore 64 Porn arrives in high-res
Sample image is not 8bit source (Score:2)
Re:Sample image is not 8bit source (Score:4, Informative)
Yeah, it's from Super Mario World, i.e. the SNES - and on the SNES [wikipedia.org] "Sprites can be 8 × 8, 16 × 16, 32 × 32, or 64 × 64 pixels, each using one of eight 16-color palettes", which would make the dolphin a 4-bit sprite.
Re: (Score:2)
They mention that form of approach in the introduction and reference Hqx specifically in Section 2: Previous work.
As mentioned two posts up there's an image of the paper (so text search blows) here: http://imgur.com/a/gRXPJ [imgur.com]
...such as 2001's HQ3X (Score:2)
...such as 2001's HQ3X (http://www.hiend3d.com/hq3x.html) which both awesome and as common as dirt ten years later.
Article author Sebastian Anthony looks like a tech nitwit with the "at long last, we might be able to play Super Mario Bros. on a big screen without stretching our beloved plumber's pixels to breaking point" comment.
Re:This is news? Anyone else run a NES emulator? (Score:5, Informative)
RTFP - they list the various existing upscaling algorithms, and why they're not perfect:
Several later algorithms are based on the same idea, but use more sophisticated logic to determine the colors of the 2x2 block. The best known ones are Eagle [Unknown 1997], 2xSaI [Ohannessian 1999], and Scale2x [Mazzoleni 2001], which use larger causal neighborhoods and blend colors. Several slightly different implementations exist under different names, such as SuperEagle and Super2xSaI. An inherent limitation of all these algorithms is that they only allow upscaling by a factor of two. Larger magnication canbe achieved by applying the algorithm multiple times, each time doubling the resolution. This strategy, however, signicantly reduces quality at larger upscaling factors, because the methods assume non-antialiased input, while producing antialiased output.
The latest and most sophisticated evolution of this type of algorithms is the hqx family [Stepin 2003]. This algorithm examines 3x3 pixel blocks at a time and compares the center pixel to its 8 neighbors. Each neighbor is classied as being either similar or dissimilar in color, which leads to 256 possible combinations. The algorithm uses a lookup table to apply a custom interpolation scheme for each combination. This enables it to produce various shapes, such as sharp corners, etc. The quality of the results is high. However, due to its strictly local nature, the algorithm cannot resolve certain ambiguous patterns and is still prone to produce staircasing artifacts. Lookup tables exist only for 2, 3, and 4 magnication factors.
whereas what they did here is:
Our goal in this work is to convert pixel art images to a resolution-independent vector representation, where regions with smoothly
varying shading are crisply separated by piecewise-smooth contour curves.
Seriously, just look at the whale image linked from TFS.
Re: (Score:2)
I'm pretty sure that's a dolphin, actually... OK, so maybe their algorithm isn't perfect yet ^_^
Re: (Score:2)
If you read the paper (which is presently slashdotted) you'll see that they compare their results to existing pixel-smoothers, and theirs definitely look better in pretty much every case.
So it's not "Hey, we invented this new thing no one ever thought of!" so much as it's "Hey, we invented this better way of doing something that already exists."
I, for one, look forward to having better graphics in emulators.
Re: (Score:2)
I've posted the CORAL link to the PDF under the original article.
Re: (Score:2)
Re: (Score:3)
Yes, this is news. Most scientific and technological advancement is iterative improvement. This paper describes an improvement in the state-of-the-art for vectorizing pixel art (which is not the same as pixel-smoothing, though a vectorized image is usually re-rasterized for display).
Seriously, read the paper. The algorithm described will require some basic understanding of splines and graph alg
Re: (Score:2)
This is news? Anyone else run a NES on those dozen or so emulators that already has pixel-smoothing options?
One suspects that if /. posted an article about a new engineering method that will allow us to finally build a cheap, working space elevator, you'd post, "This is news? Haven't we had ways of making cables for centuries?"
Re:Alright... (Score:4, Informative)
Re: (Score:3)
No, not really. From the paper:
Various commercial tools, such as Adobe Live Trace [Adobe, Inc.2010] and Vector Magic [Vector Magic, Inc. 2010], perform automatic vectorization of raster images. The exact nature of the underlying algorithms is not disclosed, however, they generally do not perform well on pixel art images, as is evidenced by the comparisons in this paper and in our supplementary material.
Re: (Score:3)
Re: (Score:2)
No it would make it worse, as pixels look better than cartoonish vectors.
Re: (Score:2)
let me patch you up until you can get back too the first grade.
Please return your grammar Nazi card on the way out.
Re: (Score:2)
And yet you didn't.
Re:Mostly good for Mario (Score:4, Funny)
Re: (Score:2)
Re: (Score:2)
I am stumped as to why everybody seems to think it can be applied in emulators with the same ease as pixel resamplers where.
If this would be used properly, it should evaluate each bitmap *on its own* before it is part of the screen, and then instead draw the resulting vector image to the screen, ofcourse matching the pixel coordinates. You cannot simply apply this to a any screenshot or video as it would give undesired results most of the time.
I would very much like to see a HD youtube video that shows how