Star Trek Fan Tests Using AI Upscaling On ‘Deep Space Nine’ Clips

A technologically inclined Star Trek fan known as “Captain Robau” was experimenting with using machine learning algorithms (which is commonly called “AI” in marketing buzzwords) to upscale art assets on the video game Final Fantasy VII. He had such good results with it, that on a whim he decided to do some tests on his favorite Trek show, Deep Space Nine. There are full details on his blog, he also plans to do a follow up with even more technical details in the coming days. The results show promise, and we’ve embedded a few examples below.

As many fans know, there are two Star Trek shows stuck in standard definition, Deep Space Nine and Voyager, and there’s no indication that CBS will do what they did with The Next Generation, a total remaster from ground up, due to high cost and changing markets.

Like many fans, Captain Robau has lamented the fact that DS9 is stuck in the 1990s, but then he thought of using AI upscaling based on his experiences with Final Fantasy:

Just like Final Fantasy 7, of which I am upscaling the backgrounds, textures, and videos in Remako mod, DS9 was also relegated to a non-HD future. While the popular Original Series and The Next Generation were mostly shot on film, the mid 90s DS9 had its visual effects shots (space battles and such) shot on video.

While you can rescan analog film at a higher resolution, video is digital and can’t be rescanned. This makes it much costlier to remaster this TV show, which is one of the reasons why it hasn’t happened.

This is where neural networks could come in, I thought. With tools like AI Gigapixel, I knew it might be possible the low definition frames of DS9 can be scaled up to a higher definition such as 1080p or 4K. It would never be the same as proper remastering, but it would a step in the good direction.

For a first test he tried out still frames from the the beloved episode “The Visitor.” Click to open each image in a new tab, so you can see the differences in details for yourself.

Original image from “The Visitor”

Upscaled image from “The Visitor”

Testing the process on actual video

Captain Robau then decided to use the opening five minutes of the fan favorite season six episode “Sacrifice of Angels” to test the machine learning’s algorithms on moving pictures, not just still frames. You can see the machine learning upscaling at work below in a side-by-side comparison video. Be sure to watch it full screen and at 1080p.

DS9 opening credits in UltraHD

And to get really ambitious he tried out upsampling the opening credits to 4k. As the author notes on his blog:

This nearly melted my computer, as it is a lot more intense to upscale than 1080p so I’ll stick to this single video for 4K examples of DS9 Enhanced.

Not a substitute for a proper remaster

As Captain Robau notes, this is not really going to achieve the same results that a proper remastering would. Upscaling is  making educated guesses at deriving higher quality image data from a lower quality source. The example videos show just how variable the end results can be, in some scenes there’s a noticeable uptick in detail, and others it’s nearly as soft. That’s primarily due to the fact the source is simply lacking information to do much with. The source is still limited to the poor color (chroma) and brightness (luminance) information inherent in recording things to NTSC video tape. There are also artifacts from the early CGI that can be seen, since they exist in the source, they’re passed on to the upsampled copy too.

While the results can be quite impressive, even with the best algorithms at work, the old saying “garbage in, garbage out” still holds true. While not a perfect apples-to-apples comparison, check out the AI upscaled image of DS9 and compare it to when the space station was shown on TNG in season 6 as part of “Birthright,” Next Gen of course was given a full remaster that started in 2012.

The Deep Space Nine model has a ton of detail, much of it was not clearly seen before Next Gen was remastered. One of the most obvious reveals from the remaster is that there’s periodic yellow detailing on the station, which was previously lost — turned into a muddy shade of grey in standard definition. Because the AI upscale only has the old standard definition version to use as a reference, the yellow detail is not present.

AI upscaled footage of the space station

CBS remastered footage of the space station from TNG “Birthright”

What’s next?

Captain Robau has made it clear this was just a test and he has no intention of trying to apply this type of process to full episodes. And for the reasons outlined above, it is clear that this process would not – and could never – achieve the kind of results seen with the Star Trek: The Next Generation remastering, which was a meticulously labor-intensive project. However, the tests show that marginal improvements could be made to the standard definition version of the show. Feeding the learning algorithm more examples and, ideally, higher quality images too, could in theory help narrow the gap a bit as well. Perhaps in the coming years, it would be possible for CBS to use a process like this for both Deep Space Nine and Voyager, which would be significantly less expensive than the process used for TNG. This enhanced product would not really be a true HD release but could add value for syndication and streaming sales, especially in the years to come.

109
Leave a Reply

23 Comment threads
86 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
38 Comment authors
  Subscribe  
newest oldest
Notify me of
DetroitBORG

I would truly be more interested in a DS9 restoration on CBS All Access than another new Star Trek series.

dennycranium

Love your screen name….

Dlope67

Yup, I’d be totally down with that. (and i’d buy it too). Also, totally love the screen name.

Karl

Not perfect, but not bad. The videos looks solid enough on my phone, at least

Professor Spock

The visual effects look pretty good in HD. It needs a bit more work.

Ambassador Sybok

The opening credits sequence looks beautiful, but then again, that Seasons 4-7 redo of the opening always looked nice. It’s Seasons 1-3 that looked rather rough in my latest re-watch of the series.

At least someone’s trying, though!

Snugglepuffxlz

I guess I wasn’t paying attention all this time. I thought they were going to remaster all the footage they were going to use in this documentary. I feel so let down right now.

FrostUK

Uh, this has nothing to do with the documentary.

John Shannon

Every clip of DS9 in the doc is being remastered in HD. They can’t likely crowd-fund the entire series (the way the coming movie was). Don’t be too disappointed. It may well be the spark that gets DS9 remastered. You never know. 😊

I’m excited to see it.

Clips are restored not remastered and we will never see the entire series in HD, money talks and profits even more. The TOS and Next gen did poorly in blu-ray sales TOS did better but next gen flopped in sales.I get it from a business standpoint. The cost was thru the roof to do those 2 series. You can’t expect a company to justify that cost when the profits won’t be there. To be fair the cost of one season was high, but the cost doing those series was real costly. Its a business and we as fans would want that, but unless they know the sales will be there at their price point ,forget it.Besides they say physical media is down , I don’t believe that but that’s what they think.

Gregg E.

What would have been nice is if they’d gone ahead with the HD-DVD + DVD ‘flippies’ for Seasons 2 and 3 of TOS after Toshiba pulled the plug on HD-DVD. At the least they should’ve made as many HD-DVD sets as they did for Season 1. Everyone who bought a Season 1 set would’ve wanted 2 and 3 to match. But instead, everyone who wanted all three seasons in HD had to wait for the Blu-Ray release then buy Season 1 a second time.

Disinvited

Gregg E.,

Why buy Season 1 again? Was their some sort of Mission Impossible self-destruct of the HD DVD players and discs because my player still works?

Disinvited

Gregg E.,,

“Was their …” should be “Was there …”

Arathorn

You’re in for a rough life if that’s what lets you down…

A34

Control is real.

Afterburn

This is exactly what I was talking about in a previous article, that I got shredded and insulted for. While this isn’t true HD, there may be a day coming when the same kind of tech can achieve better results, for a fraction of the cost of the kind of remastering required today.

Mark Z

There’s a bit of extra “quality” to be had in the original SD video, but not much. Noise can be reduced, a finer layer of grain can be added, edges can be sharpened a bit, fancy neural net upscaling can be applied, but the returns diminish very quickly. You just can’t create more actual detail without rescanning the film.

Afterburn

With current technology, no.

A34

20 years from now. Maybe.

Vulcan Soul

“You just can’t create more actual detail without rescanning the film.”

Yes you can! Check out some of the latest ML image results and you will be very astonished!

Jonathan

I look forward to time when AI Will be ale to transform original series or even the animated series into Discovery style of look :D

kmart

Jonathan,
Just turn down the brightness, add visual distortion by watching while holding a cigarette lighter and a piece of blue cellophane in front of your eyes, you’ll get your DISCOVERY look to ruin the earlier series just fine.

What are you talking about?

kmart

VS,
ML is what exactly? Is it some machine-vision style interpolation? If so, I don’t think the results would be consistently satisfactory at all.

Vulcan Soul

I mean machine learning, and GANs specifically. Granted, the technology is not there for *universal* application yet, but the basic idea is to feed tons of data (DS9 high res pictures, in this case) and then the algorithms will insert the necessary details based upon these inputs. They would add the yellow stripes to the station, for example, instead of doing a simple edge enhancement. These algorithms have improved so tremendously just in the past five years, that I don’t see why in another five years AI-assisted up-scaling of TV series won’t be feasible both financially and quality-wise. Check out letsenhance dot io for the beginnings!

Vulcan Soul

Can’t edit anymore – the link is letsenhance dot io

No you can’t .When blu-ray first came out they tried rescaning a film or use an old DVD master and the results were horrible. As they say, you can’t polish a turd. You have to go back the original 35 mm film and in the case of Next gen they had to redo the FX because they were done on video tape.Companies don’t think ahead. Christ we have 4K tvs and there pushing 8K already. Some TV networks at least locally are broadcasting in 720p!!CNN 720p where NBC 1080p. No 4K except netflix and amazon prime so why should movie studios think any different. They could care less.

Vulcan Soul

“When blu-ray first came out they tried rescaning a film or use an old DVD master and the results were horrible”.

We are obviously talking about the near-future and not the past… When Bluray came out, that’s like ancient history for technology!

Disinvited

Apparently there’s redundant copies of frames which could be used to reduce errors:

https://www.newscientist.com/article/mg12717313.900-technology-dallas-becomes-quality-viewing-/

True ,but again cost ,sales and profits are everything with these companies. As much as we love Trek and these series I don’t believe the market is there. The best thing to do is to let CBS and Paramount know we want this series in HD. If enough people do that maybe , just maybe they might do that. But as far as we know just to get these clips in HD they had to get a go fund me thing going, what’s that tell you ? As much as CBS and Paramount say Star Trek is there cash cow it’s BS. If it was this series would have been in HD along time ago.

Disinvited

Thomas P Vinelli,

Re: what’s that tell you ?

That if lower-tiered Hollywood studios, like Paramount and CBS don’t get creative in their production financing and take advantage of such new financing possibilities such as Go fund Me, they ain’t going to be here.

Allen

Deep space nine is fine the way it is. I would rather see Voyager get redefined imageries

Get in line. :-P

Billy

Me too! Would love to see Voyager get the treatment

albatrosity

No what really needs an overhaul is TAS !

Timbo

Albatrosity, I so agree with that. Filmation did the best they could, but lack of facial expression, and stiffness in general really takes it’s toll. TAS could potentially be transformed into something that would look MUCH more like a fourth season of TOS. It would all depend on how well done is was, but I’d sure be open to seeing what a dedicated team might come up with.

I believe it would be a reasonable compromise to remaster from film just the parts of the show that had no CGI visual effects, and work out CGI scenes with this smart upscaling technique. It wouldn’t be as shiny as TNG, but wouldn’t be much behind Enterprise’s own HD masters, which have a lot of CGI effects in SD in the first two seasons. That could prove to be cost-effective.

And you would pay how much for this wouldn’t be as shiny as TNG stuff.Nope if I have to pay close to 100 bucks a season at release time I want it done right.

Disinvited

Thomas P Vinelli,

Unfortunately, I’m old enough to remember when unauthorized 16mm film dupes of a syndicated reel distribution of a first series episode went for twice that, so I wouldn’t balk at that price. But I am a bit of a film archivist purist, so if they cut corners on that also being available to blow the wad on the new VFX, the value to me decreases.

dennycranium

This looked surprisingly good to me on my 4K monitor.
This exercise was to see what would happen if he applied the algorithms.
Thanks for sharing these, it was fun to see them.
I’d love to see a remaster of the series, just for streaming sake.
I’m not sure if there would be a return on investment for CBS.
For those of us who love the show we would probably pay to buy BluRay or give it a binge on our streaming service. I doubt it would draw in many new fans due to its 4X3 aspect ratio.
Watching anything on a 4X3 aspect ratio takes me out of the story.
I suspect new viewers would have the same response.

John Shannon

Awww, I’m in love with these HD clips. <3<3<3
C'mon CBS. DS9 and Voyager deserve the same love as TNG.

In your dreams ,sad to say , won’t happen.Sales were really bad with Nex gen, TOS was better.But Next Gen was the killer. If DS9 was to be in HD it would have happened already. Money talks and everybody else walks The cost of doing these series ,7 seasons about 22, 23 episodes nah ain’t going happen.

Disinvited

Matt Wright,

Does anyone know how DS9, back in the day, was converted to PAL for syndication to the UK and other PAL markets? PAL has better color and higher resolution than NTSC.

Vulcan Soul

I called this out a while ago that this will be the way how DS9 and Voyager get remastered in a few years! And no, it would not be the SAME as a proper re-compositing, but it could be just as GOOD! Have you guys seen how it is now possible to turn summer into winter and night into day in videos? And this is only the beginning! Especially if fed with large numbers of high-quality production stills and photos of DS9 (or just googling them on its own), a ML algorithm could deliver a very convincing result and fill in the blanks on its own, in a few years of time.

As Deckard said, ENHANCE!

kmart

Enhance has to be more than just turning a knob. You have to have more info to squeeze out of the image. The picture Deckard was enhancing had some kind of holographic data in it, because you could track as well as zoom, changing perspective to see things that wouldn’t be visible in a straight capture, so that is an excellent example of how the original capture sets the limits on what is possible later with the image (unless you’re tweaking to create new data, which would be cheating and/or massively expensive on a global scale.)

When you say, change night into day, all I’m remembering is how bad it looked when the Veridian star went out and they just darkened the day plate of Picard’s closeup … there are a lot of values to address when transforming an image or stripping it into a different environment, which is why so much of what was well-shot originally in A WRINKLE IN TIME turned to crap when they comped elements shot in one lighting situation into vastly different environments, against the intent when the shooting was first done.

Vulcan Soul

I think one point to remember is that this is not a forensic investigation and as such the Enhance analogy shouldn’t be taken at face value. What I was comparing is the idea of lossless zooming, but how we go about it is different. We do not need to “squeeze out more information” from the original but instead can add new information from other sources with real HD material – DS9 high rez shots, TNG’s Birthright and the coming DS9 documentary. What GANs do is “hallucinate” details into images based upon the images they have been trained on.

As for looking around the corners, have you seen the recent examples of reconstructing the original image based on shadows on a wall?

kmart

I did some interviews a few years back about the plenoptic capture processes, and even that sounds a bit too removed from conventional photography and cinematography to suit me, so these processes you cite, while of interest by themselves, sound … I dunno, inappropriate or unfair to apply to finished works. Then again, I’m no fan of TOS-remastered either, so in general I’m not in favor of anybody tampering with a finished work.

The one exception — and I said this back in the 90s — was I thought recompositing original FX elements using newer techniques would be okay, because you’re still limiting yourself to the tools of the times. The 21st century final cut of BLADE RUNNER had all the matte composite shots redone digitally using original elements (which the director himself had paid to have preserved — a far cry from the weird situation with TMP effects elements, where parties offer up radically different accounts about them), and that was a pretty transparent process that didn’t damage the look or change it in any significant way (the original work was so good that the recomps probably weren’t even necessary, something I don’t say about almost any other film outside of 2001.)

Vulcan Soul

Oh, so we are arguing the ethics. Granted, AI upscaling is not strictly only using the original work, but as described, using HD background material from the same show, isn’t it still much more faithful and respectful to the original than recomposition and even CGI replacements a la TMP or TOS could ever be? In fact TMP is a good example as it changed the original considerably AND the new effects itself are outdated already again. AI upscaling avoids both of these issues – the content stays exactly the same, and similiar to vectorization, the end result can be upscaled to any target, limited only by computer power.

kmart

Hmm. Well the TMP upgrade was out of date from the first render IMO. To have the necessary resolution, they’d have been better off animating high quality still photographs of the ship (a la 2001 for many shots) … the large-format stills shot at the end of TMP are amazing in their detail (I used to have a 16×20 photo that seemed to have more detail than I ever actually saw in the film), and if you shot the results in 35mm, you’d have been good to go for the next couple or three decades at least.

So if we proceed with the smart tech solution, do we get the cinematographer and VFX supe in to sign off on the look as being in keeping with their intentions? Or is it a producer’s call? And what sorts of rights are there in terms of the original work, and royalties? If it is a new image derived from the original work, are the royalties the same, or reduced?

I guess I’m seeing the glass as half-empty here, but that’s my kneejerk takeaway.

Tyrannicus

While this isn’t quite up to the level needed for an effects heavy scifi show like DS9 or VOY, this would be a great way to upres shows that otherwise wouldn’t or couldn’t be remastered to HD. My girlfriend was a big fan of Dr. Quinn Medicine Woman and Xena…maybe shows like that could be upgraded with this technique.

But the question is how much will they charge a season and are you willing to pay top dollar? Because they will try to suck your wallet dry as much as they can get away with. Since this product would be upscaled and the results not quite HD nah, not interested. There still asking top dollar for DVD seasons of DS9 480p

Jason Voorhees

1) it all looks exactly the same as before. I lol at anyone delusional enough to squint their eyes and tell themselves that there is a difference.
2) *every* digital tv rescales everything that arrives at any non-native resolution, and in real time.

Theoden_of_ODU

What quality did you view it at? It looks the same at 480 on YouTube, but at 1080, there was a subtle difference in most of the scenes. Not really enough to go ga-ga over, but certainly not “delusional”.

It really only seemed to benefit the space shots, where extra definition was given to the ships. The closeups really didn’t provide any real room for improvement.

I’ve said elsewhere in this thread that this is a half-measure. It’s probably not worth the time investing it over a full remastering from the source elements, especially when we can see the night-and-day differences in the remastered shots done for the DS9 documentary.

d3bug

I would like to see CBS get together with this guy to obtain his AI process and see if they can apply it to the original source material. Then we might see dramatic improvements… possibly even 4K without a lot of hands on effort.

stevedog

They don’t need to ask — he already told us. He just plugged each frame (as a batch) into AI Gigapixel, at pretty much default settings.

Webguest

I don’t know. It gets kind of a CGI look. Not sure if I like that.

I honestly don’t see a big difference and I have a 1080p monitor.To truly get a HD master you would have to do a total restore Like was done with TNG ,not a remaster which are 2 different things.And CBS will never do that. DS9,Voyager, 480p is a far as it will ever go. Even Enterprise was done horrible and that was in HD.

Your agoniser please JJ Abrams

How about one for Star Trek: The Motion Picture 2001 version?
Way, way overdue

skyjedi

It looks like video not film. That is the problem with upscaling it does not create a sharper more defined image.

I also think the different was minimal. I see it in contrast, some level of dark or light tones. But the problem with DS9 was the fuzzy look, the tarnished or bleeding colors. CGI or not , it lacks the clean and crisp look of HD. I don’t think an AI can compensate for that. I am surprised that it’s less complicated to process an old filmed video, then a digital video like DS9. I would have though the filters would be easier to control and applied.

kmart

Once you put a filter on something shot live it is hard to remove that effect if it isn’t consistent. RED PLANET was a movie shot with a red filter, and then the vfx companies (I think 11 of them in the end) had to wait while one vendor came up with a way to extract that look so that a uniform red could be applied in post for all the shots. So on-set filtration is a problem regardless of there being a film camera or not. And DS9’s prob is that it was shot in glorious 35mm but finished in SD, so whatever gorgeousness was there in the image just goes to mush right out of the gate. DS9 for me is almost like radio, because sometimes the looks is so bad on DVD and streaming that I close my eyes or try to sit further away to not be distracted by how terrible it looks. And that’s not good, because it undercuts all the great early VFX work on the ships and the often awesome stuntwork in the fights. I’d probably pay twenty bucks just to see WAY OF THE WARRIOR as intended (and since that one would only have phaser blasts to recreate, because all of the ships were still practical and shot on 35mm, it would be a helluva test bed.)

Riker001

I really don’t see much of a difference…but that’s just me.

Thomas W.

You can achieve the same effect by simply increasing sharpness and contrast.

https://www.youtube.com/watch?v=3RRGcsqhiJ4