Gamma correction

https://github.com/MiSTer-devel/Main_MiSTer/wiki

Moderators: Mug UK, Zorro 2, Greenious, spiny, Sorgelig, Moderator Team

Threepwood
Atari maniac
Atari maniac
Posts: 79
Joined: Thu Jan 10, 2019 10:06 am

Re: Gamma correction

Postby Threepwood » Tue Oct 01, 2019 11:49 am

JamesF wrote:Dell U2410 IPS with wide gamut CCFL backlight calibrated with i1 Display Pro in srgb color space, ICC in windows, 3DLUT in MadVR.
2002 Sony Trinitron KV-29FX30, also calibrated.


Thank James. I had a quick look and while that Dell was a nice display when it hit the market 10 years ago, it does not feature a hardware LUT and as soon as you attach your MiSTer (or anything else than the OS you calibrated it with) your calibration is gone. Same for the Sony.

I assume you use the i1Profiler included with the i1Display Pro (I use that colorimeter, too)? If not, take a look at DisplayCAL (OpenSource), you can get even better results with that.

EDIT:

JamesF wrote:Again, these 1.2-1.4 number which result in 2.64 to 3.08 gamma is a ridiculously wrong thing to do,, especially if it is baked into the core.
I understand you advocate for this because for some reason it looks good on your display to your eyes (very subjective), but we don't want something like this permanent in a core especially if it is a purely cosmetic feature and done wrong (imo).


I guess with "we" you mean "I" and your impression is purely subjective, too, when you use the MiSTer. Your displays are not calibrated outside of Windows/Mac OS and more likely to display wrong colors, due to the white point calibration that requires you to setup RGB individually and then are corrected by a LUT. Besides, since hardly any user will use a real calibrated display it is good to have a few options to choose from individually. That is the point: It is an optional feature that you can as well leave off if you don't like it.
Last edited by Threepwood on Tue Oct 01, 2019 11:58 am, edited 1 time in total.

JamesF
Captain Atari
Captain Atari
Posts: 153
Joined: Sat Dec 15, 2018 6:46 am

Re: Gamma correction

Postby JamesF » Tue Oct 01, 2019 11:57 am

Yep, using DisplayCAL and MadVR for years now, vastly more accurate than all the other options.
HCFR+MadVR for TV's, it's also freeware.

Anyway, I hope paulb-nl can make that polynomial gamma curve work, as it is the 'right' way to do it (imo).
From there, we can adjust the parameters to look even 'better', or closer to a CRT.
Hopefully Sorgelig can think of something that works outside of the core, like a filter.

EDIT:
WE, as 'all mister users',, but as I can only speak for myself, yes, I mean 'I'.
I would have though that someone that knows a thing or two about display calibration would understand that a gamma of 3.0 is the wrong thing to do.

@paulbnl
How do you set your Brightness level on your CRT, with a blackscreen?

Threepwood
Atari maniac
Atari maniac
Posts: 79
Joined: Thu Jan 10, 2019 10:06 am

Re: Gamma correction

Postby Threepwood » Tue Oct 01, 2019 12:29 pm

JamesF wrote:I would have though that someone that knows a thing or two about display calibration would understand that a gamma of 3.0 is the wrong thing to do.


In an ideal world where everybody had a good, calibrated screen that displays colors accurately forever and out of the box, I would agree to you. Sadly there are a variety of displays, technologies and different assumptions with which the games for the individual MiSTer cores were made. Those assumptions are made with consumer devices in mind, not usually professional equipment (same in audio production where audio is mixed using studio monitors AND an cheap radio speaker, because it has to sound good on those, too). The correction is relative to what the display shows, not to a fixed reference point and as paulbnl pointed out, byuu even recommends a gamma correction of 1.5 and his example images look way better to me, too.

Therefore I would like to repeat my question @paulbnl: "Would it be possible to integrate a reference image, like many games use them to roughly calibrate gamma (e.g. The Mark of the Ninja and many horror games) that says "adjust gamma till [object] is barely visible"?

JamesF wrote:Anyway, I hope paulb-nl can make that polynomial gamma curve work, as it is the 'right' way to do it (imo).
From there, we can adjust the parameters to look even 'better', or closer to a CRT.
Hopefully Sorgelig can think of something that works outside of the core, like a filter.


Whatever proves to be the best and also convenient solution in the end, I am all for it and I trust paulbnl and the devs to make the best decision.

User avatar
kitrinx
Captain Atari
Captain Atari
Posts: 175
Joined: Wed Sep 26, 2018 6:03 am

Re: Gamma correction

Postby kitrinx » Wed Oct 02, 2019 4:09 am

I'm not a pro with the fancy calibrations and color corrections and things, but gamma is IMHO the responsibility of the TV, not MiSTer. If your TV can't do it and you care a lot about gamma, then you should probably get a better display. Features aren't free in FPGA. They take up space and make it harder to build stable cores, so adding them "just because" isn't really a great choice. If you feel like it really must be part of mister, the correct place for this is as part of the framework, with a standard setting along with the filters. It doesn't belong as a core specific setting, and it *definitely* doesn't belong baked in.

Threepwood
Atari maniac
Atari maniac
Posts: 79
Joined: Thu Jan 10, 2019 10:06 am

Re: Gamma correction

Postby Threepwood » Wed Oct 02, 2019 6:01 am

kitrinx wrote:I'm not a pro with the fancy calibrations and color corrections and things, but gamma is IMHO the responsibility of the TV, not MiSTer. If your TV can't do it and you care a lot about gamma, then you should probably get a better display


We covered these arguments already. No LCD display allows for the necessary gamma curve corrections and MiSTer is the device emulating old hardware where games were created with different gamma curves in mind than you will find today on the market. Therefore it is the responsibility of the emulating device or software to do the corrections that definetly no consumer display can do and I am not aware of any other LCD that can.

In the article paulbnl posted earlier, byuu describes what needs to be done to fix this in emulation. MiSTer is, in my humble opinion, the best emulation device there is and it would be great if it would exceed or at least keep up with software emulation in all areas to preserve the authenticity of the look and feel of the games.

It is a great idea paulbnl had and looks much closer to what it was supposed to look. You can also check byuu's article where screenshots show the severity of the fault in colors and gamma without correction.

kitrinx wrote:It doesn't belong as a core specific setting, and it *definitely* doesn't belong baked in.


It needs to be an option that can be set per core as you can see with byuu's article. Different old display technologies (Game Boy vs CRT) need different corrections. Where it is best implemented and how, is for the devs to decide. It sounds like you already have a solid idea on how to do this and filter settings sound like a good option.

paulbnl
Atari freak
Atari freak
Posts: 74
Joined: Wed Oct 24, 2018 9:43 am

Re: Gamma correction

Postby paulbnl » Wed Oct 02, 2019 9:44 pm

JamesF wrote:Again, these 1.2-1.4 number which result in 2.64 to 3.08 gamma is a ridiculously wrong thing to do,, especially if it is baked into the core.
I understand you advocate for this because for some reason it looks good on your display to your eyes (very subjective), but we don't want something like this permanent in a core especially if it is a purely cosmetic feature and done wrong (imo).


Can you elaborate on what does not look good specifically? Like some specific spots in games? It seems like you are calling it wrong just because the numbers are high. Maybe you are just used to everything looking greyish and shadow detail being very bright.

I have added a build with your polynomial curve in the same link as the previous builds. It seems like a minor difference to me compared to 1.3. The bright shades are slightly brighter.

Threepwood wrote:Would it be possible to integrate a reference image, like many games use them to roughly calibrate gamma (e.g. The Mark of the Ninja and many horror games) that says "adjust gamma till [object] is barely visible"?


That is above my pay grade :) Maybe that is something for the 240p test suite.

JamesF
Captain Atari
Captain Atari
Posts: 153
Joined: Sat Dec 15, 2018 6:46 am

Re: Gamma correction

Postby JamesF » Thu Oct 03, 2019 7:07 am

By 'looks wrong' I mean the middle shades are much too dark compared to pure white,, pure white is used for text and plenty other things (clouds, eyes, etc..). That's the false effect of too high gamma value.
To fix this abrupt transition we use polynomial gamma to gradualy return to 2.2 from around 70% input to 100% input.

The 'poly' gamma curve is not a minor difference, it looks much more natural and doesn't transition from from middle to bright tones so abruptly, much closer to a CRT.
It's the 0% to 20% input (0-50 from 255) we want to concentrate on, not the entire gamma curve, since a CRT gamma is overall closer to 2.2-2.4 as mentioned earlier but with darker blacks.
Remember, we are trying to slightly reverse the darker side of sRGB gamma curve of LCDs, not change the entire gamma curve.

Please try 2.9_3.8 poly values as that will be closer to a gamma of 2.6 at 0-20% input, and 2.2 at 75-100% input.
As 3.15_4.4 produces a gamma of 2.8 at 0-20% and 2.2 at 75%-100%,, and imo that's slightly too dark compared to my Trinitron, but already much better than the pure 1.2-1.3 values.
You can just replace the 1.2 and 1.3 values with the two poly equations, since as explained above, we don't want to touch the brighter shades.
Maybe: Off, Poly 1.2, Poly 1.3, and 1.3.

Image

PS.
What I do is look at the first black step in the grey ramp in 240p test suite.
Then Yoshi's Island first level mountains in the background.
Also, my Dell U2410 has a gamma curve closer to the Ploy already, so it looks 'right' in the OFF setting and does not need correction,, that tells us not all LCD monitors calibrated equally from the factory.

Threepwood
Atari maniac
Atari maniac
Posts: 79
Joined: Thu Jan 10, 2019 10:06 am

Re: Gamma correction

Postby Threepwood » Thu Oct 03, 2019 10:37 am

JamesF wrote:What I do is look at the first black step in the grey ramp in 240p test suite.
Then Yoshi's Island first level mountains in the background.


I assume you take screenshots on the MiSTer that you then view on your PC?

EDIT: This test suit was written in 2014. Is there any information wether or not it was written with adjustments for CRT Gamma curves in mind? I would think this outputs unscewed test patterns that look normal on a LCD, too, while games had certain ranges boosted to compensate for a CRT's gamma curve. In that case using the 240p test suite would be useless for our use case here.

JamesF wrote:Also, my Dell U2410 has a gamma curve closer to the Ploy already, so it looks 'right' in the OFF setting and does not need correction,, that tells us not all LCD monitors calibrated equally from the factory.


How old is your monitor by now?

EDIT: Attached you find two comparisons of gamma levels using Yoshi's Island and Super Metroid (screenshots taken on MiSTer with the latest core modified by paulbnl)
You do not have the required permissions to view the files attached to this post.

paulbnl
Atari freak
Atari freak
Posts: 74
Joined: Wed Oct 24, 2018 9:43 am

Re: Gamma correction

Postby paulbnl » Thu Oct 03, 2019 6:28 pm

I am going to make it like the scaling filters so we can just use txt files for the different curves.

It is better than having fixed curves in the core.

Threepwood
Atari maniac
Atari maniac
Posts: 79
Joined: Thu Jan 10, 2019 10:06 am

Re: Gamma correction

Postby Threepwood » Thu Oct 03, 2019 7:26 pm

paulbnl wrote:I am going to make it like the scaling filters so we can just use txt files for the different curves.

It is better than having fixed curves in the core.


Awesome, thanks for the hard work!

JamesF
Captain Atari
Captain Atari
Posts: 153
Joined: Sat Dec 15, 2018 6:46 am

Re: Gamma correction

Postby JamesF » Thu Oct 03, 2019 9:46 pm

paulbnl wrote:I am going to make it like the scaling filters so we can just use txt files for the different curves.

Thank you.

User avatar
BitsNStuff
Atari freak
Atari freak
Posts: 62
Joined: Tue Oct 16, 2018 7:55 am
Contact:

Re: Gamma correction

Postby BitsNStuff » Fri Oct 04, 2019 9:20 am

That sounds fantastic, paulbnl. Thanks for putting the work in.

Sorgelig
Ultimate Atarian
Ultimate Atarian
Posts: 5282
Joined: Mon Dec 14, 2015 10:51 am
Location: Russia/Taiwan

Re: Gamma correction

Postby Sorgelig » Fri Oct 04, 2019 9:34 am

I just want to remind: if it will use LUT for gamma correction it's very unlikely will be integrated into framework as it will produce a lot of timing violataions at 148MHz. For framework integration this LUT needs to be highly optimized with pipe-lining to cope with high HDMI data rate.

paulbnl
Atari freak
Atari freak
Posts: 74
Joined: Wed Oct 24, 2018 9:43 am

Re: Gamma correction

Postby paulbnl » Fri Oct 04, 2019 10:18 am

Sorgelig wrote:I just want to remind: if it will use LUT for gamma correction it's very unlikely will be integrated into framework as it will produce a lot of timing violataions at 148MHz. For framework integration this LUT needs to be highly optimized with pipe-lining to cope with high HDMI data rate.


Sure I can add pipe-lining. With the SNES I haven't seen any timing differences with using it in the framework or in the core.

As I understand it it just needs to work on CLK_VIDEO because the LUT will be used on the input to the scaler so for the SNES it will only be at 42Mhz.

Sorgelig
Ultimate Atarian
Ultimate Atarian
Posts: 5282
Joined: Mon Dec 14, 2015 10:51 am
Location: Russia/Taiwan

Re: Gamma correction

Postby Sorgelig » Fri Oct 04, 2019 10:38 am

Because direct video requires high super-sampling clock, some cores may output video at up to 100MHz (or even a little higher).
But surely it can be guarded by clock enabler as it's on input side where clock enabler is still valid. Though i don't know how timings will be calculated

yxkalle
Atarian
Atarian
Posts: 5
Joined: Fri Oct 04, 2019 3:08 pm

Re: Gamma correction

Postby yxkalle » Fri Oct 04, 2019 3:10 pm

Doesn't it make more sense to apply the LUT before the scaler, or is that simply not possible?

softtest1
Atarian
Atarian
Posts: 9
Joined: Tue Apr 30, 2019 6:37 pm

Re: Gamma correction

Postby softtest1 » Sat Oct 05, 2019 1:58 pm

Maybe you can make it so that we have two filter files; one for gamma and one for scaling. Both would be selected in the menu the same way, and the gamma filter would be applied before the scaling filter.

paulbnl
Atari freak
Atari freak
Posts: 74
Joined: Wed Oct 24, 2018 9:43 am

Re: Gamma correction

Postby paulbnl » Sat Oct 05, 2019 6:55 pm

I have updated the link with updated cores. You'll need to copy the Gamma folder to the sdcard and replace the MiSTer binary for the Gamma correction option to appear under Scaling Filter.

The txt files for the gamma curves are just 256 lines of values 0-255.

The Genesis is now also included because it has a 100Mhz video clock. I haven't seen negative changes to the timings in the Timing analyzer.

https://drive.google.com/file/d/12FrBLf ... sp=sharing


yxkalle wrote:Doesn't it make more sense to apply the LUT before the scaler, or is that simply not possible?


softtest1 wrote:Maybe you can make it so that we have two filter files; one for gamma and one for scaling. Both would be selected in the menu the same way, and the gamma filter would be applied before the scaling filter.


It already works that way :wink:

JamesF
Captain Atari
Captain Atari
Posts: 153
Joined: Sat Dec 15, 2018 6:46 am

Re: Gamma correction

Postby JamesF » Sat Oct 05, 2019 7:42 pm

This is absolutely amazing.
I hope to see this merged to main.

Matlab script for Poly Gamma:

Code: Select all

in = 0:1:255;
a = ((in/255) .^ 2.9);
b = ((in/255) .^ 3.8);
Out = 255 * ((2 * a - b) .^ (1/2.2));
Y = round(Out);
Y1=Y(:)


Script for normal Gamma:

Code: Select all

in = 0:1:255;
Out = 255 * ((in/255) .^ 1.1);
Y = round(Out);
Y1=Y(:)


Use here if you don't have matlab:
https://www.tutorialspoint.com/matlab/try_matlab.php
Just copy the result and paste in the gamma text file.
.
You do not have the required permissions to view the files attached to this post.

softtest1
Atarian
Atarian
Posts: 9
Joined: Tue Apr 30, 2019 6:37 pm

Re: Gamma correction

Postby softtest1 » Sun Oct 06, 2019 1:45 am

paulbnl wrote:It already works that way :wink:


Thank you. I tried it now and it works really well.

JamesF
Captain Atari
Captain Atari
Posts: 153
Joined: Sat Dec 15, 2018 6:46 am

Re: Gamma correction

Postby JamesF » Sun Oct 06, 2019 3:56 pm

@paulbnl
Are you going to PR it so Sorgelig can review it?

Threepwood
Atari maniac
Atari maniac
Posts: 79
Joined: Thu Jan 10, 2019 10:06 am

Re: Gamma correction

Postby Threepwood » Sun Oct 06, 2019 6:03 pm

Indeed, I hope it makes it into the main release and all cores get updated over time. I tried the Genesis core with different games yesterday and it works amazing. The games really benefit from gamma adjustment.

paulbnl
Atari freak
Atari freak
Posts: 74
Joined: Wed Oct 24, 2018 9:43 am

Re: Gamma correction

Postby paulbnl » Mon Oct 07, 2019 11:20 am

JamesF wrote:Are you going to PR it so Sorgelig can review it?


Sure, I just had to figure out how to add scrolling to the system menu but now that's working properly I have sent the pull requests.

bhamadicharef
Atariator
Atariator
Posts: 22
Joined: Tue Jul 18, 2017 8:31 am
Location: Singapore

Re: Gamma correction

Postby bhamadicharef » Tue Oct 08, 2019 3:19 am

For those interested in research and implementation aspects
there is this old IEEE paper, PDF is online ...

A Flexible Architecture for Precise Gamma Correction
Dong-U Lee, Ray C. C. Cheung, and John D. Villasenor
IEEE Transactions on Very Large Scale Integration (VLSI) Systems
Year: 2007 | Volume: 15, Issue: 4 | Journal Article | Publisher: IEEE
https://www.researchgate.net/publicatio ... Correction

Best regards
Brahim HAMADI CHAREF:: Singapore

Sorgelig
Ultimate Atarian
Ultimate Atarian
Posts: 5282
Joined: Mon Dec 14, 2015 10:51 am
Location: Russia/Taiwan

Re: Gamma correction

Postby Sorgelig » Wed Oct 09, 2019 4:44 pm

Is it possible to re-use ADV chip output converter dsp for gamma correction? It's not LUT of course..


Return to “MiSTer”

Who is online

Users browsing this forum: No registered users and 18 guests