controller ports: minimizing input lag

https://github.com/MiSTer-devel/Main_MiSTer/wiki

Moderators: Mug UK, Zorro 2, spiny, Greenious, Sorgelig, Moderator Team

hyperterminal
Atari maniac
Atari maniac
Posts: 91
Joined: Sun Jul 09, 2017 1:43 pm

controller ports: minimizing input lag

Postby hyperterminal » Wed Nov 21, 2018 1:19 pm

I read that the USB ports on the DE10-Nano are hooked up to the microcontroller rather than directly to the FPGA (the same applies to the DB9 ports of the MiST). With the latest I/O board 5.5 which adds another USB port instead of the SPDIF port, the USB ports are connected directly to the FPGA, which is said to decrease input lag.

How can this be if all inputs are managed by the ARM CPU and the Linux OS? There shouldn't be such a big difference, should there? Has anyone measured the differences in input lag? Is there a negative effect on input lag if I do not connect the controller directly to the PCB but through an OTG hub, together with other devices?

Sorgelig
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 3175
Joined: Mon Dec 14, 2015 10:51 am
Location: Russia/Taiwan

Re: controller ports: minimizing input lag

Postby Sorgelig » Wed Nov 21, 2018 2:59 pm

1) it's not "instead of SPDIF". SPDIF is still there, just merged with 3.5mm audio jack.
2) I/O pins on new connector are core specific. It's not direct drop-in replacement for USB controller.

it's up to specific core how to use these pins. I have no plans to add support direct joystick connection for all cores. Though, some cores may get such function if authors would like.

hyperterminal
Atari maniac
Atari maniac
Posts: 91
Joined: Sun Jul 09, 2017 1:43 pm

Re: controller ports: minimizing input lag

Postby hyperterminal » Wed Nov 21, 2018 3:07 pm

Sorgelig wrote:1) it's not "instead of SPDIF". SPDIF is still there, just merged with 3.5mm audio jack.
2) I/O pins on new connector are core specific. It's not direct drop-in replacement for USB controller.

it's up to specific core how to use these pins. I have no plans to add support direct joystick connection for all cores. Though, some cores may get such function if authors would like.

Ok thank you for the clarification. Is there a list of cores that support the I/O pins on the new connector?

Sorgelig
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 3175
Joined: Mon Dec 14, 2015 10:51 am
Location: Russia/Taiwan

Re: controller ports: minimizing input lag

Postby Sorgelig » Wed Nov 21, 2018 3:09 pm

Here:



:)

kitrinx
Atariator
Atariator
Posts: 28
Joined: Wed Sep 26, 2018 6:03 am

Re: controller ports: minimizing input lag

Postby kitrinx » Wed Nov 21, 2018 6:38 pm

Not true! Altair 8800 uses serial IO for serial connection I think ;)

The USB input is pretty fast. Considering serial I/O input has to be added on a per-core basis and would likely be a hassle to keep coordinated, until someone can prove to me that there's an practical and appreciable (note: appreciable. A few ms is not appreciable) difference in latency between serial vs USB, I'm going to keep advocating just to use USB and not worry about it.

ijor
Hardware Guru
Hardware Guru
Posts: 3629
Joined: Sat May 29, 2004 7:52 pm
Contact:

Re: controller ports: minimizing input lag

Postby ijor » Wed Nov 21, 2018 9:15 pm

I hope we are not reopening a can of worms :)

kitrinx wrote:The USB input is pretty fast. Considering serial I/O input has to be added on a per-core basis and would likely be a hassle to keep coordinated, until someone can prove to me that there's an practical and appreciable (note: appreciable. A few ms is not appreciable) difference in latency between serial vs USB, I'm going to keep advocating just to use USB and not worry about it.


I agree that without some kind of connector standardization it would be difficult. But I understand somebody is already working on a low latency expansion board.

I'm not sure the extra lag is "not appreciable". According to your own tests it is about half a video frame, although more tests are probably needed to have a better estimate. But latency doesn't add linearly in cases like this:

Let's assume that the USB input latency is indeed in the order of half a frame. Let's also assume that games typically poll the input once per frame. That means that statistically, half of the times the extra latency has no effect at all, and the other half of the time, it adds a full extra frame of latency. If this is really very significant or not, I admit I'm not really such a gamer to make the best judgement.

Btw, and just for the records, it is not serial. At least not for the typical old school digital joystick, it's parallel.

Edit: Wrong stats
Fx Cast: Atari St cycle accurate fpga core

kitrinx
Atariator
Atariator
Posts: 28
Joined: Wed Sep 26, 2018 6:03 am

Re: controller ports: minimizing input lag

Postby kitrinx » Wed Nov 21, 2018 9:50 pm

ijor wrote:I'm not sure the extra lag is "not appreciable". According to your own tests it is about half a video frame, although more tests are probably needed to have a better estimate. But latency doesn't add linearly in cases like this:


It's true that it was around there, but that was also measuring it through a McCthulu board which is known to have a good amount of lag of its own. It is pretty tolerable even with that less than ideal circumstance. It's also not explored how much that could be reduced further with just tweaks to the polling of main. As it stands now, I'm dubious that most humans would be able to discern a difference. I agree though, lets not open a can of worms.

flain
Retro freak
Retro freak
Posts: 13
Joined: Sat Nov 03, 2018 6:21 am

Re: controller ports: minimizing input lag

Postby flain » Thu Nov 22, 2018 3:08 am

I haven't even looked at the MiSTer linux instance yet, but could we possibly just increase the USB polling rate from there? I have experience doing it for PC mice but haven't tried game controllers. Typically with a USB mouse, either the mouse can increase polling via harwdare, or the host can increase polling frequency via software. The mouse simply fills up its buffer and the host just reads it every poll interval regardless of what the mouse is doing.

This is the latency of different polling intervals (125hz is usually default):

Polling rate Response time
125 Hz 8 ms
250 Hz 4 ms
500 Hz 2 ms
1000 Hz 1 ms

Can we SSH into the linux instance? If so i might have a play around when i get home from work today.

Are there any implications on the FPGA side if i increase USB polling rates in the linux instance?

User avatar
Chol
Atarian
Atarian
Posts: 9
Joined: Sat Oct 27, 2018 12:53 pm
Location: Germany

Re: controller ports: minimizing input lag

Postby Chol » Thu Nov 22, 2018 8:49 am

SSH is documented in the Wiki: https://github.com/MiSTer-devel/Main_MiSTer/wiki/Network-access

There were some Kernel options for Joystick polling added to Linux 4.12: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=933bfe4d271ef5931bc7513a1239751ed251db04

But our MiSTer seems to be based on the older Linux 4.5. The Altera Kernel is 4.20, so an upgrade should be possible?

Sorgelig
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 3175
Joined: Mon Dec 14, 2015 10:51 am
Location: Russia/Taiwan

Re: controller ports: minimizing input lag

Postby Sorgelig » Thu Nov 22, 2018 10:23 am

Specific feature can be backported to 4.5. It doesn't mean it needs to be upgraded to 4.20.
Altera has many kernel versions in their github. It doesn't mean they are all working. Most stable version is 4.5 and i have no plans to upgrade it as it will bring a lot of new problems.
If you want to play with bug fixing on new kernel - you are welcome. When it will be polished to level of 4.5 then MiSTer will migrate to it. Upgrading just for higher version number is wasting the time. For embedded systems (and MiSTer is embedded system) usually only one version is used for the whole life time. Specific improvements can be cherry-picked and backported if it's really required.

Sorgelig
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 3175
Joined: Mon Dec 14, 2015 10:51 am
Location: Russia/Taiwan

Re: controller ports: minimizing input lag

Postby Sorgelig » Thu Nov 22, 2018 10:40 am

And before doing any changes to USB polling the effect should be estimated.
As it's already told above and it's very well known fact, consoles poll the joysticks ONCE per frame. So, once per 16 or 20 milliseconds. If current lag provided by input system is less than 16ms, then tweaking USB for higher polling speed won't bring any improvements.

ijor
Hardware Guru
Hardware Guru
Posts: 3629
Joined: Sat May 29, 2004 7:52 pm
Contact:

Re: controller ports: minimizing input lag

Postby ijor » Thu Nov 22, 2018 11:24 am

Sorgelig wrote:As it's already told above and it's very well known fact, consoles poll the joysticks ONCE per frame. So, once per 16 or 20 milliseconds. If current lag provided by input system is less than 16ms, then tweaking USB for higher polling speed won't bring any improvements.


It will. As I said in my previous message, the USB lag will increase the overall latency by one full frame on some percentage of the events.

Take the case of the joystick event happening at time 0, and the console polling the port at time 500us, then even a single extra millisecond latency would mean that the console will get the event at time 1ms. Too late for the console software. So the game would get the event on the next frame, instead of this one. Of course, most of the time you probably won't be so unlucky. Sometimes the console will poll the joystick at, say, 10 ms after the actual hardware event, then if the extra latency is less than 10ms, it would be completely harmless.

So let's assume that the total additional latency (the whole USB hardware chain including wireless and hub, the USB kernel, the Linux application software, etc) is indeed less than one full frame. That would mean that some percentage of the time it will have zero effect, and other times it would add one full frame. The respective percentage depends on the added latency. The closer to a full frame, the more times it will add one extra full frame of latency, and vice versa.
Fx Cast: Atari St cycle accurate fpga core

flain
Retro freak
Retro freak
Posts: 13
Joined: Sat Nov 03, 2018 6:21 am

Re: controller ports: minimizing input lag

Postby flain » Thu Nov 22, 2018 2:58 pm

.

Sorgelig
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 3175
Joined: Mon Dec 14, 2015 10:51 am
Location: Russia/Taiwan

Re: controller ports: minimizing input lag

Postby Sorgelig » Thu Nov 22, 2018 3:55 pm

USB polling rate is not in sync with core joystick polling rate. In this case ANY non-zero latency may delay the input to the next frame.

I still state that input latency for retro consoles is too exaggerated. Human also has latency between eyes and fingers.

Whatever. I/O board provide an option for custom input, so it's up to core developer to implement DB9 input. As i've explained earlier, this won't be a main input as it's crippled and limited.

No one with PS4 or XBOne thinks about direct DB9 input. Why MiSTer should? Ridiculous...

witchmaster
Atari freak
Atari freak
Posts: 57
Joined: Wed Jun 15, 2011 10:50 am

Re: controller ports: minimizing input lag

Postby witchmaster » Thu Nov 22, 2018 5:51 pm

Less input lag is of course a good thing but a low latency HDMI scaler (in the works as I understand) will affect the perceived input lag much more than faster USB polling (or direct DB9 input). At the moment if using HDMI, the total lag is 0-8ms for the USB controls, 33.4ms (two frames) for HDMI scaling, plus the lag of the TV which can vary from 9ms to 100+ms. The lag caused by USB polling is small in the scheme of things.

hyperterminal
Atari maniac
Atari maniac
Posts: 91
Joined: Sun Jul 09, 2017 1:43 pm

Re: controller ports: minimizing input lag

Postby hyperterminal » Thu Nov 22, 2018 6:17 pm

Sorgelig wrote:No one with PS4 or XBOne thinks about direct DB9 input. Why MiSTer should? Ridiculous...

I would prefer DB15 input for NeoGeo/Supergun controllers.

Games for PS4 and XBOne are programmed with the respective limitations of these systems in mind. They are usually more forgiving with regards to inputs.

Here we are not talking about the lag that is inherent to the original games and systems, but about lag that is added on top of it.

Games from the 80s rely heavily on reactions and precise controls. Any lag that can be avoided should be avoided. 1 Frame might not be much for itself but it adds up when you use an LCD screen and have aged a little, so that your reflexes have gotten slower than they used to be.

Thats my opinion on this subject. Sorgelig, I don't want to annoy you. I am very thankful for what you have done so far.

witchmaster
Atari freak
Atari freak
Posts: 57
Joined: Wed Jun 15, 2011 10:50 am

Re: controller ports: minimizing input lag

Postby witchmaster » Thu Nov 22, 2018 10:31 pm

ijor wrote:It will. As I said in my previous message, the USB lag will increase the overall latency by one full frame on some percentage of the events.

Take the case of the joystick event happening at time 0, and the console polling the port at time 500us, then even a single extra millisecond latency would mean that the console will get the event at time 1ms. Too late for the console software. So the game would get the event on the next frame, instead of this one. Of course, most of the time you probably won't be so unlucky. Sometimes the console will poll the joystick at, say, 10 ms after the actual hardware event, then if the extra latency is less than 10ms, it would be completely harmless.

So let's assume that the total additional latency (the whole USB hardware chain including wireless and hub, the USB kernel, the Linux application software, etc) is indeed less than one full frame. That would mean that some percentage of the time it will have zero effect, and other times it would add one full frame. The respective percentage depends on the added latency. The closer to a full frame, the more times it will add one extra full frame of latency, and vice versa.


hyperterminal wrote:Here we are not talking about the lag that is inherent to the original games and systems, but about lag that is added on top of it.

Games from the 80s rely heavily on reactions and precise controls. Any lag that can be avoided should be avoided. 1 Frame might not be much for itself but it adds up when you use an LCD screen and have aged a little, so that your reflexes have gotten slower than they used to be.

USB polling at 125Hz will not add one frame of input lag, it adds at most 8ms, the game itself is responsible for the possible one frame lag. Even if you use a controller with zero lag, you can press a button just after the game has polled the controller and the input lag will then be one frame. So with a zero lag controller (game at 60fps) the input lag will be between 0 and 16.7ms and with a USB controller (assuming 125Hz polling) the input lag will be between 0 and 24.7ms.

flain
Retro freak
Retro freak
Posts: 13
Joined: Sat Nov 03, 2018 6:21 am

Re: controller ports: minimizing input lag

Postby flain » Thu Nov 22, 2018 11:19 pm

Interestingly ps4 controllers when plugged into a PC via USB are by default 250hz so 4ms input lag, so yes people don't really question console input lag. I think this lag issue comes up because retropie lag is TERRIBLE, when you have lag throughout the chain it all adds up to eventually be noticeable and you have people looking to remove it everywhere. People who have experienced input lag on snes classic and those consoles also have been burnt by bad lag and so look to FPGA solutions as one way to remove lag.

flain
Retro freak
Retro freak
Posts: 13
Joined: Sat Nov 03, 2018 6:21 am

Re: controller ports: minimizing input lag

Postby flain » Thu Nov 22, 2018 11:34 pm

witchmaster wrote: Even if you use a controller with zero lag, you can press a button just after the game has polled the controller and the input lag will then be one frame. So with a zero lag controller (game at 60fps) the input lag will be between 0 and 16.7ms and with a USB controller (assuming 125Hz polling) the input lag will be between 0 and 24.7ms.


This is VERY different per game. If you want some examples of laggy input games looks at mortal kombat 2 on consoles and super mario world and others. They have more built in game lag to the point where if you keep adding lag in the chain it gets to a point where you can't even play them anymore. It does get that bad for SMW, maybe not on MiSTer but on emulators it does. There are plenty of youtube videos giving examples of this.

witchmaster
Atari freak
Atari freak
Posts: 57
Joined: Wed Jun 15, 2011 10:50 am

Re: controller ports: minimizing input lag

Postby witchmaster » Fri Nov 23, 2018 6:56 am

flain wrote:Interestingly ps4 controllers when plugged into a PC via USB are by default 250hz so 4ms input lag, so yes people don't really question console input lag. I think this lag issue comes up because retropie lag is TERRIBLE, when you have lag throughout the chain it all adds up to eventually be noticeable and you have people looking to remove it everywhere. People who have experienced input lag on snes classic and those consoles also have been burnt by bad lag and so look to FPGA solutions as one way to remove lag.

Games on PS4/XBone have an input lag usually of 80ms and above (input lag of tv/monitor excluded) but as somebody else said in this thread, games on those sysems take this into account. Like you say, lag adds up, so if it's possible to remove lag somewhere in the chain, that's great. Input lag caused by USB polling is, however, small.

flain wrote:
witchmaster wrote: Even if you use a controller with zero lag, you can press a button just after the game has polled the controller and the input lag will then be one frame. So with a zero lag controller (game at 60fps) the input lag will be between 0 and 16.7ms and with a USB controller (assuming 125Hz polling) the input lag will be between 0 and 24.7ms.


This is VERY different per game. If you want some examples of laggy input games looks at mortal kombat 2 on consoles and super mario world and others. They have more built in game lag to the point where if you keep adding lag in the chain it gets to a point where you can't even play them anymore. It does get that bad for SMW, maybe not on MiSTer but on emulators it does. There are plenty of youtube videos giving examples of this.

True, I assumed an ideal game in my example. :)

ijor
Hardware Guru
Hardware Guru
Posts: 3629
Joined: Sat May 29, 2004 7:52 pm
Contact:

Re: controller ports: minimizing input lag

Postby ijor » Fri Nov 23, 2018 11:41 am

witchmaster wrote:Less input lag is of course a good thing but a low latency HDMI scaler (in the works as I understand) will affect the perceived input lag much more than faster USB polling (or direct DB9 input). At the moment if using HDMI, the total lag is 0-8ms for the USB controls, 33.4ms (two frames) for HDMI scaling, plus the lag of the TV which can vary from 9ms to 100+ms. The lag caused by USB polling is small in the scheme of things.
...
USB polling at 125Hz will not add one frame of input lag, it adds at most 8ms, the game itself is responsible for the possible one frame lag. Even if you use a controller with zero lag, you can press a button just after the game has polled the controller and the input lag will then be one frame. So with a zero lag controller (game at 60fps) the input lag will be between 0 and 16.7ms and with a USB controller (assuming 125Hz polling) the input lag will be between 0 and 24.7ms.


It doesn't matter which component is really responsible, we are not here to "blame" anybody. What matters here is only the result. So if the USB input has an 8ms lag, then the net effect at 60Hz would be exactly as I said: 50% no extra latency, 50% one full frame extra latency (this doesn't contradict a max latency of one frame+8ms).

Btw, the only true measurement we have so far indicated the 8ms lag average, not maximum. Polling is just one component of the lag. The USB hardware alone has latency on the order of 1ms. And that doesn't even count the Linux software latency. Unfortunately the user that posted the measurements didn't mention maximum (or minimum) values. But I guess that once in a while you get events with very high latency.

I do agree however, that video output latency is more significant (except again, that some input events might get very high latency). Low latency scaler is not an universal solution. It will not work only on all monitors, and might not be possible to implement on every single core. Unless you are ready to get video tear as a sacrifice for low latency.

The actual scaler latency should be variable, not constant. If you are not using vsync, it would oscillate between one and two frames. That is an intrinsic consequence of using triple buffering and should be about the same on either scaler.
Fx Cast: Atari St cycle accurate fpga core

kitrinx
Atariator
Atariator
Posts: 28
Joined: Wed Sep 26, 2018 6:03 am

Re: controller ports: minimizing input lag

Postby kitrinx » Fri Nov 23, 2018 1:48 pm

ijor wrote:Btw, the only true measurement we have so far indicated the 8ms lag average, not maximum. Polling is just one component of the lag. The USB hardware alone has latency on the order of 1ms. And that doesn't even count the Linux software latency. Unfortunately the user that posted the measurements didn't mention maximum (or minimum) values. But I guess that once in a while you get events with very high latency.


The values ranged from around 3ms to around 17ms. Of course this is eyeballing on a scope, trying to use only the samples with very low bounce on closing the joystick button. Good news though, from the look of it Sorg added a kernel patch to specify HID polling frequency the same way you can adjust mouse polling frequency, so maybe this debate can be put to bed. I'm sure people will still quibble over the +1ms from 1000hz polling believing they are the Flash, but that 6% (1 in 17) chance of adding a frame is probably dwarfed by their gamepad itself at that point.

witchmaster
Atari freak
Atari freak
Posts: 57
Joined: Wed Jun 15, 2011 10:50 am

Re: controller ports: minimizing input lag

Postby witchmaster » Fri Nov 23, 2018 9:43 pm

ijor wrote:It doesn't matter which component is really responsible, we are not here to "blame" anybody. What matters here is only the result. So if the USB input has an 8ms lag, then the net effect at 60Hz would be exactly as I said: 50% no extra latency, 50% one full frame extra latency (this doesn't contradict a max latency of one frame+8ms).

Btw, the only true measurement we have so far indicated the 8ms lag average, not maximum. Polling is just one component of the lag. The USB hardware alone has latency on the order of 1ms. And that doesn't even count the Linux software latency. Unfortunately the user that posted the measurements didn't mention maximum (or minimum) values. But I guess that once in a while you get events with very high latency.

Sorry for bragging about this, I just find the subject of input lag so interesting. The net effect of a 50% chance of an added frame of latency for an average of 8ms usb polling can't be correct though. The usb controller polling and game controller polling are running independently from each other right? In that case, there should only be about a 25% chance of an added frame, and that will always happen on a "late" button press. (usb polling will fluctuate between being just after the polling in the game to just before).

kitrinx wrote:The values ranged from around 3ms to around 17ms. Of course this is eyeballing on a scope, trying to use only the samples with very low bounce on closing the joystick button. Good news though, from the look of it Sorg added a kernel patch to specify HID polling frequency the same way you can adjust mouse polling frequency, so maybe this debate can be put to bed. I'm sure people will still quibble over the +1ms from 1000hz polling believing they are the Flash, but that 6% (1 in 17) chance of adding a frame is probably dwarfed by their gamepad itself at that point.

Cool, the MISTer is turning into perfection! :) The 6% would be about 3% according to my reasoning above, and only at the player's slowest 3%.

ijor
Hardware Guru
Hardware Guru
Posts: 3629
Joined: Sat May 29, 2004 7:52 pm
Contact:

Re: controller ports: minimizing input lag

Postby ijor » Sat Nov 24, 2018 1:00 am

witchmaster wrote:
ijor wrote:So if the USB input has an 8ms lag, then the net effect at 60Hz would be exactly as I said: 50% no extra latency, 50% one full frame extra latency (this doesn't contradict a max latency of one frame+8ms).

Sorry for bragging about this, I just find the subject of input lag so interesting. The net effect of a 50% chance of an added frame of latency for an average of 8ms usb polling can't be correct though. The usb controller polling and game controller polling are running independently from each other right? In that case, there should only be about a 25% chance of an added frame, and that will always happen on a "late" button press. (usb polling will fluctuate between being just after the polling in the game to just before).


I think your math is wrong. In first place, see from the quote that I said "8ms lag", not average. And what I meant was really half a frame, which is actually the same because kitrinx reported an average of 8ms-9ms. I intentionally omitted the word average to make the math simpler. In second place, it is not 8ms maximum, it is 8ms-9ms average because, again, the polling interval is not the only component that contributes to the USB latency. So even on an "early" button press, you could still get an extra frame if the latency on this specific press happens to be higher than the average.

I think that it is clear that if the input latency is constantly half a frame, then 50% of the times it would add one extra frame of latency, right? Being more realistically it is not constant, but an average. This makes the math more complicated but obviously it won't be better. Actually it might be even worse because some of events would have a very high latency. Kitrinx just mentioned that some events reached 17ms latency! more than one full frame, which might make the full combined latency being two full frames!

Anyway, if we want to be serious about this, we really need to perform more measurements with multiple different controllers. I can't say I am sure, but I suspect with some controllers the latency would be worse.
Fx Cast: Atari St cycle accurate fpga core

flain
Retro freak
Retro freak
Posts: 13
Joined: Sat Nov 03, 2018 6:21 am

Re: controller ports: minimizing input lag

Postby flain » Sun Nov 25, 2018 12:33 am

I guess we still have the question - will increasing the usb polling make any difference at all? I only ask this because its a very easy thing to do, you edit one .c file and recompile the linux kernel :) It's not a big undertaking like the FPGA scaler development :).

The other question is - does this effect or cause problems to the FPGA side?

Here is a comparison of different controller types
https://www.reddit.com/r/RocketLeague/c ... fo_in_the/


Return to “MiSTer”

Who is online

Users browsing this forum: No registered users and 5 guests