4

I am emulating a screen with Xvfb, and I want to capture it with ffmpeg. In order to prevent frame dropping or duplication, I would like to capture at the exact refresh rate of the screen, but it seems that Xvfb does not … have this?

xrandr confirms that it's 0.0:

$ Xvfb :123 -ac -nolisten tcp -screen 0 1920x1080x24 &
$ DISPLAY=:123 xrandr
xrandr: Failed to get size of gamma for output screen
Screen 0: minimum 1 x 1, current 1920 x 1080, maximum 1920 x 1080
screen connected 1920x1080+0+0 0mm x 0mm
   1920x1080      0.00*

At which rate are the images then drawn to the screen? ffmpeg shows 29.97 fps for this input when using x11grab:

$ ffmpeg -f x11grab -i :123
ffmpeg version 4.3.5-0+deb11u1 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/usr --extra-version=0+deb11u1 --toolchain=hardened --libdir=/usr/lib/aarch64-linux-gnu --incdir=/usr/include/aarch64-linux-gnu --arch=arm64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
[x11grab @ 0xaaaadaa0c2a0] Stream #0: not enough frames to estimate rate; consider increasing probesize
Input #0, x11grab, from ':123':
  Duration: N/A, start: 1669816070.219332, bitrate: 1988667 kb/s
    Stream #0:0: Video: rawvideo (BGR[0] / 0x524742), bgr0, 1920x1080, 1988667 kb/s, 29.97 fps, 1000k tbr, 1000k tbn, 1000k tbc

But when I encode using the "native" framerate at realtime, I get more like ~17-21 fps.

$ ffmpeg -progress - -nostats -f x11grab -re -i :123 -f null /dev/null
…
frame=198
fps=17.80
stream_0_0_q=-0.0
bitrate=N/A
total_size=N/A
out_time_us=11111100
out_time_ms=11111100
out_time=00:00:11.111100
dup_frames=0
drop_frames=0
speed=0.999x
progress=end

It appears to me that the display does not even have a native frame rate, rather letting me sample as often as I want from it.

So, my question is: is that correct? If so, does it go up to "infinite" fps as long as the processing power is large enough?

slhck
  • 463
  • 1
  • 6
  • 22
  • This is something I'd like to know too. If anyone can answer it, please do. – Nav Jan 20 '23 at 14:21
  • @Nav I've placed a bounty on the question to draw attention. – slhck Jan 20 '23 at 19:43
  • The answer is Yes, there is no default/native refresh rate for the [`vfbScreen`](https://github.com/freedesktop/xorg-xserver/blob/master/hw/vfb/InitOutput.c#L70) – don_crissti Jan 22 '23 at 13:40
  • Frame rate is somewhat a hardware concept from scan driven devices, and xvfb doesn't have any hardware to drive. Alternately, framerate might be the rate at which a particular scene can be rendered, and this depends on the complexity of the scene and the speed of the cpu/gpu. – user10489 Jan 22 '23 at 20:39
  • Open to any answer that explains this and gives a paragraph or so of background information. – slhck Jan 23 '23 at 08:17
  • Thank you very much for setting up the bounty, slhck. Thanks to don and user10489 too. Y'all have a bounty to be claimed :-) There is however the reality that Xvfb would receive frames from some source, and that would either be at a rate of a certain number of frames per second, right? Wouldn't the nature of the source of data matter? – Nav Jan 23 '23 at 09:52
  • @don_crissti Would you like to answer? – slhck Jan 26 '23 at 20:26
  • Or @user10489 … whoever comes first. – slhck Jan 26 '23 at 20:26

1 Answers1

1

Frame rate comes from movie projectors, where you have an image in a box ("frame") and it is flashed on the screen for some time, and then blanked, and another frame flashed up for some time, and your persistence of vision makes this look like objects moving in a picture.

The first graphical video displays for computers used oscilloscopes to do vector drawing. The 'scope screen had a hold time, over which the vector lines drawn on it would slowly fade. The driver could draw and redraw different lines at different frequencies and rates to vary brightness, so some portion of the display would be refreshed at some interval, but other parts at other intervals and rates to get a different brightness. Those didn't particuarly have a frame rate.

Later, they created raster screens, where the same electron beam in the 'scope would scan the screen, but now in a rectangular pattern, lines left to right and then lines below them, either sequential or interlaced, turning the beam on and off (or varying the intensity) to give a black and white image. If the image was interlaced with a factor of 2, you had a 'half frame' rate, where half of the lines on the screen were drawn, and then the other half in between were drawn (every other line in the same half frame). This interlaced half frame rate doesn't exactly translate to a frame rate either, as motion could occur differently on the two half frames.

Early TV's had lines, but not really pixels, as it is analog within the line, but when this was brought to computers, the line became pixels, and video cards had pixel rates and refresh rates, where refresh rate is how long it takes to paint two half frames...which eventually looks like a frame rate. Especially when you get higher pixel rates so that you can paint what was two half frames in the same time as a single half frame, and then move from interlaced to progressive sequential lines.

Then we went from tube monitors to LCD monitors -- which suddenly no longer have a refresh rate, as they are not using hold time of phosphor anymore, and there is no electron beam, and so no scan rate. But the video signal format design is still using the old pixel scan format. So do LCD's have a frame rate? Sort of. They have a max pixel rate they can accept a stream of sequential pixels to put on the screen, which limits the speed of updating the entire sccreen, and thus how fast a moving object can be updated.

Now, you have LCD screens and other format screens that no longer need a pseduo analog signal to accept pixels -- they use a fully digital format containing compression and complex commands. So a screen can be told to just refresh a small region rather than the whole screen, and now frame rate is gone again.

And getting back to your question, does Xvfb have a frame rate? It's a fully virtual device with no hardware -- so there is nothing to constrain the rate it can "display" things. And the X11 protocol supports complete drawing commands rather than just raw pixel raster updates, so there's no frame rate there either.

So what about the source? The source of the image can send drawing commands limited only by the speed of your cpu and gpu, so any frame rate there can be variable, depending on those speeds plus the complexity of what it is trying to update.

So some games have a frame rate, because they want to render an entire scene and then give you the whole picture at once, rather than letting you see the scene be drawn in pieces, so typically it will double buffer -- displaying one buffer while drawing on the hidden buffer, and then swap. This creates a frame rate just like movie films. The game could just render that as fast as the hardware allows, but again, that can vary quite a bit. Many games might want to synchronize internal game state with what is displayed, so it sets a frame rate and artificially limits the hardware to that rate. (Or, if the hardware can't keep up, you skip frames and get something lower but still synchronized.)

Xvfb doesn't have a natural frame rate. It may have a maximum frame rate. Your recording software may have a maximum frame rate limited by the pipeline from capturing the image, processing it, and writing it to disk.

Ideally, you would be able to get hints from xvfb and only save a frame after there are changes made and they have been fully drawn, but this isn't a frame rate, this would be frames with variable timing. Or just save the sequence of drawing commands with no frames.

Presumably, you want a frame rate because you picked a video codec to record with that uses a fixed frame rate rather than frames with variable times or drawing primitives.

Your best bet might be to just pick an arbitrary frame rate corresponding to the output file size you want, that isn't so slow that it looks jumpy.

user10489
  • 5,834
  • 1
  • 10
  • 22
  • In retrospect, this could be an XY problem. Rather than ask what framerate to record with, maybe you should ask if there is a better video codec and corresponding recording software that can record frames with variable times. – user10489 Jan 27 '23 at 12:29