Looking for the latest and fastest way to extract single frames from video.

Guitarmonster

Member
Joined
Mar 24, 2023
Messages
19
Programming Experience
10+
I have a media player project that I am developing in c# and am looking for something that can give me the ability to extract single frames from a video as quickly as possible. The entire purpose is simply for my code to analyze a tv show episode, and search for spots where commercials are going to begin and end.

Here is what I have done so far. I am using Accord.Net to open a video file, then iterate through one frame per second converting that frame to a Bitmap object, then establishing a 10x10 pixel grid to "sample" the pixels and measure their brightness. The purpose of this is for my code to look for the "drop out" where the screen fades to black, leading to the commercial part of the video. So far my test code works nearly perfectly and is highly accurate, however, using Accord.Net is extremely slow, especially for MKV (h264) files.

There is a discussion of this project at the following thread:
Aforge only working for certain videos, when extracting frames as bitmaps

At this point I am playing around in Windows Presentation Foundation using the .Net framework 4.8. If there is a newer application type I should be using please let me know and I will switch. I am looking for performance to be number one. So far WPF has demonstrated great performance with playing video as well as applying pixel shader effects to the video, but I am certainly open to a better option if needed.

The real thing is finding a way to replace Accord.Net, which uses FFMPEG to extract data. Under the Accord.Net framework, there is supposed to be a way to extract single frames as raw bitmaps, which another designer tells me should be faster. However, it seems that this specific override in Accord either has a design bug, or has not yet been completed, so the only option is to extract actual Bitmap objects themselves.

I'm a very well seasoned programmer, but graphics is not my biggest subject.

As a side note, I am also considering analyzing the audio of the file instead and looking for audio drop outs that are consistent with a program change, then having my code do second confirmation by analyzing only those select frames as well.
 
FFMPEG can be used to inspect for black frames in command line mode, which should also be possible to do and read programmatically (process redirect output). Here are two examples I found, you'll have to research these further for options/parameters:
C#:
ffmpeg -i "V:\sample.mp4" -vf blackframe=amount=98:threshold=32 -f null -
output:
[Parsed_blackframe_0 @ 00000224d3c18c00] frame:30761 pblack:98 pts:30791761 t:513.196017 type:p last_keyframe:30736
C#:
ffmpeg -i "V:\sample.mp4" -vf "blackdetect=d=0.05:pix_th=0.10" -an -f null -
output:
[blackdetect @ 000002213003d140] black_start:513.096 black_end:514.631 black_duration:1.53487
Both methods took about 1 second to process 1 minute video when I tested.
 
FFMPEG can be used to inspect for black frames in command line mode, which should also be possible to do and read programmatically (process redirect outout). Here are two examples I found, you'll have to research these further for options/parameters:
C#:
ffmpeg -i "V:\sample.mp4" -vf blackframe=amount=98:threshold=32 -f null -
output:
[Parsed_blackframe_0 @ 00000224d3c18c00] frame:30761 pblack:98 pts:30791761 t:513.196017 type:p last_keyframe:30736
C#:
ffmpeg -i "V:\sample.mp4" -vf "blackdetect=d=0.05:pix_th=0.10" -an -f null -
output:
[blackdetect @ 000002213003d140] black_start:513.096 black_end:514.631 black_duration:1.53487
Both methods took about 1 second to process 1 minute video when I tested.

THIS is the way -Yoda

I just threw together some simple C# code to implement the second option, which by what I have found is specifically designed for this very reason. So I've been reinventing the wheel. I tweaked out the settings and read the output, then tested the results by entering in the timestamp in seconds to go to the specific spots and so far it's dead on accurate. The scanning is really fast as well, although it seems MKV files are a little slower, but overall it's not too bad. There are a few false positives here and there, but I can see that those are all very low duration sections of the video, which I can avoid by further playing around with the settings.

C#:
public void findCommercialSpots(Uri Source)
{

    string strArguments = "-i \"" + Source.OriginalString + "\" -vf \"blackdetect=d=0.1:pix_th=0.10\" -an -f null -";

    var process = new Process
    {
        StartInfo = new ProcessStartInfo
        {
            FileName = "C:\\Applications\\ffmpeg\\ffmpeg.exe",
            Arguments = strArguments,
            UseShellExecute = false,
            RedirectStandardOutput = true,
            CreateNoWindow = false,
            RedirectStandardError = true
            },
        EnableRaisingEvents = true
        };

    Console.WriteLine("Starting operation");

    process.Start();

    Console.WriteLine(process.StandardError.ReadToEnd());

    Console.WriteLine("Operation complete");

}

Although I found it odd, I did have to set RedirectStandardError on the process since ffmpeg outputs as errors instead of standard output. I learned that after watching it running and getting no output at all.

The code resulted in this output, which I will now write some code to scrape out the data and use it to bookmark the database:

C#:
ffmpeg version N-110188-g6d31619af2-20230408 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 12.2.0 (crosstool-NG 1.25.0.152_89671bf)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-w64-mingw32- --arch=x86_64 --target-os=mingw
32 --enable-gpl --enable-version3 --disable-debug --disable-w32threads --enable-pthreads --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --en
able-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --disable-libpulse --enable-libvmaf --disable-libxcb --disable
-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-chromaprint --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-
ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libkvazaar --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable
-libopus --enable-librist --enable-libssh --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --disable-libmfx --enable-libvpl --enable-openal --en
able-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --enabl
e-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --disable-libdrm --disable-vaapi --enable-li
bvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-
libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags= --extra-libs=-lgomp --extra-version=20230408
  libavutil      58.  6.100 / 58.  6.100
  libavcodec     60.  9.100 / 60.  9.100
  libavformat    60.  4.101 / 60.  4.101
  libavdevice    60.  2.100 / 60.  2.100
  libavfilter     9.  5.100 /  9.  5.100
  libswscale      7.  2.100 /  7.  2.100
  libswresample   4. 11.100 /  4. 11.100
  libpostproc    57.  2.100 / 57.  2.100
Input #0, avi, from 'C:\Test\The-Transformers-S01E01.avi':
  Duration: 00:22:28.23, start: 0.000000, bitrate: 1335 kb/s
  Stream #0:0: Video: mpeg4 (Simple Profile) (XVID / 0x44495658), yuv420p, 640x480 [SAR 1:1 DAR 4:3], 1195 kb/s, 25 fps, 25 tbr, 25 tbn
  Stream #0:1: Audio: mp3 (U[0][0][0] / 0x0055), 44100 Hz, stereo, fltp, 128 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (mpeg4 (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf60.4.101
  Stream #0:0: Video: wrapped_avframe, yuv420p(progressive), 640x480 [SAR 1:1 DAR 4:3], q=2-31, 200 kb/s, 25 fps, 25 tbn
    Metadata:
      encoder         : Lavc60.9.100 wrapped_avframe
[blackdetect @ 000000cf9002df00] black_start:0 black_end:1.84 black_duration:1.840x
[blackdetect @ 000000cf9002df00] black_start:33.96 black_end:35.08 black_duration:1.12
[blackdetect @ 000000cf9002df00] black_start:348.2 black_end:348.24 black_duration:0.04
[blackdetect @ 000000cf9002df00] black_start:501.32 black_end:502.8 black_duration:1.48
[blackdetect @ 000000cf9002df00] black_start:638.68 black_end:638.72 black_duration:0.04
[blackdetect @ 000000cf9002df00] black_start:681.92 black_end:681.96 black_duration:0.04
[blackdetect @ 000000cf9002df00] black_start:880.44 black_end:880.48 black_duration:0.04
[blackdetect @ 000000cf9002df00] black_start:892.04 black_end:892.8 black_duration:0.76
[blackdetect @ 000000cf9002df00] black_start:978.08 black_end:978.12 black_duration:0.04
[blackdetect @ 000000cf9002df00] black_start:1011.68 black_end:1011.72 black_duration:0.04
[blackdetect @ 000000cf9002df00] black_start:1067.16 black_end:1067.2 black_duration:0.04
[blackdetect @ 000000cf9002df00] black_start:1278 black_end:1278.52 black_duration:0.52
[blackdetect @ 000000cf9002df00] black_start:1298.28 black_end:1298.44 black_duration:0.16
[blackdetect @ 000000cf9002df00] black_start:1339.68 black_end:1339.8 black_duration:0.12
frame=33705 fps=3037 q=-0.0 Lsize=N/A time=00:22:28.16 bitrate=N/A speed= 121x
video:15799kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[blackdetect @ 000000cf9002df00] black_start:1347.28 black_end:1348.16 black_duration:0.88
 
Great, for reference found general options here ffmpeg Documentation and filters settings here: FFmpeg Filters Documentation
Duration was set very low in that example, default is 2 seconds.
It's true that ffmpeg sends logging to stderr.
I also played with cropping filter to process a smaller part of frame, but not much time was saved, and could get less reliable results.
C#:
-vf crop=100:100,blackdetect=d=1.0:pix_th=0.10
 
Great, for reference found general options here ffmpeg Documentation and filters settings here: FFmpeg Filters Documentation
Duration was set very low in that example, default is 2 seconds.
It's true that ffmpeg sends logging to stderr.
I also played with cropping filter to process a smaller part of frame, but not much time was saved, and could get less reliable results.
C#:
-vf crop=100:100,blackdetect=d=1.0:pix_th=0.10

With how fast this is processing I have no need to crop. Video files only have to be scanned one time, then the data is stored in the database. The great part is I planned on a media scanning engine anyway for the purpose of extracting video data (resolution, duration, etc.) With this option I'm killing two birds with one stone.

The only thing I really need to tweak is the duration of the black areas, which tends to be slightly different between shows. So the solution is for each show to be treated differently with different settings. I can have it load the first episode or better yet, sample a few random episodes and allow me to test preview it's settings and tweak them. After confirming the settings I can then commit and run the scan over every file.

I'll create a scan queue that will process videos while the television is turned off, preferably using multiple threads if ffdshow will allow.

I've even tested it with a few movie files and it seems to find chapters fairly well, with some creative tweaking of course as every movie is done differently.
 
Back
Top Bottom