Question Picturebox not refresh when the form is in background (not focus)

[shafick]

Purpose: Mirror a window of an external process (directx game) by capturing images from this window and presenting the images in a picturebox using a timer.

Problem: When my application is in focus, everything happens fine! But if I switch the focus to the Directx Game window the picturebox stops refreshing the images. When I focus on my application again, everything returns to normal.

Limitation: I need the picturebox to continue updating even though my application is in the background.

Important: This does not occur with windows such notepad, calc, browser, etc. Only with the Directx Game Window.

Timer to capture images and show in picturebox:

private void timer_picBOX_refresh_Tick(object sender, EventArgs e)
{
   pictureBox1.BackgroundImage = PrintScreen.CaptureWindow(GAME_MainHandle);            
   pictureBox1.Refresh();            
}

Class to capture specific window:

public class class_ScreenCapture
    {
        public Image CaptureScreen()
        {
            return CaptureWindow(User32.GetDesktopWindow());
        }
     
        /// <summary>
        /// Creates an Image object containing a screen shot of a specific window
        /// </summary>
        public Image CaptureWindow(IntPtr handle, int imgX = 0, int imgY = 0, int largura = 0, int altura = 0)
        {
            // get te hDC of the target window
            IntPtr hdcSrc = User32.GetWindowDC(handle);
            // get the size
            User32.RECT windowRect = new User32.RECT();
            User32.GetWindowRect(handle, ref windowRect);

            if (largura == 0 || altura == 0)
            {
                largura = windowRect.right - windowRect.left;
                altura = windowRect.bottom - windowRect.top;
            }

            // create a device context we can copy to
            IntPtr hdcDest = GDI32.CreateCompatibleDC(hdcSrc);
            // create a bitmap we can copy it to,
            // using GetDeviceCaps to get the width/height
            IntPtr hBitmap = GDI32.CreateCompatibleBitmap(hdcSrc, largura, altura);
            // select the bitmap object
            IntPtr hOld = GDI32.SelectObject(hdcDest, hBitmap);
            // bitblt over
            GDI32.BitBlt(hdcDest, 0, 0, largura, altura, hdcSrc, imgX, imgY, GDI32.SRCCOPY);
            // restore selection
            GDI32.SelectObject(hdcDest, hOld);
            // clean up 
            GDI32.DeleteDC(hdcDest);
            User32.ReleaseDC(handle, hdcSrc);

            // get a .NET image object for it
            Image img = Image.FromHbitmap(hBitmap);
            // free up the Bitmap object
            GDI32.DeleteObject(hBitmap);

            return img;
        }        

        /// <summary>
        /// Helper class containing Gdi32 API functions
        /// </summary>
        private class GDI32
        {

            public const int SRCCOPY = 0x00CC0020; // BitBlt dwRop parameter

            [DllImport("gdi32.dll")]
            public static extern bool BitBlt(IntPtr hObject, int nXDest, int nYDest,
                int nWidth, int nHeight, IntPtr hObjectSource,
                int nXSrc, int nYSrc, int dwRop);
            [DllImport("gdi32.dll")]
            public static extern IntPtr CreateCompatibleBitmap(IntPtr hDC, int nWidth,
                int nHeight);
            [DllImport("gdi32.dll")]
            public static extern IntPtr CreateCompatibleDC(IntPtr hDC);
            [DllImport("gdi32.dll")]
            public static extern bool DeleteDC(IntPtr hDC);
            [DllImport("gdi32.dll")]
            public static extern bool DeleteObject(IntPtr hObject);
            [DllImport("gdi32.dll")]
            public static extern IntPtr SelectObject(IntPtr hDC, IntPtr hObject);
        }

        /// <summary>
        /// Helper class containing User32 API functions
        /// </summary>
        private class User32
        {
            [StructLayout(LayoutKind.Sequential)]
            public struct RECT
            {
                public int left;
                public int top;
                public int right;
                public int bottom;
            }

            [DllImport("user32.dll")]
            public static extern IntPtr GetDesktopWindow();
            [DllImport("user32.dll")]
            public static extern IntPtr GetWindowDC(IntPtr hWnd);
            [DllImport("user32.dll")]
            public static extern IntPtr ReleaseDC(IntPtr hWnd, IntPtr hDC);
            [DllImport("user32.dll")]
            public static extern IntPtr GetWindowRect(IntPtr hWnd, ref RECT rect);

        }
    }

 

[jmcilhinney]

Have you done anything to diagnose the issue? Have you checked whether the Tick event of the Timer is still being raised and handled? Have you checked that that CaptureWindow method is being executed as expected? Don't just read the code. Debug it.

Also, is there a particular reason that you're using the BackgroundImage property of the PictureBox instead of the Image property?

 

[shafick]

Have you done anything to diagnose the issue? Have you checked whether the Tick event of the Timer is still being raised and handled? Have you checked that that CaptureWindow method is being executed as expected? Don't just read the code. Debug it.

Also, is there a particular reason that you're using the BackgroundImage property of the PictureBox instead of the Image property?

Like I said, when the form is focused, everything works correctly. The problem only occurs when the form loses focus (the picturebox stay static and does not change the image). When I focus back on the form, everything returns to normal.

 

[jmcilhinney]

Like I said, when the form is focused, everything works correctly. The problem only occurs when the form loses focus (the picturebox stay static and does not change the image). When I focus back on the form, everything returns to normal.

I know you said that. I read it the first time. I'm not sure why you though that it would be a good idea to repeat what you'd already posted and ignore everything I said.

 

[shafick]

I know you said that. I read it the first time. I'm not sure why you though that it would be a good idea to repeat what you'd already posted and ignore everything I said.

Sorry if it seemed offensive, some linguistic forms may seem aggressive to others when translated by online translator (without interpretation). In my language it is common to start a sentence with, "like I said". But this is no problem. Sry.

The whole code of my application continues to work when it is in the background, the logical part does not freeze. What gets frozen is the visual part (UI). As soon as I turn the focus to the window of my application it returns to be "alive" and from the images it is clear that the code continued to run while it was in the background.

This freezing does not occur when I focus on windows like webbrowser, notepad, calculator, etc. When these window types are focused the visual UI of my application continues "alive".

Freezing only occurs when I focus on the game window (directx game). When the game window gets focus, I can see the window of my application frozen. The problem is caused by any pattern that this type of window imposes on windows like winform type, etc.

In my opinion the beginning of the way is to know why this type of window prevents winform window from continuing to refresh the look.

Do you know when you open a game window and the mouse is limited to the edges of this window? And is necessary to press ALT+TAB to out of the window? It's this kind of window I'm talking about.

 

[Yann]

Did you get a solution to the problem? I'm having a similar problem.

In my case, I have two Forms. In each Form a live camera stream is shown. I am using a timer to update the pictureBox.

Problem: only de active (focused) Form is being updated, the second one is like frozen. When I change to the other Form, then the first Form gets frozen.

First attempt: I used events, triggered when a new frame was captured. This would update both Forms simultaneously. The problem was that the event handling was too slow, reducing my camera's FPS from 60 to 10.

Some help would be very appreciated.

Thx

 

[Skydiver]

How are you showing the two forms? Are both forms modeless forms being displayed using Form.Show(), or are you using something else?

Show us your the relevant parts of your code.

 

[Yann]

Hi skydiver,

Unfortunately I do not have the code with me right now, but I can explain my code in a general way.

In my main method I use:
Application.Run(new ApplicationContext(new Monitor1(), new Monitor2()));

This method starts both Forms. This two Forms inherit variables, methods and the Form Design from MonitorBase. The show() method is called on each instance (of Monitor1 and Monitor2) after the cameras have started streaming.

Right now I have a private timer in MonitorBase, which triggers the protected GrabFrame() method. This method is overriden in Monitor1 and Monitor2 to update the pictureBox with the grabed frame, respectively.

I can change from one form to the other using the numpad. Pushing 6, I call the select() on the instance of Monitor2. Pushing 4, calls select() on the instance of Monitor1.

I hope my explanation brings some clarity.

 

[Skydiver]

Works fine for me. I don't have a picture box, but I do have label which I update every 500 ms.

using System;
using System.Windows.Forms;

abstract class MonitorBase : Form
{
    protected Label _timeLabel;

    public MonitorBase(string caption)
    {
        Text = caption;

        _timeLabel = new Label()
        {
            Dock = DockStyle.Fill,
            AutoSize = true,
            TextAlign = System.Drawing.ContentAlignment.MiddleCenter,
        };

        Controls.Add(_timeLabel);

        var timer = new Timer() { Interval = 500 };
        timer.Tick += (o, e) => GrabFrame();
        timer.Start();
    }

    protected abstract void GrabFrame();
}

class Monitor : MonitorBase
{
    int _id;

    public Monitor(int id) : base($"Monitor {id}")
        => _id = id;

    protected override void GrabFrame()
        => _timeLabel.Text = $"{_id}: {DateTime.Now}";
}

class Context : ApplicationContext
{
    public Context(MonitorBase mon1, MonitorBase mon2)
        : base(mon1)
    {
        mon2.Show();
        mon1.Show();
    }
}

class Program
{
    [STAThread]
    static void Main()
    {
        Application.EnableVisualStyles();
        Application.SetCompatibleTextRenderingDefault(false);
        Application.Run(new Context(new Monitor(1), new Monitor(2)));
    }
}

 

[Skydiver]

And to eliminate the chance that it's the keypress handling that is interfering with the screen updates, here's an the code above modified to handle keypresses:

using System;
using System.Collections.Generic;
using System.Windows.Forms;

abstract class MonitorBase : Form
{
    protected readonly Label _timeLabel;

    public Keys Key { get; }

    public MonitorBase(string caption, Keys key)
    {
        Text = caption;
        Key = key;

        _timeLabel = new Label()
        {
            Dock = DockStyle.Fill,
            AutoSize = true,
            TextAlign = System.Drawing.ContentAlignment.MiddleCenter,
        };

        Controls.Add(_timeLabel);

        var timer = new Timer() { Interval = 500 };
        timer.Tick += (o, e) => GrabFrame();
        timer.Start();
    }

    protected abstract void GrabFrame();
}

class Monitor : MonitorBase
{
    readonly int _id;

    public Monitor(int id, Keys key)
        : base($"Monitor {id}", key)
    {
        _id = id;
    }

    protected override void GrabFrame()
        => _timeLabel.Text = $"{_id}: {DateTime.Now}";
}

class Context : ApplicationContext
{
    Dictionary<Keys, MonitorBase> _keyMonitor = new Dictionary<Keys, MonitorBase>();

    public Context(MonitorBase mon1, MonitorBase mon2)
        : base(mon1)
    {
        RegisterMonitor(mon1);
        RegisterMonitor(mon2);

        mon2.Show();
        mon1.Show();
    }

    void RegisterMonitor(MonitorBase monitor)
    {
        _keyMonitor[monitor.Key] = monitor;

        monitor.KeyPreview = true;
        monitor.KeyDown += Monitor_KeyDown;
    }

    void Monitor_KeyDown(object sender, KeyEventArgs e)
    {
        if (_keyMonitor.ContainsKey(e.KeyCode))
        {
            _keyMonitor[e.KeyCode].Select();
            e.Handled = true;
        }
    }
}

class Program
{
    [STAThread]
    static void Main()
    {
        Application.EnableVisualStyles();
        Application.SetCompatibleTextRenderingDefault(false);
        Application.Run(new Context(new Monitor(1, Keys.NumPad4),
                                    new Monitor(2, Keys.NumPad6)));
    }
}

 

[Yann]

Hi Skydiver,

You were right, my timer was also working. The problem resides indeed somewhere else. Let me recap.

In my parent Form class I used one timer to update the pictureBox of each child. Because my camera streams 60 fps, my timer should be triggering all 10 milliseconds (sampling freq > 1.2* system freq).

Problem: At that rate only one pictureBox gets updated. As if the pictureBox update would take to much time to update and the application couldn't follow. If I set the rate to 100 milliseconds both pictureBoxes get updated flowlessly....unfortunately too slow for my goal.

Because the problem is more related to the slow pictureBox update, I will open a new post. But before that, I want to leave my performed tests also in here.

Test: reading a couple of pages on the internet I found out the pictureBox tool ist designed mostly for static images. I tried the next alternatives (with unsuccessful results):

- EMGU library + COM, using the imageBox tool
- tried embedding a Windows Media Player, but the tool is only for 32bit projects
- WPF instead of WinForms. Apparently it uses the GPU but the CPU usage is still high.
- Isolate each child into his own thread.

I am sure that the problem relies on the update of the pictureBox/imageBox, cause I receive the camera frames at 16 milliseconds (60 fps). The only step between the newframe (as pointer) and the pictureBox.Image = bitmap, is the instantiation of the bitmap using the pointer:
Bitmap bitmap = new Bitmap(...., frameData).

Why does the update of a pictureBox need that much CPU? My 7 cores are for both childs at around 85%.

Bye