RenderDoc Mac GL Progress : Mesh Viewer Window

I wired up the Mesh Viewer window code (BufferViewer.cpp) with a copy of the code from the Texture Viewer window code ie.

void BufferViewer::UI_CheckResize()
{
#if defined(RENDERDOC_PLATFORM_APPLE)
  m_Ctx.Replay().AsyncInvoke(lit("UI_CheckResize"), [this](IReplayController *) {
    if(m_Output && m_Output->GetPendingResizeMT())
    {
      GUIInvoke::blockcall(this, [this]() { UI_SetViewContextMT(); });
    }
  });
#endif // #if defined(RENDERDOC_PLATFORM_APPLE)
}

void BufferViewer::UI_SetViewContextMT()
{
#if defined(RENDERDOC_PLATFORM_APPLE)
  if(m_Output)
    m_Output->SetViewContextMT();
#endif // #if defined(RENDERDOC_PLATFORM_APPLE)
}

and it worked first time

Next steps are to iron out the resize events not always triggering the UI updates I need. Baldur found this Apple API in use in the MoltenVk source code which allows any thread to run a block of Objective-C code on the main thread which is precisely what I need to do. The caveat is I need to make sure the ReplayThread is not running any commands and is blocked from running commands whilst these main thread commands execute.

RenderDoc Mac GL Update

Made steady progress this morning. The flickering was the thumbnails never being reused (the CALayer returned by GetHandle() changes on GL but was constant on Vulkan). At one point there was 128 thumbnails when there should have been 2. Fixing that makes performance much nicer and removes most of the flickering.

Overall it is working well. One thing not quite right is the main texture window/pane resize seems to be required to trigger things properly ie. resize on the pixel context pane with the thumbnails not being picked up as a resize event properly. I need to debug that.
Still a bit of flickering in the video but the UI feels much nicer to use today than it did yesterday (even in a debug build)  

RenderDoc Mac GL Progress

Working to overcome the design discrepency between MacOS which needs the OS UI APIs to be called from the main thread and RenderDoc which performs OS UI APIs on its Replay thread. These are the NSOpenGLContext APIs need to be called on the main thread:

setView:view
context update

The Qt UI provides events on the main thread which could be used however the required data is on the Replay thread and private to the RenderDoc platform and API implementations. Also these main thread API calls need to be done in co-ordination with the Replay thread ie. the Replay thread can’t be rendering when these calls are peformed on the main thread, it needs some kind of lock stepping. RenderDoc already has delegate methods to invoke function calls on the Replay thread from the main thread and vice-versa. One way to achieve the required lock stepping is to have the Replay thread perform a blocking invoke of a method on the main thread. Whilst tackling this also sorted out the high DPI scaling issue which meant the cursor hover position did not line up with the image being viewed.

Adding a quick version of this blocking invoke allows the Texture Window rendering to work in RenderDoc Mac GL.

Here is a quick video showing the first prototype version running.

This is very much a prototype and there is lots more still to do. This only has the Texture Window working and not the Mesh Window. I need to debug the cross thread delegation to make sure it is optimal and only calling the APIs when required.

RenderDoc Mac GL Debugging

Whilst experimenting and debugging the RenderDoc Mac GL rendering issue (which looked similar to the Vulkan high DPI issue) I discovered that my fix for Vulkan high DPI needs changing. The fix is a calling a main thread only API on the replay thread. A quick fix should be quite easy, not quite sure if the quick fix will handle the scenario of the RenderDoc window starting on a high DPI monitor and being moved to a normal DPI monitor (or vice-versa), that scenario currently works.

Spent some time exploring the RenderDoc Mac GL TextureViewer rendering, it is not as previously suspected a high DPI issue. The raw GL rendering is not correct on Mac in the Texture Viewer. The background pattern which should be squares are not squares and the pixel context cursor is not the correct shape. Both of them are somehow not being corrected for the aspect ratio of the window. I have checked the raw texture contents and that is as expected and the right dimensions. This incorrect rendering is also why the cursor position and the hover coordinates are not lining up in the TextureViewer display.

RenderDoc GL TextureViewer showing squares are not squares.

TextureViewer Output for the same Event ID, showing the raw data is correct and the rendering of it is the problem.

Whilst experimenting with comparing GL rendering with Vulkan rendering I discovered a Vulkan crash if you open a Vulkan capture, close it, open a Vulkan capture then RenderDoc replay crashes in vkSubmit. It might be similar to the shutdown crash I found in the RenderDoc Vulkan automated tests. To debug that further I think I need to get MoltenVk source debugging working.

Quick flash update on RenderDoc Mac GL progress

Very quick update from progress on home coding this morning (I do not sleep much and use the early morning hours to code when I can) working on RenderDoc Mac GL version. Very quickly (hackily) wired up the code throught the various interfaces to allow the setView call on the NS OpenGL context to happen on the main thread. First results are promising (note I have not yet implemented high DPI fixes for OpenGL):

Quick video showing the texture viewer display updating when different rendering events are selected

Home Coding Updates from the past few days

My previous RenderDoc upstream PR (Apple fixes related to RPATH, Python support) was merged. Started the next RenderDoc upstream PR which has some minor RenderDoc fixes which have been found whilst working on the Apple version ie. Apple specific fixes, common code which is relying on GL extensions which Apple do not support.

Whilst testing one of the changes got side tracked by an odd Apple error about not being able to inject into the process (despite the injection working in the automated tests). The error is perfectly legitimate and makes sense you can not inject an x64 process with an arm64 dylib. The background was the app I was trying to capture was was my own test app (not the RenderDoc test app). I had built this app locally on an M1 Mac Book Air and it made no immediate sense that it was being compiled as an x64 application and not an arm64 application. It turns out when running “make” from an x64 app (in my case Sublime Text) then the process environment is x64 and not arm64 ie. uname, arch, platform all return x64. If I built the same project from the command line then it was an arm64 application. I have now hard coded my test makefile to make arm64 target. I think this means Sublime Text must be an x64 application running via Rosetta 2.

I explored the Metal + GL rendering interop texture idea from Apple and it is elegant. I was also impressed with the Apple documentation for once. My previous experiences with Apple documnetation have been quite poor when searching for details on OpenGL, dynamic library loading and hooking. However I do not think helps with the RenderDoc problem as the NSOpenGlContext is still used for the OpenGL rendering and it has to call setView from the main thread which is the current problem for RenderDoc Mac GL version.

Now exploring the callstack of when setView is called from the non-main thread (the RenderDoc replay thread). Trying to think about how to change the RenderDoc UI code to work with the Apple rules that “setView” must be called on the main thread.

Made a proposed PR with the high DPI fix for RenderDoc Mac Vulkan UI, which was making it hard to use the TextureView window with the data at the cursor position not matching what you are seeing.

Before the high DPI fix

After the high DPI fix

Home Coding

For the past few weeks have started coding at home again on the RenderDoc for Mac project. I am using an M1 Mac Book Air as my development platform.

Getting RenderDoc for Mac up and running again locally with a goal to get the automated tests for OpenGL and Vulkan running has involved learning about:

• cmake 3.19 differences in how it finds the python module compared to earlier cmake versions.
• cmake configuration for RPATH setup in Mac executables and dynamic libraries
• Python importing of dynamic libraries (on non-Windows platforms it will only load *.so libraries whereas Mac default is *.dylib).
• MoltenVk installation
• nuklear, Cocoa, Metal and OpenGL on Mac to get a minimal framework to run the tests (like GLFW/Xlib/XCB/ but reduced in scope and implemented as a single source and header file).

The current status for the RenderDoc OpenGL automated tests on Mac are the most basic tests are passing on Mac locally (the branch is not merged). Almost all of the OpenGL tests require OpenGL 4.2 or higher however the Mac only supports OpenGL 4.1 which means changing the existing tests to make them run for Mac which is not desirable because of its impact to non-Mac platforms.

The current status for the RenderDoc Vulkan automated tests on Mac is better than the OpenGL tests with over 20 tests passing. The automated tests have found:

• RenderDoc support for MSAA is not working correctly on Mac (verified when using the RenderDoc UI with a sample vulkan application).
• crashes in the MoltenVk driver where it tries to de-reference invalid pointers which according to the Vullkan specification should not be accessed (I believe other Vulkan drivers in the past had similar problems).
• a RenderDoc shutdown crash in the MoltenVk driver where the internal Metal image pointer is null during a clear image command.

Now the automated tests have verified the basic RenderDoc capture and replay code for Mac OpenGl, the next steps are to tackle the RenderDoc UI integration. The immediate challenge is RenderDoc UI has a main thread for all its UI work and a Replay thread which does all of the OpenGL rendering. On Mac calls to the OS windowing libraries must only be made on the main thread which is at odds with how the RenderDoc UI is implemented and wants to work…