RenderDoc Mac GL Unity Editor Testing

Looking for larger OpenGL applications to test with I chose the Unity Editor application. Had a short detour to work out how to get Unity Editor to run in OpenGL mode instead of Metal, turns out the Apple Silicon version of Unity Editor (and standalone) only support the Metal API. Running the Unity Editor on an Intel Mac supports the OpenGL API. Not much to say except it worked first time, capturing Unity Editor from RenderDoc Mac and then replaying it on Intel or Apple Silicon RenderDoc UI. Here are some screenshots of it replaying in RenderDoc UI:

I have not experienced any crashes or problems during the capturing or replay.

The next steps are to review the changes to support running Apple UI calls on the main thread and open a PR to upstream them.

RenderDoc Mac DisplayRendererPreview Support

Whilst using the remote server feature for renderdoccmd, I tried out a few more of the command line tool options to see if they worked on Apple.

The thumb option would generate an image preview from the replay of a capture. It just worked ie.

Next was the replay option which should display a window showing the preview from the replay of the capture. This required platform specific implementation of this method

void DisplayRendererPreview(IReplayController *renderer, TextureDisplay &displayCfg, uint32_t width,
uint32_t height, uint32_t numLoops)

I had various bits of Cocoa window handling code from the nuklear_cocoa implementation that was done to support the RenderDoc Mac automated tests. This requirement was simpler, it only needed a window, the GL context and drawing would all be handled by the renderdoc library code.

I started from scratch and ended up implementing these APIs

extern void *cocoa_windowCreate(int width, int height, const char *title);
extern void *cocoa_windowGetView(void *cocoaWindow);
extern void *cocoa_windowGetLayer(void *cocoaWindow);
extern bool cocoa_windowShouldClose(void *cocoaWindow);
extern bool cocoa_windowPoll(unsigned short &appleKeyCode);

and then the preview code (which was adapted from the linux platform version of the preview code) became

void DisplayRendererPreview(IReplayController *renderer, TextureDisplay &displayCfg, uint32_t width, uint32_t height, uint32_t numLoops)
{
  void *cocoaWindow = cocoa_windowCreate(width, height, "renderdoccmd");
  void *view = cocoa_windowGetView(cocoaWindow);
  void *layer = cocoa_windowGetLayer(cocoaWindow);
  IReplayOutput *out =
      renderer->CreateOutput(CreateMacOSWindowingData(view, layer), ReplayOutputType::Texture);

  out->SetTextureDisplay(displayCfg);

  uint32_t loopCount = 0;

  bool done = false;
  while(!done)
  {
    if(cocoa_windowShouldClose(cocoaWindow))
    {
      break;
    }

    unsigned short appleKeyCode;
    if(cocoa_windowPoll(appleKeyCode))
    {
      // kVK_Escape
      if(appleKeyCode == 0x35)
      {
        break;
      }
    }

    renderer->SetFrameEvent(10000000, true);
    out->Display();

    usleep(100000);

    loopCount++;

    if(numLoops > 0 && loopCount == numLoops)
      break;
  }
}

I even included support for pressing the Escape key to close the preview window.

The Apple specific code is quite straight forward and basic

#import <Cocoa/Cocoa.h>
#import <QuartzCore/CAMetalLayer.h>

struct cocoa_Window
{
  NSWindow *nsWindow;
  NSView *nsView;
  bool shouldClose;
};

@interface cocoa_WindowApplicationDelegate : NSObject
@end

@implementation cocoa_WindowApplicationDelegate 

 - (void)applicationDidFinishLaunching:(NSNotification *)notification
{
  @autoreleasepool
  {
    NSEvent *event = [NSEvent otherEventWithType:NSEventTypeApplicationDefined
      location:NSMakePoint(0, 0)
      modifierFlags:0
      timestamp:0
      windowNumber:0
      context:nil
      subtype:0
      data1:0
      data2:0];
    [NSApp postEvent:event atStart:YES];
    [NSApp stop:nil];
  }
}

@end

@interface cocoa_WindowDelegate : NSObject
{
  cocoa_Window *window;
}

- (instancetype)initWithCocoaWindow:(cocoa_Window *)initWindow;

@end

@implementation cocoa_WindowDelegate 

- (instancetype)initWithCocoaWindow:(cocoa_Window *)initWindow
{
  self = [super init];
  assert(self);
  window = initWindow;
  return self;
}

- (BOOL)windowShouldClose:(id)sender
{
  window->shouldClose = true;
  return NO;
}
@end

void *cocoa_windowCreate(int width, int height, const char* title)
{
  cocoa_Window* window = (cocoa_Window*)calloc(1, sizeof(cocoa_Window));

  @autoreleasepool
  {
    [NSApplication sharedApplication];
    [NSApp setActivationPolicy:NSApplicationActivationPolicyRegular];
    [NSApp activateIgnoringOtherApps:YES];

    id appDelegate = [[cocoa_WindowApplicationDelegate alloc] init];
    [NSApp setDelegate:appDelegate];

    if(![[NSRunningApplication currentApplication] isFinishedLaunching])
      [NSApp run];
  }

  NSRect contentRect;
  contentRect = NSMakeRect(0, 0, width, height);

  window->nsWindow = [[NSWindow alloc]
      initWithContentRect:contentRect
      styleMask:NSWindowStyleMaskTitled | NSWindowStyleMaskClosable | NSWindowStyleMaskMiniaturizable 
      backing:NSBackingStoreBuffered
      defer:NO];
  assert(window->nsWindow);

  window->nsView = [[NSView alloc] initWithFrame:contentRect];
  assert(window->nsView);
  [window->nsView setLayer:[CAMetalLayer layer]];
  [window->nsView setWantsLayer:YES];

  id windowDelegate = [[cocoa_WindowDelegate alloc] initWithCocoaWindow:window];
  assert(windowDelegate);

  [window->nsWindow center];
  [window->nsWindow setContentView:window->nsView];
  [window->nsWindow makeFirstResponder:window->nsView];
  [window->nsWindow setTitle:@(title)];
  [window->nsWindow setDelegate:windowDelegate];

  [window->nsWindow orderFront:nil];
  [window->nsWindow makeKeyAndOrderFront:nil];
  return (void *)window;
}

void *cocoa_windowGetView(void* cocoaWindow)
{
  cocoa_Window *window = (cocoa_Window*)cocoaWindow;
  assert(window);
  return (void*)(window->nsView);
}

void *cocoa_windowGetLayer(void* cocoaWindow)
{
  cocoa_Window *window = (cocoa_Window*)cocoaWindow;
  assert(window);
  return (void*)(window->nsView.layer);
}

bool cocoa_windowShouldClose(void* cocoaWindow)
{
  cocoa_Window *window = (cocoa_Window*)cocoaWindow;
  assert(window);
  return window->shouldClose;
}

bool cocoa_windowPoll(unsigned short& appleKeyCode)
{
  bool keyUp = false;
  @autoreleasepool
  {
    for(;;)
    {
      NSEvent *event = [NSApp nextEventMatchingMask:NSEventMaskAny
                              untilDate:[NSDate distantPast]
                              inMode:NSDefaultRunLoopMode
                              dequeue:YES];
      if(event == nil)
        break;

      if ([event type] == NSEventTypeKeyUp) 
      {
        appleKeyCode = [event keyCode];
        keyUp = true;
      }

      [NSApp sendEvent:event];
    }
  }
  return keyUp;
}

Here is the preview output when replaying the Karting tutorial (GL API)

Here is the preview output when replaying a test sample (Vulkan API)

RenderDoc Mac Keyboard Support

The RenderDoc library has keyboard support ie. F12 to trigger a capture in the hooked application. This had not been implemented in the Mac version. Locally I had code on an old branch with RenderDoc Mac capture key support. This implementation was very simple and queried the keyboard directly which worked but it meant pressing F12 on a different application would also trigger a capture ie. the keyboard events were not focussed to the application. Did some Apple API searching using the Dash tool and found something that looked like it would do what I wanted, which was to add an event listener to an existing application/window addLocalMonitorForEventsMatchingMask.

It did what it said on the tin and just worked. The Apple specific code ended up being straightforward and is now in PR review for upstreaming.

#import <Cocoa/Cocoa.h>

#define MAX_COUNT_KEYS (65536)
static bool s_keysPressed[MAX_COUNT_KEYS];
static id s_eventMonitor;

void apple_InitKeyboard()
{
    s_eventMonitor = [NSEvent addLocalMonitorForEventsMatchingMask:
    (NSEventMaskKeyDown|NSEventMaskKeyUp)
    handler:^(NSEvent *incomingEvent) {
        NSEvent *result = incomingEvent;
        //NSWindow *targetWindowForEvent = [incomingEvent window];
        //if (targetWindowForEvent == _window) 
        {
            unsigned short keyCode = [incomingEvent keyCode];
            if ([incomingEvent type] == NSEventTypeKeyDown) 
            {
                s_keysPressed[keyCode] = true;
            }
            if ([incomingEvent type] == NSEventTypeKeyUp) 
            {
                s_keysPressed[keyCode] = false;
            }
        }
        return result;
    }];
}

bool apple_IsKeyPressed(int appleKeyCode)
{
    return s_keysPressed[appleKeyCode];
}

The non-Apple specific code is more lines of code because have to translate between the RenderDoc key codes and the Apple virtual key codes which means lines of code like this:

switch(key)
{
    case eRENDERDOC_Key_A: appleKeyCode = kVK_ANSI_A; break;
    case eRENDERDOC_Key_B: appleKeyCode = kVK_ANSI_B; break;
    case eRENDERDOC_Key_C: appleKeyCode = kVK_ANSI_C; break;
    case eRENDERDOC_Key_D: appleKeyCode = kVK_ANSI_D; break;
    case eRENDERDOC_Key_E: appleKeyCode = kVK_ANSI_E; break;
    case eRENDERDOC_Key_F: appleKeyCode = kVK_ANSI_F; break;
    case eRENDERDOC_Key_G: appleKeyCode = kVK_ANSI_G; break;
    case eRENDERDOC_Key_H: appleKeyCode = kVK_ANSI_H; break;
    case eRENDERDOC_Key_I: appleKeyCode = kVK_ANSI_I; break;
    case eRENDERDOC_Key_J: appleKeyCode = kVK_ANSI_J; break;
    case eRENDERDOC_Key_K: appleKeyCode = kVK_ANSI_K; break;

It would be nice if OS’s could agree on keyboard scancodes and give applications a unified cross-platform value to test against.

RenderDoc Mac GL Testing using Unity

I wanted to test RenderDoc Mac GL with a larger project than the small samples I had been using A Unity application made sense to me. I am familar with using Unity on Mac and OpenGL. Hit a stumbling block the Apple Silicon support in Unity does not support OpenGL (it is Metal only). This meant I needed to use an Intel version of the RenderDoc library for capturing and replay. This turned out to be an opportunity to test/use/verify on Mac the RenderDoc remote server feature. I had never needed to use this feature on desktop (only used it when using RenderDoc on Android platform).

I switched to using an Intel Mac. For the Unity test project I downloaded the Unity Learn Karting Microgame. Changed the project to use the OpenGL API and built the standalone application. On the Intel Mac compiled the RenderDoc Mac GL code from the development branch. Starting the remote server on the Intel Mac was straight forward “renderdoccmd remoteserver“.

On the ARM Mac in the RenderDoc UI connected to the remote server via its IP address ie. Tools menu, Manage Remote Servers. From the UI could then launch applications remotely as if they were local (had to accept some Apple dialogs on the Intel machine about granting access to certain folders) and then it all just worked

I also made captures locally on the Intel Mac using RenderDoc UI and transferred those capture files to the ARM Mac and replayed them locally on the ARM Mac without any problems.

Here is an example of replaying locally on ARM Mac (with the original capture being made on Intel Mac)

RenderDoc Mac GL Testing with OpenGL Samples Pack

A couple of small Apple specific RenderDoc PRs have already been upstreamed (thanks to Baldur for reviewing and merging them). I wanted to do more testing before making a PR with the threading fixes for RenderDoc Mac GL. The change is currently isolated to Apple specific files in the RenderDoc platform code (the gl_common.h change is deleting some Apple specific code) ie.

To get some OpenGL test cases I used the OpenGL Samples Pack project. First hurdle was the FreeImage static library included in the project was compiled for Intel. I am working on ARM (M1 Mac Book Air [which is very nice BTW]). Solved this by getting the FreeImage source (at the correct version) and then modifying its makefiles to build an ARM64 target version of the static library and include that in the universal static library (Apple has a nice system for this using the lipo tool).

I tested RenderDoc Mac GL using about 30 different OpenGL samples. If the sample worked outside of RenderDoc (not all of the samples work on Apple OpenGL) then I was able to capture and replay the sample without significant errors/problems and the replay results matched the result of running the sample outside of RenderDoc. This process was about an hour long with a single launch of RenderDoc UI and then launching and capturing and replaying one GL sample after another.

During the testing I got errors from Qt when docking panes which is a specific Qt message and it seems harmless (I think this error is not specific to RenderDoc Mac version). I got one replay error where the shader did not bind for the postvs and I also got one crash in Apple dispatch_async method which did not make any sense. I was not able to reproduce either of these crashes or replay errors.

These OpenGL samples are unit test focussed and not very demanding. Some of the samples wrote to the depth buffer and it was good to see the depth buffer working during replay and visible in the UI.

RenderDoc Mac Vulkan Crash Debugging

During testing and using RenderDoc Mac UI with Vulkan captures I discovered a 100% crash which happened the second time a Vulkan capture was opened. The message in Xcode was

2021-03-25 06:22:15.603812+0000 qrenderdoc[62009:956064] *** -[CaptureMTLDepthStencilState retain]: message sent to deallocated instance 0x6008ae823fd0

I could not recreate this outside of Xcode and I traced it down to having "GPU Frame Capture" enabled in Xcode. Changing that option to Disabled and now the crashes have gone away.

I have seen various changes in behaviour/crashes/asserts/errors when having Xcode GPU Frame Capture enabled when running with the MoltenVk driver.

RenderDoc Mac GL & Vulkan Main Thread Checker Asserts

Spent a bit of time running RenderDoc Mac from Xcode with the "Main Thread Checker" option enabled.

On Vulkan this triggered due to a previous fix I did to support high DPI. I was calling an NSView method to convert layer size ie.

CGSize viewScale = [view convertSizeToBacking:layer.bounds.size];

This was changed to use the layer scale to avoid the use of the view object from the Replay thread ie.

const CGFloat scaleFactor = layer.contentsScale;
width = layer.bounds.size.width * scaleFactor;
height = layer.bounds.size.height * scaleFactor;

On RenderDoc Mac GL the layer bounds do not get automatically updated when the view resizes. This means have to get the window size from the view object, which is not allowed on the Replay thread.

To solve this implemented a simple caching system with the Replay thread reading the width and height from a cache keyed by the NSView pointer address. The cache is updated on the main thread via a dispatch_async call. The data ownership between the main thread and the Replay thread is clear with the Replay thread only ever reading from the cache and the main thread only ever writing to the cache. In addition the Replay thread is responsible for choosing which cache entry that the main thread should update. This simplicity and clarity of data ownership per thread is intended to avoid any threading problems ie. race conditions. The code ended up being nice and simple:

static void viewGetWindowSizeMT(NSView *view, int index)
{
  CALayer *layer = view.layer;
  assert([layer isKindOfClass:[CALayer class]]);

  const NSRect contentRect = [view frame];
  const CGFloat scaleFactor = layer.contentsScale;
  assert(index >= 0 && index < s_WindowCount);

  int width = contentRect.size.width * scaleFactor;
  int height = contentRect.size.height * scaleFactor;
  s_WindowWidths[index] = width;
  s_WindowHeights[index] = height;
}

void getAppleWindowSize(void *viewHandle, int &width, int &height)
{
  NSView *view = (NSView *)viewHandle;
  assert([view isKindOfClass:[NSView class]]);

  int index = getWindowIndex(view);
  if (index == -1)
  {
    index = s_WindowCount;
    assert(index >= 0 && index < MAX_WINDOWS_COUNT);
    s_WindowIndexes[index] = view;
    s_WindowWidths[index] = 0;
    s_WindowHeights[index] = 0;
    ++s_WindowCount;
  }
  width = s_WindowWidths[index];
  height = s_WindowHeights[index];
  dispatch_async(dispatch_get_main_queue(), ^(void)
  {
    viewGetWindowSizeMT(view, index);
  });
}

As an unexpected bonus this has fixed the problem on RenderDoc Mac GL where resizing the thumbnail pane did not update the rendering views to the correct aspect ratio.

With both of this changes I have not been able to trigger the Xcode main thread checker asserts when using RenderDoc Mac with GL or Vulkan replays.

Debugging with Xcode

My setup for debugging RenderDoc Mac using Xcode is to have Xcode launch qrenderdoc and automatically attach to it. When I first started doing this I had a problem where the process would quit with an error about unknown command line argument ie.

QTRD 057176: [04:33:05]       qrenderdoc.cpp( 197) - Error   - "Unknown options: N, S, D, o, c, u, m, e, n, t, R, e, i, s, i, o, n, s, D, e, b, u, g, M, o, d, e."

Until today I had worked around this problem by commenting out the code which would quit if an unknown command line arugment was found.

I was a bit confused about why Xcode was adding extra command lines which were not set in the project and looking at the argc, argv parameters passed to main there are three arguments, the extra two arguments being passed are

-NSDocumentRevisionsDebugMode
YES

WIth a bit of internet searching tracked down the source of the extra command line arguments to the "Allow debugging when browsing versions" option in the Xcode Run options for the qrenderdoc target

Disabling that option prevents the extra command line arugments being added and makes the debugging process much smoother without an interuption on every launch of qrenderdoc from Xcode.

RenderDoc Mac GL : New approach to solve the threading issue

After long and fruitful discussions with Baldur it was clear the previous approach or any approaches which required changes to the RenderDoc interfaces or RenderDoc common code is not desired and would not be able to be upstreamed. Today I tried a new approach which is entirely in the RenderDoc CGL Platform layer.

To recap, the challenge is from Replay thread in the RenderDoc CGL Platform layer there are some Apple UI Object function calls which must be made on the main thread i.e.

[view setWantsBestResolutionOpenGLSurface:true];
[context setView:view];
[context update];

The challenge was the data context and view are private to and owned by the RenderDoc CGL Platform layer and there is no way to expose that to the main thread without changing the RenderDoc interfaces (which is undesirable).

The Apple main thread dispatch offers a solution to this using dispatch_async ie.

void contextUpdateMT(NSOpenGLContext* context)
{
  [context update];
}

void scheduleContextUpdate(NSOpenGLContext* context)
{
  dispatch_async(dispatch_get_main_queue(), ^(void)
  {
    contextUpdateMT(context);
  });
}

The problem is the context object can be in use on the Replay thread and there is a race condition here if the main thread calls methods on this object whilst the Replay thread is using it.

The current solution is to track the active context object on the Replay thread and then on the main thread if the dequeued context object is in use by the Replay thread, the main thread command is rescheduled. This idea of rescheduling the main thread command is a polling type approach instead of locking approach which is intended to avoid a potential deadlock between the main and Replay thread.

The current simplistic approach uses a mutex lock around the tracking of the active context object to prevent the Replay thread switching to the same context that the main thread is operating on.

The code ends up looking like this

static void scheduleContextSetView(NSOpenGLContext* context, NSView* view)
{
  dispatch_async(dispatch_get_main_queue(), ^(void)
  {
    contextSetViewMT(context, view);
  });
}

static void scheduleContextUpdate(NSOpenGLContext* context)
{
  dispatch_async(dispatch_get_main_queue(), ^(void)
  {
    contextUpdateMT(context);
  });
}

void contextUpdateMT(NSOpenGLContext* context)
{
  pthread_mutex_lock(&s_ActiveNSGLContextMutex);
  if (s_ActiveNSGLContext != context)
  {
    [context update];
  }
  else
  {
    scheduleContextUpdate(context);
  }
  pthread_mutex_unlock(&s_ActiveNSGLContextMutex);
}

void contextSetViewMT(NSOpenGLContext* context, NSView* view)
{
  pthread_mutex_lock(&s_ActiveNSGLContextMutex);
  if (s_ActiveNSGLContext != context)
  {
    [view setWantsBestResolutionOpenGLSurface:true];
    [context setView:view];
    [context update];
  }
  else
  {
    scheduleContextSetView(context, view);
  }
  pthread_mutex_unlock(&s_ActiveNSGLContextMutex);
}

then on the Replay thread when setting the current contect

pthread_mutex_lock(&s_ActiveNSGLContextMutex);
s_ActiveNSGLContext = context;
pthread_mutex_unlock(&s_ActiveNSGLContextMutex);

And it appears to just work (video).

The breakthrough which made this happen was the consistency of the RenderDoc Gl rendering implementation which calls a method to make the platform context the current context before doing any rendering calls (without that consistency this approach would not work).

The changed files are here and all restricted to RenderDoc CGL Platform specific code or Apple specific code in the RenderDoc command code.

RenderDoc Mac GL Threading

Did a quick experiment using the Apple dispatch API by adding this code block to the RenderDoc CGL platform code in the NSGL_createContext objective-C method which is called on the Replay thread

  dispatch_async(dispatch_get_main_queue(), ^(void)
  {
    NSGL_LogText("Main Thread NSOpenGLContext");
  });

The method worked first time and saw this output in the RenderDoc diagnostic log (MT is the result of NSThread.isMainThread on the execution thread and QT is the result of NSThread.isMainThread on the submission thread).

Core     PID  33462: [19:45:47]       cgl_platform.cpp(44) - Log     - CGL: MT:1 QT:0 Main Thread NSOpenGLContext

Next steps are to work out the relationship with ReplayManager::run to explore how to make this work. I am thinking about adding an API named something like “start rendering”, “end rendering” to IReplayController, which would be implemented in the platform layer and would be null operation on most platforms. The start and end calls would be called in the ReplayManager::run method and would prevent the replay manager loop from executing commands at the same time as the main thread is executing UI dispatch commands. The main thread dispatch blocks could do a Try Get Lock to guarantee executing commands only when the replay thread is idle. The one piece I need to get clear in my mind is how to reschedule the commands if the replay thread is not idle. I want to try and use a polling and reschedule approach to avoid deadlocks.

I still need to debug is renderdoccmd replay to see if it runs on the main thread or a helper thread. Right now renderdoccmd replay does not do much on Mac because the Mac window code has not been implemented in it.

As an aside I was looking at a drawio extension to Visual Studio Code which looks like it makes it quite simple to create diagrams and integrate them live into markdown documents in Visual Studio Code. I am going to give it a go to draw out the threading model design for RenderDoc main thread and replay thread.