Calling Convention for Notifications

The preceding discussion covered the COM RPC debugging architecture in terms of six debugger-notification APIs (DebugORPC...()). However, rather than being actual API-entry points in a static-linked or dynamically-linked library, these notifications use a somewhat unusual calling convention to communicate with the notification implementations, which are found inside debugger products. This somewhat strange calling convention is used for the following reasons:

Two of the six notifications need to start and attach the debugger if it is not already attached to the relevant process.
The convention used transitions into the debugger code with the least possible disturbance of the debuggee's state and executes the minimal amount of debuggee code. This increases robustness of debugging.
The debugger is necessarily equipped to deal with concurrency issues of other threads executing in the same process. Therefore, it is important to transition to the debugger as fast as possible to avoid inadvertent concurrency problems.

The actual calling convention used is by its nature inherently processor and operating-system specific. On Win32 implementations, the default calling convention for notifications takes the form of a software exception, which is raised by a call to the RaiseException Win32 API:

VOID RaiseException(
   DWORD  dwExceptionCode,      // exception code
   DWORD  dwExceptionFlags,      // continuable exception flag
   DWORD  cArguments,         // number of arguments in array
   CONST DWORD *  lpArguments    // address of array of arguments
   );

As used here, the arguments to this raised exception call in order are:

dwExceptionCode: An exception code EXCEPTION_ORPC_DEBUG (0x804F4C45) is used. The debugger should recognize this exception as a special one indicating an COM RPC debug notification.
dwExceptionFlags: This is zero to indicate a continuable exception.
cArguments: One
lpArguments: The array contains one argument. This argument is a pointer to a structure which contains the notification specific information that the COM RPC system passes to the debugger. The definition of this structure ORPC_DBG_ALL is given below. The same structure is used for all the notifications. The structure is just the union of the arguments of the six debugger notification APIs. For a particular notification, not all the fields in the structure are meaningful and those that are not relevant have undefined values; details on this are below:

 typedef struct ORPC_DBG_ALL  {
   BYTE *            pSignature;
   RPCOLEMESSAGE *   pMessage;
   const IID *          iid;
   void*               reserved1;
   void*               reserved2;
   void*               pInterface;
   IUnknown *         pUnkObject;
   HRESULT         hresult;
   void *            pvBuffer;
   ULONG            cbBuffer;   
   ULONG *            lpcbBuffer; 
   void *             reserved3;
   } ORPC_DBG_ALL;

The pSignature member of this structure points to a sequence of bytes which contains:

A four-byte sanity-check signature of the ASCII characters "MARB" in increasing memory order.23.
A 16-byte GUID indicating which notification this is. Each of the six notifications defined here has a different GUID. More notifications and corresponding GUIDs can be defined in the future and be known not to collide with existing notifications.
A four-byte value which is reserved for future use. This value is NULL currently.

The notifications specified here pass their arguments by filling in the appropriate structure members. See each notification description for details.

Using software exceptions for COM debugging notifications is inconvenient for in-process debugging. In-process debuggers can alternately get these notifications via direct calls into the debugger's code. The debugger which wants to be notified by a direct call passes in an IOrpcDebugNotify interface in the LPORPC_INIT_ARGS argument to DllDebugObjectRPCHook. If this interface pointer is available, COM makes the debug notifications by calling the methods on this interface. The methods all take an LPORPC_DBG_ALL as the only argument. The information passed in this structure is identical to that passed when the notification is done by raising a software exception.