C++ Calling Convention with naked functions

By | July 26, 2019

Calling conventions are a standardized methods for c++ compilers how to pass parameters to called function. MS Visual Studio gives possibility to use four of them: cdecl, fastcall, stdcall and vectorcall. The specific of every calling convention is hidden in function prolog and not visible for high level language programmer. This article makes an attempt to explain differences between these calling conventions using c++ code examples with naked functions.
What is naked function? A naked function is the function which is declared with naked attribute that makes the compiler to generate function code without prologue and epilogue. In MS Visual Studio the naked attribute is valid for x86 and ARM platform. For g++ it is only supported for ARM compiler only, but frankly to say I never used naked in GNU compilers.

So if we are using naked function with parameters in c++ we need to write prologue and epilogue by ourselves and the prologue and epilogue implementations will varies for different calling convention.
Calling Convertion
cdecl. The parameters are passed through the stack (the last parameter in the function definition is pushed in the stack first) and prologue needs to retrieve them from the stack before using:


#include "stdafx.h"
__declspec(naked) void naked_cdecl(_TCHAR* a, _TCHAR* b, _TCHAR* c)
{
   // wprintf(L”Parameters: %s, %s, %s\n”, a, b, c); in naked function parameters are not initialized
   _asm {
      mov eax, [esp + 4]
      mov [a], eax
      mov eax, [esp + 8]
      mov [b], eax
      mov eax, [esp + 12]
      mov [c], eax
   }
   // Now you can print
   wprintf(L"Parameters: %s, %s, %s\n", a, b, c);
   _asm
   {
      ret
   }
}
int _tmain(int argc, _TCHAR* argv[])
{
      _TCHAR a[] = L"aaaaaa";
      _TCHAR b[] = L"bbbbbb";
      _TCHAR c[] = L"cccccc";
      naked_cdecl(a, b, c);
      return 0;
}

stdcall. Similar as cdecl, the parameters are passed through the stack (the last parameter in the function definition is pushed in the stack first). The prologue is the same as for cdecl but epilogue is different, we need to return from function with stack adjustment in other words return from function and add 12 to stack pointer (esp register value), 4 byte for every parameter. In our case we have 3 parameters: 4 * 3 = 12:


#include "stdafx.h"
__declspec(naked) void naked_stdcall(_TCHAR* a, _TCHAR* b, _TCHAR* c)
{
   // wprintf(L”Parameters: %s, %s, %s\n”, a, b, c); in naked function parameters are not initialized
   _asm {
      mov eax, [esp + 4]
      mov [a], eax
      mov eax, [esp + 8]
      mov [b], eax
      mov eax, [esp + 12]
      mov [c], eax
   }
   // Now you can print
   wprintf(L"Parameters: %s, %s, %s\n", a, b, c);
   _asm
   {
      ret 0xC
   }
}
int _tmain(int argc, _TCHAR* argv[])
{
      _TCHAR a[] = L"aaaaaa";
      _TCHAR b[] = L"bbbbbb";
      _TCHAR c[] = L"cccccc";
      naked_stdcall(a, b, c);
      return 0;
}

fastcall.The fastcall convention is mixed, the first 2 parameters are passed through ecx and edx registers, the third one and subsequent through stack. Prologue looks rather complicated, I did not write it by myself like in 2 previous examples and borrowed it from disassembler:


#include "stdafx.h"
__declspec(naked) void naked_fastcall(_TCHAR* a, _TCHAR* b, _TCHAR* c)
{
   // wprintf(L"Parameters: %s, %s, %s\n", a, b, c); in naked function paramers are not initialized
   _asm {
      push ebp
      mov ebp, esp
      sub esp, 0D8h
      push ebx
      push esi
      push edi
      push ecx
      lea edi, [ebp – 0D8h]
      mov eax, 0CCCCCCCCh
      rep stos dword ptr es : [edi]
      pop ecx
      mov [b], edx
      mov [a], ecx
   }
   // Now you can print
   wprintf(L”Parameters: %s, %s, %s\n”, a, b, c);
   _asm
   {
      mov esp, ebp
      pop ebp
      ret 4
   }
}
/*void naked_fastcall(_TCHAR* a, _TCHAR* b, _TCHAR* c)
{
wprintf(L”Parameters: %s, %s, %s\n”, a, b, c);
}*/
int _tmain(int argc, _TCHAR* argv[])
{
   _TCHAR a[] = L"aaaaaa";
   _TCHAR b[] = L"bbbbbb";
   _TCHAR c[] = L"cccccc";
   naked_fastcall(a, b, c);
   return 0;
}

vectorcall. According to Microsoft documentation the vectorcall uses more registers for arguments than fastcall. However for particular example I did not find any differences in calling convention between vectorcall and fastcall.

Leave a Reply

Your email address will not be published. Required fields are marked *