Calling Conventions (__cdecl, __stdcall, …etc)

Calling convention is considered as a low-level scheme of how subroutines (i.e. functions) receive parameters from their callers and how they return a result, that seems somehow related to the programming language implementation and the compiler design.

  • The calling conventions vary in:
    • Where parameters, return values and return addresses are placed( in registers, call stack, a mix of both, or in other memory structures)  ?
    • The order in which actual arguments for formal parameters are passed.
    • How a return value is delivered from the callee back to the caller (on the sack, in a register, or within the heap) ?
    • How the task of setting up for and cleaning up after a function cal is divided between the caller and the callee.

From this  sense, calling conventions categorized in 3 types (Caller cleanup, Callee cleanup, Either Caller or Call cleanup). Not all conventions are available on all supported platforms, and some conventions use platform-specific implementations, in this article we’ll mention briefly some calling conventions use when programming x86 architecture microprocessor, and supported by visual C/C++ compilers.

Keyword Stack cleanup Parameter passing
__cdecl

Caller

Pushes parameters on the stack, in reverse order (right to left)
__clrcall

n/a

Load parameters onto CLR expression stack in order (left to right).
__stdcall

Callee

Pushes parameters on the stack, in reverse order (right to left)
__fastcall

Callee

Stored in registers, then pushed on stack
__thiscall

Callee

Pushed on stack; this pointer stored in ECX
__vectorcall

Callee

Stored in registers, then pushed on stack in reverse order (right to left)

We’ll pick one of these calling convention to talk about here which is (__cdecl) , and we can revise msdn articles in this topic from here.

__cdecl : stands for (C declaration) Is the default calling convention for C/C++ programs, subroutine’s arguments are passed on the stack. Integer values and memory addresses are returned in the EAX register, (EAX, ECX, EDX) registers are reserved for the caller and the reset reserved for the callee. In context of the C programming language, function arguments are pushed on the stack in reverse order, for example

int callee(intintint);
int caller(void)
{
	int ret;
	ret = callee(1, 2, 3);
	ret += 5;
	return ret;

The produced assembly code on Intel-x68 machines will be as follows:

push	ebp
mov 	ebpesp
push	3
push	2
push	1
call	callee
add 	esp, 12
add 	eax, 5
pop 	ebp
ret

The caller cleans the stack after the function call returns. The __cdecl  as we mentioned is usually the default calling convention for x-86 C/C++ compilers, although many compilers provide options to change the calling convention used, by manually define a function to be _cdecl, as follows

void _cdecl funct();

note: the _cdecl modifier must be included in the function prototype and declaration. Because the stack is cleaned up by the caller in this calling convention, it can be used simply be used to make vararg functions, by writing something like this:

int __cdecl system(const char *);
// Example of the __cdecl keyword on function pointer
typedef BOOL(__cdecl *funcname_ptr)(void * arg1const char * arg2DWORD flags, ...);

I found recently this summarized tutorial on code project, highly recommend and worth reading.

References: