Tuesday, July 25, 2017

Starting a PInvoke command-line tool

It frustrates me that people continue to misuse rundll32 to try to run arbitrary functions. Unfortunately, there's no convenient alternative - nobody wants to compile a full C/C++ program just to run one function, and using P/Invoke in PowerShell is gnarly. If it's easy to do the wrong thing and hard to do the right thing, people will probably do the wrong thing, so the solution here is to make the right thing easy.

Yesterday I started poking around with .NET reflection and P/Invoke; I managed to get my program invoking an arbitrary parameterless function. Today I built on that. I now have these features working:

  • Numeric parameters are accepted in hex or decimal, from 16 to 64 bits long. The native specifier corresponds to the .NET IntPtr type in that it becomes a 32- or 64-bit field as appropriate for the system.
  • A return value can be accepted if its type is given (default is none). Currently only 32-bit or wider numeric types are supported for returns.
  • The calling convention can be specified: stdcall (default), cdecl, or thiscall.
  • Pointers to structures can be easily passed with the blockptr specifier. blockptr takes a list of parameters just like a function, creates a structure containing them, gets a pointer to that structure, and passes that to the main function. These can be nested.
For example, this command line sets the window text color to a pretty blue and returns whether the function worked:

user32.dll!SetSysColors /return int (int 1, blockptr(int 8), blockptr(int 0xAA4400))

No comments:

Post a Comment