Have you tried it using normal backslashes? It'd be really unusual for someone to make a Windows .exe file and then have it not understand paths with backslashes. So unusual that I would complain to them. 
Something like this will convert the current path to using forward slashes, and insert it into a command to be run. (Your example has two paths, so I'm not sure if this is exactly what you want, but the rest should be easy.)
function OnClick(clickData)
{
var cmd = clickData.func.command;
var path = new String(DOpus.FSUtil.Resolve(clickData.func.sourcetab.path));
path = path.replace(/\\/g,'/');
var cmdLine = 'calibredb.exe catalog "' + path + '/Documents/CalCat 0.ALL.csv" --with-library="' + path + '/Main Calibre/0.All"';
DOpus.Output(cmdLine);
// cmd.RunCommand(cmdLine);
}
That will output what it would run to the script log (or Button Editor output panel if you run it from there, which is what I'd recommend to save time).
If it looks right, comment out the DOpus.Output line near the bottom, and un-comment the cmd.RunCommand line below that, to make it actually run the command.
