This is all how that exe was compiled.
When you compile a COM component or executable it can be set to dynamic binding. Therefore when this executable is started, then embedded code will look in the registry in order to locate the runtime.
So if Net Express 5.1 is installed under C:\Program Files (x86)\NX51 and Visual COBOL Server under C:\Program Files (x86)VC , both have unique registry settings.
HKEY_LOCAL_MACHINE\SOFTWARE\Micro Focus\NetExpress\5.1\COBOL\5.1\Environment
HKEY_LOCAL_MACHINE\SOFTWARE\Micro Focus\Visual COBOL\2.3\COBOL\Environment
So if it's compiled dynamically bound the registry is used.
If it's not dynamically bound then the normal Windows search PATH is used. This in your case would present a problem, as the very first cblrtsm.dll it find it will load.
Be very careful of making sure there are no runtime files sitting in Windows\system32, as these will get loaded before the location in the registry.
I often come across poor installations when vendors have deployed cblrtsm.dll and other runtime files in WIndows\system32 on customer sites. I end up moving those applicable files into the vendors own solution in order that both products work.
I can't vouch if CBL_EXEC_RUN_UNIT will work in your scenario but it's a really easy little demo for you to create to prove the point.
The thing to remember out of all of this......you cannot have 2 different runtimes loaded into the same process. I don't know if CBL_EXEC_RUN_UNIT will still have NX51 runtime as a parent process I've not tried.
Neil