[Vm-dev] Identify console executable and standard one

[Alistair Grant]

Hi Eliot & Subbu,

On 16 April 2018 at 05:50, K K Subbu <kksubbu.ml at gmail.com> wrote:
>
> On Sunday 15 April 2018 11:42 PM, Eliot Miranda wrote:
>>>
>>>
>>> On Windows in sqFileStdioHandlesInto() in sqWin32FilePrims.c:
>>>
>>> files[0].sessionID = thisSession;
>>> files[0].file = GetStdHandle(STD_INPUT_HANDLE);
>>> files[0].writable = false;
>>> files[0].lastOp = 0; /* unused on win32 */
>>> files[0].isStdioStream = GetConsoleMode(files[0].file, &mode) != 0;
>>> AddHandleToTable(win32Files, files[0].file);
>>
>>
>> This could be causing problems, because AFAIA
>> GetConsoleMode(files[0].file, &mode) is no longer reliable.  Can you test?
>
>
> I believe _isatty(fd) from io.h is the syscall on Windows:
>
>  https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/isatty
>
> HTH .. Subbu

Right, _isatty sounds like the more sensible call.  Just looking at
the docs, the return value of GetConsoleMode() is simply whether it
succeeded or not.

However, in some quick tests it appears that both _isatty() and
GetConsoleMode() return the correct value when called from a cmd.exe
terminal window (I guess GetConsoleMode() fails if there isn't a
terminal).  They both fail when called from a cygwin shell (mintty).
It looks like vim (yes, the text editor) has some MIT licensed code
that does the check correctly from within mintty.

I think my time will be very limited over the next few days, but I'll
try and narrow this down eventually.


Cheers,
Alistair