Have you tried enabling logging or debugging?
You can do by opening the server widget from the Wolrd 
menu, open..., RFB/VNCServer. The best is to pin it.
The widget has a debugging submenu where you can enable logging and 
debugging, and you can open the server log too.

Seeing those messages, it seems your client is trying to detect the 
encoding and/or color depth. Setting them before connecting should help 
in this case.

Levente

On Wed, 7 Sep 2016, Yoshiki Ohshima wrote:

> In the console, there are bunch of messages like this:
>
> RFB: caught ConnectionClosed: Connection close while waiting for data.
> RFB: caught ConnectionClosed: Connection close while waiting for data.
> RFB: caught ConnectionClosed: Connection close while waiting for data.
> RFB: caught ConnectionTimedOut: send data timeout; data not sent
> RFB: caught ConnectionClosed: Connection close while waiting for data.
> RFB: caught ConnectionClosed: Connection close while waiting for data.
> RFB: caught ConnectionClosed: Connection close while waiting for data.
> RFB: caught ConnectionTimedOut: send data timeout; data not sent
> RFB: caught ConnectionClosed: Connection close while waiting for data.
>
> It basically reset the connection every some milliseconds and
> reconnect.  I browsed code and RFBSocket>>waitForSendDoneFor: looks
> suspicious; but some number fudging in that method on my end did not
> help.
>
> (This excessive OS resource use may explain my other issue of getting
> to the state of the VM cannot write or read a file...)
>
> On Tue, Sep 6, 2016 at 11:52 PM, Yoshiki Ohshima
> <Yoshiki.Ohshima at acm.org> wrote:
>> With a 5.0 based image and a VM with VM version: " 'Croquet Closure
>> Cog VM [CoInterpreterPrimitives VMMaker.oscog-eem.1044] Squeak Cog
>> Spur 5.0.3248'", and RFB-ul.13, I still get the error.
>>
>> (I wanted to mail the SqueakDebug.log but I have been encountering
>> another problem.  The VM somehow loses an ability to read and write
>> files and the debugger only shows decompiled code and calling
>> Debugger>>storeLog manually does not produce a file.)
>>
>> There is no reason to stick with a 5.0 image so I'll move on to 5.1
>> and see what happens.
>>
>> On Fri, Sep 2, 2016 at 4:27 PM, Yoshiki Ohshima <Yoshiki.Ohshima at acm.org> wrote:
>>> Thanks.  I was not aware of those newer versions.  I'll give it a try.
>>>
>>>
>>> On Fri, Sep 2, 2016 at 3:59 PM, Levente Uzonyi <leves at caesar.elte.hu> wrote:
>>>> Hi Yoshiki,
>>>>
>>>> Your symptom doesn't ring a bell. I don't know what settings your client
>>>> program has, but with other clients it usually helps to set the color space
>>>> to 24-bits and the encoding to ZRLE.
>>>>
>>>> You can try to enable logging on RFBServer to see what happens when your
>>>> client gets disconnected.
>>>>
>>>> We have improved the RFB package over the years[1], but I don't think any of
>>>> our changes would be related to your problem. (If your image is 5.0, then
>>>> RFB-ul.13 is the last one that should work. RFB-ul.14 is for 5.1.)
>>>>
>>>> Levente
>>>>
>>>> [1] http://leves.web.elte.hu/squeak/
>>>>
>>>>
>>>> On Fri, 2 Sep 2016, Yoshiki Ohshima wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I'm trying to use the RFB server RFB-ikp.8.mcz from
>>>>>
>>>>> http://www.squeaksource.com/RFB/
>>>>>
>>>>> with the Mac OS ScreenShare client.
>>>>>
>>>>> The Mac OS client can connect to the server but after 10 seconds or
>>>>> so, the server raises an error.
>>>>>
>>>>> Has somebody tried it lately?  I am on Squeak 5.0 and 5.0 CogSpur VM.
>>>>> Is the newer version better at running RFB?
>>>>>
>>>>> (The client has some other issues.  I'll track them down later.)
>>>>>
>>>>> --
>>>>> -- Yoshiki
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> -- Yoshiki
>>
>>
>>
>> --
>> -- Yoshiki
>
>
>
> -- 
> -- Yoshiki
>
>