[Pharo-project] [Vm-dev] Re: Can OSProcess functionality be implemented using FFI instead of plugin?

Sun Jan 17 02:25:29 UTC 2016

On Sat, Jan 16, 2016 at 11:02 PM, Eliot Miranda <eliot.miranda at gmail.com>
wrote:

>
>
> On Sat, Jan 16, 2016 at 6:00 AM, Mariano Martinez Peck <
> marianopeck at gmail.com> wrote:
>
>>
>> Hi all,
>>
>> Sorry for reviving an old thread but I thought it was better to continue
>> the discussion here because of the context.
>> As you may have read, the other day I released a first approeach to a
>> subset of OSProcess based on FFI (posix_spwan() family of functions):
>>
>> https://github.com/marianopeck/OSSubprocess
>>
>>  And with that in mind, I wanted to share a few things with you. The main
>> 2 problems I found with implementing this with FFI was:
>>
>> 1) We have all already agree and discussed that fork+exec cannot be done
>> in separate FFI calls. So at the very min you need either a plugin method
>> that does the fork()+exec() OR wrapping a lib like posix_spwan()
>>
>> 2) The other main problem, is, as you all said (and mostly  Nicolas), is
>> the problems with the preprocessor (constants, macros, etc).
>>
>> With all that said, I was able to get my stuff working. However, I am
>> still using some primitives of OSProcess plugin because of 2).
>>
>> I read Eliot idea and what I don't like is the need of a C compiler in
>> the user machine. I think that's a high constrain. Then Igor suggested that
>> WE (developers and maintainers of a certain tool) are the ones that
>> compiles the little C program to extract constant values etc and then WE
>> provide as part of our source code, some packages with some SharedPool
>> depending on the platform/OS. And Igor approach looked a bit better to me.
>>
>
>
>
> You misunderstand the proposal.
>

I think I did. But let me confirm that below ;)

> The C compiler is needed /only when changing the set of constants/, i.e.
> when /developing/ the interface.  The C compiler is /not/ needed when
> deploying.
>
> The idea is to
> a) at development time, e.g. when a new variable is added to a SharedPool
> containing platform constants, a C program is autogenerated that outputs in
> some format a description of the names and values of all the constants
> defined in the pool.  One convenient notation is e.g. STON.  For the
> purposes of this discussion let's assume we're using ston, but any format
> the image an parse (or indeed a shared object the image can load on teh
> current pkatform) will do.  The output of the autogenerated C program would
> be called something like <SharedPoolName>.<PlatformName>.ston, e.g.
> UnixConstants.MacOSX64.ston or UnixConstants.Linux32.ston.  The ston files
> can easily be parsed by facilities in the Smalltalk image.
>
> b) when deploying the system to a set of platforms one includes all the
> relevant platform-specific ston files.
>
>
OK. But let me ask something. Below you said "be it a plugin or a dll
doesn't matter". To autogenerate the C program, I must know which header
files to include for each platform and probably a few others things. For
example, besides exporting the value,  I would also like to export the
sizeof(). At that depends how was the VM compiled, right?   So...my
question is...if such a autogenerated C code could be part of the VM
building (considering all the settings being assume when building), cannot
I reuse the knowledge the VM already has? Like which header files
to include, if it was compiled 32 bits or 64 bits, which C compiler to use,
etc..

What I mean is if it would be easier if I take the SharedPool at VM
building time, and from there I autogenerate (and run) the C code that
would generate the output. Then, when we "deploy" the VM, we can deploy it
with relevant platform specific ston files as you said.

> c) at startup the image checks its current platform.  If the platform is
> the same that it was saved on, no action is taken.  But if the platform as
> changed then the relevant ston file is selected, parsed, and the values for
> the variables in the shared pool updated to reflect the values of the
> current platform.
>
> So the C compiler is only needed when developing the interface, not when
> deploying it.
>
>
OK

>
>> Then Nicolas made a point that if we plan to manage all that complexity
>> at the image level it may become a hell too.
>>
>> So.... what if we take a simpler (probably not better) approach and we
>> consider the "c program that exports constants and sizes" a VM Plugin?
>> Let's say we have a UnixPreprocessorPlugin (that would work for OSX, Linux
>> and other's Unix I imagine for the time being) which provides a function
>> (that is exported) which answers an array of arrays. For each constant, we
>> include the name of the constant, the value, and the sizeof().  Then from
>> image side, we simply do one FFI call, we get the large array and we adapt
>> it to a SharedPool or whatever kind of object representing that info.
>>
>
>
>
> This is what I suggestred in teh first place.  That what is autogenerated
> is a shared object (be it a plgin or a dll doesn't matter, it is machine
> code generated by a C compiler form an autogenerated C program compiled
> with the platform's C compiler) that can be loaded at run-time and
> interrogated to fetch the values of a set of variables
>

OK, got it. But still, it would be easier if the "platform" in this case is
the "machine where we build the VM we will then distribute" right? i mean,
I would like to put this in the CI jobs that automatically builds the VM,
and not myself building for each platform.

*I mean, my main doubt is if this job of autogenerating C code, compile it,
run it, export text file, and distribute text file with the VM, could be
done as part of the VM building. *

> .  But I think that the textual notation suggested above is simpler.  The
> test files are easier to distribute and change.  Shared objects and
> plugins have a habit of going stale, and there needs to be metadata in
> there to describe the set of constants etc, which is tricky to generate and
> parse because it is binary (pointer sizes, etc, etc).  Instead a simple
> textual format should be much more robust.  One could even edit by hand to
> add new constants.  It would be easy to make the textual file a versioned
> file.  Etc, etc.
>
>

OK. Got it. And do you think using X Macros for the autogenerated C (from
the SharedPool) is a good idea?
And then I simply write a text file out of it.

>
>> I know that different users will need different constants. But let's say
>> the infrastructure (plugin etc) is already done. And let's say I am a user
>> that I want to build something with FFI and I need some constants that I
>> see are not defined. Then I can simply add the ones I need in the plugin,
>> and next VM release will have those. If Cog gets moved to Github, then this
>> is even easier. Everybody can do a PR with the constants he needs. And in
>> fact, if we have the infrastructure in place, I think that we each of us
>> spend half an hour, we may have almost everything we need.
>>
>> For example, I can add myself all those for signals (to use kill() from
>> FFI), all those from fcntl (to make none blocking pipes), all those from
>> wait()/waitpid() family (so that I can do a waitpid() with WNOHANG), etc
>> etc etc.
>>
>> I know it's not the best approach but it's something that could be done
>> very easily and would allow A LOT of stuff to be moved to FFI just because
>> we have no access to preprocess constants or sizeof()  (to know how to
>> allocate). I also know this won't cover macros and other stuff. But still.
>>
>> If you think this is a good idea, I can spend the time to do it.
>>
>> Cheers,
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Thu, May 10, 2012 at 10:09 AM, Nick Ager <nick.ager at gmail.com> wrote:
>>
>>> <snip>
>>> Well, like opendbx, maybe because opengl has quite standard interface...
>>> </snip>
>>>
>>> and
>>>
>>> <snip>
>>> It's not that it's not doable, it's that we gonna reinvent gaz plant
>>> and it gonna be so boring...
>>> I'd like to see a proof of concept, even if we restrict to libc, libm,
>>> kernel.dll, msvcrt.dll ...
>>> </snip>
>>>
>>> <snip>
>>> Is the unix style select()
>>> ubiquitous or should I use WaitForMultipleObject() on Windows? Are
>>> specification of read/write streams implementation machine independant
>>> (bsd/sysv/others...)
>>> </snip>
>>>
>>> Perhaps *a* way forward is to try to find existing projects which have
>>> already created cross-platform abstractions for platform specific
>>> functionality. Then we can use FFI to access that interface in a similar
>>> way to OpenGL and OpenDBX. For example NodeJs works across unixes - perhaps
>>> they have a useful cross-platform abstraction, boost  has abstractions of
>>> IPC etc
>>>
>>> Nick
>>>
>>
>>
>>
>> --
>> Mariano
>> http://marianopeck.wordpress.com
>>
>>
>
>
> --
> _,,,^..^,,,_
> best, Eliot
>

-- 
Mariano
http://marianopeck.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20160116/d221ca21/attachment-0001.htm