Support code structure (was RE: [Slightly OT] Source code management)

Andreas Raab Andreas.Raab at gmx.de
Wed Feb 6 22:09:06 UTC 2002


Tim,

[Warning: This is going to be long but you've asked for it ;-) Also it's
not as much about giving answers as it is about explaining and asking
questions]

> An explanation of your problems with the tree structure would be very
> helpful at this point. 

"Problems" might be too hard a word word. There are no problems insofar
as I could get this structure to work (as I mentioned in various prior
emails I can live with every reasonable structure and "reasonable" in
this context means that it's actually possible to implement it).

My (more psychological) "problem" with this structure is that it doesn't
go together at all with my working style. The key to approaching this is
to understand that I _do_ write C code  - such as the 3D engine for
instance (which is partly written in plain C), and various other bits
and pieces (like the Win32 support code). The current tree structure
puts everything _way_ out of reach - where I now have to switch between
usually two directories (win32 <-> generated, or win32 <-> MumblePlugin)
which are on the same level I suddenly find myself needing to navigate
various tree structures. Let's take an example: Not too long ago I fixed
a problem which interrelated issues in the drop plugin, 3D accelerator
plugin, and the "core" support code (this related to some weirdness
initializing COM directly which is used by all of the three). In the
structure that's implemented at SF it would require me to navigate
between all of the following directories:

Cross/plugins/B3DAcceleratorPlugin/    (headers)
Cross/plugins/DropPlugin/              (headers)
Win32/vm/                              (basic support code)
Win32/plugins/B3DAcceleratorPlugin/    (implementation)
Win32/plugins/DropPlugin/              (implementation)
Win32/misc/                            (makefile)

It eats up my short-term memory chunks just to remember those
directories (six of them so it's full ;-) whereas in my current
structure I can just play "trial and error" (using Emacs and ensuring
there's always at least one window on win32 and one on generated) and
concentrate on solving the problem. [BTW, these examples are not at all
uncommon - I just fixed a network problem which would've been similarly
interrelated due to some synchronization issues; and
implementing+testing a new plugin -like UUID which I just did- works in
a similar "distributed" fashion].

I know that you and many other people will now say "but it's cleanly
structured what could be wrong?" and I agree it _is_ cleanly structured.
Just in a way that prevents me from getting any work done ;-( I think
the issue I find most annoying about the structure is the conceptual
distance between the cross platform and any platform specific support
code. In other words, if I need to switch from any kind of
cross-platform code (like FilePlugin.h) to the actual implementation
(like sqWin32FilePrims.c) then I have to go from
Cross/plugins/FilePlugin/FilePlugin.h to
Win32/plugins/FilePlugin/sqWin32FilePrims.c which is a directory jump
across SIX levels. That's a huge conceptual distance; by the time I got
there I have already forgotten what it was I wanted to do...

So let me bounce the question: What is wrong with the structure that Ian
and I are using for years?! Since I think that most people don't care a
lot about hacking C code, why not set up the structure in a way that
does work for those people who do?!

> The only example problem that's been explicitly
> discussed so far was the smooshing of all the internal plugins' code
> into the same place as the core vm code, but that was fixed by Lex's
> suggestion of a subtree for the internal plugins.

That's certainly a problem although I don't care as much about it as
long as it's not about any external libaries. Mainly because there's
usually some pretty decent style used (e.g., have MumblePlugin.h and
MumblePlugin.c) so that there's hardly ever anything to worry about. I
think the only worrysome issue is to mix up any external libraries in
this set of sources - people writing those libraries can't foresee that
they might be put  together by other users and so they really can't
enforce a scheme like we have for MumblePlugin. But see below.

Another problem that I've mentioned before is the issue of copying files
around (though one might say this is a VMMaker and not a structural
problem). But you know all about that and I don't need to repeat this
here.

> I recall that you
> suggested that the jpeglib (and mpeglib?) libraries ought to be
> separated out, which has merits of cleanliness but possibly 
> complicates makefiles. 

Perhaps so but I think that the advantages outweigh the possible
complications by far. One important thing to keep in mind (and libjpeg
is an excellent example for this) is that various libraries do need
specific configurations for the appropriate platform target. libjpeg for
instance uses jconfig.h for all sorts of stuff. While it may for some
libraries be possible to use generic definitions that hold across all
platforms Squeak is running on, other will not be as easy - libmpeg for
example, which needs to be told what the "right" RGBA format is. That's
one of the reasons why I think these libraries should be separate. If
they are, then you might just have a little local makefile that says "cd
libjpeg; configure; make" and all will be well no matter what platform
you're on. A second reason for wanting the separation is that we should
really link against libraries if somehow possible - it makes life a lot
less painful in particular if you might have to add a few crude hacks in
order to get by. A third reason for wanting to have it separate is that
it makes upgrading much easier; if you want to upgrade to libjpeg7.0 you
just dump the old and put in the new library. And a fourth reason is
that it makes the distinction between the "basic" library and any
"library support code" (like the jmem_data_src|dst files that Juan
addded) a lot simpler. In other words, you know what is intrinsic part
of the library and what isn't - which means that you also know what code
you most likely have to look at if something goes wrong (well, assuming
that the library maintainers have their act together ;-)

So basically, I want to leave C libraries alone and bundled to the
maximum amount possible. Don't mess with them unless absolutely
necessary. Putting them into a subdirectory is only one solution - I
don't mind where they go, it just seemed easy enough to have an include
"libName/mumble.h" because that allows all sorts of fancy setups[*1].
Once we start mixing up the library souce code with any kind of
generated code we're in for trouble. Just consider the current example
of "error.c" vs. "jerror.c". Well, I wouldn't (and I didn't ;-) bother
to look if there's a file by that name in those external sources and I
bet you most other people wouldn't look either. That's what those
libjpeg maintainers are for, right?! ;-)

[*1] One of the setups I'm favouring in this respect is to have one
single "libs" directory that contains _all_ the libraries we might use
(libjpeg, libmpeg, libffi etc.) Since we're never really going to touch
any of these libraries it feels right to take them out of the central
path of development and even give them separate make files. All you'd
have to do then is that you build all your libraries (which could be
done beforehand so you really don't have to build those source files at
all) and either "ln-s" the directories from there or just set your VPATH
to include the "libs" directory.

BTW, another (though minor) point to make here is that on some platforms
the relevant support libraries might already be installed. If I'm not
mistaken then every Linux has a libjpeg.so so we'd just need to link
against it rather than having to include the entire library per se. The
above approach allows for this in a rather favourable way (e.g., ln-s
the import library for weak linking instead of actually building the
lib).

> What other problems do we need to solve?

Most definitely that of diverging source code across platforms. E.g., I
use an old version of libmpeg. The one at source forge doesn't work for
Windows. But I don't see a reason in just not delivering an mpeg plugin
if an older version of the library works well. How do we deal with this
issue?!

You might say this is a CVS question, use tags, etc. but that is not
true. It's primarily a structural problem considering the stated goals
of the central repository (namely one download for _all_ sources and
builds). Simply because the current structure is unable to deal with the
fact that Windows uses a different version of libmpeg than Mac uses.

The general problem here is that any structure would need to support a
possible branch and merge intrinsically or else you cannot build
different VMs from the same sources. A simple way to do this would be
just ln-s some "canonical" source into a platform specific build tree.
If you need to branch for some reason you just break the link and use a
local file. As an example, libjpeg could have everything ln-sed (is that
a verb? what the hell - it's american english ;-) into the platform dirs
and have a platform-specific copy of jconfig.h (which is not linked). It
might not be as easy considering that some platforms don't have links
but if SF and CVS support it, it might still be possible to implement
something along these lines. 

In the worst case you could in fact have completely independent sources
- and we do have those already. Notice the (somewhat ridiculous) #ifdef
NO_STD_FILESUPPORT in the file primitives. I needed them in order to be
able to override the "cross-platform" version of the file primitives.
Now take out the word "cross" and make it "canonical" and you solve
various problems at once.

[access rights]
> That would seem to be something that cvs/rcs/(most serious source
> repositories) ought to be able to handle adequately, surely.

Well, as far as I know (and nobody told me otherwise sofar) it does not.
>From the best of my knowledge you can _not_ grant restricted access
rights in the way that I think is needed. What I'm talking about is a
way to prevent _me_ from having access to anything but the Win32 parts
of the support code. I don't need any and I don't want any. Similarly,
if Ian thinks that some person is great Unix hacker or if John thinks
some other person is a great Mac hacker or if you think yet another
person is a great Acorn hacker I would not like the idea of implicitly
passing the access rights to the Windows support code as well.

I've stated this point over and over and over again and part of my
reluctance to do anything at all at source forge is that the risk of me
just screwing up yours or Lex's or John's latest work is just way to
large to be ignored. I've screwed up before and I am _not_ an
experienced SF user and I _do_ and _will_ make more mistakes.

So far I have yet to see a solution to that problem. The statement that
it "ought" to be possible is just not enough for me.

> Though, yes, that is something of a tautology since any system
> that didn't do it wouldn't count as a serious system.

So what does that tell us about all those systems ;-)

> Keep the list of people permitted to write files short (in fact I've
> publically suggested that it ought to _not_ include anyone 
> that actively develops vm code so it has to go through a review
> before acceptance - perhaps only Goran should be so deified?)
> and do regular, preferably automated, build attempts to find
> (some kinds of) problems sooner rather than later.

Yes, these are all good thoughts. 
<Kinda off-topic for this thread but in the mood of the original one>
However, you may want to consider some of the contrary arguments about
giving write access to _more_ people rather than less. I've had a couple
of good arguments with Rob on this issue and although he didn't convince
me on various of the core points I think he (and others) are right in
such that there ought to be a way of having a "quick turnaround" time if
need be (and you can't put a person with access rights to determine if
there is any need - that person could just as well integrate the
modifications). Also there's a huge psychological effect behind this
which shouldn't be ignored too easily. That's why I was interested in
arch in the first place. If I'm not mistaken then it _should_ be
possible to say "get me all the sources from the canonical directory
plus the hacks from X, Y, and Z" and have X, Y, and Z be completely
different servers. Thus, the "need" factor is taken away from the person
who's controlling access in the first place - and that's certainly a
good think in situations where one just can't respond for some reason or
other.
</off-topic>

Cheers,
  - Andreas





More information about the Squeak-dev mailing list