Marcel,

Considering that the average user does run multiple apps and that
therefore both memory and disk space are dynamic resources your
"recommendations" for the user to first see how much disk space and
memory is available sound just ridiculous. In the end, your scheme risks
a hard crash if you do any real work in a real-life environment and
that's most definitely not my understanding of a stable application. 

In fact, I'd go sofar and argue that automatic lazy virtual memory
management is one of the most stupid ideas that I've ever heard of. If
the memory subsystem returns you a supposedly valid pointer to a memory
block you just allocated why should you be unable to use it? If it
cannot grant the request it can return Null and have the client deal
with the problem. With automatic lazy VM you have to be aware that at
_any_ time you access memory you must be able to recover from a memory
fault. You should also consider that just catching an exception is not
enough by far - if that happens to you in the midst of trying to write
to a forwarding block in the middle of full GC you are just screwed.

Requiring the user to "quit and restart" an application in order to get
some memory back is ... how do I put this ... not exactly close to
reality. What if you want to run a server app? Would you require
Comanche users to "quit and restart" just because you had a temporary
large request?! BFD? Yes, I'd say so.

And your argument that "the system becomes unusable long before" is not
always true. I can give you very simple examples that will trigger
almost all of the effects I'm describing _without_ thrashing memory too
much (if your OS is reasonable efficient at paging). Here's a simple
one: Start four Squeak's with -memory 512 and in each of them do:

MemoryHog := (1 to: 100) collect:[:x| Form extent: 1024 at 1024 depth: 32].
MemoryHog := nil.
Smalltalk garbageCollect.

A non-issue?! Not when you're doing (for example) large image
manipulation at various levels. I do. BTW, the above can be done simpler
as

Form extent: 11000 at 11000 depth: 32.

Takes a couple of seconds but _my_ system is perfectly usable. Before,
during, and after I do this.

So let's just say that I still think there's a huge difference between
what I was describing and that Mach behavior. That you haven't seen any
of these problems in about eight years doesn't mean they don't exist -
all it means is that _YOU_ haven't seen them.

Cheers,
  - Andreas

> -----Ursprüngliche Nachricht-----
> Von: squeak-dev-admin at lists.squeakfoundation.org 
> [mailto:squeak-dev-admin at lists.squeakfoundation.org] Im 
> Auftrag von Marcel Weiher
> Gesendet: Freitag, 1. Februar 2002 22:41
> An: squeak-dev at lists.squeakfoundation.org
> Betreff: Re: AW: Dynamic system memory use
> 
> 
> 
> On Friday, February 1, 2002, at 09:29 , Andreas Raab wrote:
> 
> >> Also, I started implementing the grow/shrink primitives, but then
> >> couldn't figure out how such an implementation differs from simply
> >> starting Squeak with -memory 512, in the presence of a lazy
> >> VM subsystem.
> >
> > It differs in various ways.
> 
> Really?
> 
> >  First of all, let me repeat what's actually
> > the expected behavior of any implementation for dynamic memory
> > management functions:
> >
> > On startup we _reserve_ an (arbitrarily high) amount of 
> address space.
> 
> This is exactly what doing a large malloc() does on Mach.
> 
> > The only reason we do this is to prevent the stupid C allocator from
> > giving away memory with addresses in the specified region 
> so that Squeak
> > is guarantueed to get a linear chunk if requested.
> 
> Exactly.
> 
> >  At this point, we
> > don't take a single byte of the memory, all we do is 
> _reserving address
> > space_.
> 
> This is exactly what doing a large malloc() does on Mach.
> 
> >  When we call one of the shrink/grow functions, then we actually
> > request (continuous) portions of memory out of the reserved address
> > space.
> 
> You get this on Mach (the 'grow') just by using the memory 
> you allocated.
> 
> >  Now we need it and we know we need it. Growing may fail due to
> > insufficient resources and that's fine; we can recover from such a
> > failure quite nicely (e.g., you'll just get a low space warning).
> 
> I haven't had that sort of failure in about 8 years (not that much HD 
> space on those Next boxes), but you probably get a signal 
> that you could 
> catch (or a message to your Mach exception port that you could also 
> catch).
> 
> However, the system will have become quite unusable long before that.
> 
> > Compare that to your lazy VM subsystem: For one thing, your 
> VM subsystem
> > may require backing store for the entire reserved address 
> space which is
> > a significant waste of disk space (the current Windows VM 
> reserves about
> > 1.5GB of memory and I tend to run multiple images at the same time).
> 
> Mach doesn't do that.
> 
> > Secondly, if it does _not_ provide backing store, then your 
> VM subsystem
> > may randomly fail at some unspecified point of actually 
> accessing the
> > memory it promised to you.
> 
> Yes.  However, this is currently difficult to prevent on 
> Mach.  One way 
> that I have implemented is swapping to your own private swapfile.  
> However, this means you have to pre-allocate this file for all the 
> memory you want to use, which is very limiting, performane-wise.
> 
> >  Meaning that instead of a low space warning
> > you'll get a core dump (if at all).
> 
> The easy way to prevent this is to set the -memory parameter to less 
> than your available disk space.  Incidentally, your system will have 
> ground to a halt thrashing long before you're out of 
> disk-space.  Sorry, 
> this is just a non-issue.
> 
> >  Thirdly, since the VM subsystem can
> > hardly monitor whether we still write to pages of memory 
> (which, after a
> > shrink we do _not_) it is unable to release that portion of 
> memory to
> > the OS or other apps.
> 
> If they are not used, they well get paged out to disk and just sit 
> unused in some swapfile.  If you really need the space back, quit and 
> restart your image.  BFHD.
> 
> >  Fourthly, since Squeak has no knowledge about how
> > much memory is actually available from the system without going to
> > lengthy swapping,
> 
> Huh?
> 
> >  there is no way of optimizing large data set
> > manipulations based on the amount of physical memory
> 
> Just set your -memory parameter to the amount of physical memory 
> available.  What exactly is the problem?
> 
> >  (which can be a big
> > advantage due to the unusually low localization of memory access
> > patterns in OOP systems). So there's a _huge_ difference 
> between a lazy
> > VM subsystem and the behavior described above.
> 
> Not really.  The system is slightly more elegant, definitely.  But in 
> practical terms I would wager that the differences are negligible.
> 
> > What I would recommend is looking at your common C 
> allocator and find a
> > way of reserving address space (is there a function to 
> create a new heap
> > in Linux?)
> 
> This function is called malloc() on Mac OS X. ;-)  (Actually, 
> it is the 
> behavior of the underlying vm_allocate(), but a large malloc() will 
> essentially just call-through.)
> 
> >  and allocate/free continous portions from it (used to be
> > called sbrk()).
> 
> No need to be condescending...
> 
> Anyway, here's what 'man sbrk' says:
> 
> 
> BRK(2)                    System Programmer's Manual          
>           
> BRK(2)
> 
> NAME
>       brk, sbrk - change data segment size
> 
> SYNOPSIS
>       #include <unistd.h>
> 
>       char *
>       brk(const char *addr)
> 
>       char *
>       sbrk(int incr)
> 
> DESCRIPTION
>       The brk and sbrk functions are historical curiosities left over 
> >from ear-
>       lier days before the advent of virtual memory management.
> 
> ...
> 
> That's actually quite funny. ;-)  The explanation is that sbrk() and 
> friends make assumptions about a specific layout of 
> contiguous segments 
> that is simply not valid on Mach.
> 
> Regards,
> 
> Marcel
> 
> 
>