One idle thought I had today was whether one couldn't entirely get rid of float-valued NaNs and instead introduce a well-known NaN instance that *isn't* an instance of Float and consequently also isn't a Number, doesn't do any arithmetic etc. etc. etc.
Another possibility is to have the primitives fail and the fallback code signal a Notification. The notification would by default return a NaN float. It might be okay to use a NaN object too;; however it should behave the same way as IEEE NaNs and would have to be interoperable with FFI and FloatArrays (does Squeak have them?).
I don't think it makes sense to use Integer or Fraction NaNs. In that case you do not have infinities either, and raising hard errors makes more sense. Note that GNU Smalltalk does not even raise a ZeroDivide for floating-point division:
st> 1 / 0 Object: 1 error: The program attempted to divide a number by zero ZeroDivide(Exception)>>signal (AnsiExcept.st:216) SmallInteger(Number)>>zeroDivide (AnsiExcept.st:1534) SmallInteger>>/ (SmallInt.st:277) UndefinedObject>>executeStatements (a String:1) nil
st> 1.0 / 0.0 Inf
What I have personally no clue about is whether anyone ever uses float-valued NaNs for anything useful.
It allows delaying error checking to after the end of the computation. Some errors that do not affect the result would be silently ignored; most errors would result in a NaN. Of course Smalltalk floats are so slow that this might not even make a difference (unless you use FFI to interface with external libraries).
With floating-point, I decided that the best course of action is "assume the IEEE-754 people are always right" (and that means W. Kahan mostly).
Paolo