[Vm-dev] [long] Squeak and High-DPI

Mon May 2 19:04:55 UTC 2016

Hi all,
[this is a longish post, you might want to use this toc]

[TOC]
- [INTRO]
- [TERMINOLOGY]
  - [DPI]
  - [DEFAULT DPI]
- [STATUS]
  - [WINDOWS DPI]
  - [OSX CARBON DPI]
  - [OSX COCOA DPI]
- [SUGGESTIONS]      <<<< PLEASE COMMENT
  - [OSX COCOA PATCH]
  - [VM/IMAGE INTERFACE]

[INTRO]

For some time now, OSX and also Windows support screens of different pixel per inch (DPI, :P).

The approaches are slightly different; but both support a fallback mode where the pixel content of non-aware applications is automatically scaled to look like on "default dpi"[See DEFAULT DPI]. Effectively, we end up with pixel doubling.

One way or another, this means Squeak looks somewhat pixelated on new hardware. 

These are some thoughts of mine on how to address this.

[TERMINOLOGY]
  [DPI] Actually "dots-per-inch" but freely used in multiple senses by different people. For displays it used to mean "how many pixels are there approximately in an inch?", but for a long time now, The [DEFAULT DPI] that OSes report usually was fixed/defaulted (Win: 96, OS X: 72), _regardless_ of the actual screen's dpi.

  [DEFAULT DPI] On OS X, the system reports default DPI as 72 dpi and Retina typically as 144 dpi, _although_ the "actual" dpi is different. For example, My MacBook Pro 15" is reported as  144dpi, but https://www.sven.de/dpi/ calculates that I have 220.53 pixel to the inch. That's why Apple is moving away from DPI and just talking about a "scale factor"[1],[2] and the abstract "point" principle, moving away from actual pixels for "user content"[3]. On Windows, the default dpi is 96 dpi, but Windows always made it a little easier to change that. 

  Squeak also has a default dpi, namely 96.0 dpi.

[STATUS]
  [WINDOWS DPI]

  Windows supports different levels of DPI awareness[4], available depending on OS Version[5].

  Long story short, with calling a special function[6] or (As we did/do/can do) setting a key in the executables ".manifest" file to "true" or "true/PM", Windows treats our Application as DPI-Aware[7].

  Currently the effect of enabling this for our VMs means:
   - Looks ok on non-High-DPI screens
   - Looks tiny on High-DPI screens
   - Moving the VM between High-DPI and non-High-DPI screens changes the outer frame size of the application, the pixel size reported _to_ the VM remains the same.

  [OSX CARBON DPI]

  DPI-awareness and High-DPI are not possible with the QuickDraw-API the Carbon VM used to use.

  [OSX COCOA DPI]

  OS X now has a Device-in-pixel mode and a Userspace-in-"point" mode. When opted-in via Info.plist file, on AppKit-Level, one can convert between those with +[NSScreen backingScaleFactor], -convertRectToBacking:, -convertRectFromBacking and similar. On Quartz2D/CoreGraphics-Level, CGContextGetUserSpaceToDeviceSpaceTransform(), CGContextConvertRectToDeviceSpace(), and CGContextConvertRectToUserSpace() can be used. Both approaches are slightly different. OpenGL needs an additional Opt-in.

  In contrast (currently) to Windows, when moving between high- and non-high-DPI, the outer frame size of the application stays the same but the pixels of the backing store change.

  Up until now, the Cocoa branch is not DPI-aware.

[SUGGESTIONS]

  [OSX COCOA PATCH]
  Attached is a patch that makes the VM High-DPI ready. PLEASE COMMENT.
  Summary:
   - Add Info.plist entry to state whether we want High DPI (NSHighResolutionCapable)
     THIS IS DISABLED BY DEFAULT (as with the Windows VM's manifest)
   - All Views:
     -add -sqScreenSize for the Pixel-dimensions to report to Squeak (and use for calculation in Squeak coordinates)
     -add -sqMousePosition: to convert Mouse coordinates from the View's system (maybe high-dpi) to Squeak coordinates
   - OpenGL view:
     - Opt-in for High resolution (-setWantsBestResolutionOpenGLSurface:) (overridden by Info.plist's entry)
     - Make texture size/tracking rects based on -sqScreenSize
   - Quartz2D/CoreGraphics view:
     - Work around deprecation of [[NSGraphicsContext currentContext] graphicsPort] in OS X 10.10
     - Draw image in -sqScreenSize size, and position correctly in the View's coordinate system
     - (remove some duplication from OpenGL view)
   - iPhone SqueakUIView:
     - use Fallbacks for -sqScreenSize/-sqMousePosition:, no High-DPI yet.
  Note: this patch is against the Cog/platform/ios directory, Interpreter needs similar treating.

  [VM/IMAGE INTERFACE]
  With the current Windows VM and the Cocoa VM (+patch as above), users can Opt-in for High-DPI.

  However, the Image does not know that its high pixel count is on a small display (hello Etoys!). I _want_ the image to know and to be able to react.

  I think that the image should treat DPI changes similar screen size changes. Here are my ideas:
   - The VM exposes screen size as prim 106, screen depth as 'primitiveScreenDepth'
     * We should expose 'primitivePixelPerInch' with a fallback of '^ 96.0'.
     * 96.0 dpi shall be the [DEFAULT DPI] of Squeak.
       > For OS X that would mean, on non-high-dpi it shall report 96.0 dpi (not 72 dpi), on high-dpi it shall report 192.0 dpi (not 144)
       > Windows already has 96.0 dpi as [DEFAULT DPI], it shall reuse that and adhere to [8]
   - The image regularly checks the display size (DisplayScreen class>>checkForNewScreenSize).
     * We should do the same for DPI (Morphic main loop etc.)
   - There's already a client of DPI changes, TextStyles.
     * The image shall store changed dpi settings to TextStyles>>pixelsPerInch:
   - Projects can react to display size changes via #displaySizeChanged, But only fonts can react to DPI changes via #pixelsPerInchChanged
     * We shall make project able to react to dpi changes, maybe via #pixelsPerInchChanged
   - The standard tools are currently pixel-oriented and use hard-coded pixel values
     * I am unsure. 
     * We could introduce conventions to act dpi-dependent (integers: pixels with autoscaling (pixels / default dpi * actual dpi or so), floats or scaled: dpi-based, via TextStyle class>>pointsToPixels:).
     * We could introduce actual units and say "this tool is 3 cm wide" and calculate pixels via #pointsToPixels:.
     * We could eliminate hard-coded values and base tools on font sizes. I tried that with the FontImporterTool (for example: FontImporterTool>>#buttonHeight, #filenameHeight. This may be hard and not very traceable, tho.
    - We could move the dpi api from TextStyle to somewhere more general.

I know there are some rough edges here and there; I'd like to hear your suggestions.

Let's make Squeak look nice and sharp :)

Best
	-Tobias

PS: Thanks to Marcel for some valuable suggestions and help to get Quartz transformations right.

[1]: https://developer.apple.com/library/mac/documentation/Cocoa/Reference/ApplicationKit/Classes/NSScreen_Class/index.html#//apple_ref/occ/instm/NSScreen/backingScaleFactor
[2]: https://developer.apple.com/library/ios/documentation/UIKit/Reference/UIView_Class/index.html#//apple_ref/occ/instp/UIView/contentScaleFactor
[3]: https://developer.apple.com/library/mac/documentation/GraphicsAnimation/Conceptual/HighResolutionOSX/Explained/Explained.html#//apple_ref/doc/uid/TP40012302-CH4-SW2
[4]: https://msdn.microsoft.com/en-us/library/windows/desktop/dn280512(v=vs.85).aspx
[5]: https://msdn.microsoft.com/en-us/library/dn469266(v=vs.85).aspx#high_dpi_features_in_windows
[6]: https://msdn.microsoft.com/en-us/library/dn302122(v=vs.85).aspx
[7]: http://www.squeakvm.org/svn/squeak/branches/Cog/build.win32x86/squeak.cog.spur/Squeak.exe.manifest
[8]: https://msdn.microsoft.com/en-us/library/dn469266(v=vs.85).aspx#dpi_and_the_desktop_scaling_factor

-------------- next part --------------
A non-text attachment was scrubbed...
Name: osx-highdpi.patch
Type: application/octet-stream
Size: 19320 bytes
Desc: not available
Url : http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20160502/d4ad2a40/osx-highdpi-0001.obj