[OpenSmalltalk/opensmalltalk-vm] Freetype broken for build.win32x86/pharo.cog.spur (#319)
Here is what I found...  Its also broken in HEAD.
I'm not certain, but it seems if a previous Freetype 2.6.5 compile succeeded, then the Freetype 2.9.1 compile will succeed - but not in a fresh Cygwin install. So to reproduce... ``` Delete C:\cygwin Reinstall using [installCygwin.bat](https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/scripts/installCy...) $ git clone --depth 30 https://github.com/OpenSmalltalk/opensmalltalk-vm.git $ git checkout -b freetype-broke-here 7ac7303 $ cd opensmalltalk-vm/build.win32x86/pharo.cog.spur $ ./mvm -d ```
Hi Ben,
I'm not certain, but it seems if a previous Freetype 2.6.5 compile succeeded, then the Freetype 2.9.1 compile will succeed - but not in a fresh Cygwin install. So to reproduce...
I'm afraid you're losing me in this sentence, sorry. Are you saying you succeeded in building 2.9.1 after 2.6.5? But that's my current situation, and it doesn't work.
You seem to be saying that doing a fresh install of cygwin doesn't help, so why do it?
Is this just showing how to reproduce the error?
Can you clarify?
Thanks, Alistair
Are you saying you succeeded in building 2.9.1 after 2.6.5?
Yes. This was a once-off experience... that tried build Windows HEAD and it failed, then I went to an earlier commit and that built okay, then I returned to HEAD and that now built okay. I'm in the middle of trying to replicate that, but haven't yet, hence "seems".
A fresh cygwin help reproduce the error since it ensures nothing historical is accidentally making the HEAD build. i.e. to avoid a "works for me" scenario.
Okay, here is my experiment!
Broken commit on its own... ``` 253 mkdir ~/7ac7303 254 cd ~/7ac7303 255 git clone --depth 30 https://github.com/OpenSmalltalk/opensmalltalk-vm.git 256 cd opensmalltalk-vm/ 257 git checkout 7ac7303 258 grep url third-party/freetype2.spec ==>freetype2_spec_download_url:=http://ftp.igh.cnrs.fr/pub/nongnu/freetype/freetype-2.9.1.tar.gz 259 cd build.win32x86/pharo.cog.spur 260 ./mvm -d 281 history FAILURE! ``` Now in another directory, first the good commit then broken commit on top of that. ``` 270 mkdir ~/b83f9ecd 271 cd ~/b83f9ecd 272 git clone --depth 30 https://github.com/OpenSmalltalk/opensmalltalk-vm.git 273 cd opensmalltalk-vm/ 274 git checkout b83f9ecd 275 grep url third-party/freetype2.spec ==>freetype2_spec_download_url:=http://ftp.igh.cnrs.fr/pub/nongnu/freetype/freetype-2.6.5.tar.gz 276 cd build.win32x86/pharo.cog.spur 277 ./mvm -d SUCCESS AS EXPECTED
278 git checkout 7ac7303 279 grep url ../../third-party/freetype2.spec ==>freetype2_spec_download_url:=http://ftp.igh.cnrs.fr/pub/nongnu/freetype/freetype-2.9.1.tar.gz 280 ./mvm -d SUCCESS!!! 281 history ```
Hi Ben,
Can you check which version of libfreetype.dll you end up with after the final build. I followed basically the same steps as above (build b83f9ecd followed by 7ac73031f) and get:
FT2Library current libraryVersion " 2.6.5"
P.S. I hope I've done something wrong!
Thanks again for looking in to this.
Cheers, Alistair
I also get 2.6.5. So the reason it succeeded is that it used the cached library and didn't even try to build 2.9.1. So secondary issue, there seems something wrong with make's dependency on third-party/freetype2.spec.
Yep. And vmLatest is still 2.6.5, so it's not just us.
That seems a little strange though. IIUC vmLatest should be have been built on the CI with a fresh environment.
this is weird. I remember I cleaned the cache to have a proper build.
Just recording one observed post-build difference between v2.6.5 and v2.9.1... ``` Ben@BENGINEER ~/Repos/freetype265/opensmalltalk-vm/build.win32x86/pharo.cog.spur $ find . -name ftconfig.h ./builddbg/third-party/freetype-2.6.5/build/include/freetype/config/ftconfig.h ./builddbg/third-party/freetype-2.6.5/builds/amiga/include/config/ftconfig.h ./builddbg/third-party/freetype-2.6.5/builds/vms/ftconfig.h ./builddbg/third-party/freetype-2.6.5/include/freetype/config/ftconfig.h ``` ``` Ben@BENGINEER ~/Repos/freetype291/opensmalltalk-vm/build.win32x86/pharo.cog.spur $ find . -name ftconfig.h ./builddbg/third-party/freetype-2.9.1/builds/amiga/include/config/ftconfig.h ./builddbg/third-party/freetype-2.9.1/builds/vms/ftconfig.h ./builddbg/third-party/freetype-2.9.1/include/freetype/config/ftconfig.h ``` And the broken build says... ``` -- Installing: .../opensmalltalk-vm/.thirdparty-cache/windows/i386/include/freetype2/ft2build.h
file INSTALL cannot find ./builddbg/third-party/freetype-2.9.1/build/include/freetype/config/ftconfig.h ```
I normalised the LOGD files to remove the version info then diff'd them... ``` $ sed 's/2.6.5/X.X.X/g' LOGD.265 | sed 's/freetype265/freetypeXXX/g' > LOG.265xxx $ sed 's/2.9.1/X.X.X/g' LOGD.291 | sed 's/freetype291/freetypeXXX/g' > LOG.291xxx $ diff LOGD.265x LOGD.291x
4130c4130 < configure: autobuild timestamp... 20181212-153854 ---
configure: autobuild timestamp... 20181212-154649
5789d5788 5796,5801d5794 < -- The CXX compiler identification is GNU 6.4.0 < -- Check for working CXX compiler: /usr/bin/i686-w64-mingw32-g++.exe < -- Check for working CXX compiler: /usr/bin/i686-w64-mingw32-g++.exe -- works < -- Detecting CXX compiler ABI info < -- Detecting CXX compiler ABI info - done < -- Detecting CXX compile features < -- Detecting CXX compile features - done
5803,5807c5796,5800 < -- Checking for module 'harfbuzz>=0.9.7' < -- < -- Creating directory /home/Ben/Repos/freetypeXXX/opensmalltalk-vm/build.win32x86/pharo.cog.spur/builddbg/third-party/freetype-X.X.X/build/include/freetype/config < -- Creating file /home/Ben/Repos/freetypeXXX/opensmalltalk-vm/build.win32x86/pharo.cog.spur/builddbg/third-party/freetype-X.X.X/build/include/freetype/config/ftconfig.h < -- Creating file /home/Ben/Repos/freetypeXXX/opensmalltalk-vm/build.win32x86/pharo.cog.spur/builddbg/third-party/freetype-X.X.X/build/include/freetype/config/ftoption.h ---
-- Could NOT find harfbuzz (missing: HARFBUZZ_INCLUDE_DIRS HARFBUZZ_LIBRARIES) -- Could NOT find ZLIB (missing: ZLIB_LIBRARY ZLIB_INCLUDE_DIR) -- Could NOT find PNG (missing: PNG_LIBRARY PNG_PNG_INCLUDE_DIR) -- Could NOT find ZLIB (missing: ZLIB_LIBRARY ZLIB_INCLUDE_DIR) -- Could NOT find BZip2 (missing: BZIP2_LIBRARIES BZIP2_INCLUDE_DIR)
5809a5803,5808
CMake Warning: Manually-specified variables were not used by the project:
WITH_PNG
5880d5877 [BEN==>From release ntoes I think these file differences are okay/intended] < -- Installing: /home/Ben/Repos/freetypeXXX/opensmalltalk-vm/.thirdparty-cache/windows/i386/include/freetype2/freetype/ftautoh.h < -- Installing: /home/Ben/Repos/freetypeXXX/opensmalltalk-vm/.thirdparty-cache/windows/i386/include/freetype2/freetype/ftcffdrv.h < -- Installing: /home/Ben/Repos/freetypeXXX/opensmalltalk-vm/.thirdparty-cache/windows/i386/include/freetype2/freetype/ftttdrv.h < -- Installing: /home/Ben/Repos/freetypeXXX/opensmalltalk-vm/.thirdparty-cache/windows/i386/include/freetype2/freetype/ttunpat.h ---
-- Installing: /home/Ben/Repos/freetypeXXX/opensmalltalk-vm/.thirdparty-cache/windows/i386/include/freetype2/freetype/ftdriver.h -- Installing: /home/Ben/Repos/freetypeXXX/opensmalltalk-vm/.thirdparty-cache/windows/i386/include/freetype2/freetype/ftparams.h
5924,5929c5920,5925 < -- Installing: /home/Ben/Repos/freetypeXXX/opensmalltalk-vm/.thirdparty-cache/windows/i386/include/freetype2/freetype/config/ftconfig.h ---
CMake Error at cmake_install.cmake:35 (file): file INSTALL cannot find "/home/Ben/Repos/freetypeXXX/opensmalltalk-vm/build.win32x86/pharo.cog.spur/builddbg/third-party/freetype-X.X.X/build/include/freetype/config/ftconfig.h".
make[2]: *** [Makefile:62: install] Error 1 make[1]: *** [../third-party/Makefile.freetype2:25: /home/Ben/Repos/freetypeXXX/opensmalltalk-vm/.thirdparty-cache/windows/i386/bin/libfreetype.dll] Error 2 make[1]: Leaving directory '/home/Ben/Repos/freetypeXXX/opensmalltalk-vm/build.win32x86/pharo.cog.spur' make: *** [../common/Makefile:157: debug] Error 2
5924,5929c5920,5925 < -- Installing: /home/Ben/Repos/freetypeXXX/opensmalltalk-vm/.thirdparty-cache/windows/i386/include/freetype2/freetype/config/ftoption.h < -- Installing: /home/Ben/Repos/freetypeXXX/opensmalltalk-vm/.thirdparty-cache/windows/i386/lib/libfreetype.dll.a < -- Installing: /home/Ben/Repos/freetypeXXX/opensmalltalk-vm/.thirdparty-cache/windows/i386/bin/libfreetype.dll < -- Installing: /home/Ben/Repos/freetypeXXX/opensmalltalk-vm/.thirdparty-cache/windows/i386/lib/cmake/freetype/freetype-config.cmake < -- Installing: /home/Ben/Repos/freetypeXXX/opensmalltalk-vm/.thirdparty-cache/windows/i386/lib/cmake/freetype/freetype-config-noconfig.cmake < cp -f /home/Ben/Repos/freetypeXXX/opensmalltalk-vm/.thirdparty-cache/windows/i386/bin/libfreetype.dll builddbg/vm ```
[harfbuzz commits](http://git.savannah.gnu.org/cgit/freetype/freetype2.git/log/?qt=grep&q=h...) since 2016-07-12... ``` 2018-05-08 | [autofit] Avoid potential SEGV if running out of memory. 2018-04-10 | Modernize CMake build. 2018-03-01 | * builds/unix/configure.raw: Need HarfBuzz 1.3.0 or newer. 2017-10-14 | [afshaper] Delay creating `hb_set' objects until needed. 2016-08-07 | [sfnt] Fix `FT_Get_Advance' for bitmap strikes. ```
harfbuzz package has available versions 1.4.8-1, 1.7.4-1, 1.7.6-1. May as well choose the latest. Installed harfbuzz-1.7.6-1.
Just to isolate the error away from the overarching pharo build, so its easier to iterate on... ``` $ mkdir -p /tmp/freetype291 && cd /tmp/freetype291
$ cp ~/Repos/freetype291/opensmalltalk-vm/build.win32x86/pharo.cog.spur/builddbg/third-party/freetype-2.9.1/Toolchain-cross-mingw32-cygwin.cmake \ /tmp/freetype291/Toolchain-cross-mingw32-cygwin.cmake
$ wget http://ftp.igh.cnrs.fr/pub/nongnu/freetype/freetype-2.9.1.tar.gz
$ tar xf freetype-2.9.1.tar.gz && cd freetype-2.9.1
$ mkdir build && cd build
$ cmake ..\ -DCMAKE_TOOLCHAIN_FILE=../Toolchain-cross-mingw32-cygwin.cmake \ -DCMAKE_INSTALL_PREFIX=/home/Ben/Repos/freetype265/opensmalltalk-vm/.thirdparty-cache/windows/i386 \ -DWITH_PNG=OFF \ -DBUILD_SHARED_LIBS=ON \ -DCMAKE_C_FLAGS="-m32 -static-libgcc -static-libstdc++" -- The C compiler identification is GNU 6.4.0 -- Check for working C compiler: /usr/bin/i686-w64-mingw32-gcc.exe -- Check for working C compiler: /usr/bin/i686-w64-mingw32-gcc.exe -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Found PkgConfig: /usr/bin/pkg-config.exe (found version "0.29.1") -- Could NOT find harfbuzz (missing: HARFBUZZ_INCLUDE_DIRS HARFBUZZ_LIBRARIES) -- Could NOT find ZLIB (missing: ZLIB_LIBRARY ZLIB_INCLUDE_DIR) -- Could NOT find PNG (missing: PNG_LIBRARY PNG_PNG_INCLUDE_DIR) -- Could NOT find ZLIB (missing: ZLIB_LIBRARY ZLIB_INCLUDE_DIR) -- Could NOT find BZip2 (missing: BZIP2_LIBRARIES BZIP2_INCLUDE_DIR) -- Configuring done -- Generating done CMake Warning: Manually-specified variables were not used by the project: WITH_PNG -- Build files have been written to: /tmp/freetype291/build/freetype-2.9.1/build
$ make
$ make CMake Error at cmake_install.cmake:35 (file): file INSTALL cannot find "/tmp/freetype291/build/freetype-2.9.1/build/include/freetype/config/ftconfig.h". ```
Confirmed broken upstream https://savannah.nongnu.org/bugs/?func=detailitem&item_id=54846
@estebanlm, Making just this change http://git.savannah.gnu.org/cgit/freetype/freetype2.git/commit/?id=660afb5ce...... ``` diff --git a/CMakeLists.txt b/CMakeLists.txt index ad8ded0..6f3fef7 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -229,9 +229,12 @@ if (UNIX) endif () string(REPLACE "/undef " "#undef " FTCONFIG_H "${FTCONFIG_H}") - file(WRITE "${PROJECT_BINARY_DIR}/include/freetype/config/ftconfig.h" - "${FTCONFIG_H}") +else() + file(READ "${PROJECT_SOURCE_DIR}/include/freetype/config/ftconfig.h" + FTCONFIG_H) endif () +file(WRITE "${PROJECT_BINARY_DIR}/include/freetype/config/ftconfig.h" + "${FTCONFIG_H}") ``` fixes the build ``` -- Installing: /home/Ben/Repos/freetype265/opensmalltalk-vm/.thirdparty-cache/windows/i386/include/freetype2/freetype/config/ftconfig.h Successfully remade target file 'install'. ``` How do you want to handle applying this patch? Perhaps tar up the patched code and put it on a server of our own? e.g. http://files.pharo.org/thirdparty/freetype-2.9.1+660afb5.tar.gz
Ran image 71433 with VM built above. All tests pass except: * Monticello-Tests... tests get stuck on error "PrimitiveFailed: primitive #sizeOf: in File class failed" * Tests-Traits... get stuck on "Instance of C2 class did not understand #localMethodDict"
Next step is to modify ***opensmalltalk-vm/build.win32x86/third-party/Makefile.freetype2*** to patch its CMakeLists.txt after the "tar x" (but might be a couple of days before I can get to this.)
@bencoman, I'm really grateful for you doing this. Thank you very much :)
Likewise. Esteban can I leave it to you to approve the pull request? I don't want to interfere; I'm afraid of breaking things.
@estebanlm, Does **scripts/installCygwin.bat** get run each CI build? My PR will include adding installation of the "patch" package to this script.
yes it should
To summarise our Discord chat, `patch` command was not installed by ***installCygwin.bat*** but is required so it can be invoked from ***Makefile.filetree*** as you suggest :).
Sorry, I didn't test run ***installCygwin.bat*** and appveyor barfed the PR. I'll force push a fix in a moment.
This has been merged in
Closed #319.
Unfortunately, after all this hard and appreciated work, we now have another CI problem: the build times out...
https://ci.appveyor.com/project/OpenSmalltalk/vm/builds/21318435/job/b3npekw...
Build execution time has reached the maximum allowed time for your plan (60 minutes).
We cannot re-build every library we need by ourself, it does not scale... Or we build them in a different project
Maybe we may ask:
https://help.appveyor.com/discussions/problems/16983-hitting-60min-build-tim...
Unfortunately, after all this hard and appreciated work, we now have another CI problem: the build times out... https://ci.appveyor.com/project/OpenSmalltalk/vm/builds/21318435/job/b3npekw...
Build execution time has reached the maximum allowed time for your plan (60 minutes).
I think something else must have gone wrong. I tested caching-disabled here... https://ci.appveyor.com/project/bencoman/opensmalltalk-vm/builds/21290200 to force a full rebuild of all third-party dependencies and the max job time was 32 minutes (with timeout per-job, not overall.)
We cannot re-build every library we need by ourself, it does not scale... Or we build them in a different project
I do believe there would be value in that, particularly to speed up building locally for the first time. But this failure for build 1.0.1585 seems like just an aberration. Five subsequent builds have succeeeded... https://ci.appveyor.com/project/OpenSmalltalk/vm/history
well, it should not be the build of freetype who adds so much time (and yes, time to build was ~30-40 minutes last time I checked). If needed, for different reasons, in the Pharo win64 build I started to put third party libraries in a repository and download them from there... maybe we can do the same for the 32bit build?
On Wed, 2 Jan 2019 at 20:14, Esteban Lorenzano notifications@github.com wrote:
well, it should not be the build of freetype who adds so much time (and yes, time to build was ~30-40 minutes last time I checked). If needed, for different reasons, in the Pharo win64 build I started to put third party libraries in a repository and download them from there... maybe we can do the same for the 32bit build?
I was curious about the different arrangement for win64. Could you point to that repo? cheers -ben
is in the makefiles for pharo.cog.spur :)
http://files.pharo.org/vm/pharo-spur64/win/third-party/
On Wed, 2 Jan 2019 at 20:41, Esteban Lorenzano notifications@github.com wrote:
is in the makefiles for pharo.cog.spur :)
Cool. But what is the heritage of those files? Is there a version controlled makefile that built them?
That's the advantage of using cygwin pre-built libraries: they are compiled for us and security patches are applied sooner than if we manage them by ourself.
And that's the drawback: we have no control on exact version (apart major version), and library can change from one build to another. But it's the same if you use pre-built libraries from macosx and linux (libc, libm, etc...). We have to trust...
vm-dev@lists.squeakfoundation.org
-
akgrant43 -
Ben Coman -
Ben Coman
-
Eliot Miranda
-
Esteban Lorenzano
-
Nicolas Cellier