Porting Edict to Win32

The better portion of the last 6 weeks have been spent porting Edict and the 10 years of nerdcruft support libraries to Windows and introducing proper packaging for distribution. Given that I almost exclusively develop under, and for, Linux there are a few cases where my platform assumptions are broken under Windows. A few of the major sticking points I encountered are detailed below.

Hardware

Windows IO is appalling. Particularly if you rely on random IO; doubly so if you still use a HDD.

Unfortunately software development is particularly weighted towards touching many small files. `Edict' is composed of ~1500 files, and the vendored dependencies another ~800; around half of which generate object files. As a result the performance of my IDE tends to be constrained by random IO. Interestingly, debug build times were also IO limited due to the size of the binaries and associated symbols.

Thankfully I had an old SSD used for development work in an old machine laying in a drawer. It doesn’t have a great capacity but just about anything in the last few years is overkill for a single project.

As an aside: Linux’s bcache block device, which provides a fast IO cache, is far better than it has any right to be (and I eagerly await the upstreaming of bcachefs). It has totally transformed how my desktop performs under Linux. It is unconscionable that Windows does not provide this by default. Intel does, but it’s architecture specific and size restricted, which boggles the mind.

MSYS2

A small part of my core code relies upon specific extensions found in gcc and clang, or features not implemented by MSVC. Rather than transition to Win32 and MSVC I opted to use the MSYS2 environment to provide the necessary tools: gcc, cmake, git, pgk-config, etc; and transition to MSVC at my option later.

PATH

While there are some noticeable differences in the runtime behaviour of MSYS’s tools they are close enough to a familiar environment that existing scripting shouldn’t need much attention.

However the PATH environment variable did trip me up a few times. One quick method of addressing the MSYS tools is to prepend/append the MSYS paths to the system or user PATH. Unfortunately in my case CLion requires 'MinGW Makefiles' which result in hard errors when sh is in your path. The workaround for this was to point CMAKE_PREFIX_PATH at the MSYS tools for Windows targets rather than modify the system PATH.

Be careful that your PATH refers to the correct tool for your architecture. pkg-config in particular may appear to operate correctly but search only for 32 bit libraries if you have installed versions of the tool for multiple architectures.

sh

Shell scripts may more effort to support under Windows than is warranted. A shell may not be present and differences in filesystem behaviours can rapidly trip up some tools (with file locking being the most common culprit).

Given I already use Python as a hard dependency in other locations of Edict I finished the incremental transition from Shell to Python that had already started.

Packaging

In an attempt to keep platform specifics down I use CPack for generating packages. I can generate NSIS for Windows, .deb for Linux, and TXZ for ad-hoc testing from the one set of definitions.

It was quite quick to get a trivial TXZ created if you’re not too picky about filesystem layouts.

However it does seem that the CPack documentation and functionality is 90% complete.

  • Why do installed binaries sometimes have hardcoded prefixes?

  • What effect do half the CPACK variables actually have?

I’m still not quite sure what the new hotness is for open Windows installers. I suspect 'WIX' is recommended. But while constructing an NSIS config is pretty painful, it works, it’s scriptable, and it’s one less thing to relearn.

Vendoring

My development system happens to have most of Edict’s dependencies already installed to support other applications I use outside of work. This makes it somewhat annoying to consistently ensure that build or runtime dependencies are consistently in the build and package configurations. Using a strict continuous integration setup to protect against this is still on my ever growing TODO list but should dovetail nicely with recent work I’ve done for testing under GitLab with docker.

To ensure reproducibility, and some level of build time configuration, I’ve changed all my dependencies to submodules which are built at configure time. This exposed some annoyances in their build and runtime behaviour.

  • zlib has a tendency to modify files, and minizip dumps autoconf files in the source tree. This necessitates changes to the 'ignore' attribute of the submodules if we want a clean git status.

  • assimp can’t be build out of tree under Windows, and can’t load a subset of model formats when built with any explicit build configuration.

  • freetype-2.9.1 produces various link time errors under MSYS.

It honestly would have been less time consuming to finish the equivalent nerdcruft libraries than it would have taken to ensure these libraries configured, built, and were discovered under MSYS (particularly assimp). However it’s done now. The replacement work may form a weekend project over the coming months.

Vulkan

While I don’t currently use Vulkan nerdcruft has a set of independent C++ bindings and a library/layer loader created over the course of some experiments last year. I spend a small period porting this library until sufficient complexities emerged that it wasn’t worth the investment for this project.

Enumeration of platforms under Windows felt unjustifiably complex. Under Linux (and presumably other POSIX systems) we scan a hardcoded set of locations for JSON configuration files. Under Windows we scan PnP registry keys for values that point to the actual configuration files. Perhaps my distaste for this approach comes from my disdain for the Windows registry (and associated lack of support in my platform libraries).

Coupled with the various platform issues outlined above it wasn’t worth the time investment for this project. Some improvements are being made over weekends, but it’s unclear when I’ll get to finishing this proper.

Win32

The Windows platform APIs are totally different to every other platform I intend to support.

There are some components which almost necessarily need to be written for each platform, e.g. `How do we query the current application’s path?'

Others are similar across POSIX but different under Windows, e.g. `How do we memory map a file?' Thankfully most of these concerns fall neatly into some form of abstraction we already provide.

IO

Path differences, binary IO, and file locking were unreasonably annoying. A good number of differences in configuration and build differences come down to either:

separators

Given ':' is used as a drive separator it can’t be unambiguously used as a separator for list. Unlike every other system I deal with. PATH and friends all have to be special cased for one system now (and disappointingly there isn’t a distinct library search path variable like LD_LIBRARY_PATH). And the use of '\' as a directory separator means we now need to be on the lookout for these characters creeping into other areas of the build given they, again, won’t work on any other system I encounter.

character encoding

The use of UCS-2 for character encoding, while laudable given it’s early adoption, complicates platform agnostic code given the prevalence of UTF-8 on every other system. My current approach is to rely on the C++ u8string accessor for paths, but this approach will be insufficient and needs revisiting when proper i18n and l10n support is added to Edict.

More insidious is the need to opt in to binary IO (i.e. decline the transformation of newlines). While the above problems tend to result in hard build or runtime errors, forgetting to set O_BINARY on one occasion may result in data corruption which might only be detected much further along in the pipeline.

My current approach is to introduce an assertion that binary mode has been selected in all IO wrapper objects. While this complicates any hypothetical support requirement for Windows line endings I’m not convinced that enabling this behaviour will ever be a valid use case for my projects.

And lastly, the default behaviour to lock access to files differences substantially from the UNIX model (where files can be readily moved and deleted at any point after their creation irrespective of open file handles). It’s possible to work around this behaviour in my own code, but more troublesome is the behaviour of external tooling; we can’t change, for example, the locking mode of Python’s tempfile module despite how thoroughly it broke the assumptions of some of my unit tests.

Symbol renaming

There appears to a tendency for developers to know what symbol names I desire and define them with macros deep inside some critical Windows header, e.g. NEAR and FAR for pointers vs clipping planes, and OPAQUE for blending. This can result in some bugs that are difficult to track down or necessitate a great deal of renaming within your own projects. Given I have a hard enough time remembering many enumeration names without going to my second choice name I opted for a third approach: nuke their macro and replace it with my own.

Yes, it’s incredibly dangerous; not because of symbols redefinitions, but the chance of silently using invalid values. This is mitigated somewhat by consistent use of strong typing in these definitions such that there’s a limited risk of inadvertent casting to a common type (like int), and the Windows specifics tend to be very tightly controlled and auditable.

Multithreading

I’ve created a few special purpose threading primitives outside those of the standard libraries. Some of these can be efficiently implemented using standard library primitives (e.g. ticketlock uses std::atomic), others may be implemented less efficiently in terms of standard library primitives or by using system primitives (e.g. semaphores using std::condition_variable and std::atomic);

This would have been incredibly simple if there was no need to support Windows 7 as future iterations provide WaitOnAddress which is a superset of the futex functionality these primitives are implemented with under Linux. Unfortunately it’s a hard call to drop Windows 7 support and so we need to resort to constructing these structures from older Win32 APIs.

Having written a few iterations of this functionality over the years it wasn’t terrifically time consuming but it does require a fair amount of concentration for the duration.

CL/GL

Our OpenGL and, to a lesser extent, OpenCL binding generator needed a variety of small improvements.

Some amount of the generated code was hacked together under the assumption that the required platform was GLX; statically specified headers, some type overrides, and functionality to select the desired platform (GLX/WGL/etc) needed to be modified. Now we have the ability to select the platform provider for OpenGL and Vulkan (as the binding generator is largely a shared codebase for the Khronos APIs).

The primary complication of platform selection under Windows has been avoiding symbol clashes. The Windows headers include definitions for OpenGL functions under GDI which tend to be included frequently enough that clashes can’t be avoided. The simplest solution to this is namespacing all generated OpenGL functions. But it does have the drawback that it’s quite possible to accidentally reference the Windows symbols rather than our own.

To mitigate this problem I’ve imposed a strict requirement to never refer to any header that we wrap outside of one canonical wrapper header; i.e. if we want OpenGL functions we include <cruft/gl/gl.hpp> rather than <GL/gl.h>. This allows us to contain any platform specific mitigations or hacks in the one place.

Future

Now that I can provide installers or tarballs to testers on a more popular platform it offers a greater variety of systems I can get feedback on before more public availability. The coming development cycle will focus on more efficient means of testing feedback for platform issues, and the expansion of gameplay elements in the hope I can have something more concrete to show to interested parties at GCAP.

Comments

comments powered by Disqus