Ternaris Laboratoryhttp://ternaris.com/lab/2015-09-04T17:42:00+02:00Technical Report: Nix on Windows2015-09-04T17:42:00+02:00Ternaristag:ternaris.com,2015-09-04:lab/nix-on-windows.html<div class="contents topic" id="contents">
<p class="topic-title first">Contents</p>
<ul class="auto-toc simple">
<li><a class="reference internal" href="#cross-platform-deployment-with-nix" id="id4">1 Cross-platform deployment with Nix</a><ul class="auto-toc">
<li><a class="reference internal" href="#nix-the-purely-functional-package-manager" id="id5">1.1 Nix — The Purely Functional Package Manager</a></li>
<li><a class="reference internal" href="#the-nix-store" id="id6">1.2 The Nix Store</a></li>
<li><a class="reference internal" href="#the-standard-environment" id="id7">1.3 The Standard Environment</a></li>
</ul>
</li>
<li><a class="reference internal" href="#compiling-on-and-for-windows" id="id8">2 Compiling on and for Windows</a><ul class="auto-toc">
<li><a class="reference internal" href="#id1" id="id9">2.1 Mingw-w64</a></li>
<li><a class="reference internal" href="#id2" id="id10">2.2 Cygwin</a></li>
</ul>
</li>
<li><a class="reference internal" href="#challenges-on-cygwin" id="id11">3 Challenges on Cygwin</a><ul class="auto-toc">
<li><a class="reference internal" href="#finding-dlls" id="id12">3.1 Finding DLLs</a></li>
<li><a class="reference internal" href="#dll-relocation-and-cygwin-s-fork" id="id13">3.2 <span class="caps">DLL</span> Relocation and Cygwin’s fork</a></li>
<li><a class="reference internal" href="#automagic-exe-suffixing" id="id14">3.3 Automagic .exe Suffixing</a></li>
</ul>
</li>
<li><a class="reference internal" href="#reproducible-cygwin-environments" id="id15">4 Reproducible Cygwin Environments</a><ul class="auto-toc">
<li><a class="reference internal" href="#custom-cygwin-mirror" id="id16">4.1 Custom Cygwin Mirror</a></li>
<li><a class="reference internal" href="#installation-and-cygwin-package-management" id="id17">4.2 Installation and Cygwin Package Management</a></li>
<li><a class="reference internal" href="#remote-access" id="id18">4.3 Remote Access</a></li>
<li><a class="reference internal" href="#do-not-copy-cygwins" id="id19">4.4 <span class="caps">DO</span> <span class="caps">NOT</span> <span class="caps">COPY</span> <span class="caps">CYGWINS</span></a></li>
</ul>
</li>
<li><a class="reference internal" href="#nix-on-cygwin" id="id20">5 Nix on Cygwin</a><ul class="auto-toc">
<li><a class="reference internal" href="#preparations" id="id21">5.1 Preparations</a></li>
<li><a class="reference internal" href="#package-build-and-install-nix" id="id22">5.2 Package, Build and Install Nix</a><ul class="auto-toc">
<li><a class="reference internal" href="#fix-missing-library-function-definitions" id="id23">5.2.1 Fix Missing Library Function Definitions</a></li>
<li><a class="reference internal" href="#fix-library-file-extension-and-directory" id="id24">5.2.2 Fix Library File Extension and Directory</a></li>
<li><a class="reference internal" href="#fix-perl-extension" id="id25">5.2.3 Fix Perl Extension</a></li>
</ul>
</li>
<li><a class="reference internal" href="#standard-environment-hooks" id="id26">5.3 Standard Environment Hooks</a><ul class="auto-toc">
<li><a class="reference internal" href="#dll-search-path-wrappers" id="id27">5.3.1 <span class="caps">DLL</span> Search Path Wrappers</a></li>
<li><a class="reference internal" href="#runtime-dependency-non-detection" id="id28">5.3.2 Runtime Dependency Non-Detection</a></li>
<li><a class="reference internal" href="#dll-rebasing-to-enable-cygwin-s-fork" id="id29">5.3.3 <span class="caps">DLL</span> Rebasing to Enable Cygwin’s fork()</a></li>
</ul>
</li>
<li><a class="reference internal" href="#nixpkgs-fixes-a-beginning" id="id30">5.4 Nixpkgs fixes — A Beginning</a></li>
</ul>
</li>
<li><a class="reference internal" href="#summary-and-outlook" id="id31">6 Summary and Outlook</a><ul class="auto-toc">
<li><a class="reference internal" href="#store-level-dll-rebasing" id="id32">6.1 Store Level <span class="caps">DLL</span> Rebasing</a></li>
<li><a class="reference internal" href="#proper-runtime-dependency-detection" id="id33">6.2 Proper Runtime Dependency Detection</a></li>
<li><a class="reference internal" href="#pure-standard-environment" id="id34">6.3 Pure Standard Environment</a></li>
<li><a class="reference internal" href="#environment-variable-length" id="id35">6.4 Environment Variable Length</a></li>
<li><a class="reference internal" href="#batch-files-instead-of-shebang-wrappers" id="id36">6.5 Batch Files Instead of Shebang Wrappers</a></li>
<li><a class="reference internal" href="#native-windows-binaries" id="id37">6.6 Native Windows Binaries</a></li>
</ul>
</li>
</ul>
</div>
<p><em>Nix is a package manager for Unix-like systems, which supports the installation of multiple versions of an application and which can run in parallel to whatever package manager a system uses natively (apt, pacman, yum, …). As other package managers, it needs a compiler toolchain to generate packages. Mingw-w64 and Cygwin provide free open source toolchains on Windows, both having their trade-offs and benefits: Cygwin offers a <span class="caps">POSIX</span> emulation needed by many applications, but binaries generated with Cygwin require the Cygwin runtime library; Mingw-w64 generates native Windows binaries without providing a <span class="caps">POSIX</span> emulation. In this report we describe how the Nix package manager realizes the installation of multiple versions of an application, what challenges Cygwin poses to these mechanisms, how we addressed, what should be done next, and how Mingw-w64 could be integrated to support management of native windows applications.</em></p>
<div class="section" id="cross-platform-deployment-with-nix">
<h2><a class="toc-backref" href="#id4">1 Cross-platform deployment with Nix</a></h2>
<p>Cross-platform deployment means to exactly reproduce an application with its dependencies across different target platforms, like <span class="caps">GNU</span>/Linux, Mac <span class="caps">OS</span> X or Windows. An application — or more specifically a release of an application — is built and tested with specific versions of dependencies, like dynamic libraries, interpreter language modules, databases and command line tools.</p>
<p>For the deployment of multiple applications or releases of one application, it is essential that multiple versions of dependencies can be installed in parallel without interference. To minimize disk and <span class="caps">RAM</span> usage it is desirable that identical dependencies are shared between applications.</p>
<p>The Nix Package Manager solves these problems and the remainder of this section gives a brief introduction of its features and mechanisms.</p>
<div class="section" id="nix-the-purely-functional-package-manager">
<h3><a class="toc-backref" href="#id5">1.1 Nix — The Purely Functional Package Manager</a></h3>
<p><a class="reference external" href="http://nixos.org/nix">Nix</a> is a package manager for Unix-like systems, which can run in parallel to whatever package manager the system uses natively (apt, pacman, yum, …). As of <a class="reference external" href="http://hydra.nixos.org/release/nix/nix-1.7">version 1.7</a> it was available for <span class="caps">GNU</span>/Linux and Mac <span class="caps">OS</span> X (Darwin). <a class="reference external" href="http://nixos.org/nixos/about.html">NixOS</a> is a Linux distribution that uses it natively as its sole package manager.</p>
<p>Nix is a purely functional package manager in that it treats the files of packages like values of functions in purely functional programming languages such as Haskell: The functions describe how packages are built, they don’t have side-effects, and they never change after they have been built.</p>
<p>A collection of such build functions is provided as the Nix Packages Collection (<a class="reference external" href="https://github.com/NixOS/nixpkgs">Nixpkgs</a>). At its core Nixpkgs has a set of functions that provide a compiler toolchain and a minimal set of system tools across platforms, described in its own section: <a class="reference internal" href="#the-standard-environment">The Standard Environment</a>.</p>
<p>Nix supports installation of multiple versions of one application and even multiple, differently compiled builds of one version; sticking to the Filesystem Hierachy Standard (<a class="reference external" href="http://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard"><span class="caps">FHS</span></a>) would lead to collisions. Instead nix installs the files generated by its build functions into a database, described in its own section: <a class="reference internal" href="#the-nix-store">The Nix Store</a>.</p>
</div>
<div class="section" id="the-nix-store">
<h3><a class="toc-backref" href="#id6">1.2 The Nix Store</a></h3>
<p>Nix uses functions in a purely functional sense to describe its packages. The values of these functions are called <em>outputs</em> and are stored as subdirectories of the Nix Store (<tt class="docutils literal">/nix/store</tt>). An <em>output</em> in the store holds the content of a package that would usually be installed to <tt class="docutils literal">/usr</tt>. In that regard, the store resembles <tt class="docutils literal">/opt</tt> with the possibility to have multiple versions and even multiple builds of the same version installed in parallel. This is achieved by using a cryptographic hash as a prefix for the store output.</p>
<div class="highlight"><pre><span class="go">% ls -l /nix/store/w08118q0kp26qdcz5cdl2rx6chghawam-hello-2.9</span>
<span class="go">dr-xr-xr-x 2 root root 4096 Jan 1 1970 bin/</span>
<span class="go">dr-xr-xr-x 5 root root 4096 Jan 1 1970 share/</span>
<span class="go">% ls -l /nix/store/w08118q0kp26qdcz5cdl2rx6chghawam-hello-2.9/bin</span>
<span class="go">-r-xr-xr-x 2 root root 30845 Jan 1 1970 hello*</span>
</pre></div>
<p>The cryptographic hash used as a prefix is calculated over the function describing the build of a package as well as all its build-time dependencies, represented by similar functions. Variations in any of these will lead to a different hash and therefore a different store location.</p>
<p>To make sure that a binary in the store will load the correct dynamic libraries, Nix hardcodes absolute store paths of dependencies into build binaries.</p>
<div class="highlight"><pre><span class="go">% ldd =hello</span>
<span class="go"> linux-vdso.so.1 (0x00007fffbc1fe000)</span>
<span class="go"> libc.so.6 => /nix/store/i11d0d4015p0vbdnjq7lb509v9pwp049-glibc-2.19/lib/libc.so.6 (0x00007f55d1263000)</span>
<span class="go"> /nix/store/i11d0d4015p0vbdnjq7lb509v9pwp049-glibc-2.19/lib/ld-linux-x86-64.so.2 (0x00007f55d1610000)</span>
</pre></div>
<p>Based on such and other references of store locations, Nix performs automatic runtime dependency detection and supports copying a package and its dependencies from one machine to another: If a store path of package A occurs in any file of package B, package A is a runtime dependency of B.</p>
</div>
<div class="section" id="the-standard-environment">
<h3><a class="toc-backref" href="#id7">1.3 The Standard Environment</a></h3>
<p>One essential component for building packages with Nix is the standard environment, mostly consisting of:</p>
<ul class="simple">
<li>A working compiler toolchain (gcc, libc, binutils) and</li>
<li>a minimal set of system tools (e.g. curl, coreutils, tar, bzip).</li>
</ul>
<p>Each platform (Linux/Darwin) has a unique standard environment. It builds the base of Nixpkgs and is provided by the <tt class="docutils literal">stdenv</tt> package, which resolves to different implementations depending on the Nix <tt class="docutils literal">system</tt> variable.</p>
<p>Most Linux distributions including NixOS are handled by <tt class="docutils literal">stdenvLinux</tt>, which provides a so-called <em>pure</em> standard environment. It has no external dependencies (impurities) to programs or libraries provided by the host system outside of the Nix Store.</p>
<p>In contrast to that, <tt class="docutils literal">stdenvNative</tt> is another standard environment, which creates wrapper shell scripts in the store to use the native compiler toolchain provided by the host system. Its dependency on programs and libraries outside of the store make it impure. We will see implications of this later on.</p>
<p>Darwin uses a mixture of these two approaches were the native C library is used but apart from that the toolchain is built by Nix.</p>
</div>
</div>
<div class="section" id="compiling-on-and-for-windows">
<h2><a class="toc-backref" href="#id8">2 Compiling on and for Windows</a></h2>
<p>Our target platforms are:</p>
<ul class="simple">
<li>Windows Vista (32bit/64bit),</li>
<li>Windows 7 (32bit/64bit),</li>
<li>Windows 8 and 8.1 (32bit/64bit),</li>
<li>Windows Server 2008 and 2008 R2 (32bit/64bit),</li>
<li>Windows Server 2012 and 2012 R2 (64bit).</li>
</ul>
<p>We investigated two projects that provide free open source 32bit and 64bit toolchains for Windows: <a class="reference external" href="http://mingw-w64.yaxm.org/">Mingw-w64</a> and <a class="reference external" href="https://cygwin.com/">Cygwin</a>.</p>
<div class="section" id="id1">
<h3><a class="toc-backref" href="#id9">2.1 Mingw-w64</a></h3>
<p>"Mingw-w64 brings free software toolchains to Windows". It is a collection of headers and provides pre-packaged toolchains to compile native Windows binaries. While more and more projects are using it, it does not provide full <span class="caps">POSIX</span> <span class="caps">API</span> functionality and it is up to the projects and their build systems to support Mingw-w64.</p>
<p>One hard requirement for us was <span class="caps">POSIX</span> <span class="caps">API</span> functionality of the target system. We outline in <a class="reference internal" href="#native-windows-binaries">Native Windows Binaries</a> how Mingw-w64 could be facilitated.</p>
</div>
<div class="section" id="id2">
<h3><a class="toc-backref" href="#id10">2.2 Cygwin</a></h3>
<p>"Cygwin is a large collection of <span class="caps">GNU</span> and Open Source tools which provide functionality similar to a Linux distribution on Windows." In contrast to Mingw-w64, it has a <span class="caps">DLL</span> that "provides substantial <span class="caps">POSIX</span> <span class="caps">API</span> functionality" on Windows at its core. The emulated <span class="caps">POSIX</span> <span class="caps">API</span> functionality has some drawbacks, explained in <a class="reference internal" href="#challenges-on-cygwin">Challenges on Cygwin</a>,</p>
<p>Running 32bit Cygwin on a 64bit Windows works well with the exception of Windows Server 2012 and its R2, which seem to be <a class="reference external" href="https://cygwin.com/ml/cygwin/2014-10/msg00502.html">unsuitable for <span class="caps">WOW64</span></a>.</p>
<p>As a consequence Windows Server 2012 (R2) can be used to build and run 64bit Cygwin binaries, but not 32bit.</p>
</div>
</div>
<div class="section" id="challenges-on-cygwin">
<h2><a class="toc-backref" href="#id11">3 Challenges on Cygwin</a></h2>
<p>The combination of Windows, Cygwin, and Nix introduces challenges caused by Windows’ Portable Executable Format (<span class="caps">PE</span>). <em><span class="caps">PE</span></em> is to Windows, what <span class="caps">ELF</span> is to Linux. It is the file format used for <tt class="docutils literal">.dll</tt> and <tt class="docutils literal">.exe</tt> files on Windows, and it has some peculiar properties that cause major problems in Cygwin environments.</p>
<p>Further, Cygwin goes to great length to ease execution of <em><span class="caps">PE</span></em> from within Cygwin’s Unix-like environment.</p>
<div class="section" id="finding-dlls">
<h3><a class="toc-backref" href="#id12">3.1 Finding DLLs</a></h3>
<p>Windows has no simple, reliable way to load DLLs from absolute filesystem locations, which is inherently needed to make Nix’ concept of the store work.</p>
<p>Windows executables and shared libraries reference shared libraries they depend on only by name, not by full path as on some other platforms. This means that the <span class="caps">GCC</span> <tt class="docutils literal"><span class="pre">--rpath</span></tt> flag is useless on Windows. When <tt class="docutils literal"><span class="pre">C:\foo\bar.exe</span></tt> requests library <tt class="docutils literal">baz</tt> Windows follows certain <a class="reference external" href="http://msdn.microsoft.com/en-us/library/windows/desktop/ms682586.aspx">search rules</a> to find and load a file named <tt class="docutils literal">baz.dll</tt>. Usually it looks first in the directory of the executable itself (<tt class="docutils literal"><span class="pre">C:\foo\</span></tt>) and if unsuccessful it will at some point also try all directories contained in the <tt class="docutils literal"><span class="caps">PATH</span></tt> environment variable.</p>
<p>This property makes library versioning particularly difficult to achieve and leads to the infamous <a class="reference external" href="http://en.wikipedia.org/wiki/DLL_Hell"><span class="caps">DLL</span> Hell</a>.</p>
</div>
<div class="section" id="dll-relocation-and-cygwin-s-fork">
<h3><a class="toc-backref" href="#id13">3.2 <span class="caps">DLL</span> Relocation and Cygwin’s fork</a></h3>
<p>Windows has no equivalent to the <span class="caps">POSIX</span> <tt class="docutils literal">fork()</tt> system call and Cygwin goes to great length to implement it on top of Windows’ <tt class="docutils literal">CreateProcess</tt>. Cygwin’s <tt class="docutils literal">fork()</tt> implementation creates the necessity that all DLLs are loaded to unique pre-defined addresses by all processes.</p>
<p>When Windows runs a program that loads multiple DLLs to the same, intersecting or otherwise occupied address, the libraries have to be relocated to different random addresses. This is a common procedure and every <span class="caps">DLL</span> provides a <tt class="docutils literal">.reloc</tt> section that describes how to update address references in the <span class="caps">DLL</span> code. Apart from minor load time penalties these <span class="caps">DLL</span> relocations do not interfere in any way with normal Windows programs.</p>
<p>However, Cygwin programs are not simple Windows programs. Cygwin implements a <span class="caps">POSIX</span> layer in Windows, and some <span class="caps">POSIX</span> calls do not map easily to the Win32 <span class="caps">API</span>. Cygwin’s implementation of <tt class="docutils literal">fork()</tt> is not a lightweight copy-on-write approach as its equivalents on e.g. Linux. Instead Cygwin creates a new instance of the running program and copies all data and state from the parent to the new child process. For this to work it is necessary that the forked binaries and all their libraries have been loaded to the same non-colliding virtual addresses in the parent as well as in the child process. If Windows <span class="caps">DLL</span> relocation happens, the <tt class="docutils literal">fork()</tt> will most likely fail with an error like "address space already occupied".</p>
</div>
<div class="section" id="automagic-exe-suffixing">
<h3><a class="toc-backref" href="#id14">3.3 Automagic .exe Suffixing</a></h3>
<p>Binary executables in Windows have a <tt class="docutils literal">.exe</tt> filename extension. As this would break many programs and shell scripts, Cygwin implements automatic filename globbing when it comes to handling executables. If there is a file named <tt class="docutils literal">foo.exe</tt> in the current working directory, calling <tt class="docutils literal">./foo</tt> will actually run <tt class="docutils literal">foo.exe</tt>. Similarly, running <tt class="docutils literal">vim foo</tt> would open <tt class="docutils literal">foo.exe</tt> in vim for editing. This default behavior saves many programs from needing special patches or being broken on Windows.</p>
<p>While this makes it tricky to create a file <tt class="docutils literal">foo</tt>, while a file <tt class="docutils literal">foo.exe</tt> already exists, it also can be exploited: If there is a <tt class="docutils literal">foo.exe</tt> in the current directory and at the same time an executable shell script called <tt class="docutils literal">foo</tt>, then calling <tt class="docutils literal">./foo</tt> will run the shell script. The shell script effectively shadows the executable and can act as a transparent wrapper script, a feature that we exploit to improve Cygwin’s Nix compatibility.</p>
</div>
</div>
<div class="section" id="reproducible-cygwin-environments">
<h2><a class="toc-backref" href="#id15">4 Reproducible Cygwin Environments</a></h2>
<p>Cygwin has no fixed release cycles and its update procedure resembles a rolling release distribution. Nix main feature is reproducibility. Nix on Cygwin at least so far is impure in that it uses components outside the store provided by Cygwin. We need a way to make Cygwin environments reproducible.</p>
<p>Our requirements are:</p>
<ul class="simple">
<li>one Windows machine to host an arbitrary amount of Cygwin environments for 32bit and 64bit,</li>
<li>control which Cygwin package versions are installed so we can reliably reproduce Cygwin environments,</li>
<li>Nix packaged for Cygwin to be installable like any other Cygwin package,</li>
<li>installation of Cygwin packages via command line,</li>
<li>and ssh access to each of the environments.</li>
</ul>
<div class="section" id="custom-cygwin-mirror">
<h3><a class="toc-backref" href="#id16">4.1 Custom Cygwin Mirror</a></h3>
<p>By setting up a <a class="reference external" href="https://sourceware.org/cygwin-apps/package-server.html">Cygwin mirror</a>, we can freeze the state of all Cygwin packages and subsequently control which packages are updated and when. It also allows us to add our own packages to the repository. Cygwin mirrors use the perl script <a class="reference external" href="https://sourceware.org/viewvc/cygwin-apps/genini/">genini</a> to create index files containing metadata of all available packages. There is one for 32bit <tt class="docutils literal">x86/setup.ini</tt> and one for 64bit <tt class="docutils literal">x86_64/setup.ini</tt>.</p>
<p>We use three folders: <tt class="docutils literal">upstream</tt> tracks an upstream mirror, <tt class="docutils literal">custom</tt> contains our own packages and <tt class="docutils literal">cygwin</tt> merges the former two and forms our Cygwin mirror.</p>
<dl class="docutils">
<dt>upstream</dt>
<dd><p class="first">is synced via rsync with an <a class="reference external" href="https://cygwin.com/mirrors.html">upstream mirror</a> (around <span class="caps">60GB</span>).</p>
<div class="highlight"><pre><span class="k">for</span> x in x86 x86_64<span class="p">;</span> <span class="k">do</span>
rsync -vazP <span class="si">${</span><span class="nv">RSYNC_MIRROR</span><span class="si">}</span>/<span class="nv">$x</span>/ upstream/<span class="nv">$x</span>/
<span class="k">done</span>
</pre></div>
<p class="last">By omitting the <tt class="docutils literal"><span class="pre">--delete</span></tt> flag and putting the <tt class="docutils literal"><span class="pre">upstream/*/setup.*</span></tt> index files under version control we can follow updates while being able to go back in case something breaks.</p>
</dd>
<dt>custom</dt>
<dd><p class="first">will contain our custom packages and we use <tt class="docutils literal">genini</tt> to create its <tt class="docutils literal">setup.ini</tt> index files.</p>
<div class="highlight"><pre><span class="k">for</span> x in x86 x86_64<span class="p">;</span> <span class="k">do</span>
perl genini --arch<span class="o">=</span><span class="nv">$x</span> --recursive --output <span class="nv">$x</span>/setup.ini <span class="nv">$x</span>/setup.ini <span class="nv">$x</span>/release
<span class="k">done</span>
</pre></div>
<p class="last">It is important to call genini in the directory containing <tt class="docutils literal">x86</tt> and <tt class="docutils literal">x86_64</tt> with relative paths like above as otherwise paths in the generated files will be incorrect.</p>
</dd>
<dt>cygwin</dt>
<dd><p class="first">is the root of the custom Cygwin mirror. It is a symlink farm that merges <tt class="docutils literal">upstream</tt> and <tt class="docutils literal">custom</tt>. As genini is very slow, we create the index files by concatenating while omitting what would be an in-between header. Cygwin also looks for a bzip2 version named <tt class="docutils literal">setup.bz2</tt>.</p>
<div class="last"><div class="highlight"><pre><span class="k">for</span> x in x86 x86_64<span class="p">;</span> <span class="k">do</span>
cat upstream/<span class="nv">$x</span>/setup.ini > cygwin/<span class="nv">$x</span>/setup.ini
tail -n+7 custom/<span class="nv">$x</span>/setup.ini >> cygwin/<span class="nv">$x</span>/setup.ini
bzip2 -c cygwin/<span class="nv">$x</span>/setup.ini > cygwin/<span class="nv">$x</span>/setup.bz2
<span class="k">done</span>
</pre></div>
</div></dd>
</dl>
</div>
<div class="section" id="installation-and-cygwin-package-management">
<h3><a class="toc-backref" href="#id17">4.2 Installation and Cygwin Package Management</a></h3>
<p>To bootstrap a Cygwin environment we use <a class="reference external" href="https://github.com/ternaris/bootstrap-cygwin/blob/master/bootstrap-cygwin.bat">a batch</a> file calling Cygwin’s <tt class="docutils literal">setup.exe</tt>. Next to it are a skel directory to be used as Cygwin’s <tt class="docutils literal">/etc/skel</tt> and the 32bit and 64bit installers.</p>
<pre class="literal-block">
bootstrap-cygwin.bat
setup-x86_64.exe
setup-x86.exe
skel/
</pre>
<p>The bootstrap script creates the <tt class="docutils literal">mintty.bat</tt> batch file, which starts a shell in the new Cygwin environment by executing its <tt class="docutils literal">mintty.exe</tt> as login shell (parameter <tt class="docutils literal">-</tt>) and setting the Cygwin directory as window title.</p>
<div class="highlight"><pre>start <span class="s2">"%~dp0"</span> <span class="s2">"%~dp0bin\mintty.exe"</span> -
</pre></div>
<p>Especially with ssh access (see <a class="reference internal" href="#remote-access">Remote Access</a>), it helps a lot if shell prompts indicate the Cygwin directory they belong to, as set in <a class="reference external" href="https://github.com/ternaris/bootstrap-cygwin/blob/master/skel/.bashrc">.bashrc</a>.</p>
<div class="highlight"><pre><span class="nb">export </span><span class="nv">PS1</span><span class="o">=</span><span class="s1">'\[\e]0;\w\a\]\n(cygroot='</span><span class="k">$(</span>cygpath -w -a /<span class="k">)</span><span class="s1">') \[\e[32m\]\u@\h \[\e[33m\]\w\[\e[0m\]\n\$ '</span>
</pre></div>
<p>For Cygwin package management we found it handy to put the Cygwin installer for the architecture (<tt class="docutils literal"><span class="pre">setup-x86.exe</span></tt> or <tt class="docutils literal"><span class="pre">setup-x86_64.exe</span></tt>) as <tt class="docutils literal">setup.exe</tt> into the Cygwin root folder and to create a wrapper <tt class="docutils literal">setup.bat</tt> with some default parameters. This is handled by <tt class="docutils literal"><span class="pre">bootstrap-cygwin.bat</span></tt> (see above).</p>
<pre class="literal-block">
$ ls -1 /setup.*
setup.bat
setup.exe
$ cat /setup.bat
"%~dp0setup.exe" -q -X -O -s http://custom.cygwin/mirror -R "%~dp0" -l "c:\cygwin-cache" %*
</pre>
<p>This enables easy execution from Windows Explorer. To have it available from a shell within the Cygwin environment we created two aliases: one for querying available packages and the other to simply run <tt class="docutils literal">/setup.bat</tt>.</p>
<div class="highlight"><pre><span class="k">if</span> <span class="nb">test</span> <span class="k">$(</span>uname -m<span class="k">)</span> <span class="o">=</span> <span class="s2">"x86_64"</span><span class="p">;</span> <span class="k">then</span>
<span class="nv">cygarch</span><span class="o">=</span>x86_64
<span class="k">else</span>
<span class="nv">cygarch</span><span class="o">=</span>x86
<span class="k">fi</span>
<span class="nb">export</span> <span class="nv">$cygarch</span>
<span class="nb">alias </span><span class="nv">cygsetup</span><span class="o">=</span><span class="s1">'/setup.bat'</span>
<span class="nb">alias </span><span class="nv">cygquery</span><span class="o">=</span><span class="s1">'cat /cygdrive/c/cygwin-cache/<MIRDIR>/${cygarch}/setup.ini |grep -i --color'</span>
</pre></div>
</div>
<div class="section" id="remote-access">
<h3><a class="toc-backref" href="#id18">4.3 Remote Access</a></h3>
<p>To register <span class="caps">SSH</span> as a Windows service, Cygwin has the <tt class="docutils literal"><span class="pre">ssh-host-config</span></tt> configure script. We enhanced it to allow for multiple Cygwins to provide simultaneous <span class="caps">SSH</span> access, by accepting a port number and service name. Our patches for this and for <tt class="docutils literal">cygserver</tt> (see below) are submitted and integrated upstream.</p>
<pre class="literal-block">
$ ssh-host-config --port 32001 --name cygwin-x86-32001-sshd
$ cygrunsrv --start cygwin-x86-32001-sshd
</pre>
<p><a class="reference external" href="https://cygwin.com/cygwin-ug-net/using-cygserver.html">Cygserver</a> provides Cygwin applications with services which require security arbitration or which need to persist while no other Cygwin application is running. Similar to the <span class="caps">SSH</span> services there is a script (<tt class="docutils literal"><span class="pre">cygserver-config</span></tt>) to configure the Windows service, which with our patches supports now an explicit name to allow for multiple cygserver services per Windows machine.</p>
<pre class="literal-block">
$ cygserver-config --name cygwin-x86-32001-cygserver
$ cygrunsrv --start cygwin-x86-32001-cygserver
</pre>
</div>
<div class="section" id="do-not-copy-cygwins">
<h3><a class="toc-backref" href="#id19">4.4 <span class="caps">DO</span> <span class="caps">NOT</span> <span class="caps">COPY</span> <span class="caps">CYGWINS</span></a></h3>
<p>We encountered that Windows Explorer does not preserve permissions correctly when copying Cygwins, that effectively led to the umask being ignored.</p>
</div>
</div>
<div class="section" id="nix-on-cygwin">
<h2><a class="toc-backref" href="#id20">5 Nix on Cygwin</a></h2>
<p>In this section we describe fixes to the Nix package manager itself to compile on Cygwin, hooks to create an impure standard environment for Cygwin and give an overview of fixes we did to Nixpkgs.</p>
<div class="section" id="preparations">
<h3><a class="toc-backref" href="#id21">5.1 Preparations</a></h3>
<p>There are a couple of things to ensure or be aware of before actually starting with Nix on Cygwin:</p>
<dl class="docutils">
<dt>git and unix line endings</dt>
<dd><p class="first">Before using git in any way, e.g. and especially to check out <a class="reference external" href="https://github.com/NixOS/nixpkgs">nixpkgs</a> git needs to be told not to be smart about line endings. Our bootstrap handles this via a skeleton file.</p>
<pre class="literal-block">
$ git config --global core.eol lf
</pre>
<p class="last">If for some reason you do not want to set this globally for your Cygwin user, at the very least you have to set it for checkouts of Nixpkgs.</p>
</dd>
</dl>
<p><span class="caps">XXX</span>: Is this still needed? If so, there should be a bug/feature request for Nix/Nixpkgs</p>
<dl class="docutils">
<dt>allowBroken</dt>
<dd><p class="first">At the moment of writing, nix considers its support for Cygwin to be broken and has to be told that you are aware of that. Our bootstrap handles this via skel file.</p>
<pre class="last literal-block">
$ cd; mkdir -p .nixpkgs; echo "{ allowBroken = true; }" >> .nixpkgs/config.nix
</pre>
</dd>
</dl>
<p><span class="caps">XXX</span>: Is this still needed?</p>
</div>
<div class="section" id="package-build-and-install-nix">
<h3><a class="toc-backref" href="#id22">5.2 Package, Build and Install Nix</a></h3>
<p>To install Nix on Cygwin we decided to use Cygwin’s cygport source packaging tool. Cygport is inspired by Gentoo’s Portage package managing system and borrows some of its ideas and syntax for defining packages. The only item we have to provide is a nix.cygport file, available in <a class="reference external" href="https://github.com/ternaris/cygports">ternaris/cygports</a>:</p>
<div class="highlight"><pre><span class="nv">NAME</span><span class="o">=</span><span class="s2">"nix"</span>
<span class="nv">VERSION</span><span class="o">=</span>1.8
<span class="nv">RELEASE</span><span class="o">=</span>1
<span class="nv">CATEGORY</span><span class="o">=</span><span class="s2">"Devel"</span>
<span class="nv">SUMMARY</span><span class="o">=</span><span class="s2">"The purely functional package manager"</span>
<span class="nv">DESCRIPTION</span><span class="o">=</span><span class="s2">"Nix is a powerful package manager for Linux and other Unix systems that makes package management reliable and reproducible. It provides atomic upgrades and rollbacks, side-by-side installation of multiple versions of a package, multi-user package management and easy setup of build environments."</span>
<span class="nv">HOMEPAGE</span><span class="o">=</span><span class="s2">"http://nixos.org/nix/"</span>
<span class="nv">SRC_URI</span><span class="o">=</span><span class="s2">"http://nixos.org/releases/nix/nix-1.8/nix-1.8.tar.xz"</span>
<span class="nv">DEPEND</span><span class="o">=</span><span class="s2">"bison curl flex gcc-core gcc-g++ libbz2-devel libsqlite3-devel make openssl-devel patch perl perl-DBD-SQLite perl-WWW-Curl pkg-config"</span>
<span class="nv">REQUIRES</span><span class="o">=</span><span class="s2">"</span><span class="nv">$DEPEND</span><span class="s2">"</span>
src_compile<span class="o">()</span> <span class="o">{</span>
<span class="nb">cd</span> <span class="si">${</span><span class="nv">S</span><span class="si">}</span>
./configure --prefix<span class="o">=</span>/usr --sysconfdir<span class="o">=</span>/etc --disable-init-state
./config.status --quiet --file Makefile.config
sed -i Makefile.config <span class="se">\</span>
-e <span class="s2">"s,^libdir = /usr/lib,libdir = /usr/bin,"</span> <span class="se">\</span>
-e <span class="s2">"s,^perllibdir = .*</span><span class="nv">$,</span><span class="s2">perllibdir = </span><span class="k">$(</span>perl -E <span class="s1">'use Config; print $Config{vendorarch};'</span><span class="k">)</span><span class="s2">,"</span>
make
<span class="o">}</span>
src_install<span class="o">()</span> <span class="o">{</span>
<span class="nb">cd</span> <span class="si">${</span><span class="nv">S</span><span class="si">}</span>
<span class="c"># XXX: maybe we can configure already to the $pkgdir locations, but it feels that up there should be the after-install-locations. currently, this leads to a couple of things being relinked on make install.</span>
sed -i Makefile.config <span class="se">\</span>
-e <span class="s2">"s,dir = /usr,dir = </span><span class="si">${</span><span class="nv">pkgdir</span><span class="si">}</span><span class="s2">/usr,"</span> <span class="se">\</span>
-e <span class="s2">"s,prefix = /usr,prefix = </span><span class="si">${</span><span class="nv">pkgdir</span><span class="si">}</span><span class="s2">/usr,"</span>
cyginstall <span class="nv">profiledir</span><span class="o">=</span><span class="si">${</span><span class="nv">pkgdir</span><span class="si">}</span>/etc/profile.d
<span class="o">}</span>
</pre></div>
<p>Some patches to Nix’ sources were needed as outlined below and <a class="reference external" href="https://github.com/NixOS/nix/pull/408">merged upstream</a>:</p>
<ol class="arabic simple">
<li><span class="caps">GCC</span> complained that several standard <span class="caps">BSD</span> or <span class="caps">POSIX</span> functions, e.g. <tt class="docutils literal">srandom</tt>, are not defined.</li>
<li>Libraries (DLLs) were built with a <tt class="docutils literal">.so</tt> file extension instead of the Windows standard <tt class="docutils literal">.dll</tt> extension and they were installed to the <tt class="docutils literal"><span class="pre">${prefix}/lib</span></tt> directory which is fine on Linux and Darwin, however, on Windows, <span class="caps">DLL</span> files have to be installed to the <tt class="docutils literal"><span class="pre">${prefix}/bin</span></tt> directory.</li>
<li>The Nix <tt class="docutils literal">Store</tt> Perl extension did not compile due to missing Perl symbols.</li>
</ol>
<div class="section" id="fix-missing-library-function-definitions">
<h4><a class="toc-backref" href="#id23">5.2.1 Fix Missing Library Function Definitions</a></h4>
<p><span class="caps">GCC</span> complained about missing definitions of standard functions in some source files. This happened mainly for functions usually found in <tt class="docutils literal">stdlib.h</tt>, which is not always included explicitly by the Nix sources, as it is pulled in by other headers on Linux/Darwin. Adding the explicit <tt class="docutils literal">#include</tt> statements where appropriate partially solves the problem.</p>
<p>However, some functions are still not found, even with all headers in place. As it turns out culprit is the <tt class="docutils literal"><span class="pre">-std=c++0x</span></tt> gcc flag, which sets the language standard and implicitly enables <tt class="docutils literal">__STRICT_ANSI__</tt>, which in turn prevents the Cygwin headers from defining some vital functions. Unsetting it by adding <tt class="docutils literal"><span class="pre">-U__STRICT_ANSI__</span></tt> to the <tt class="docutils literal"><span class="caps">CFLAGS</span></tt> variable concludes this fix.</p>
</div>
<div class="section" id="fix-library-file-extension-and-directory">
<h4><a class="toc-backref" href="#id24">5.2.2 Fix Library File Extension and Directory</a></h4>
<p>This requires a straightforward patch to the make files to conditionally set file extension and installation directory of shared libraries.</p>
</div>
<div class="section" id="fix-perl-extension">
<h4><a class="toc-backref" href="#id25">5.2.3 Fix Perl Extension</a></h4>
<p>Nix’ Perl extensions use internal Perl functions found in the corresponding <tt class="docutils literal">perl.dll</tt>. On Windows we cannot create libraries with unresolved symbols, which makes it necessary to link the extension module against <tt class="docutils literal">perl.dll</tt>. To address this issue we discover the path of <tt class="docutils literal">perl.dll</tt> on Windows and add it to <span class="caps">LDFLAGS</span> of the store target in the corresponding Makefile.</p>
</div>
</div>
<div class="section" id="standard-environment-hooks">
<h3><a class="toc-backref" href="#id26">5.3 Standard Environment Hooks</a></h3>
<p>After experimenting with various approaches we decided to use <tt class="docutils literal">stdenvNative</tt>. We outline the steps towards a pure environment in <a class="reference internal" href="#pure-standard-environment">Pure Standard Environment</a>.</p>
<p>At some point in the past Nixpkgs had support for the Cygwin platform. Over time this port became dysfunctional and it was subsequently removed, leaving only a few code remnants in the Nixpkgs tree. One of the first of these remnants we encountered was a hook for <tt class="docutils literal">stdenvNative</tt> that set some platform specific options and most notably disabled shared libraries by default. For our new Cygwin support we decided to reevaluate all previous changes in the spirit of staying as close as possible to Nix on NixOS for easier maintenance in the future.</p>
<p>In some aspects, however, Cygwin is fundamentally different from other Nix platforms. As described in <a class="reference internal" href="#challenges-on-cygwin">Challenges on Cygwin</a>, Windows <em><span class="caps">PE</span></em> format causes two major challenges:</p>
<ol class="arabic simple">
<li>DLLs cannot be reliably referenced by absolute paths</li>
<li>DLLs need to have unique pre-defined addresses</li>
</ol>
<p>We address these challenges by <a class="reference external" href="https://github.com/ternaris/nixpkgs/tree/cygwin/pkgs/stdenv/cygwin">hooks</a> to Nixpkgs’ <tt class="docutils literal">stdenvNative</tt>.</p>
<div class="section" id="dll-search-path-wrappers">
<h4><a class="toc-backref" href="#id27">5.3.1 <span class="caps">DLL</span> Search Path Wrappers</a></h4>
<p>As described in <a class="reference internal" href="#finding-dlls">Finding DLLs</a>, Windows looks for DLLs in the <tt class="docutils literal"><span class="caps">PATH</span></tt> environment variable and does not support referencing them by absolute path, as needed by nix.</p>
<p>To emulate library references by absolute path we have to make sure that on program execution, all DLLs direct or indirect dependencies are present in the <tt class="docutils literal"><span class="caps">PATH</span></tt> variable. To this end we added a <a class="reference external" href="https://github.com/ternaris/nixpkgs/blob/cygwin/pkgs/stdenv/cygwin/wrap-exes-to-find-dlls.sh">postInstall hook</a> to <tt class="docutils literal">stdenvNative</tt> that creates wrapper shell scripts for all store output’s executable, exploiting Cygwin’s <cite>Automagic .exe Suffixing</cite>.</p>
<div class="highlight"><pre>postFixupHooks+<span class="o">=(</span>_cygwinWrapExesToFindDlls<span class="o">)</span>
_cygwinWrapExesToFindDlls<span class="o">()</span> <span class="o">{</span>
find <span class="nv">$out</span> -type l <span class="p">|</span> <span class="k">while</span> <span class="nb">read </span>LINK<span class="p">;</span> <span class="k">do</span>
<span class="nv">TARGET</span><span class="o">=</span><span class="s2">"</span><span class="k">$(</span>readlink <span class="s2">"</span><span class="si">${</span><span class="nv">LINK</span><span class="si">}</span><span class="s2">"</span><span class="k">)</span><span class="s2">"</span>
<span class="c"># fix all non .exe links that link explicitly to a .exe</span>
<span class="k">if</span> <span class="o">[[</span> <span class="si">${</span><span class="nv">TARGET</span><span class="si">}</span> <span class="o">==</span> *.exe <span class="o">]]</span> <span class="o">&&</span> <span class="o">[[</span> <span class="si">${</span><span class="nv">LINK</span><span class="si">}</span> !<span class="o">=</span> *.exe <span class="o">]]</span><span class="p">;</span> <span class="k">then</span>
mv <span class="s2">"</span><span class="si">${</span><span class="nv">LINK</span><span class="si">}</span><span class="s2">"</span> <span class="s2">"</span><span class="si">${</span><span class="nv">LINK</span><span class="si">}</span><span class="s2">.exe"</span>
<span class="nv">LINK</span><span class="o">=</span><span class="s2">"</span><span class="si">${</span><span class="nv">LINK</span><span class="si">}</span><span class="s2">.exe"</span>
<span class="k">fi</span>
<span class="c"># generate complementary filenames</span>
<span class="k">if</span> <span class="o">[[</span> <span class="si">${</span><span class="nv">LINK</span><span class="si">}</span> <span class="o">==</span> *.exe <span class="o">]]</span><span class="p">;</span> <span class="k">then</span>
<span class="nv">_LINK</span><span class="o">=</span><span class="s2">"</span><span class="si">${</span><span class="nv">LINK</span><span class="p">%.exe</span><span class="si">}</span><span class="s2">"</span>
<span class="nv">_TARGET</span><span class="o">=</span><span class="s2">"</span><span class="si">${</span><span class="nv">TARGET</span><span class="p">%.exe</span><span class="si">}</span><span class="s2">"</span>
<span class="k">else</span>
<span class="nv">_LINK</span><span class="o">=</span><span class="s2">"</span><span class="si">${</span><span class="nv">LINK</span><span class="si">}</span><span class="s2">.exe"</span>
<span class="nv">_TARGET</span><span class="o">=</span><span class="s2">"</span><span class="si">${</span><span class="nv">TARGET</span><span class="si">}</span><span class="s2">.exe"</span>
<span class="k">fi</span>
<span class="c"># check if sould create complementary link</span>
<span class="nv">DOLINK</span><span class="o">=</span>1
<span class="k">if</span> <span class="o">[[</span> <span class="si">${</span><span class="nv">_TARGET</span><span class="si">}</span> <span class="o">==</span> *.exe <span class="o">]]</span><span class="p">;</span> <span class="k">then</span>
<span class="c"># the canonical target has to be a .exe</span>
<span class="nv">CTARGET</span><span class="o">=</span><span class="s2">"</span><span class="k">$(</span>readlink -f <span class="s2">"</span><span class="si">${</span><span class="nv">LINK</span><span class="si">}</span><span class="s2">"</span><span class="k">)</span><span class="s2">"</span>
<span class="k">if</span> <span class="o">[[</span> <span class="si">${</span><span class="nv">CTARGET</span><span class="si">}</span> !<span class="o">=</span> *.exe <span class="o">]]</span><span class="p">;</span> <span class="k">then</span>
<span class="nv">CTARGET</span><span class="o">=</span><span class="s2">"</span><span class="si">${</span><span class="nv">CTARGET</span><span class="si">}</span><span class="s2">.exe"</span>
<span class="k">fi</span>
<span class="k">if</span> <span class="o">[</span> ! -e <span class="s2">"</span><span class="si">${</span><span class="nv">CTARGET</span><span class="si">}</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
<span class="nb">unset </span>DOLINK
<span class="k">fi</span>
<span class="k">fi</span>
<span class="k">if</span> <span class="o">[</span> -e <span class="s2">"</span><span class="si">${</span><span class="nv">_LINK</span><span class="si">}</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
<span class="c"># complementary link seems to exist</span>
<span class="c"># but could be cygwin smoke and mirrors</span>
<span class="nv">INO</span><span class="o">=</span><span class="k">$(</span>stat -c%i <span class="s2">"</span><span class="si">${</span><span class="nv">LINK</span><span class="si">}</span><span class="s2">"</span><span class="k">)</span>
<span class="nv">_INO</span><span class="o">=</span><span class="k">$(</span>stat -c%i <span class="s2">"</span><span class="si">${</span><span class="nv">_LINK</span><span class="si">}</span><span class="s2">"</span><span class="k">)</span>
<span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="si">${</span><span class="nv">INO</span><span class="si">}</span><span class="s2">"</span> -ne <span class="s2">"</span><span class="si">${</span><span class="nv">_INO</span><span class="si">}</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
<span class="nb">unset </span>DOLINK
<span class="k">fi</span>
<span class="k">fi</span>
<span class="c"># create complementary link</span>
<span class="k">if</span> <span class="o">[</span> -n <span class="s2">"</span><span class="si">${</span><span class="nv">DOLINK</span><span class="si">}</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
ln -s <span class="s2">"</span><span class="si">${</span><span class="nv">_TARGET</span><span class="si">}</span><span class="s2">"</span> <span class="s2">"</span><span class="si">${</span><span class="nv">_LINK</span><span class="si">}</span><span class="s2">.tmp"</span>
mv <span class="s2">"</span><span class="si">${</span><span class="nv">_LINK</span><span class="si">}</span><span class="s2">.tmp"</span> <span class="s2">"</span><span class="si">${</span><span class="nv">_LINK</span><span class="si">}</span><span class="s2">"</span>
<span class="k">fi</span>
<span class="k">done</span>
find <span class="nv">$out</span> -type f -name <span class="s2">"*.exe"</span> <span class="p">|</span> <span class="k">while</span> <span class="nb">read </span>EXE<span class="p">;</span> <span class="k">do</span>
<span class="nv">WRAPPER</span><span class="o">=</span><span class="s2">"</span><span class="si">${</span><span class="nv">EXE</span><span class="p">%.exe</span><span class="si">}</span><span class="s2">"</span>
<span class="k">if</span> <span class="o">[</span> -e <span class="s2">"</span><span class="si">${</span><span class="nv">WRAPPER</span><span class="si">}</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
<span class="c"># check if really exists or cygwin smoke and mirrors</span>
<span class="nv">INO</span><span class="o">=</span><span class="k">$(</span>stat -c%i <span class="s2">"</span><span class="si">${</span><span class="nv">EXE</span><span class="si">}</span><span class="s2">"</span><span class="k">)</span>
<span class="nv">_INO</span><span class="o">=</span><span class="k">$(</span>stat -c%i <span class="s2">"</span><span class="si">${</span><span class="nv">WRAPPER</span><span class="si">}</span><span class="s2">"</span><span class="k">)</span>
<span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="si">${</span><span class="nv">INO</span><span class="si">}</span><span class="s2">"</span> -ne <span class="s2">"</span><span class="si">${</span><span class="nv">_INO</span><span class="si">}</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
<span class="k">continue</span>
<span class="k">fi</span>
<span class="k">fi</span>
mv <span class="s2">"</span><span class="si">${</span><span class="nv">EXE</span><span class="si">}</span><span class="s2">"</span> <span class="s2">"</span><span class="si">${</span><span class="nv">EXE</span><span class="si">}</span><span class="s2">.tmp"</span>
cat ><span class="s2">"</span><span class="si">${</span><span class="nv">WRAPPER</span><span class="si">}</span><span class="s2">"</span> <span class="s"><<EOF</span>
<span class="s">#!/bin/sh</span>
<span class="s">export PATH=$_PATH${_PATH:+:}\${PATH}</span>
<span class="s">exec "\$0.exe" "\$@"</span>
<span class="s">EOF</span>
chmod +x <span class="s2">"</span><span class="si">${</span><span class="nv">WRAPPER</span><span class="si">}</span><span class="s2">"</span>
mv <span class="s2">"</span><span class="si">${</span><span class="nv">EXE</span><span class="si">}</span><span class="s2">.tmp"</span> <span class="s2">"</span><span class="si">${</span><span class="nv">EXE</span><span class="si">}</span><span class="s2">"</span>
<span class="k">done</span>
<span class="o">}</span>
</pre></div>
<p>Creating these scripts can be tricky, as writing to the file <tt class="docutils literal">foo</tt> in presence of <tt class="docutils literal">foo.exe</tt> in the same directory will overwrite the contents of the latter executable. Symlinks to executables need special attention, too. If a program provides <tt class="docutils literal">foo.exe</tt> and a symlink <tt class="docutils literal"><span class="pre">bar.exe->foo.exe</span></tt>, the hook has to create a wrapper <tt class="docutils literal">foo</tt> and also a complementary symlink <tt class="docutils literal"><span class="pre">bar->foo</span></tt>, otherwise programs whose behavior depends on the executable name (e.g. busybox) will break. Some build system will create constructs like <tt class="docutils literal"><span class="pre">bar->foo.exe</span></tt>, which our hook sanitizes.</p>
</div>
<div class="section" id="runtime-dependency-non-detection">
<h4><a class="toc-backref" href="#id28">5.3.2 Runtime Dependency Non-Detection</a></h4>
<p>In order to detect store location A to be a runtime dependency of B, the Nix store relies on A’s path being mentioned in B. On Linux and Darwin, binaries reference their dependencies by absolute paths and therefore enable runtime detection. On Windows this is not possible as outlined in <a class="reference internal" href="#finding-dlls">Finding DLLs</a>. As a quick and <a class="reference external" href="https://github.com/ternaris/nixpkgs/blob/cygwin/pkgs/stdenv/cygwin/all-buildinputs-as-runtimedep.sh">dirty solution</a> we turn all build-time dependencies of package to be runtime dependencies as well.</p>
<div class="highlight"><pre><span class="c"># On cygwin, automatic runtime dependency detection does not work</span>
<span class="c"># because the binaries do not contain absolute references to store</span>
<span class="c"># locations (yet)</span>
postFixupHooks+<span class="o">=(</span>_cygwinAllBuildInputsAsRuntimeDep<span class="o">)</span>
_cygwinAllBuildInputsAsRuntimeDep<span class="o">()</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">[</span> -n <span class="s2">"</span><span class="nv">$buildInputs</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
mkdir -p <span class="s2">"</span><span class="nv">$out</span><span class="s2">/nix-support"</span>
<span class="nb">echo</span> <span class="s2">"</span><span class="nv">$buildInputs</span><span class="s2">"</span> >> <span class="s2">"</span><span class="nv">$out</span><span class="s2">/nix-support/cygwin-buildinputs-as-runtime-deps"</span>
<span class="k">fi</span>
<span class="k">if</span> <span class="o">[</span> -n <span class="s2">"</span><span class="nv">$nativeBuildInputs</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
mkdir -p <span class="s2">"</span><span class="nv">$out</span><span class="s2">/nix-support"</span>
<span class="nb">echo</span> <span class="s2">"</span><span class="nv">$nativeBuildInputs</span><span class="s2">"</span> >> <span class="s2">"</span><span class="nv">$out</span><span class="s2">/nix-support/cygwin-buildinputs-as-runtime-deps"</span>
<span class="k">fi</span>
<span class="o">}</span>
</pre></div>
<p>This works well, but results in superfluous runtime dependencies. See <a class="reference internal" href="#proper-runtime-dependency-detection">Proper Runtime Dependency Detection</a> for a better proposal.</p>
</div>
<div class="section" id="dll-rebasing-to-enable-cygwin-s-fork">
<h4><a class="toc-backref" href="#id29">5.3.3 <span class="caps">DLL</span> Rebasing to Enable Cygwin’s fork()</a></h4>
<p>As described in <a class="reference internal" href="#dll-relocation-and-cygwin-s-fork"><span class="caps">DLL</span> Relocation and Cygwin’s fork</a>, <span class="caps">DLL</span> relocation is lethal to Cygwin’s <tt class="docutils literal">fork()</tt>. Key to a working <tt class="docutils literal">fork()</tt> call is to avoid address collisions when loading DLLs by minimizing the probability of relocation.</p>
<p>Windows DLLs can define an <tt class="docutils literal">ImageBase</tt> in the <tt class="docutils literal">IMAGE_OPTIONAL_HEADER</tt> section of their <span class="caps">PE</span> header. Its value defines the <span class="caps">DLL</span>’s preferred base address when it is loaded into the virtual address space of a running process.</p>
<p>Cygwin makes sure that all DLLs it installs are <em>rebased</em> to unique addresses using <tt class="docutils literal">ImageBase</tt> and that the address space beginning at <tt class="docutils literal">ImageBase</tt> is large enough to hold the complete library. Rebasing happens automatically whenever Cygwin’s <tt class="docutils literal">setup.exe</tt> installs or updates a package.</p>
<p>Binutils’ <tt class="docutils literal">ld</tt> linker accepts an option called <tt class="docutils literal"><span class="pre">--enable-auto-image-base</span></tt> for <span class="caps">DLL</span> creation. When enabled, <tt class="docutils literal">ld</tt> automatically generates a pseudo-unique value for <tt class="docutils literal">ImageBase</tt> based on a hash of the library name. The rationale behind this approach is that differently named DLLs will end up at different base addresses. It does not account for which parts of the address space are claimed by other libraries and collisions are still possible albeit somewhat less likely to occur. Cygwin’s global solution does a better job ensuring non-overlapping address spaces.</p>
<p>However, Cygwin is not aware of libraries compiled and installed via Nix. For choosing and setting ImageBase in newly compiled DLLs we wrote two more postInstall hooks specific to <a class="reference external" href="https://github.com/ternaris/nixpkgs/blob/cygwin/pkgs/stdenv/cygwin/rebase-i686.sh">32bit</a> and <a class="reference external" href="https://github.com/ternaris/nixpkgs/blob/cygwin/pkgs/stdenv/cygwin/rebase-x86_64.sh">64bit</a></p>
<div class="highlight"><pre>postFixupHooks+<span class="o">=(</span>_cygwinFixAutoImageBase<span class="o">)</span>
_cygwinFixAutoImageBase<span class="o">()</span> <span class="o">{</span>
find <span class="nv">$out</span> -name <span class="s2">"*.dll"</span> <span class="p">|</span> <span class="k">while</span> <span class="nb">read </span>DLL<span class="p">;</span> <span class="k">do</span>
<span class="k">if</span> <span class="o">[</span> -f /etc/rebasenix.nextbase <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
<span class="nv">NEXTBASE</span><span class="o">=</span><span class="s2">"</span><span class="k">$(</span></etc/rebasenix.nextbase<span class="k">)</span><span class="s2">"</span>
<span class="k">fi</span>
<span class="nv">NEXTBASE</span><span class="o">=</span><span class="si">${</span><span class="nv">NEXTBASE</span><span class="k">:-</span><span class="nv">0x200000000</span><span class="si">}</span>
<span class="nv">REBASE</span><span class="o">=(</span><span class="sb">`</span>/bin/rebase -i <span class="nv">$DLL</span><span class="sb">`</span><span class="o">)</span>
<span class="nv">BASE</span><span class="o">=</span><span class="si">${</span><span class="nv">REBASE</span><span class="p">[2]</span><span class="si">}</span>
<span class="nv">SIZE</span><span class="o">=</span><span class="si">${</span><span class="nv">REBASE</span><span class="p">[4]</span><span class="si">}</span>
<span class="nv">SKIP</span><span class="o">=</span><span class="k">$((</span><span class="o">((</span><span class="nv">$SIZE</span>>>16<span class="o">)+</span><span class="m">1</span><span class="o">)</span><<<span class="m">16</span><span class="k">))</span>
<span class="nb">echo</span> <span class="s2">"REBASE FIX: </span><span class="nv">$DLL</span><span class="s2"> </span><span class="nv">$BASE</span><span class="s2"> -> </span><span class="nv">$NEXTBASE</span><span class="s2">"</span>
/bin/rebase -b <span class="nv">$NEXTBASE</span> <span class="nv">$DLL</span>
<span class="nv">NEXTBASE</span><span class="o">=</span><span class="s2">"0x`printf %x </span><span class="k">$((</span><span class="nv">$NEXTBASE</span><span class="o">+</span><span class="nv">$SKIP</span><span class="k">))</span><span class="s2">`"</span>
<span class="nb">echo</span> <span class="nv">$NEXTBASE</span> > /etc/rebasenix.nextbase
<span class="k">done</span>
<span class="o">}</span>
</pre></div>
<p>It keeps a global base address counter, whose initial value is set to be slightly above the address space taken by Cygwin’s most important library <tt class="docutils literal">cygwin1.dll</tt>, which is specific to 32bit/64bit. For each new library the postInstall hook rebases the library to the address saved in the counter and increments the counter by the size of the library, rounded up to the next multiple of 64kB. This ensures unique non-overlapping areas for all DLLs in the store, without the necessity for additional or global rebasing scripts. This approach has some drawbacks, as address space cannot be reclaimed by the hook when software is uninstalled from the store. At least on 32bit Windows the hook will run out of valid addresses sooner rather than later. Also, on Windows, one program cannot load multiple shared libraries with the same name at the same time. Therefore, it would be possible to rebase different versions of the same library to the same base address and therefore to better utilize the available address space.</p>
<p>See <a class="reference internal" href="#store-level-dll-rebasing">Store Level <span class="caps">DLL</span> Rebasing</a> for a better proposal.</p>
</div>
</div>
<div class="section" id="nixpkgs-fixes-a-beginning">
<h3><a class="toc-backref" href="#id30">5.4 Nixpkgs fixes — A Beginning</a></h3>
<p>Nixpkgs used to have support for Cygwin, which was dropped at some point. Before starting our work we flagged occurences of old Cygwin specifics to be investigated. For 4 packages we could remove the old Cygwin specifics and there remain 49 such todo markers in packages that we did not investigate so far.</p>
<p>All our patches to Nixpkgs are being discussed and are meanwhile merged as <a class="reference external" href="https://github.com/NixOS/nixpkgs/pull/6101">pull request</a>. Here we’d like to give a brief overview.</p>
<ul class="simple">
<li>In three cases (gnugrep, gnum4, gnumake) we decided to disable failing tests, but only, because they were already disabled for at least one other platform.</li>
<li>In case a newer version of a package had Cygwin support, we decided to upgrade for all platforms (boehmgc, gettext)</li>
<li>In case of bash we had to downgrade from 4.2 to 4.1, the latest version with support for Cygwin. We did so only for Cygwin, the other platforms remain on 4.2. It looks like Cygwin support will be back with bash 4.3.</li>
<li>We encountered 4 packages (cpio, findutils, gnutar, gzip) that ship an older version of gnulib, which gets confused by the Cygwin headers. Small package-specific <tt class="docutils literal">preConfigure</tt> hooks solve the issues.</li>
<li>For gawk, gettext, ncurses, and zip it was sufficient to specify <cite>configure</cite> flags specific to Cygwin.</li>
<li>For 12 packages we use Cygwin specific patches taken from Cygwin itself: bash, boehmgc, coreutils, gettext, libffi, libiconv, mysql, openssl, pkgconfig, popt, python27, w3m.</li>
<li>Nixpkgs provides a <tt class="docutils literal">substituteInPlace</tt> command which is implemented using bash’s pattern substitution. At least with bash-4.1 we experienced this to be seriously slow and propose to use <tt class="docutils literal">sed</tt> instead, along with an alias <tt class="docutils literal"><span class="pre">sed-i</span></tt> that maps to <tt class="docutils literal">sed <span class="pre">-i""</span></tt> for gnused and <tt class="docutils literal">sed <span class="pre">-i</span> ""</tt> for bsdsed. Alternatively, we could require gnused to be installed.</li>
</ul>
<p>And there is the category of weird fixes, most notably <tt class="docutils literal">asciidoc</tt>:</p>
<div class="highlight"><pre><span class="gi">+ makeFlags = if stdenv.isCygwin then "DESTDIR=/." else null;</span>
</pre></div>
<p>Without this the destination directory would be prefixed with <tt class="docutils literal">//</tt>, which thanks to Cygwin becomes <tt class="docutils literal">\\</tt> and is subsequently interpreted as a network path.</p>
</div>
</div>
<div class="section" id="summary-and-outlook">
<h2><a class="toc-backref" href="#id31">6 Summary and Outlook</a></h2>
<p>There is a cygport to install Nix on Cygwin and a impure standard environment allows installation of some packages contained in the Nix Packages Collection. In addition to investigate and fix the remaining packages, we propose:</p>
<ul class="simple">
<li><span class="caps">DLL</span> rebasing in the store to enable binary caches for Cygwin,</li>
<li>detection of real runtime dependencies</li>
<li>a pure standard environment to eliminate interference with the Cygwin host system,</li>
<li>reduction of environment variable length to remain within Cygwins limits,</li>
<li>investigation of batch files as an alternative or supplement to shebang wrappers to enable execution from outside of Cygwin, and</li>
<li>Mingw-w64 as an alternative or supplement to Cygwin to create native Windows binaries.</li>
</ul>
<div class="section" id="store-level-dll-rebasing">
<h3><a class="toc-backref" href="#id32">6.1 Store Level <span class="caps">DLL</span> Rebasing</a></h3>
<p>Currently, we use a simple, non-intrusive postInstall hook that rebases all DLLs as part of the build process (see <a class="reference internal" href="#dll-rebasing-to-enable-cygwin-s-fork"><span class="caps">DLL</span> Rebasing to Enable Cygwin’s fork()</a>. Therefore, rebasing is only run when building a package locally, but will not affect packages that are installed through binary caches.</p>
<p>This renders distribution of precompiled packages effectively useless at the moment. Additionally, address space cannot be reused when libraries are removed from the store.</p>
<p>A better solution would track addition and removal of packages to and from the store and perform rebase (and defragmentation) as part of a package’s build. For that to work, the nix store would need support for platform specific hooks, which would be executed after adding and before removing software from the store. This solution would move the current code to a more appropriate place and make all steps reversible.</p>
<p>We believe that such a solution would greatly improve the long term health of a Nix on Cygwin installation, as running out of address space would become much more unlikely and distributed binaries would work out of the box on every machine, as the Nix Store would take care of rebasing regardless of a package’s origin.</p>
</div>
<div class="section" id="proper-runtime-dependency-detection">
<h3><a class="toc-backref" href="#id33">6.2 Proper Runtime Dependency Detection</a></h3>
<p>Currently, we simply turn all build dependencies into runtime dependencies (see <a class="reference internal" href="#runtime-dependency-non-detection">Runtime Dependency Non-Detection</a>), which results in superfluous runtime dependencies.</p>
<p>The <a class="reference internal" href="#dll-search-path-wrappers"><span class="caps">DLL</span> Search Path Wrappers</a> already mention dependencies by absolute path. As a side effect these dependencies would already be detected. However, a proper solution should not rely on side effects of another hook.</p>
<p>A proper runtime dependency detection should scan all binaries (.dll and .exe) of a package and map the referenced library names to absolute store locations, based on the currently set <tt class="docutils literal"><span class="caps">PATH</span></tt> variable.</p>
</div>
<div class="section" id="pure-standard-environment">
<h3><a class="toc-backref" href="#id34">6.3 Pure Standard Environment</a></h3>
<p>Some of the package issues we encountered, were caused by some build systems discovering libraries belonging to the host Cygwin environment.</p>
<p>We would like to investigate a pure standard environment, where everything — including the Cygwin <span class="caps">DLL</span> — is served from the Nix Store.</p>
<p>This environment could enforce purity much like <tt class="docutils literal">stdenvLinux</tt> does on NixOS. With an almost empty <tt class="docutils literal">/</tt> there would be nothing to find for badly behaved packages. Such a "self-hosted" solution would not require the explicit installation of Cygwin anymore, but could be distributed as a small tarball consisting of not much more than a <tt class="docutils literal">/nix</tt> directory.</p>
</div>
<div class="section" id="environment-variable-length">
<h3><a class="toc-backref" href="#id35">6.4 Environment Variable Length</a></h3>
<p>Windows limits the maximum size of user-defined environment variables to <a class="reference external" href="https://msdn.microsoft.com/en-us/library/windows/desktop/ms682653">32767 characters</a>.</p>
<p>Nix’ way of handling Python packages, for example, stores each dependency individually in the Nix store and assembles the Python environmnet via the <tt class="docutils literal"><span class="caps">PYTHONPATH</span></tt> variable set in shebang wrapper scripts. Assuming an average 100 characters per Python package store path, this starts to be an issue in case of more than 300 dependencies.</p>
<p>Different forms of packaging that do not assemble Python environments via <tt class="docutils literal"><span class="caps">PYTHONPATH</span></tt> need to be investigated and likewise for other languages.</p>
<p>Up until Windows Server 2003 and Windows <span class="caps">XP</span> there was an additional limit on the whole block of all environment variables, which does not exist anymore in newer versions.</p>
</div>
<div class="section" id="batch-files-instead-of-shebang-wrappers">
<h3><a class="toc-backref" href="#id36">6.5 Batch Files Instead of Shebang Wrappers</a></h3>
<p>Nix in general and we especially for Nix on Cygwin make heavy use of shebang wrappers to set environment variables for an application, e.g.:</p>
<div class="highlight"><pre><span class="c">#!/bin/sh</span>
<span class="nb">export </span><span class="nv">PATH</span><span class="o">=</span>/nix/store/3hshx30bkahwyxs2y7dkiilzf72ggva7-cc-native/bin:<span class="si">${</span><span class="nv">PATH</span><span class="si">}</span>
<span class="nb">exec</span> <span class="s2">"</span><span class="nv">$0</span><span class="s2">.exe"</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
</pre></div>
<p>Cygwin supports shebang well, but it is not easily possible to execute these wrappers from outside Cygwin. To support this, generation of Windows batch files could be an alternative. However, if Windows’s command line prompt is involved in their execution, this would create an even smaller <a class="reference external" href="http://support.microsoft.com/en-us/kb/830473">limit of 8191 characters</a> for the <a class="reference internal" href="#environment-variable-length">Environment Variable Length</a>).</p>
</div>
<div class="section" id="native-windows-binaries">
<h3><a class="toc-backref" href="#id37">6.6 Native Windows Binaries</a></h3>
<p>As outlined in <a class="reference internal" href="#compiling-on-and-for-windows">Compiling on and for Windows</a> we evaluated two projects providing free open source toolchains: Cygwin and Mingw-w64.</p>
<p>While our initial focus was to provide a full <span class="caps">POSIX</span>-environment based on Cygwin it would be very interesting to attempt the integration of Mingw-w64 to create native Windows binaries, which can be run outside of Cygwin.</p>
<table border="1" class="docutils">
<colgroup>
<col width="38%" />
<col width="20%" />
<col width="42%" />
</colgroup>
<thead valign="bottom">
<tr><th class="head">Build Environment</th>
<th class="head">Toolchain</th>
<th class="head">Runtime Environment</th>
</tr>
</thead>
<tbody valign="top">
<tr><td>Cygwin</td>
<td>Cygwin</td>
<td>Cygwin</td>
</tr>
<tr><td>Cygwin</td>
<td>Mingw-w64</td>
<td>native</td>
</tr>
<tr><td>Mingw-w64</td>
<td>Mingw-w64</td>
<td>native</td>
</tr>
</tbody>
</table>
<p>Our current solution is represented by the first row: Cygwin is used as build environment and toolchain to create binaries that need Cygwin also as runtime environment.</p>
<p>As the next step, we could use Nix in a Cygwin build environment with Mingw-w64 toolchain to build native Windows programs, which could then be run natively. A Cygwin build environment could support this native mode as well as the Cygwin runtime.</p>
<p>We then could maybe even go further and build Nix itself with Mingw-w64. This would require to create alternate paths for quite a lot of <span class="caps">POSIX</span> specific Nix code. However, this would get rid of some of the complexities of the current build environment and may lead to the creation of the ultimate Windows build tool.</p>
</div>
</div>