From 88de1da361c4439a8e5b32f153ffbd782cca6563 Mon Sep 17 00:00:00 2001 From: Tyson Whitehead Date: Mon, 5 Feb 2018 17:43:55 -0500 Subject: [PATCH] Rewrite of README for new overlay framework --- README.md | 329 +++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 238 insertions(+), 91 deletions(-) diff --git a/README.md b/README.md index 61a2408..b925f13 100644 --- a/README.md +++ b/README.md @@ -1,119 +1,196 @@ # Overview -This package contains a collection of Nix expressions created by -SHARCNET staff for various software packages. You can use them to -install the associated packages into your Nix environment. - -These directions assume you have loaded the Nix module (this enables -your Nix environment and makes the various Nix commands available) +This repository overlays the [upstream nixpkgs-channels +repository](https://github.com/NixOS/nixpkgs-channels) with fixes and +additional packages to create a repository suitable for use on Compute +Canada HPC clusters. It is available to use by default on Compute +Canada clusters. It's layout is modelled after the [upstream +nixpkgs](https://github.com/NixOS/nixpkgs). -```sh -module load nix +# Using + +This repository overlayed onto the latest [upstream +nixpkgs-channel](https://github.com/NixOS/nixpkgs-channels) is the +default channel on Compute Canada clusters. To update to the latest +version (i.e., the last one for which all overlays built successfully) +do + +```bash +nix-channel --update ``` -# Obtaining the latest expressions +This is safe to do as the prior version can always be returned to by +doing a rollback -If you don't yet have a copy of *~/nix-nixpkgs-sharcnet*, you will -need to clone a copy (use your SHARCNET credentials) - -```sh -cd ~ -git clone https://git.sharcnet.ca/nix/nixpkgs-sharcnet.git +```bash +nix-channel --rollback ``` -If you already have a copy, you can update to the latest version +The `nix-env` command can be used to install, update, or remove +packages from the environment. -```sh -cd ~/nix-nixpkgs-sharcnet -git pull +## Installing + +To install (or update) a package, specify the packages attribute to +`nix-env` using the `--attr` option + +```bash +nix-env --install --attr attribute ``` -# Using an expression - -Once you have a copy of the expressions you can use `nix-env` to -install, update, or remove them from your environment. - - -## Installing an expression - -To install (or update) a package pass the corresponding file (replace -*package.nix* in what follows) to `nix-env` using the `--file` option - -```sh -nix-env --install --file package.nix -``` - -You can also copy or symlink the file into your *~/.nix-defexpr* -directory and then refer to it as a top-level attribute (replace -*package* with the base name of the copied file) - -```sh -nix-env --install --attr package -``` +where `attribute` is a placeholder for the attribute that designates +the desired package (e.g., `emacs`). ## Removing -To remove a package you first need to figure out its name (generally, -but not necessarily, the same as the name of the Nix expression file) +Removing a package requires knowing its name. Get this by querying +the current environment -```sh +```bash nix-env --query ``` -Then you (replace *package* with the package name) +Then it can be uninstalled with the `--uninstall` options -```sh -nix-env --uninstall package +```bash +nix-env --uninstall name ``` +where `name` is a placeholder for the name returned by `--query` +(e.g., emacs). + ## Reverting -To undo the last install/update/remove command +These commands are always safe to do as they can always be undone by +doing a rollback -```sh +```bash nix-env --rollback ``` -# Creating an expression +## Using -The best way to create a new package is to pillage existing package -expressions. This includes the ones in [this -repository](https://git.sharcnet.ca/nix/nixpkgs-sharcnet) (click the -repository tab to see the files) and all the various [upstream -ones](https://git.sharcnet.ca/nix/nixpkgs-sharcnet/tree/master) (click -on a package and then following the Nix expression link). The -[packaging manual](https://nixos.org/nixpkgs/manual/) details all the -various options you will see in these files. +A development environment consisting of a given set of packages can be +entered using the `nix-shell` command with the `--packages` option -## Building - -A package can be build by running (replace *file.nix* with the name of -your file) - - -```sh -nix-build file.nix +```bash +nix-shell --packages attribute ... ``` -If the build succeeds, the output will be symlinked into -the current directory as *result*. +Dependencies of non-installed packages are not tracked, so it is not +guaranteed that the specified packages will remain on the system for +more than the current day (nor does the fact that they sometimes +remain for more than a day imply they will always remain). Should +they be removed, rerunning the command will temporarily reinstall them +again. -## Debugging a build -A simulated build environment can be entered by running (replace -*file.nix* with the name of your file) +# Developing -```sh -nix-shell --pure file.nix +To add to this this repository, `git clone` it into a local directory. + +```bash +git clone https://git.computecanada.ca/nix/ccpkgs.git +cd ccpkgs ``` -which gives you a shell in the build environment described by the -epxression. The default build and install locations need to be -overriden to somewhere you have write permission (this uses */tmp*) +As it is an overlay, it also requires a matching release of the +upstream `nixpkgs-channels` repository. This should be cloned into +the *upsteam* subdirectory (if this directory does not it exist it +will use the current channel if it is the correct version) + +```bash +git clone https://github.com/NixOS/nixpkgs-channels.git -b nixos-$(< .version) upstream +``` + +## Layout + +Longer term overlays, like package additions, are put in the *pkgs* +subdirectory. Shorter term overlays, like temporary upstream +workarounds, are placed in the *temporary* subdirectory. The +*pkgs/all-packages.nix* and *temporary/all-temporary.nix* files are +applied as [overlays](https://nixos.org/nixpkgs/manual/#chap-overlays) +on top of the the upstream nixpkgs-channel. This means attributes +defined in them (first *all-temporary.nix* and then +*all-packages.nix*) will replace attributes defined in +nixpkgs-channel. + +Inside both of these files, the default scope is the final attribute +set (i.e., after all overlays have been applied). The final, prior +(before the file's overlays are applied), and next (after the file's +overlays are applied) attribute sets are also available under the +top-level attributes `pkgs`, `super`, and `self`. Consider the +`emacs` attribute for example. `emacs` and `pkgs.emacs` refer to the +final `emacs` definition. The prior (before this overlay) and next +(after this overlay) definition are also available though +as`super.emacs` and `self.emacs`, respectively. + +## Adding + +Packages are added by creating an appropriate *pkgs/.../package.nix* +or, if more multiple files are required, +*pkgs/.../package/default.nix* expression, where `package` is a +placeholder for a suitably descriptive (e.g., `emacs`), add then +adding a `package = callPackage ...` line into +*pkgs/all-packages.nix*. This makes available under the attribute +`package`. + +The easiest way to create the nix expression, is to look through the +existing overlay expressions, find one that is suitably close to what +is desired, and then duplicate and modify it. [Upstream +nixpkgs](https://github.com/NixOS/nixpkgs) expressions can also be +used as a starting point as they also follow the exact same format. + +More information on creating packages can be found in the [Nixpkgs +Contributor Guide](https://nixos.org/nixpkgs/manual/). More +information about the nix expression language itself can be found in +the [Nix Package Manager +Guide](https://nixos.org/nix/manual/#ch-expression-language). + +## Testing + +Nix expressions can be build from the top-level directory by using the +`nix-build` command. For example + +```bash +nix-build --attr attribute +``` + +where `attribute` is replaced with the appropriate attribute (e.g., +`emacs`). It is highly recommended to do this before pushing any +changes to the master repository as any breakage will prevent further +releases of the channel to the clusters until fixed. + +## Debugging + +The `nix-shell` command can be used to enter a non-sandboxed version +of the build environment from the top-level directory. Passing the +`--pure` option causes it to purge the environment. RHEL and Fedora +do not guard against multiple evaluations of the */etc/profile.d* +scripts though, so the environment will re-initialize itself from +*~/.bashrc* script unless it is modified to not source */etc/bashrc* +when `__ETC_PROFILE_SOURCED` is set + +```bash +sed -i -e 's|^\(\s*\)\(\. /etc/bashrc\)\s*$|\1test -n "$__ETC_PROFILE_SOURCED" \|\| \2|' ~/.bashrc +``` + +With this one-time change, the purged simulated build environment for +can be entered by running -```sh +```bash +nix-shell --pure --attr attribute +``` + +`NIX_ENFORCE_PURITY` should be set in the entered environment to +enable additional checks against system libraries making it into the +build. The build and install locations should also be reset to +somewhere writable (e.g., */tmp*) + +```bash +export NIX_ENFORCE_PURITY=1 NIX_BUILD_TOP=/tmp/nix-$USER mkdir "$NIX_BUILD_TOP" @@ -123,20 +200,90 @@ out=$PWD/out prefix=$out ``` -Then you can invoke the standard nix build phases and use the fact -that you are in a shell to diagnose whatever goes wrong (setting -`NIX_ENFORCE_PURITY` enables various checks along the way to ensure no -host bits get into the results as these would not be available in the -real sandboxed build) +After this, the standard [build +phases](https://nixos.org/nixpkgs/manual/#sec-stdenv-phases) can be +manually invoked and debugged - -``` -export NIX_ENFORCE_PURITY=1 - -unpackPhase +```bash +eval "${unpackPhase:-unpackPhase}" cd "$sourceRoot" -configurePhase -buildPhase -installPhase -``` \ No newline at end of file +eval "${patchPhase:-patchPhase}" +eval "${configurePhase:-configurePhase}" +eval "${buildPhase:-buildPhase}" +eval "${installPhase:-installPhase}" +``` + +where the above evaluates the `*Phase` environment variable if it +exists (the phase is overridden) or otherwise calls the standard +`*Phase` function (the phase is not overridden). In some extreme +cases it can be useful to look at the [actual +implementation](https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/setup.sh) +of these functions and the `genericBuild` function which ties +them all together. + + +# Deployment + +Both this repository and nixpkgs-channel are regularly checked for +updates by the [hydra server](https://hydra.sharcnet.ca). If a newer +version of either of these are found, the hydra server automatically +clones them and attempts to build all the overlays defined in +*pkgs/all-packages.nix* and temporary/all-temporary.nix* and a channel +tarball too. The Compute Canada clusters monitor this process, and, +if it succeeds, download the channel tarball and make it available as +a new release. + +This means deployment is as simple as doing + +```bash +git push +``` + +waiting for a while (see the [hydra +server](https://hydra.sharcnet.ca/) to check on progress), and then +doing + +```bash +nix-channel --update +``` + +## Normal Changes + +As mentioned in the [developing section](#testing), new or changed +attribute should first be test built before pushing changes upstream + +```bash +nix-build --attr attribute +``` + +where `attribute` is replaced with the appropriate attribute (e.g., +`emacs`). + +## Sweeping Changes + +The top-level *release.nix* expression contains the builds performed +by the hydra server. If significant changes have been made, these can +be manually tested before pushing + +```bash +nix-build release.nix --attr tested +``` + +A build of the channel tarball can also be done + +```bash +nix-build release.nix --attr channel +``` + +The resulting channel tarball can even be imported as new channel +and used the same as the default channel + +```bash +nix-channel --add "file://$(echo "$PWD"/result/tarballs/*)" channel +nix-channel --update channel +nix-env --install --attr channel.attribute +``` + +where `channel` is a place holder for an arbitrary new channel name +(e.g., `test`) and `attribute` for some attribute (e.g., `emacs`).