Syzygial/ccpkgs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Rewrite of README for new overlay framework

Browse Source
This commit is contained in:
Tyson Whitehead
2018-02-05 17:43:55 -05:00
parent 1cb8792d05
commit 88de1da361
1 changed files with 238 additions and 91 deletions
Download Patch File Download Diff File Expand all files Collapse all files

329
README.md
View File

@@ -1,119 +1,196 @@
# Overview # Overview
This package contains a collection of Nix expressions created by This repository overlays the [upstream nixpkgs-channels
SHARCNET staff for various software packages. You can use them to repository](https://github.com/NixOS/nixpkgs-channels) with fixes and
install the associated packages into your Nix environment. additional packages to create a repository suitable for use on Compute
Canada HPC clusters. It is available to use by default on Compute
These directions assume you have loaded the Nix module (this enables Canada clusters. It's layout is modelled after the [upstream
your Nix environment and makes the various Nix commands available) nixpkgs](https://github.com/NixOS/nixpkgs).
```sh # Using
module load nix
This repository overlayed onto the latest [upstream
nixpkgs-channel](https://github.com/NixOS/nixpkgs-channels) is the
default channel on Compute Canada clusters. To update to the latest
version (i.e., the last one for which all overlays built successfully)
do
```bash
nix-channel --update
``` ```
# Obtaining the latest expressions This is safe to do as the prior version can always be returned to by
doing a rollback
If you don't yet have a copy of *~/nix-nixpkgs-sharcnet*, you will ```bash
need to clone a copy (use your SHARCNET credentials) nix-channel --rollback
```sh
cd ~
git clone https://git.sharcnet.ca/nix/nixpkgs-sharcnet.git
``` ```
If you already have a copy, you can update to the latest version The `nix-env` command can be used to install, update, or remove
packages from the environment.
```sh ## Installing
cd ~/nix-nixpkgs-sharcnet
git pull To install (or update) a package, specify the packages attribute to
`nix-env` using the `--attr` option
```bash
nix-env --install --attr attribute
``` ```
# Using an expression where `attribute` is a placeholder for the attribute that designates
the desired package (e.g., `emacs`).
Once you have a copy of the expressions you can use `nix-env` to
install, update, or remove them from your environment.
## Installing an expression
To install (or update) a package pass the corresponding file (replace
*package.nix* in what follows) to `nix-env` using the `--file` option
```sh
nix-env --install --file package.nix
```
You can also copy or symlink the file into your *~/.nix-defexpr*
directory and then refer to it as a top-level attribute (replace
*package* with the base name of the copied file)
```sh
nix-env --install --attr package
```
## Removing ## Removing
To remove a package you first need to figure out its name (generally, Removing a package requires knowing its name. Get this by querying
but not necessarily, the same as the name of the Nix expression file) the current environment
```sh ```bash
nix-env --query nix-env --query
``` ```
Then you (replace *package* with the package name) Then it can be uninstalled with the `--uninstall` options
```sh ```bash
nix-env --uninstall package nix-env --uninstall name
``` ```
where `name` is a placeholder for the name returned by `--query`
(e.g., emacs).
## Reverting ## Reverting
To undo the last install/update/remove command These commands are always safe to do as they can always be undone by
doing a rollback
```sh ```bash
nix-env --rollback nix-env --rollback
``` ```
# Creating an expression ## Using
The best way to create a new package is to pillage existing package A development environment consisting of a given set of packages can be
expressions. This includes the ones in [this entered using the `nix-shell` command with the `--packages` option
repository](https://git.sharcnet.ca/nix/nixpkgs-sharcnet) (click the
repository tab to see the files) and all the various [upstream
ones](https://git.sharcnet.ca/nix/nixpkgs-sharcnet/tree/master) (click
on a package and then following the Nix expression link). The
[packaging manual](https://nixos.org/nixpkgs/manual/) details all the
various options you will see in these files.
## Building ```bash
nix-shell --packages attribute ...
A package can be build by running (replace *file.nix* with the name of
your file)
```sh
nix-build file.nix
``` ```
If the build succeeds, the output will be symlinked into Dependencies of non-installed packages are not tracked, so it is not
the current directory as *result*. guaranteed that the specified packages will remain on the system for
more than the current day (nor does the fact that they sometimes
remain for more than a day imply they will always remain). Should
they be removed, rerunning the command will temporarily reinstall them
again.
## Debugging a build
A simulated build environment can be entered by running (replace # Developing
*file.nix* with the name of your file)
```sh To add to this this repository, `git clone` it into a local directory.
nix-shell --pure file.nix
```bash
git clone https://git.computecanada.ca/nix/ccpkgs.git
cd ccpkgs
``` ```
which gives you a shell in the build environment described by the As it is an overlay, it also requires a matching release of the
epxression. The default build and install locations need to be upstream `nixpkgs-channels` repository. This should be cloned into
overriden to somewhere you have write permission (this uses */tmp*) the *upsteam* subdirectory (if this directory does not it exist it
will use the current channel if it is the correct version)
```bash
git clone https://github.com/NixOS/nixpkgs-channels.git -b nixos-$(< .version) upstream
```
## Layout
Longer term overlays, like package additions, are put in the *pkgs*
subdirectory. Shorter term overlays, like temporary upstream
workarounds, are placed in the *temporary* subdirectory. The
*pkgs/all-packages.nix* and *temporary/all-temporary.nix* files are
applied as [overlays](https://nixos.org/nixpkgs/manual/#chap-overlays)
on top of the the upstream nixpkgs-channel. This means attributes
defined in them (first *all-temporary.nix* and then
*all-packages.nix*) will replace attributes defined in
nixpkgs-channel.
Inside both of these files, the default scope is the final attribute
set (i.e., after all overlays have been applied). The final, prior
(before the file's overlays are applied), and next (after the file's
overlays are applied) attribute sets are also available under the
top-level attributes `pkgs`, `super`, and `self`. Consider the
`emacs` attribute for example. `emacs` and `pkgs.emacs` refer to the
final `emacs` definition. The prior (before this overlay) and next
(after this overlay) definition are also available though
as`super.emacs` and `self.emacs`, respectively.
## Adding
Packages are added by creating an appropriate *pkgs/.../package.nix*
or, if more multiple files are required,
*pkgs/.../package/default.nix* expression, where `package` is a
placeholder for a suitably descriptive (e.g., `emacs`), add then
adding a `package = callPackage ...` line into
*pkgs/all-packages.nix*. This makes available under the attribute
`package`.
The easiest way to create the nix expression, is to look through the
existing overlay expressions, find one that is suitably close to what
is desired, and then duplicate and modify it. [Upstream
nixpkgs](https://github.com/NixOS/nixpkgs) expressions can also be
used as a starting point as they also follow the exact same format.
More information on creating packages can be found in the [Nixpkgs
Contributor Guide](https://nixos.org/nixpkgs/manual/). More
information about the nix expression language itself can be found in
the [Nix Package Manager
Guide](https://nixos.org/nix/manual/#ch-expression-language).
## Testing
Nix expressions can be build from the top-level directory by using the
`nix-build` command. For example
```bash
nix-build --attr attribute
```
where `attribute` is replaced with the appropriate attribute (e.g.,
`emacs`). It is highly recommended to do this before pushing any
changes to the master repository as any breakage will prevent further
releases of the channel to the clusters until fixed.
## Debugging
The `nix-shell` command can be used to enter a non-sandboxed version
of the build environment from the top-level directory. Passing the
`--pure` option causes it to purge the environment. RHEL and Fedora
do not guard against multiple evaluations of the */etc/profile.d*
scripts though, so the environment will re-initialize itself from
*~/.bashrc* script unless it is modified to not source */etc/bashrc*
when `__ETC_PROFILE_SOURCED` is set
```bash
sed -i -e 's|^\(\s*\)\(\. /etc/bashrc\)\s*$|\1test -n "$__ETC_PROFILE_SOURCED" \|\| \2|' ~/.bashrc
```
With this one-time change, the purged simulated build environment for
can be entered by running
```sh ```bash
nix-shell --pure --attr attribute
```
`NIX_ENFORCE_PURITY` should be set in the entered environment to
enable additional checks against system libraries making it into the
build. The build and install locations should also be reset to
somewhere writable (e.g., */tmp*)
```bash
export NIX_ENFORCE_PURITY=1
NIX_BUILD_TOP=/tmp/nix-$USER NIX_BUILD_TOP=/tmp/nix-$USER
mkdir "$NIX_BUILD_TOP" mkdir "$NIX_BUILD_TOP"
@@ -123,20 +200,90 @@ out=$PWD/out
prefix=$out prefix=$out
``` ```
Then you can invoke the standard nix build phases and use the fact After this, the standard [build
that you are in a shell to diagnose whatever goes wrong (setting phases](https://nixos.org/nixpkgs/manual/#sec-stdenv-phases) can be
`NIX_ENFORCE_PURITY` enables various checks along the way to ensure no manually invoked and debugged
host bits get into the results as these would not be available in the
real sandboxed build)
```bash
``` eval "${unpackPhase:-unpackPhase}"
export NIX_ENFORCE_PURITY=1
unpackPhase
cd "$sourceRoot" cd "$sourceRoot"
configurePhase eval "${patchPhase:-patchPhase}"
buildPhase eval "${configurePhase:-configurePhase}"
installPhase eval "${buildPhase:-buildPhase}"
``` eval "${installPhase:-installPhase}"
```
where the above evaluates the `*Phase` environment variable if it
exists (the phase is overridden) or otherwise calls the standard
`*Phase` function (the phase is not overridden). In some extreme
cases it can be useful to look at the [actual
implementation](https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/setup.sh)
of these functions and the `genericBuild` function which ties
them all together.
# Deployment
Both this repository and nixpkgs-channel are regularly checked for
updates by the [hydra server](https://hydra.sharcnet.ca). If a newer
version of either of these are found, the hydra server automatically
clones them and attempts to build all the overlays defined in
*pkgs/all-packages.nix* and temporary/all-temporary.nix* and a channel
tarball too. The Compute Canada clusters monitor this process, and,
if it succeeds, download the channel tarball and make it available as
a new release.
This means deployment is as simple as doing
```bash
git push
```
waiting for a while (see the [hydra
server](https://hydra.sharcnet.ca/) to check on progress), and then
doing
```bash
nix-channel --update
```
## Normal Changes
As mentioned in the [developing section](#testing), new or changed
attribute should first be test built before pushing changes upstream
```bash
nix-build --attr attribute
```
where `attribute` is replaced with the appropriate attribute (e.g.,
`emacs`).
## Sweeping Changes
The top-level *release.nix* expression contains the builds performed
by the hydra server. If significant changes have been made, these can
be manually tested before pushing
```bash
nix-build release.nix --attr tested
```
A build of the channel tarball can also be done
```bash
nix-build release.nix --attr channel
```
The resulting channel tarball can even be imported as new channel
and used the same as the default channel
```bash
nix-channel --add "file://$(echo "$PWD"/result/tarballs/*)" channel
nix-channel --update channel
nix-env --install --attr channel.attribute
```
where `channel` is a place holder for an arbitrary new channel name
(e.g., `test`) and `attribute` for some attribute (e.g., `emacs`).