Sadiq's Knowledge Vaults

Nix

15 min read

It’s build by Elco dolstra as a part of his PhD thesis, as a purely functional software deployment model, it’s about how to deploy the software reliably
The One solution for this problem is distribute the same binary across the machines, but there are problems with this approach

  1. Binary won’t work when the compiled system and the system in which the binary is running are different
  2. Also binaries often rely on the dynamic libraries that are present in the system, often times either this dynamic library is not present in the system or there is a version mismatch

One solution for the above problems is to make the code open source and let the code compile directly from source to binary on the users machine, but still there can be version mismatch and if it’s an big company with lot of servers, we need to compile the same code on the same kind of machines, every time
Thus we need to capture the buildability and the runnability of the software to make it identical to run on all the machines reliably and as intended
For tacking this problem Elco built DCL(domain specific language) and named it “Nix”
As last until we have the same processor type and same operating system, Nix will output the same binary for the given code, not letting us worry about the steps and the inherit dependencies and there inconsistent versions

Nix Language Basics

["hi", world, 42.0] # JSON: Array
["hi" world 42.0] # Nix: List
 
# JSON: Object
{
	name: "sadiq",
	age: 21
}
# Nix: Attribute
{
	name = "sadiq";
	age = 21;
}
 
# JS: Function
function add(a, b) { # Function definition
	return a + b;
}
add(2, 3); # Function calling
# Nix: Function
add = {a, b}: a + b # Function definition
add {a=2; b=3} # Function calling
 
 
# Nix BuiltIn Functions
# Attribute names
builtins.arrtNames {a=1; b=2} # output: ["a", "b"]
 
# functions args -> returns list of all arguments that a functions accepts
let 
	f={a, b ? 40} : a + b
in
	builtins.functionArgs f
# output: {a = false; b = true;}
 
# Nix Path
./some-file.txt # the path which is not a string in Nix returns absolute path, means /home/sadiq/.../some-file.txt
 
builtins.toString ./some-file.txt # the path which is a string in Nix returns nix store path which is with it's unique hash identifier, means /nix/store/ejnbihrtobinrthibrtg-some-file.txt, here the same files with same names and same content have different paths

Derivations

It executes the command and capture the output
example of running a C file and capturing the output

derivation {
	name = "some_random_file";
	system = "aarch64-darwin";
	builder = "/bin/bash";
	src = ./main.c;
	args = [ "-c" ''
		"/bin/bash/clang $src" -o $out
		''];
}

here the output directory is created in the Nix store and put captured output there. To run the above we run nix-instantiate <nix-file>.nix, thus command just serialize the derivation in a deterministic way and gives us the nix store file path of where the code output serialized
Here when we run the serialized derivation executables then it’s called realize the derivation that is done by nix-store --realize <nix-store-path-of-serialized-derivation>, this command is by default ran in an sandbox which is an isolated empty and temporary folder with no env variables to protect system configuration
Rather running both of this commands, we can run nix-build <nix-file>.nix, this will serialize and realize the derivation in nix file and also creates a result symlink in current directory which points to nix store containing the output

But the above derivation has an error, The derivation uses the system’s local C compiler (clang from our machine). This makes the build non-reproducible, meaning different machines may produce different outputs. To Solve this problem we use following steps:

  1. Nix never trusts system tools
    • If our derivation uses /bin/clang, /usr/bin/gcc, etc., the build depends on the host machine
    • Two different machines → different compilers → different outputs → breaks reproducibility
  2. Nix builds the compiler itself
    • Nix can build Clang/GCC from source as a derivation
    • That compiler is added to the build environment through buildInputs
    • Now every system uses the exact same compiler, bit-for-bit identical
  3. Nix builds and tracks dependent libraries too
    • C program may need: libc, libm, dynamic loader, any linked runtime libs
    • Nix builds those too, and puts their paths inside the derivation environment
    • This ensures our program links only against libraries built by Nix → fully reproducible
  4. Different projects can use different versions of tools
    • Project A can use GCC 12
    • Project B can use GCC 9
    • Both versions live separately in the Nix store
    • They never override each other (no global “compiler version hell”)
  5. If anything changes, Nix rebuilds only what is necessary
    • Change in: compiler version, source code, a dependency → Nix detects it using hashes and builds a new output path
  6. Every build goes into a unique directory in the Nix store
    • Example Nix store paths: /nix/store/abc123-clang-12.0.1 /nix/store/def456-myprogram-1.0
    • Each directory is immutable (read-only)
    • If something changes → the hash changes → path changes(new file in Nix Store)
    • This is where Merkle trees come in

Q How Nix uses Merkle Trees ? #A Nix makes builds fully reproducible by giving every derivation output a cryptographic hash that is computed from all of its inputs, including the source code, build tools, compiler versions, environment variables, libraries, and the build script itself. Because each dependency also has its own hash, Nix calculates hashes recursively, this forms a Merkle-tree-like structure where the hash of a node depends on the hashes of all its children. If our program depends on Clang, libc, and some helper libraries, their hashes are included when hashing our program, so any change to a dependency automatically produces a new hash. This means that if anything changes anywhere in the dependency graph, compiler versions, library versions, build flags, or the source code, Nix generates a completely new store path, ensuring bit-for-bit reproducibility. This Merkle-tree structure guarantees that the exact versions of tools and libraries used are always known, prevents accidental interference between projects, and allows deduplication because identical hashes reuse the same store paths

Nix code that builds toolchain to build the actual code

let
  # 1. Download sources manually
  binutilsSrc = derivation {
    name = "binutils-src";
    system = "x86_64-linux";
    builder = "/bin/sh";
    args = [ "-c" ''
      mkdir -p $out
      curl -L https://ftp.gnu.org/gnu/binutils/binutils-2.40.tar.xz -o $out/binutils.tar.xz
    '' ];
  };
  
  /*
  # The above code can also be written as
  binutilsSrc = builtins.fetchTarball {
    url = "https://ftp.gnu.org/gnu/binutils/binutils-2.40.tar.xz";
    sha256 = "sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=";
  };
  
  # Here the Sha can and cannot be given, if we don't find the sha code, keep it empty string and run it and downloaded compiler will give the sha, keep that in Nix code
  */
 
  muslSrc = derivation {
    name = "musl-src";
    system = "x86_64-linux";
    builder = "/bin/sh";
    args = [ "-c" ''
      mkdir -p $out
      curl -L https://musl.libc.org/releases/musl-1.2.4.tar.gz -o $out/musl.tar.gz
    '' ];
  };
 
  gccSrc = derivation {
    name = "gcc-src";
    system = "x86_64-linux";
    builder = "/bin/sh";
    args = [ "-c" ''
      mkdir -p $out
      curl -L https://ftp.gnu.org/gnu/gcc/gcc-13.2.0/gcc-13.2.0.tar.xz -o $out/gcc.tar.xz
    '' ];
  };
 
  # 2. Build binutils
  binutils = derivation {
    name = "binutils";
    system = "x86_64-linux";
    builder = "/bin/sh";
    src = "${binutilsSrc}/binutils.tar.xz";
    args = [ "-c" ''
      tar xf $src
      cd binutils-2.40
      ./configure --prefix=$out
      make -j$(nproc)
      make install
    '' ];
  };
 
  # 3. Build musl libc
  musl = derivation {
    name = "musl";
    system = "x86_64-linux";
    builder = "/bin/sh";
    src = "${muslSrc}/musl.tar.gz";
    args = [ "-c" ''
      tar xf $src
      cd musl-1.2.4
      ./configure --prefix=$out
      make -j$(nproc)
      make install
    '' ];
  };
 
  # 4. Build GCC (bootstrap)
  gcc = derivation {
    name = "gcc-bootstrap";
    system = "x86_64-linux";
    builder = "/bin/sh";
    src = "${gccSrc}/gcc.tar.xz";
    PATH = "${binutils}/bin";
    args = [ "-c" ''
      tar xf $src
      cd gcc-13.2.0
      ./contrib/download_prerequisites
      ./configure \
        --disable-multilib \
        --enable-languages=c \
        --prefix=$out \
        --with-native-system-header-dir=${musl}/include
 
      make -j$(nproc)
      make install
    '' ];
  };
 
  # 5. Final usable toolchain
  toolchain = derivation {
    name = "pure-toolchain";
    system = "x86_64-linux";
    builder = "/bin/sh";
    args = [ "-c" ''
      mkdir -p $out/bin
      ln -s ${binutils}/bin $out/bin/binutils
      ln -s ${musl}/bin     $out/bin/musl
      ln -s ${gcc}/bin      $out/bin/gcc
      # final PATH wrapper
      cat > $out/env <<EOF
export PATH=${binutils}/bin:${gcc}/bin:${musl}/bin:\$PATH
EOF
    '' ];
  };
 
in toolchain

Nix code to run the C code using the above code as dependency

let
  toolchain = import ./toolchain.nix;
 
in derivation {
  name = "hello-world";
  system = "x86_64-linux";
  src = ./main.c;
  builder = "/bin/sh";
  PATH = "${toolchain}/env";
  args = [ "-c" ''
    mkdir build
    gcc $src -static -o build/hello
    mkdir -p $out
    cp build/hello $out/
  '' ];
}

Nixpkgs

  • nixpkgs = a giant Git repo of packages
  • Contains all packages: compilers, Rust, Python, Node, C libraries, tools, etc.
  • our Nix builds usually import `nixpkgs
  • nixpkgs = a collection of functions, not binaries
  • Every package is a function → returns a derivation
let
  pkgs = import <nixpkgs> {};
in pkgs.rustc
 
pkgs.rustc   # is a function that returns a derivation
  • nixpkgs overlays & overrides let us customize packages
  • we can modify dependencies or versions cleanly:
pkgs // {
  myRust = pkgs.rustc.override { extensions = [ "rust-src" ]; };
}

Stdenv

  • It’s basically compiler + build tools
  • It gives us a C compiler (clang or gcc), bash, core utils, make, a standard POSIX shell environment
  • stdenv.mkDerivation is how we create packages
  • Stdenv always defines 3 main phases
    • unpackPhase
    • buildPhase
    • installPhase
  • we can override any of them
    Example:
stdenv.mkDerivation {
  name = "demo";
  src = ./.;
  buildPhase = ''
    echo "building…"
  '';
}
  • Stdenv is built from a “bootstrap Stdenv”
  • That bootstrap Stdenv is built from pre-built binaries
  • Eventually it builds a full toolchain from source: the Nixpkgs world is self-hosting
  • Stdenv chooses the compiler automatically
    • On Linux → GCC by default
    • On macOS → Clang by default
    • we can also force it to take a compiler if we want: stdenv = pkgs.gccStdenv;
  • Stdenv provides hooks - scripts that run automatically during builds (e.g., patchShebangs, configure, fixup). Example: Rust does not use the autotools configure script → hooks skip that phase automatically
  • Running a Rust Project using Stdenv(3 approaches)
    1. Basic Rust build with Stdenv (using nixpkgs) 
      { pkgs ? import <nixpkgs> {} }:
 
pkgs.stdenv.mkDerivation {
  pname = "my-rust-app";
  version = "1.0";
  src = ./.;
  buildInputs = [ pkgs.rustc pkgs.cargo ];
  buildPhase = ''
    cargo build --release
  '';
  installPhase = ''
    mkdir -p $out/bin
    cp target/release/my-rust-app $out/bin/
  '';
}
      • What Stdenv does here:
        • Creates an isolated environment containing only Cargo + Rustc
        • Ensures reproducible builds
        • Ensures no accidental dependencies leak in
    2. Stdenv using a pinned nixpkgs 
      {
  description = "Rust build";
 
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05";
 
  outputs = { self, nixpkgs }: {
    packages.default = 
      nixpkgs.legacyPackages.x86_64-linux.stdenv.mkDerivation {
        pname = "rust-pkg";
        version = "1.0";
        src = ./.;
        buildInputs = with nixpkgs.legacyPackages.x86_64-linux; [
          cargo rustc
        ];
      };
  };
}
    3. Pure derivation using Rust Platform: nixpkgs has a Rust build system wrapping Stdenv: 
      pkgs.rustPlatform.buildRustPackage {
  pname = "my-rust-app";
  version = "1.0";
 
  src = ./.;
 
  cargoLock.lockFile = ./Cargo.lock;
}
      • This:
        • automatically fetches crates
        • automatically creates Cargo vendor directory
        • ensures full reproducibility

Nix Flakes

A flake is simply a source tree (typically a Git repository) containing a file named flake.nix that provides a standardized interface to Nix artifacts like packages, development environments, or NixOS configurations
The fundamental problem Nix Flakes solve is ensuring that when you evaluate the same flake code on different machines, you get identical results. Unlike traditional Nix expressions that can depend on environment variables, arbitrary files, or unversioned Git repositories, Nix Flakes enforce pure evaluation mode and lock all dependencies to exact revisions in a flake.lock file

  • Inputs are dependencies your flake needs, they can come from GitHub repositories, GitLab, local filesystems, or tarballs
  • Outputs are what your flake produces: packages, development shells, NixOS configurations, or even custom modules. Each output type is meant for a specific Nix command
  • The lock file (flake.lock) is automatically generated and pins all input dependencies to exact commit hashes, ensuring reproducibility across systems. This is similar to how package-lock.json works in npm or Cargo.lock in Rust

Here are the examples to understand Flakes:

  1. Simple Flake 
    {
  description = "My first Nix flake";
 
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
  };
 
  outputs = { self, nixpkgs }:
  let
    system = "x86_64-linux";
    pkgs = import nixpkgs { inherit system; };
  in
  {
    packages.${system}.default = pkgs.hello;
  };
}
  • inputs: Dependencies your flake needs (here we’re pulling nixpkgs)
  • outputs: A function that takes inputs and returns an attribute set with your flake’s outputs
  • packages.${system}.default: The default package for your system architecture
  • To use this flake: 
    # Enable flakes (one-time setup)
# In NixOS configuration:
nix.settings.experimental-features = [ "nix-command" "flakes" ];
 
# Create flake
nix flake init
cd project
nix build  # Builds the default package
nix flake show  # Shows all available outputs
    • nix run and nix build → look for packages.${system}.default
    • nix develop → looks for devShells.${system}.default
    • nixos-rebuild --flake → looks for nixosConfigurations.${hostname}
    • home-manager switch --flake → looks for homeConfigurations.${username}
  1. Development Shells: Creating reproducible development environments 
    {
  description = "Python development environment";
 
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };
 
  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; };
      in
      {
        devShells.default = pkgs.mkShell {
          buildInputs = with pkgs; [
            python310
            python310Packages.pip
            python310Packages.virtualenv
            nodejs
          ];
 
          shellHook = ''
            echo "Python development environment loaded"
            python --version
          '';
        };
      }
    );
}
    • We can run nix develop to activate this environment without having Python or Node installed globally
  2. Multiple Inputs and Version Pinning 
    {
  description = "Using multiple nixpkgs versions";
 
  inputs = {
    nixpkgs-stable.url = "github:nixos/nixpkgs/nixos-23.05";
    nixpkgs-unstable.url = "github:nixos/nixpkgs/nixos-unstable";
  };
 
  outputs = { self, nixpkgs-stable, nixpkgs-unstable }:
  let
    system = "x86_64-linux";
    pkgs-stable = import nixpkgs-stable { inherit system; };
    pkgs-unstable = import nixpkgs-unstable { inherit system; };
  in
  {
    packages.${system} = {
      stable-hello = pkgs-stable.hello;
      unstable-hello = pkgs-unstable.hello;
      default = pkgs-stable.hello;
    };
  };
}
    • Now we can run:
    nix run .#stable-hello     # Uses stable version
nix run .#unstable-hello   # Uses unstable version
    • After running any Nix command with your flake, a flake.lock file is automatically created:
    nix flake metadata  # Shows current locked versions
nix flake update    # Update inputs to latest available versions
nix flake update --update-input nixpkgs  # Update specific input

Overlays in Flakes:
It allows to customize or extend nixpkgs packages. Here’s how to use them in a flake. This overlay pattern is useful when you want downstream consumers to use our customizations

{
  description = "Flake with overlays";
 
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };
 
  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs {
          inherit system;
          overlays = [ self.overlays.default ];
        };
      in
      {
        packages.default = pkgs.my-custom-package;
      }
    ) // {
      overlays.default = final: prev: {
        my-custom-package = final.callPackage ./package.nix { };
      };
    };
}

Flake Parts for Modular Flakes: For larger projects, flake-parts provides a modular framework that eliminates boilerplate. Flake-parts automatically handles per-system outputs, reducing code duplication

{
  description = "Modular flake with flake-parts";
 
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    flake-parts.url = "github:hercules-ci/flake-parts";
  };
 
  outputs = inputs@{ flake-parts, nixpkgs, ... }:
    flake-parts.lib.mkFlake { inherit inputs; } {
      systems = [ "x86_64-linux" "aarch64-linux" ];
 
      perSystem = { config, self', pkgs, system, ... }: {
        packages.default = pkgs.hello;
 
        devShells.default = pkgs.mkShell {
          buildInputs = [ pkgs.nodejs ];
        };
      };
 
      flake = {
        nixosConfigurations.my-host = nixpkgs.lib.nixosSystem {
          system = "x86_64-linux";
          modules = [ ./configuration.nix ];
        };
      };
    };
}

Input Follows (Dependency Deduplication): When we have multiple flake inputs that depend on nixpkgs, we can force them to use the same version. This dramatically reduces disk space usage and ensures no version conflicts

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    home-manager.url = "github:nix-community/home-manager/release-23.05";
    home-manager.inputs.nixpkgs.follows = "nixpkgs";
    
    # Force custom-flake to use our nixpkgs
    custom-flake.url = "path:./custom";
    custom-flake.inputs.nixpkgs.follows = "nixpkgs";
  };
 
  outputs = { self, nixpkgs, home-manager, custom-flake, ... }: {
    # Now all three use the same nixpkgs version
  };
}

Home Manager

It’s used to manage the applications configurations(like ghostty configs, obsidian configs), dot files, zshrc, environment variables. The home manager is initialized using

nix run home-manager -- init --switch .

It generates a file like this

{
	inputs = {
		nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
		home-manager = {
			url = "github:nix-community/home-manager";
			inputs.nixpkgs.follows = "nixpkgs";
		};
	};
 
	outputs = { nixpkgs, home-manager, ... }:
		let
			system = "x86_64-linux";
			pkgs = nixpkgs.legacyPackages.${system};
		in {
			homeConfigurations."sadiq" 
				= home-manager.lib.homeManagerConfiguration {
					inherit pkgs;
					modules = [ /home.nix ];
			};
		};
}

the “sadiq” is taken from system username, and to activate this configuration we run

home-manager switch --flake .#sadiq

here the ”.” is the path of the file and “sadiq” is the configuration we want to activate, there can be many configurations in a file
For adding configurations to the applications, let’s add alias for GitHub cli using nix home manager

{ config, pkgs, }: ...
{
	programs.gh = {
		enable = true;
		settings = {
			version = "1";
			aliases = {
				"as" = "auth status";
			};
		};
		gitCredentialHelper.enable = true;
		extensions = [ pkgs.gh-eco ];
	};
}

For setting behaviors for applications that use configurations in out .bashrc/.zshrc as reference, they don’t work, as the setting will apply in home-manager are set in the .bashrc generated by home-manager and that changes are not reflected in actual .bashrc, so just make the initial .bashrc tracked by home-manager by creating the symlink
We just need the below three commands to run the existing nix configurations in the new system

# github login
nix run nixpkgs#gh -- auth login
 
# cloning repo
nix run nixpkgs#gh -- repo clone MdSadiqMd/nix
 
# running flakes
nix run home-manager -- switch --flake my-nix-config#sadiq

Common Commands

# Initialize a flake
nix flake init
 
# Show all outputs
nix flake show
 
# Update dependencies to latest versions
nix flake update
 
# Update specific input
nix flake update nixpkgs
 
# Check flake metadata
nix flake metadata
 
# Build a specific package
nix build .#package-name
 
# Run a package
nix run .#package-name
 
# Activate dev shell
nix develop
 
# Run CLI impurely (for unfree packages)
nix run --impure nixpkgs#steam

Graph View