denv

containerized development environments across many runners

Develop code in identical command line interface environments across different container managers and their runners.

Containerization of code has only grown in popularity since it (almost) completely removes issues of dependency incompatibility. More recently, even developing code within a container has grown in popularity leading to several approaches.

The main problem I have with all of the alternatives is lack of support for my specific workflow. I not only use my personal computer (with docker or podman) but I also commonly use academic High Performance Computers or Clusters (HPCs) which tend to have apptainer or singularity installed due to their better support for security-focused (lack of user access) installations. This is the main origin for denv - provide a common interface for using images as a development environment across these four container managers.

Alternatives

In general, most of these alternatives are either more popular than denv or maintained by larger groups of people (or both), so I would suggest using one of these projects if they work for your use case.

• distrobox: main inspiration for denv, POSIX-sh program with a larger feature set than denv but currently restricted to docker or podman
  • I attempted to develop distrobox in such a way as to support apptainer and singularity; however, the added features mainly centered around editing the container while it is being run were not supported by the apptainer/singularity installations on the HPCs I have access to.
• devcontainers: currently restricted to docker (or podman with some tweaking), has a CLI reference implementation and tight integration with the VS Code IDE
• devpod: Codespaces but open-source, client-only and unopinionated: Works with any IDE and lets you use any cloud, kubernetes or just localhost docker. Similarly restricted to docker/podman as VS Code devcontainers - focused more on providing a GitHub Codespaces-like service that isn't locked in to VS Code and GitHub.
• devbox: "similar to a package manager ... except the packages it manages are at the operating-system level", a helpful tool based on nix, written in go
  • This is the newest project and probably most closely aligned to my goals; however, it would require understanding how to write NixPkgs for all my dependencies (which is not an easy task given how specialized so many of the packages are) and I am not currently able to functionally install it on the HPCs which use apptainer/singularity.
• toolbox: built on top of podman, similar in spirit to distrobox and devbox
• repro-env: rust-wrapper for podman, focused on specifying a manifest file which is then evolved into a lock file specifying SHAs for container images and packages, allows env to only evolve when developer desires.
• devenv: "develop in your shell. deploy containers." Again, goals aligned with this project, but relies on the NixPkgs registry or building dependencies locally with nix definitions. Requires installation of nix package manager (I think).

Quick Start

Install the latest release on GitHub.

curl -s https://raw.githubusercontent.com/tomeichlersmith/denv/main/install | sh 

Make sure the installation was successful (and you have a supported runner installed)

denv check

Initialize the current directory as a denv.

denv init <dev-image-to-use>

Open an interactive shell in the newly-created denv

denv

Usage Cheatsheet

After initialization (above), the rest of the denv-specific subcommands are housed under denv config, which allow you to

• Change the image the denv uses denv config image <image-tag>
• Pull down the image again denv config image pull
• Disable copying of all host environment variables denv config env all off
• Set environment variables to specific values denv config env copy foo=bar
• Copy environment variables from the host (if not copying all of them) denv config env copy host_foo
• Disable network connection denv config network off
• Have other directories mounted denv config mounts /full/path/to/my/other/dir
• Print the config for inspection/debugging denv config print
• Set the program denv should run if no arguments are provided denv config shell /path/to/program

See denv help or man denv for more details about denv and its subcommands.

Motivation

I began developing for a highly specialized and extremely technical mixed C++ and Python project several years ago. Besides two different languages, the project also depends on several large upstream C++ projects, each of which could take hours to build and install even on multi-core machines. This naturally led to a solution where a single set of dependencies would be built and then shared with developers using a networked filesystem (e.g. NFS). This solution had several downsides:

1. The entire development environment depended on a networked filesystem that could be slow or even completely unresponsive when under heavy load.
2. The environment was extremely delicate and required certain environment variables to have the correct values.
3. The environment was almost completely unmovable. Or in other words, if you wished to develop on a different computer or cluster, you would need to learn how to compile everything yourself.
4. In order to develop for our project, you needed to also learn how to interact with a cluster via the terminal usually meaning you had to learn SSH and a terminal editor like vim or emacs.

All of these downsides led me toward a quest to improve our developer environment with two main goals in mind: manueverability and user friendliness. Containerization fits the first goal really well; however, the ease of use of the various container managers leaves a bit to be desired thus spawning this project.

Vocabulary

Before delving deeper into the complicated world of containers and their runners, I want to make sure you and I are on the same page about the meaning of some words.

Each of the items below is a word which I define for the purposes of this manual. They may have different definitions outside of this manual.

• denv: a shortening of "development environment" and is used to refer to both the command that is being documented here denv and the environment that the command spawns when run successfully. Generally, you should read it as the environment when it is an improper noun (i.e. preceded by an article like "the" or "a") and the command otherwise. An overly convoluted example is "The denv is created by running denv.". The first use refers to the spawned environment and the second refers to the command. In this manual, I try to represent "denv" as code when it refers to the command.
  • Pronunciation: Personally, I have always pronounced "denv" as "d/ea/nv" but many folks I collaborate with read it as "dee-/ea/nv".
• container: while not technically correct, I usually think of containers as light-weight virtual machines (VMs). This gets the point across that they have a different environment and can contain software that couldn't run on the host.
• image: the data that can be used to launch a container with specific software in it. Usually, images consist of layers which are created by running different commands within the image during the build process.
  • Look into Developing the Environment to learn more about building images for use with denv in mind.
• workspace: the special directory containing all of the files that are being worked on. This is simply a shorthand for this special directory and can, for most use cases, map pretty cleanly to the "repository root directory" or (if you are using git), the directory that contains the hidden directory .git.
• runner: a program that uses a image to start up a container with which the user can interact. In this project, I use "container runner" and "container manager" interchangeably even though they aren't technically the same. The requirements on a "runner" to be a backend for denv are defined in Adding a new Runner.

Stability Promise

With the release of version 1.0, denv features a strong commitment to backwards compatibility and stability.

Future releases will not introduce backwards incompatible changes that make existing denv configurations (mostly represented by the .denv/config file whose specification is in the FILES section of the denv config manual) stop working, or break working invocations of the command-line interface.

This does not, however, preclude fixing outright bugs, even if doing so might break denv configurations that rely on their behavior.

There will never be a denv 2.0. Any desirable backwards-incompatible changes will be opt-in on a per-denv basis, so users may migrate at their leisure.

Stability promise text copied heavily from just.

Getting Started

I am going to assume that you already have a container image in mind for running as the denv for this page; however, you can look at Developing the Environment for general notes on how to create an image that is useful as a denv. A good image to use if you wish to test run denv is one of the python images. This can give you access to the latest release of python without having to spend time compiling it or even installing it on your system. In addition, python has good support for "user" installation of packages within the home directory, so you can install your favorite packages pretty quickly within the denv without having to go through the rigamarole of actually building an image yourself.

Requirements

denv is a simple wrapper around a container runner. You must have a container runner installed on the system you wish to use denv on. Generally, the runners can be separated into two groups.¹

• Personal Laptops and Desktops: docker or podman
  • Both of these are more widely used in software industry and so they are generally easier to install and use; however, they have historically required certain elevated privileges that made them undesirable for academic clusters.
  • Version requirements on docker or podman have not been investigated.
• Computing Clusters: apptainer or singularityCE
  • These are commonly chosen by computing clusters due to their ability to be very strictly configured so users can run them without elevated priveleges thus avoiding some security vulnerabilities.
  • denv requires the use of the --env flag for singularity "flavors". This was implemented in version 3.6 for singularity and so any new install of either apptainer or singularityCE should work. Legacy installations of singularity (i.e. versions older than 3.8.7) should function with denv (down to version 3.6); however, denv only tests version 3.8.7.

denv supports non-Linux operating systems indirectly through Window's Subsystem for Linux (WSL) on Windows and through Linux Virtual Machines on MacOS (spawned by Docker on Mac or Lima). denv is limited to non-ID-necessary processes on MacOS. Read through the Sidebar on Operating Systems if you want to learn more. If you wish to use graphical applications from within a denv on Windows or MacOS, you will likely need to install an X server (VcXSrv on Windows and XQuartz on MacOS are common options).

Installation

The most recent installation can be obtained by running the install script in the GitHub repository.

curl -s https://raw.githubusercontent.com/tomeichlersmith/denv/main/install | sh 

One can pass parameters to the install script by providing extra options to sh

curl -s https://raw.githubusercontent.com/tomeichlersmith/denv/main/install | \
  sh -s -- --prefix dir --next

Here, I highlight the main options that are available.

• --prefix dir allows you to decide on the location where denv should be installed
• --next says to use the HEAD of the main branch instead of the most recent release (may or may not differ)

The installation script is merely a helpful and simple script for getting the program, its helper program _denv_entrypoint, the manual, and tab completion files all in their correct locations. The manual and tab completion files are not necessary for the functionality of denv, so one can simply download denv and _denv_entrypoint from the GitHub repository (whichever release, branch, tag, or commit you wish) and put them side-by-side in some location you wish to keep them. If they do not exist in a directory pointed to by your PATH variable then you will need to specify their full path to run.

Use denv check to verify that an installation was successful and to list the runners supported by the installed denv and which ones are accessible in the current environment. An example output would be

$ denv check
Entrypoint found alongside denv
Looking for apptainer... not found
Looking for singularity... not found
Looking for podman... not found
Looking for docker... found 'Docker version 27.0.3, build 7d4bcd8' <- use without DENV_RUNNER defined
denv would run with 'docker'

Here, I can see which programs denv looked for and I am informed that it only found docker which is the one it will use by default. The extra comment about DENV_RUNNER is helpful if there are multiple runners installed - you can override denv's choice by setting DENV_RUNNER to the command you wish to use.

Usage

denv is a relatively simple program with very few commands and even fewer options, but to help you get started, lets start a denv with a dependency you would want to be completely isolated from your computer: python2.7.

First, we will work in an example workspace that can easily be cleaned up.

mkdir denv-eg
cd denv-eg

Now we can initialize the environment.

denv init python:2.7.18

This step will download the image to your computer if you do not already have it. It will also be the first step that checks if you have a supported container runner. Now we can enter the denv itself.

denv

The prompt will likely change since denv changes the hostname for the container to include the name given to the denv (in this case just "denv"). Here, we now have a terminal where the python available is Python 2.7.

python --version
# output: Python 2.7.18

This example is pretty trivial (especially given the plethora of other solutions for isolated python environments), but it does show the basic workflow of denv - users configure it to have a certain container image defining the environment in-which to develop and then users enter this environment to do their work. We can clean up the denv by exiting our workspace and deleting the directory.

exit # leave the denv shell
cd ..
rm -r denv-eg

Note: Container runners maintain image caches outside of the denv workspace so if you wish to remove the python:2.7.18 image from your system, you will need to look at the documentation for your runner on how to do that. Generally, keeping extra images in your cache is good because it saves time re-downloading image layers that have been downloaded before, so you really should only worry about deleting the image if you are running out of disk space.

Sidebar on Operating Systems

⚠️ This section is laced with sarcasm. ⚠️

Containers are a technology created from combining a few Linux kernel features. This makes them incredibly versatile and just confusing enough to be wrapped in thick layers of software and high-tech jargon. Many others have written about the technology underpinning containers¹, so I won't go into more detail here. I merely point this out to emphasize that non-Linux operating systems only access containers via Virtual Machines (VM) hosting Linux.² Specifically, this means the other two big operating systems (Mircrosoft Windoze and Mac OS X) must be accomodated with VMs.

Windoze

(Misspelling Windows as Windoze since I think its funny.)

Microsoft released Windoze Subsystem for Linux (WSL) in 2016 after finally realizing that work can only really get done within a functional operating system (like Linux-based ones). While largely expected to be apart of their Embrace Extend Extinguish methodology, it still benefits us in that we can use containers in an almost equivalent way they would be used on a Linux system with the added step of having to wait for Windoze to boot and open WSL. One of the most popular container runners docker just plainly tells people to use WSL when installing it.

With this context in mind, denv supports Windoze indirectly through WSL in the same manner as docker. If you are on Windoze, first enable WSL and then interact with docker (or whatever runner you want³) and denv within the WSL terminal.

Mac OS X

Both docker and podman support Mac OS X by spawning light weight VMs that can be kept idle while awaiting container instruction.⁴ Since Mac OS X has a more similar filesystem (and presumably kernel) to Linux, its VM is able to have a tighter integration with the host system. This has both benefits and difficulties. The benefit is that we presumably get performance improvements (I'm assuming this, I have not tested it.) The downside is that the user is not exposed directly to a terminal within the Linux kernel system like they are for Windoze (via WSL) or bare-metal Linux. This is an issue for denv (and for other container wrappers) since, as a wrapper, we are trying to connect the container and host which is difficult to do when given only limited access to the intermediary VM layer.

With this context in mind, denv has limited support for Mac OS X. Namely, denv is able to support non-ID-necessary processes within the constructed denv. For example, compiling and running a program within denv works fine but making a commit with git within it does not. The discovery of this limitation and any ongoing work is tracked in denv Issue #102.

Graphical Applications

denv ensures that the X11 apps spawned from within the container can connect with the host by passing the DISPLAY environment variable and mounting the /tmp/.X11-unix directory. For Linux-hosts and WSL, this is enough⁵; however, on MacOS additional setup is required.

If you don't plan to use a graphical application from within the a denv (or if you are just using a network-based application like Jupyter Lab), then there is no need to do this additional setup on MacOS. These instructions⁶ which basically amount to installing XQuartz⁷ and then disabling access control with xhost + (check links for specific, permanent configuration of XQuartz after installation).

Tuning the denv

While denv is not built to construct a mutable environment, it does support a few different methods for tuning its behavior to suit your specific development workflow.

Version control

The configuration of the denv is a very light file and, by default, denv init produces a .gitignore file in its config/cache directory which helps you ignore the files that may change from computer-to-computer (or user-to-user if your team is using denv).

If you are already in a git repository, you can start tracking your denv configuration along with the rest of your code by simply adding the hidden denv directory.

git add .denv
git commit -m "initialize a denv configuration"

RC Files

One of the key features of denv is mapping the workspace you are developing in to the home directory of the denv container. This allows a lot of natural specialization since many programs look into the home directory (or some subdirectory of it) for their configuration files.

Moreover, many of these RC files are light text files and can be committed into your version control like denv's own config files; thus, carrying the denv with the code that requires it.

denv copies over some "skeleton" files into the workspace to help get it started and make sure any interactive shell that is opened is configured in a reasonable way. This is one of the main locations to do workspace specialization. I've mainly worked with bash, so that is the example I show below.

Example bash_aliases

The skeleton .bashrc copied into the workspace directory sources the .bash_aliases file if it exists. This makes this file a perfect candidate for storing workspace-specific shell functions that will be available to you once you enter the denv.

For example, I can help my C++ projects follow a more ergonomic build system by wrapping CMake calls.

# in <workspace>/.bash_aliases
build() {
  cmake -B build -S . $@ && cmake --build build
}

test() {
  build && cmake --build build --target test
}

run() {
  build && ./build/program $@
}

sh and .profile

Inside the denv, we use sh -lc to run the command provided to denv. The c flag is used to specify the command being run, but of more importance to us is the l flag which forces sh to be a login shell.

For our purposes, this means that various initialization files will be sourced while launching sh and before the command is run, specifically, one of the files that can be used is at ~/.profile (or <workspace>/.profile outside the denv). Updating this file with changes to *PATH variables (like PATH, LD_LIBRARY_PATH, and PYTHONPATH) can be helpful so that executables can be run interactively (i.e. denv then my-executable) and non-interactively (i.e. denv my-executable).

Extra Mounts

By default, denv only allows the container to view the files within the workspace¹. This works for many projects; however, sometimes software being developed requires input data from outside of its workspace, this could be input data files that need to be read when running the software or perhaps another source of software to run alongwith the code being developed. denv supports both of these workflows by allowing users to specify extra mounts to be connected to the denv when it is being run.

denv config mounts /path/to/extra/data/outside/workspace

denv requires any additional mounts to be specified by their full path and to already exist. This prevents user typos as well as ensures the user knows what path will be available within the denv (i.e. symlinks outside the denv may not map properly inside the denv).

Environment Variables

The default behavior of denv is to copy all of the host environment variables into the denv so that the environment within the denv is "familiar" to the user. Sometimes, this behavior is not desirable and so users can choose to disable it and seletively copy environment variables. In addition, users can choose to set specific values of environment variables within the denv that will stay that value regardless on what the value of that variable is on the host.

Some examples of using this behavior are provided in the EXAMPLES section of the denv config manual.

Cluster Computing

Note: In this section, to avoid typing, I'm going to refer to the Apptainer/SyLabs SingularityCE/Singularity group of runners as appatainer and the Docker/Podman group of runners as docker.

On many computing clusters, apptainer is the default container runner installed and used by denv and, in order to make the usage of apptainer (via denv) the same as docker, we reference images using the OCI image name ([registry/]owner/repo:tag). For example

denv init python:3.12

is the same on personal computers with docker and remote clusters with apptainer. We achieve this mimicry by piggy-backing on the apptainer cache directory where apptainer stores an intermediate SIF image it builds from the OCI image we provide it. This has two large effects for users of denv on computing clusters.

1. If the default cache location within users' home directories is too small to hold the images you want them to be using, they should update their shell configuration to define APPTAINER_CACHEDIR (or SINGULARITY_CACHEDIR for the singularity runners) to a different location with more space, preferably a place that supports atomic rename.
2. If you plan to run many parallel jobs using the same image, you should pre-build a SIF image using apptainer pull and provide this image to denv (in denv init or denv config image) so that denv uses a frozen image during parallel processing rather than referencing the cache which the user could change while the parallel jobs are running leading to potential differences.

These comments may apply to the docker family of runners especially if more clusters adopt a configuration where both Podman and Apptainer are installed. (For example, Containers on HPC proposes a configuration.) Personally, I have yet to gain access to a cluster where Podman is installed and configured in a way usable by denv. I have accessed a cluster where both Podman and Apptainer are installed but Podman is not given access to user namespaces and thus we are unable to pretend to be the correct user in a Podman-launched container.

Command Reference Manual

This section of the websites contains the HTML-rendered copies of the manpages for denv and its major sub-commands. Since these pages are mainly focused on being render-able by the man program, their appearance on this site is slightly odd. That being said, they are still a nice reference for anyone who prefers to view the website rather than use man denv on the command line.

DENV(1)                           User Manual                          DENV(1)

NAME
       denv v1.1.1

SYNOPSIS
       denv {help|-h|–help}

       denv version

       denv init [args]

       denv config [args]

       denv check [-h, –help] [-q, –quiet]

       denv [COMMAND] [args...]

DESCRIPTION
       denv  is a light, POSIX-compliant wrapper around a few common container
       managers, allowing the user to efficiently interact with container-ized
       envorinments  uniformly  across  systems  with different installed man‐
       agers.  It has few commands, prioritizing simplicity so that users  can
       easily  and  quickly  pass their own commands to be run within the spe‐
       cialized and isolated environment.

COMMANDS
       help prints a short help message and exits.  The aliases -h and  --help
       also exist for this command.

       version prints the name and version of the currently installed denv

       init initialize a new denv.  See denv-init(1) for details.

       config manipulate the configuration of the current denv.  See denv-con‐
       fig(1) for details.

       check check the installation of denv and look for  supported  container
       runners.  See denv-check(1) for details.

       COMMAND any other command not matching one of the options above is pro‐
       vided to the configured denv to run within the  containerized  environ‐
       ment.  The rest of the command line is passed along with COMMAND so its
       args are seen as if they were run manually within the shell of the con‐
       tainer.

EXAMPLES
       denv is meant to be used after building a containerized developer envi‐
       ronment.  Look at the online manual for help getting started on  devel‐
       oping  the  environment  itself, but for these examples, we will assume
       that you already have an image built in which you wish to develop.

   Basic Start-Up
       First, we go into the directory that holds the code we wish to  develop
       and  tell  denv  that this workspace should be running a specific image
       for its developer environment.

              denv init myuser/myrepo:mytag

       Then we can open a shell in the denv.

              denv

       Now you can build and run programs from within the denv with its solid‐
       ified  set  of  software  and  tools while still editing the code files
       themselves with whatever text editor you wish outside of the denv.  The
       init  command  produces a configuration file .denv/config which you can
       share between users and so it is excluded from the  default  .gitignore
       generated  within  .denv.  All other files within .denv are internal to
       denv and can only be modified at your own risk.

SEE ALSO
       denv-init(1), denv-config(1), denv-check(1)

ENVIRONMENT
       denv tests the definition and reads the value of a few different  envi‐
       ronment  variables - allowing the user to modify its behavior in an ad‐
       vanced way without having to provide many command line arguments.

       DENV_DEBUG if set, enable xtrace in denv so the user  can  see  exactly
       what commands are being run.

       DENV_INFO  if set, print progress information updates to terminal while
       denv is running

       DENV_RUNNER set to the container manager command you wish denv to  use.
       This  should  only  be used in the case where multiple managers are in‐
       stalled and you wish to override the default denv behavior of using the
       first runner that it finds available.

       DENV_NOPROMPT disable all user prompting.  This makes the following de‐
       cisions in the places where there would be prompts.

       • denv init errors out if there  is  already  a  denv  in  the  deduced
         workspace or if a passed workspace does not exist

       • denv  init and denv config image will not pull an image if it already
         exists

       DENV_TAB_COMMANDS a space-separated list of commands to include in tab-
       completions of denv.  This is helpful if there are a set of common com‐
       mands you use within the denv.

SCRIPTING
       denv has a shebang subcommand that can be used to construct a script to
       be  run by a certain program within a constructed denv.  denv’s shebang
       consists of several lines all beginning with the shebang signal charac‐
       ters  #!.   It begins with a normal Unix shebang.  /usr/bin/env is used
       to avoid having to type the full path to denv and -S  is  used  so  the
       whitespace between denv and shebang is respected (i.e. split).

              #!/usr/bin/env -S denv shebang

       The  following lines of the script file can then contain the configura‐
       tion of the denv.  This can be done in two ways.  If you already have a
       denv  workspace  that  you  want to run inside of, you can just specify
       that

              #!denv_workspace=/full/path/to/workspace

       If you don’t have a workspace, then you will need to define the config‐
       uration  of  the denv.  At minimum, you must inform denv which image it
       should be running.

              #!denv_image=python:3

       For singularity or apptainer runners, you need to pre-build this  image
       since,  without  a workspace, denv doesn’t know where it should put the
       intermediary image file.

              #!denv_image=/full/path/to/image.sif

       Other denv configuration options can be specified in this running  mode
       as  well.   The easiest way to see the options is to inspect the output
       of denv config print which will contain the options not related to  en‐
       vironment  variables.   A full listing of available options is given by
       any config file written by denv into a .denv directory for a workspace.
       When running from a workspace (i.e. when providing denv_workspace with‐
       in the shebang lines), the other options are ignored in favor of  read‐
       ing them from the workspace configuration.

       The last line of the shebang lines is then the program that will be run
       with the file and the rest of the command line.  This  program  is  run
       within  the denv so it does not need to reside on the host system.  The
       path does not need to even be a full path like  with  the  normal  unix
       shebang.  The following examples hope to give some more context for how
       to get started with denv’s shebang.

   Workspace Example
              #!/usr/bin/env -S denv shebang
              #!denv_workspace=/full/path/to/workspace
              #!program
              script for program

   Workspace-Less Example (singularity or apptainer)
              #!/usr/bin/env -S denv shebang
              #!denv_image=/full/path/to/image.sif
              #!program
              script for program

   Workspace-Less Example (other runners)
              #!/usr/bin/env -S denv shebang
              #!denv_image=owner/repo:tag
              #!program
              script for program

RUNNER DEDUCTION
       denv does not persist what runner is being used inside of its  configu‐
       ration  for  a  specific workspace.  This is done intentionally so that
       configurations could be shared across machines that may rely on differ‐
       ent  runners;  however,  this  could lead to confusion if denv is being
       used on a machine that has multiple runners installed.  In  this  case,
       it  is  highly  suggested  to use denv check and test-run the different
       runners to see which are capable of being used by denv.

              # lists which runners it supports and which ones it has found
              denv check

       A simple test would be to make sure denv can open a shell in some ubun‐
       tu  image.   Check  different runners by using the environment variable
       DENV_RUNNER.

              denv init ubuntu:22.04
              # run this for each of the runners "found" by denv check
              DENV_RUNNER=<runner> denv

       If any of the runners do not work (i.e. open an interactive bash termi‐
       nal), please make a bug report by opening an issue for further investi‐
       gation.  However, there are some configurations  of  popular  container
       runners  that  denv does not intend to support, so you may be forced to
       use a specific container runner out of the  ones  installed.   In  this
       case,  it  is  highly recommended to define the DENV_RUNNER environment
       variable in your ~/.bashrc (or equivalent) to avoid complication.

   Automatic Deduction
       denv does make some attempts to avoid this complexity by having an  au‐
       tomatic  choosing behavior that prefers runners that are more likely to
       be configured properly.  For this reason, denv chooses to  prefer  run‐
       ners that act as emulators over the runners they are emulating (for ex‐
       ample, podman is checked before docker and apptainer is checked  before
       singularity).   In  addition, since the configuration of podman on some
       computing clusters is not supportive of denv and apptainer is installed
       on  these  clusters, apptainer is checked before podman.  This leads to
       the following order of priority currently within denv when  DENV_RUNNER
       is not defined.

       1. apptainer

       2. singularity

       3. podman

       4. docker

FILES
       This  part  of  the  manual is an attempt to list and explain the files
       within a .denv directory.

   config
       The file  storing  the  configuration  of  the  denv  related  to  this
       workspace.  While it is plain-text and you can edit it directly.  Edit‐
       ing it with the denv config set of commands is helpful for doing  basic
       typo-  and  existence-  checking.  The config file is a basic key=value
       shell file that will be sourced by denv.   See  the  FILES  section  of
       denv-config(1) for more detail.

   skel-init
       This is an empty file that, if it exists, signals to the entrypoint ex‐
       ecutable that the files from /etc/skel have been copied into  the  denv
       home directory.  This prevents accidental overwriting of files that the
       user may edit as well as saving time when starting up the container.

   images
       This is a directory that holds any image files that may be generated by
       the runner denv is using to run the container.  For some runners, it is
       helpful to explicitly build an image outside of the cache directory and
       then  run  that image file.  This directory holds those images.  It can
       be deleted if the user wishes to reclaim some disk space; however, that
       means any image that are configured to be used by denv will then be re-
       downloaded and re-built.

denv                               Aug 2024                            DENV(1)

DENV(1)                           User Manual                          DENV(1)

NAME
       denv init

SYNOPSIS
       denv init [help|-h|--help] IMAGE [WORKSPACE] [--no-gitignore] [--clean-
       env|--no-copy-all] [--force] [--name NAME] [--no-mkdir|--mkdir]  [--no-
       over|--over]

OPTIONS
       --help,  -h,  or help print a short help message for denv or one of its
       sub commands

       --no-gitignore do not generate a gitignore file when setting up  a  new
       denv configuration

       --clean-env or --no-copy-all do not enable copying of all host environ‐
       ment variables within the new denv.  Later activation (or deactivation)
       of  copying all host environment variables can be done with denv config
       env all See denv-config(1) for details on denv config.

       --force  forces  re-initialization  of  a  denv  even  if  the  current
       workspace has one

       --name  sets  the name for the denv workspace that is being initialized
       to NAME

       --[no-]mkdir don't prompt about if denv can create the workspace direc‐
       tory. Just do it (--mkdir) or not (--no-mkdir).

       --[no-]over don't prompt about if denv should overwrite a configuration
       within the workspace or override a configuration in a parent directory.
       Just do it (--over) or not (--no-over).

ARGUMENTS
       IMAGE the name of a container image to use when starting a container to
       host the developer environment

       WORKSPACE the directory where the environment should be stored and con‐
       figured, used by default as the home directory within the developer en‐
       vironment so that the environment can also have its own shell  configu‐
       ration files and ~/.local paths.  If not provided, we just use the cur‐
       rent working directory.  If provided, we make sure it exists, enter  it
       and then continue.  If WORKSPACE already has a denv (or one of its par‐
       ent directories has a denv), then the user is prompted on if they  wish
       to  overwrite (in the case that WORKSPACE itself is a denv) or override
       (otherwise) the already-existing denv.

EXAMPLES
       Print the command line help for denv init without making any  edits  to
       the filesystem or beginning the process of configuring a new denv.

              denv init help

       Create  a new denv based off the python:3.11 container image within the
       current directory, allowing all host environment variables to be copied
       into the denv when running.

              denv init python:3.11

       Same  as  above,  but do not allow the host environment variables to be
       copied into the denv.

              denv init --clean-env python:3.11

       Create a new denv based off the python:3.11 container image and set its
       name to “py311” rather than the workspace directory’s name.

              denv init python:3.11 --name py311

       Create a new denv in some other location besides the current directory.
       Since the directory has the same name as above, the denvs  will  appear
       similar even though their workspace directory (on the host) may be dif‐
       ferent names.

              denv init python:3.11 py311

       As a standalone program, denv can be used within other scripts and sup‐
       port  programs.   With this in mind, a common process is to have a denv
       configuration that can be ensured to exist for the rest  of  additional
       tasks.   The  following example uses the --no-mkdir and --no-over flags
       to  silently  ensure  that  the  present  working  directory  has   the
       python:3.11 image configured.

              denv init --no-mkdir --no-over python:3.11

       WARNING:  denv  init  does not ensure that the image is the same as the
       one passed, it just quielty refused to overwrite an existing configura‐
       tion  allowing  it  to be quickly bypassed if the configuration has al‐
       ready been made or initialize the configuration if no configuration  is
       present.

DEFAULT CONFIGURATION
       denv  init  makes the following choices on configuration that can later
       be edited by denv config if the user desires.

       • The interative shell is /bin/bash -i, change with denv  config  shell
         PROGRAM

       • No  specific environment variables are copied from the host or set to
         specific values, change with denv config  env  copy  VAR[=VAL].   The
         denv  either  copies  all host environment variables (the default) or
         none (with --clean-env).

       • No extra diretories are mounted into the denv (besides those automat‐
         ically mounted by the underlying container runner, e.g. apptainer au‐
         to-mounts /tmp), update with denv config mounts DIR.

       • The denv is connected to the host’s network, disable with denv config
         network off.

SEE ALSO
       denv(1), denv-config(1), denv-check(1)

denv                               Aug 2024                            DENV(1)

DENV(1)                           User Manual                          DENV(1)

NAME
       denv config

SYNOPSIS
       denv config [help|-h|–help]

       denv config print [env]

       denv config image <pull | IMAGE>

       denv config mounts DIR0 [DIR1 DIR2 ...]

       denv config shell SHELL

       denv config network {[yes|on|true]|[no|off|false]}

       denv config env [help|-h|–help]

       denv config env print

       denv config env all [yes|no]

       denv config env copy VAR0[=VAL0] [VAR1[=VAL1] ...]

DESCRIPTION
       Manipulate  the  configuration of a denv that already exists.  The com‐
       mands here will all fail if a  denv  hasn’t  been  created,  see  denv-
       init(1) to create a new denv.

       denv config is separated into a set of sub-commands that are focused on
       manipulating the different aspects of the  denv  configuration.   These
       correspond to the different keywords specified after `config'.

   COMMANDS
       help print a help message for denv config.  This is the command that is
       issued if no keywords are given.   It  also  has  the  aliases  -h  and
       --help.

       print  print the loaded denv configuration.  This is helpful for debug‐
       ging purposes and inspecting the denv that you currently  have  config‐
       ured.

       image  set  (and  potentially  pull) the container image that should be
       used with the denv that you currently reside in.

       mounts provide additional directories to mount  into  the  denv  during
       running.

       shell set the shell that should be executed by denv if no other command
       is given by the user.

       network enable network connection for the denv  (passing  yes,  on,  or
       true) or disable this network connection (passing no, off, or false).

       env manipulate and view the environment variables that will be provided
       to the denv at run time.

ARGUMENTS
       Below are different arguments that can be provided separated  by  which
       command they correspond to.

   print
       env  If  any argument is provided to print, then all of the environment
       variables that will be passed into the denv are printed after  the  de‐
       duced configuration.  Without an argument, print will just show the de‐
       duced configuration and only print the environment variables  if  copy-
       all is false.

   image
       IMAGE  The  provided  argument is the image tag that should be used for
       running with the denv.   If  this  argument  is  the  special  key-word
       `pull',  then it won’t change the actual image tag and instead re-down‐
       load the currently configured image from the registry.

       For apptainer and  singularity  runners,  this  IMAGE  can  also  be  a
       filesystem path to a unpacked image that is already downloaded (and un‐
       packed) somewhere else on the computer.  In this case, we  simply  sym‐
       link  the  image  to  our  image cache so denv can operate like normal.
       Work is on-going to investigate supporting this workflow for other con‐
       tainer runners <https://github.com/tomeichlersmith/denv/issues/37>.

   mounts
       DIR Each of the space-separate arguments are interpreted as a directory
       that should be included in the list of mounts for  the  container  that
       denv  spawns.  These are in addition to the mount of the denv workspace
       to the container home directory.  They are mounted into  the  container
       at  the same filesystem location that they have on the host.  These di‐
       rectories are required to be full paths so that the user  is  cognizant
       of  what  paths will be available in the container and what arent.  One
       can use realpath(1) to deduce a fullpath from  a  relative  path  in  a
       POSIX-compliant way if desired.

   shell
       SHELL the program to use as the interactive shell within the container‐
       ized environment.  No checks on what this program is or if it  is  even
       available within the container are done.  As the name implies, denv ex‐
       pects it to be some shell that the user can interact with  but  techni‐
       cally it is just the default program that is run when the user does not
       provide any arguments to denv.

   env
       The env subcommand has its own sub-commands due to the  variability  of
       defining  which  environment  variables  should be copied into the con‐
       tainerized environment.

       help print a help message for this subcommand.  This has aliases -h and
       --help as well.

       print print out the environment variables and their values that will be
       passed into the container.

       all toggle the decision on if all possible environment  variables  from
       the  host environment should be copied.  yes, true, on all mean to copy
       all possible variables from the host environment, while their  inverses
       no,  false,  off  mean  to disable this feature and only copy variables
       that are explicitly defined via the copy command below.  Variables  de‐
       fined  with  a specific value overwrite any values that would be copied
       from the environment.

       copy configure which environment variables to copy  into  the  denv  at
       runtime.   Each  of  the  space-separated arguments to this command are
       treated separated and are interpreted as a VAR  with  an  optional  VAL
       distinguished by a `=' character.

       • VAR  environment  variable  name  either in the host environment that
         should be copied into the denv (if no value is specified with an  `='
         sign)  or defined to a specific value (when a value is specified with
         an `=' sign).  These names cannot  match  special  shell  environment
         names (e.g. `HOME') or special denv names (e.g. `DENV_RUNNER').

       • VAL  environment  variable value to use instead of the value from the
         host environment.  These values cannot have the  special  characters:
         space ' ', tick '`', quote '"', or dollar-sign '$'.  Providing a val‐
         ue for a specific environment variable means that variable  does  not
         need  to  exist in the host environment.  Moreover, providing a value
         takes precedence: if a value is provided, the denv will receive  that
         value,  ignoring any value that may exist in the environment (even if
         all is toggled to on and all environment variables are being copied).

EXAMPLES
       Print out the current configuration of the denv.

              denv config print

       Change the image that the denv should use when running.  Be careful. No
       cleaning or checking of compatibility is done.  A drastic enough change
       in the image may require recompilations or even re-writes of code being
       written and developed within the denv.

              denv config image my-repo/my-image:new-tag

       Pull  down the image that is currenlty configured again.  This is help‐
       ful if the denv is using an image tag like “latest” and should  be  up‐
       dated  to  the latest release again.  Updating to the latest release is
       not done automatically because of the warnings above.

              denv config image pull

   Sharing Environment Variables
       The syntax for sharing environment variables with the  denv  is  a  bit
       terse, so it is helpful to display some examples.

       By  default  (without  --no-copy-all  or  --clean-env when running denv
       init), denv will copy all possible environment variables from the  host
       into the denv.  This means one can

              export foo=bar
              printenv foo      # prints out "bar"
              denv printenv foo # also prints "bar"

       In  some  situations,  this is over-sharing and you can disable this so
       that host environment variables are not copied into the denv anymore.

              denv config env all no
              export foo=bar
              printenv foo      # prints out "bar"
              denv printenv foo # does not print anything and returns the error code 1

       Even with copying all environment variables  disabled,  one  can  still
       copy  specific  values  from the host or set specific variables to have
       specific values for the denv.

              denv config env copy baz myfoo=mybaz
              denv printenv myfoo # prints "mybaz"
              printenv myfoo      # does not print anything and returns error code 1
              denv printenv baz   # not set in host yet so does not print anything
              export baz="hooray"
              denv printenv baz   # prints "hooray"

FILES
       The denv config command is used to safely edit the .denv/config file so
       that  the user does not accidentally break their configuration.  Never‐
       theless, this file is a regular text file and so can be edited directly
       if  the  user  wishes to do something more advanced that the basic com‐
       mands described above can handle.

       The config file is a basic key=value shell file that will be sourced by
       denv whenever the configuration is needed.  denv assumes that this con‐
       fig file defines the following shell variables for it to use.

       denv_name the name for this denv

       denv_image the image to use when running the denv

       denv_shell the program to run as a interactive shell  if  running  denv
       without any arguments

       denv_mounts  a  space separated list of extra mounts to mount into denv
       when running

       denv_env_var_copy_all a boolean flag signalling if denv should copy all
       possible  host  environment  variables  into  the  denv ("true") or not
       ("false").

       denv_env_var_copy a space-separated list of host environment  variables
       to  copy  into  the  denv.  This is ignored if denv_env_var_copy_all is
       "true".  There are some restrictions on the names of variables that can
       be used and so editing this value directly is not recomended.  Use denv
       config env copy which does this validation.

       denv_env_var_set a space-separate list of key=value pairs that will  be
       set  as  environment  variables within the denv.  These values override
       any values that could be copied from the host.  There are  restrictions
       on the names and values that can be kept here so editing this value di‐
       rectly is not recommended.  Use denv config env copy to edit this value
       while validating that the rules are followed.

       denv_network  a boolean flag signalling if denv should connect the con‐
       tainer to the host network ("true") or disable all  network  connection
       ("false").

       Since  denv  v1, this configuration is considered stable.  Any new con‐
       figuration options that are desirable to introduce new features will be
       optional and thus are not required to reside within this file.

SEE ALSO
       denv(1), denv-init(1), denv-check(1)

denv                               Aug 2024                            DENV(1)

DENV(1)                           User Manual                          DENV(1)

NAME
       denv check

SYNOPSIS
       denv check [-h|--help] [-q|--quiet] [-s|--silent] [--workspace]

OPTIONS
       --help or -h print a short help message

       --quiet or -q suppress non-error output

       --silent  or  -s  suppress all output include error messages (i.e. just
       return the exit code below)

       --workspace check to see if denv can deduce a workspace from  the  cur‐
       rent directory

EXIT CODES
       denv  check  follows  the POSIX convention of returning a non-zero exit
       code when a failure condition is encountered.

       • 0 success, denv installation is complete and  there  is  a  supported
         runner to use (or the user printed the help message)

       • 1 failure, denv cannot find the entrypoint script as an executable in
         the directory it is installed in

       • 2 failure, denv cannot find a supported runner to use

       • 3 failure, DENV_RUNNER defined to a runner that denv does not support
         or is not an available program on the machine

       • 4  failure, denv cannot find a workspace in which it can run from the
         current directory (user probably missing a denv init)

       • 127 denv check was supplied an argument it didn’t recognize

SEE ALSO
       denv(1), denv-config(1), denv-init(1)

denv                               Aug 2024                            DENV(1)

Developing the Environment

By "developing the environment" I mean changing the image denv runs in some way. Usually, this means adding a new dependency so that the code can continue to progress; however, it could also mean patching how a certain dependency is configured, where something is installed, or making quality-of-life updates to the environment.

It is important to emphasize once again that denv is not a tool for actually building the images that are used to create development environments. The reasons for this are (in rough order):

1. The separation of developing the environment from developing the code that requires the environment is helpful for isolating complicated dependency issues from the normal developer. The image build context could even be kept in a separate repository in order to enforce this isolation.
2. Avoid build repetition. Many projects I work on have dependencies that take hours to build even on fast multi-core machines. This means it can save many people time if the container image is built once for everyone and then distributed via a registry (like DockerHub).
3. Somes HPCs do not have apptainer/singularity installations that support building from a recipe file. While newer versions support building from a recipe file in a secure environment, I wanted a tool whose interface and user experince can be uniform including these runners and their common limitations.

Now that is out of the way, I have a few suggestions to make about how to develop these environments. Since my ability to build images from a recipe file with singularity or apptainer is restricted and I usually already have docker or podman installed, the images are built using docker (or podman) and then pushed to the registry from which denv can pull them.

Install Locations

Since the image being run by denv has its own root filesystem, it is usually helpful to install dependencies into system locations (like /usr/local) so that downstream projects (either more dependencies or the code that is being developed) can easily find these dependencies.

Environment Variables

Instead of expecting users to correctly write configurations into the .bashrc in their workspace, one can make heavy use of environment variables in the image definition.

In addition, all of the files in /etc/skel from the image are copied into the workspace when denv is run for the first time. This enables the image creators to update those files with any environment tuning that needs to happen at run time. For example, one could deduce where a project is installed and then add that directory to PATH if it isn't one of the standard locations.

apptainer support

Apptainer and singularity take a slightly different approach to running containers than docker or podman and so there are some restrictions on the images you should impose to ease this difference.

Take a look at Docker-Apptainer Compatibility in the apptainer docs to learn more.

distrobox

This is probably the biggest tip. Remember when I said that distrobox inspired denv and denv has a smaller feature set? Well, I think distrobox is a good companion for denv - especially for denv users who wish to develop the environment a bit.

I often open a distrobox with a specific image in order to try adding new dependencies and once that installation process is deduced, I can build the dependency into the image and use it later with denv.

Version Control

Similar to the code being developed, it is generally important to strictly version control the image used to create the denv. This helps keep track of the complicated process of aligning dependencies as well as help users know which dependencies (and the versions of them) they have access to.

How to Contribute

All contributing is helpful! Anything from correcting a spelling mistake in the documentation, adding a new example, patching bugs, or adding features is highly encouraged. Below, I've collected some notes on these various levels of contribution.

Documentation Updates

If you are writing more detailed explanation or adding in a new example, please git clone the repository and make sure the updated documentation can be built into a website by mdbook and has the format you expect.

This website actually has a helpful edit button in the top-right corner that will take you to the file on GitHub to edit and submit a pull request with any updates you wish to suggest. This is especially helpful for smaller updates that don't affect the format of the website pages.

New Examples

As far as I'm concerned, the more the merrier! If you are writing an example, please be detailed about which runner you are using, the version of denv, and how you've configured denv to aid in your workflow.

Patching Bugs or Adding Features

If you find a bug or think of a new feature to add, please open a GitHub Issue to start the discussion. This allows all collaborators to see what you plan to work on as well as potentially offer some insight on how to get going.

Code Contributions

When you work on developing denv make sure to install shellcheck add run the check and test scripts.

./ci/check # uses shellcheck to avoid common issues writing shell scripts
./ci/test <your-runner> # basic functionality checking

The GitHub workflows test all of the currently supported runners, so make sure to enable them in your fork of the repository so that they will test runners that you may not have installed on your system!

If the code being developed is anything larger than an extremely small one-line change, please open an issue and reference the issue number in your branch name. Generally, I like the format <issue_number>-short-title for example 19-connect-net was used when developing network-connection supported related to issue 19.

Version Control

When changing the denv version number, one must change it in three locations.

• denv itself at the top
• install so future pullers will get the latest version
• man/man1/denv.1 so the man page has the new version number

This is annoying to always have to remember to do, so there is a short shell script to do this for you.

./ci/set-version X.Y.Z

Writing tests

denv uses bats to run tests and pins its version (as well as the necessary plugins) using submodules. Look to these resources for the assert_* family of functions and how tests are structured and run.

• bats-core
• bats-assert
• bats-file

Adding a new Runner

I use "runner" and "manager" pretty much inter-changeably on this site because, for the purposes of denv, the difference between the two is not of great importance.

Requirements

There are two main features that a program needs to satisfy in order to be usable by denv as a container-interaction backend.

1. Download Images from a Registry
2. Check if an Image is already Downloaded
3. Run the Image as a Container

Usually container "runners" are specifically focused on doing the last task (and only that one) while "managers" can do all of these tasks and more (like building images, pushing images to a registry, tagging images, etc...).

The main reason I use manager and runner interchangeably here is because we do not use a lot of the features managers provide and so I could easily forsee denv supporting a runner whose download/check mechanism is some curl or wget shell scripting nonsense rather than a call to the runner program itself.

Running the Image

This section expands a bit on the requirements on the runner when running an image. For specifics, one will need to look at the denv source itself to see how the currently-supported runners run images as containers.

denv wishes to integrate the containerized environment with the host environment in several ways so that the user, while developing within a denv, can easily use programs within the denv as if they are installed within the host system. This leads to a few requirements on the program that runs the images.

• User Ownership: any files written when in the denv should be owned by the user once they exit the denv
• Launching GUI Programs: the user should be able to launch a GUI program from within the denv
• Network and Port Connection: the user should be able to use the host network and connect to ports on localhost within the denv
• Home Directory: the in-denv home directory should be set to the workspace outside of the denv
• Environment Variables: the user is able to configure which environment variables to pass into the denv, so the runner needs to be able to define environment variables for the spawned container at runtime

What to Test

If you are developing a new runner to be wrapped by denv, the natural next question to ask is how should I test that it is functional.

First and foremost, make sure your additions to denv still pass the non-interactive tests.

./ci/check
./ci/test <your-runner>

These can be enabled in your fork of denv so that they run automatically on GitHub when you push to a branch on your fork. In order for GitHub to be able to test your runner, you will need to install it into the GitHub runner during the testing workflow (.github/workflows/test.yml) and add it to the runners-to-test.json file in the include list.

Tracking New Releases in Test

I've written up a "dependabot" that will check the latest release of certain GitHub repositories and update the testing workflow with those releases if a new one is found. In order to follow new releases of the runner being added, you must make changes in two places of the CI infrastructure.

1. Add the runner repository (OWNER/REPO) under the track key in its entry in ci/runners-to-test.json.
2. Make sure your installation procedure will depend (and can handle) the version changing.

Right now, this is done for sylabs/singularity and apptainer/apptainer so look to those runners within the testing infrastructure as examples.

Besides the non-interactive tests, there are some additional, manual tests that I haven't figured out how to automate since they check interactions between the host and denv environments.

Make sure GUI Programs can be launched

There is a small image that can be used to test whether GUI programs can be run from within a denv. Launching from a in-denv shell, launching it directly from outside the denv, and launching without sharing environment variables should work properly.

I haven't found a quick and easy way to test this, but look at the test/gui directory for my notes on how I've tested this in the past. As a first pass, you can attempt to run the xeyes program from denv.

DENV_RUNNER=<your-runner> ./test/gui/run-gui-test

Make sure Network and Ports are Connected

My main reason for supporting this is to allow me to interact with a Jupyter Lab instance running from within the denv. The Jupyter Project has built some images with their software installed which can be used for testing.

mkdir net-test
cd net-test
denv init jupyter/datascience-notebook
denv jupyter lab --no-browser

The user should be able to access the localhost link displayed by jupyter. Note: Developers should know that these images have a very special user and entrypoint configuration specialized for jupyter lab which may cause extra complications.

The non-interactive tests do check for network connection within the container by attempting to connect a socket to a public Google DNS server. I do not expect this to be enough to guarantee network functionality especially for new runners that I am not familiar with. Any aid in more precisely testing external network connection as well as "localhost"-style connection would be an appreciated contribution.

Developing in a VM

A full virtual machine (VM) can be very helpful for developing and testing these applications. Oftentimes, developers don't want to have all of the container runners they wish to test on their bare metal system, so it is helpful to learn how to test these options within a VM.

These notes are developed for the Virtual Machine Manager that I use on Linux Mint. From casual browsing, it appears that other VM managers can do similar processes.

Mount Source Code

I don't want to install all of my code editing toolkits into a simple test VM, so I instead mount the directory my source code is in into the VM so that the VM can just see the current code I'm writing on the host.

Credit where credit is due, these notes are just parroting the tutorial given by Arindam on Debugpoint.

On the Host in the VM Manager

1. Make sure the "Enable shared memory" box is checked in the "Memory" section of the VM settings
2. Add a filesystem to the VM
   1. Select "Add Hardware" and choose "Filesystem"
   2. Driver: virtiofs (was the default for me)
   3. Source path: /full/path/to/denv/on/host (can use GUI file manager)
   4. Target path: denvsource (more of a name rather than a path as you'll see below)

Inside the VM

1. Make sure the mount point for the source code exists

mkdir -p denv

2. Mount the virtual filesystem to our mount point

sudo mount -t virtiofs denvsource denv/

3. Do a developer's install so that we can run denv like normal

cd denv/
./install -d