denv

containerized development environments across many runners

Develop code in identical command line interface environments across different container managers and their runners.

Containerization of code has only grown in popularity since it (almost) completely removes issues of dependency incompatibility. More recently, even developing code within a container has grown in popularity leading to several approaches.

The main problem I have with all of the alternatives is lack of support for my specific workflow. I not only use my personal computer (with docker or podman) but I also commonly use academic High Performance Computers or Clusters (HPCs) which tend to have apptainer or singularity installed due to their better support for security-focused (lack of user access) installations. This is the main origin for denv - provide a common interface for using images as a development environment across these four container managers.

Alternatives

In general, most of these alternatives are either more popular than denv or maintained by larger groups of people (or both), so I would suggest using one of these projects if they work for your use case.

• distrobox: main inspiration for denv, POSIX-sh program with a larger feature set than denv but currently restricted to docker or podman
  • I attempted to develop distrobox in such a way as to support apptainer and singularity; however, the added features mainly centered around editing the container while it is being run were not supported by the apptainer/singularity installations on the HPCs I have access to.
• devcontainers: currently restricted to docker (or podman with some tweaking), has a CLI reference implementation and tight integration with the VS Code IDE
• devpod: Codespaces but open-source, client-only and unopinionated: Works with any IDE and lets you use any cloud, kubernetes or just localhost docker. Similarly restricted to docker/podman as VS Code devcontainers - focused more on providing a GitHub Codespaces-like service that isn't locked in to VS Code and GitHub.
• devbox: "similar to a package manager ... except the packages it manages are at the operating-system level", a helpful tool based on nix, written in go
  • This is the newest project and probably most closely aligned to my goals; however, it would require understanding how to write NixPkgs for all my dependencies (which is not an easy task given how specialized so many of the packages are) and I am not currently able to functionally install it on the HPCs which use apptainer/singularity.
• toolbox: built on top of podman, similar in spirit to distrobox and devbox
• repro-env: rust-wrapper for podman, focused on specifying a manifest file which is then evolved into a lock file specifying SHAs for container images and packages, allows env to only evolve when developer desires.
• devenv: "develop in your shell. deploy containers." Again, goals aligned with this project, but relies on the NixPkgs registry or building dependencies locally with nix definitions. Requires installation of nix package manager (I think).

Quick Start

Install the latest release on GitHub.

curl -s https://tomeichlersmith.github.io/denv/install | sh

Make sure the installation was successful (and you have a supported runner installed)

denv check

Initialize the current directory as a denv.

denv init <dev-image-to-use>

Open an interactive shell in the newly-created denv

denv

Usage Cheatsheet

After initialization (above), the rest of the denv-specific subcommands are housed under denv config, which allow you to

• Change the image the denv uses denv config image <image-tag>
• Pull down the image again denv config image pull
• Disable copying of all host environment variables denv config env all off
• Set environment variables to specific values denv config env copy foo=bar
• Copy environment variables from the host (if not copying all of them) denv config env copy host_foo
• Disable network connection denv config network off
• Have other directories mounted denv config mounts /full/path/to/my/other/dir
• Print the config for inspection/debugging denv config print
• Set the program denv should run if no arguments are provided denv config shell /path/to/program

See denv help or man denv for more details about denv and its subcommands.

Motivation

I began developing for a highly specialized and extremely technical mixed C++ and Python project several years ago. Besides two different languages, the project also depends on several large upstream C++ projects, each of which could take hours to build and install even on multi-core machines. This naturally led to a solution where a single set of dependencies would be built and then shared with developers using a networked filesystem (e.g. NFS). This solution had several downsides:

1. The entire development environment depended on a networked filesystem that could be slow or even completely unresponsive when under heavy load.
2. The environment was extremely delicate and required certain environment variables to have the correct values.
3. The environment was almost completely unmovable. Or in other words, if you wished to develop on a different computer or cluster, you would need to learn how to compile everything yourself.
4. In order to develop for our project, you needed to also learn how to interact with a cluster via the terminal usually meaning you had to learn SSH and a terminal editor like vim or emacs.

All of these downsides led me toward a quest to improve our developer environment with two main goals in mind: manueverability and user friendliness. Containerization fits the first goal really well; however, the ease of use of the various container managers leaves a bit to be desired thus spawning this project.

Vocabulary

Before delving deeper into the complicated world of containers and their runners, I want to make sure you and I are on the same page about the meaning of some words.

Each of the items below is a word which I define for the purposes of this manual. They may have different definitions outside of this manual.

• denv: a shortening of "development environment" and is used to refer to both the command that is being documented here denv and the environment that the command spawns when run successfully. Generally, you should read it as the environment when it is an improper noun (i.e. preceded by an article like "the" or "a") and the command otherwise. An overly convoluted example is "The denv is created by running denv.". The first use refers to the spawned environment and the second refers to the command. In this manual, I try to represent "denv" as code when it refers to the command.
  • Pronunciation: Personally, I have always pronounced "denv" as "d/ea/nv" but many folks I collaborate with read it as "dee-/ea/nv".
• container: while not technically correct, I usually think of containers as light-weight virtual machines (VMs). This gets the point across that they have a different environment and can contain software that couldn't run on the host.
• image: the data that can be used to launch a container with specific software in it. Usually, images consist of layers which are created by running different commands within the image during the build process.
  • Look into Developing the Environment to learn more about building images for use with denv in mind.
• workspace: the special directory containing all of the files that are being worked on. This is simply a shorthand for this special directory and can, for most use cases, map pretty cleanly to the "repository root directory" or (if you are using git), the directory that contains the hidden directory .git.
• runner: a program that uses a image to start up a container with which the user can interact. In this project, I use "container runner" and "container manager" interchangeably even though they aren't technically the same. The requirements on a "runner" to be a backend for denv are defined in Adding a new Runner.

Stability Promise

With the release of version 1.0, denv features a strong commitment to backwards compatibility and stability.

Future releases will not introduce backwards incompatible changes that make existing denv configurations (mostly represented by the .denv/config file whose specification is in the FILES section of the denv config manual) stop working, or break working invocations of the command-line interface.

This does not, however, preclude fixing outright bugs, even if doing so might break denv configurations that rely on their behavior.

There will never be a denv 2.0. Any desirable backwards-incompatible changes will be opt-in on a per-denv basis, so users may migrate at their leisure.

Stability promise text copied heavily from just.

Getting Started

I am going to assume that you already have a container image in mind for running as the denv for this page; however, you can look at Developing the Environment for general notes on how to create an image that is useful as a denv. A good image to use if you wish to test run denv is one of the python images. This can give you access to the latest release of python without having to spend time compiling it or even installing it on your system. In addition, python has good support for "user" installation of packages within the home directory, so you can install your favorite packages pretty quickly within the denv without having to go through the rigamarole of actually building an image yourself.

Requirements

denv is a simple wrapper around a container runner. You must have a container runner installed on the system you wish to use denv on. Generally, the runners can be separated into two groups.¹

• Personal Laptops and Desktops: docker or podman
  • Both of these are more widely used in software industry and so they are generally easier to install and use; however, they have historically required certain elevated privileges that made them undesirable for academic clusters.
  • Version requirements on docker or podman have not been investigated.
• Computing Clusters: apptainer or singularityCE
  • These are commonly chosen by computing clusters due to their ability to be very strictly configured so users can run them without elevated priveleges thus avoiding some security vulnerabilities.
  • denv requires the use of the --env flag for singularity "flavors". This was implemented in version 3.6 for singularity and so any new install of either apptainer or singularityCE should work. Legacy installations of singularity (i.e. versions older than 3.8.7) should function with denv (down to version 3.6); however, denv only tests version 3.8.7.

denv supports non-Linux operating systems indirectly through Window's Subsystem for Linux (WSL) on Windows and through Linux Virtual Machines on MacOS (spawned by Docker on Mac or Lima). denv is limited to non-ID-necessary processes on MacOS. Read through the Sidebar on Operating Systems if you want to learn more. If you wish to use graphical applications from within a denv on Windows or MacOS, you will likely need to install an X server (VcXSrv on Windows and XQuartz on MacOS are common options).

Installation

The most recent version can be obtained by running the install script in the GitHub repository.

curl -s https://tomeichlersmith.github.io/denv/install | sh

One can pass parameters to the install script by providing extra options to sh

curl -s https://tomeichlersmith.github.io/denv/install | \
  sh -s -- vX.Y.Z --prefix dir --next

Here, I highlight the main options that are available.

• --prefix dir allows you to decide on the location where denv should be installed
• --next says to use the HEAD of the main branch instead of the most recent release (may or may not differ)
• vX.Y.Z allows you to choose which version of denv you want to install

The installation script is merely a helpful and simple script for getting the program, its helper program _denv_entrypoint, the manual, and tab completion files all in their correct locations. The manual and tab completion files are not necessary for the functionality of denv, so one can simply download denv and _denv_entrypoint from the GitHub repository (whichever release, branch, tag, or commit you wish) and put them side-by-side in some location you wish to keep them. If they do not exist in a directory pointed to by your PATH variable then you will need to specify their full path to run.

Use denv check to verify that an installation was successful and to list the runners supported by the installed denv and which ones are accessible in the current environment. An example output would be

$ denv check
Entrypoint found alongside denv
Looking for apptainer... not found
Looking for singularity... not found
Looking for podman... not found
Looking for docker... found 'Docker version 27.0.3, build 7d4bcd8' <- use without DENV_RUNNER defined
denv would run with 'docker'

Here, I can see which programs denv looked for and I am informed that it only found docker which is the one it will use by default. The extra comment about DENV_RUNNER is helpful if there are multiple runners installed - you can override denv's choice by setting DENV_RUNNER to the command you wish to use.

Usage

denv is a relatively simple program with very few commands and even fewer options, but to help you get started, lets start a denv with a dependency you would want to be completely isolated from your computer: python2.7.

First, we will work in an example workspace that can easily be cleaned up.

mkdir denv-eg
cd denv-eg

Now we can initialize the environment.

denv init python:2.7.18

This step will download the image to your computer if you do not already have it. It will also be the first step that checks if you have a supported container runner. Now we can enter the denv itself.

denv

The prompt will likely change since denv changes the hostname for the container to include the name given to the denv (in this case just "denv"). Here, we now have a terminal where the python available is Python 2.7.

python --version
# output: Python 2.7.18

This example is pretty trivial (especially given the plethora of other solutions for isolated python environments), but it does show the basic workflow of denv - users configure it to have a certain container image defining the environment in-which to develop and then users enter this environment to do their work. We can clean up the denv by exiting our workspace and deleting the directory.

exit # leave the denv shell
cd ..
rm -r denv-eg

Note: Container runners maintain image caches outside of the denv workspace so if you wish to remove the python:2.7.18 image from your system, you will need to look at the documentation for your runner on how to do that. Generally, keeping extra images in your cache is good because it saves time re-downloading image layers that have been downloaded before, so you really should only worry about deleting the image if you are running out of disk space.

Sidebar on Operating Systems

⚠️ This section is laced with sarcasm. ⚠️

Containers are a technology created from combining a few Linux kernel features. This makes them incredibly versatile and just confusing enough to be wrapped in thick layers of software and high-tech jargon. Many others have written about the technology underpinning containers¹, so I won't go into more detail here. I merely point this out to emphasize that non-Linux operating systems only access containers via Virtual Machines (VM) hosting Linux.² Specifically, this means the other two big operating systems (Mircrosoft Windoze and Mac OS X) must be accomodated with VMs.

Windoze

(Misspelling Windows as Windoze since I think its funny.)

Microsoft released Windoze Subsystem for Linux (WSL) in 2016 after finally realizing that work can only really get done within a functional operating system (like Linux-based ones). While largely expected to be apart of their Embrace Extend Extinguish methodology, it still benefits us in that we can use containers in an almost equivalent way they would be used on a Linux system with the added step of having to wait for Windoze to boot and open WSL. One of the most popular container runners docker just plainly tells people to use WSL when installing it.

With this context in mind, denv supports Windoze indirectly through WSL in the same manner as docker. If you are on Windoze, first enable WSL and then interact with docker (or whatever runner you want³) and denv within the WSL terminal.

Mac OS X

Both docker and podman support Mac OS X by spawning light weight VMs that can be kept idle while awaiting container instruction.⁴ Since Mac OS X has a more similar filesystem (and presumably kernel) to Linux, its VM is able to have a tighter integration with the host system. This has both benefits and difficulties. The benefit is that we presumably get performance improvements (I'm assuming this, I have not tested it.) The downside is that the user is not exposed directly to a terminal within the Linux kernel system like they are for Windoze (via WSL) or bare-metal Linux. This is an issue for denv (and for other container wrappers) since, as a wrapper, we are trying to connect the container and host which is difficult to do when given only limited access to the intermediary VM layer.

With this context in mind, denv has limited support for Mac OS X. Namely, denv is able to support non-ID-necessary processes within the constructed denv. For example, compiling and running a program within denv works fine but making a commit with git within it does not. The discovery of this limitation and any ongoing work is tracked in denv Issue #102.

Graphical Applications

denv ensures that the X11 apps spawned from within the container can connect with the host by passing the DISPLAY environment variable and mounting the /tmp/.X11-unix directory. For Linux-hosts and WSL, this is enough⁵; however, on MacOS additional setup is required.

If you don't plan to use a graphical application from within the a denv (or if you are just using a network-based application like Jupyter Lab), then there is no need to do this additional setup on MacOS. These instructions⁶ which basically amount to installing XQuartz⁷ and then disabling access control with xhost + (check links for specific, permanent configuration of XQuartz after installation).

Tuning the denv

While denv is not built to construct a mutable environment, it does support a few different methods for tuning its behavior to suit your specific development workflow.

Version control

The configuration of the denv is a very light file and, by default, denv init produces a .gitignore file in its config/cache directory which helps you ignore the files that may change from computer-to-computer (or user-to-user if your team is using denv).

If you are already in a git repository, you can start tracking your denv configuration along with the rest of your code by simply adding the hidden denv directory.

git add .denv
git commit -m "initialize a denv configuration"

RC Files

One of the key features of denv is mapping the workspace you are developing in to the home directory of the denv container. This allows a lot of natural specialization since many programs look into the home directory (or some subdirectory of it) for their configuration files.

Moreover, many of these RC files are light text files and can be committed into your version control like denv's own config files; thus, carrying the denv with the code that requires it.

denv copies over some "skeleton" files into the workspace to help get it started and make sure any interactive shell that is opened is configured in a reasonable way. This is one of the main locations to do workspace specialization. I've mainly worked with bash, so that is the example I show below.

Example bash_aliases

The skeleton .bashrc copied into the workspace directory sources the .bash_aliases file if it exists. This makes this file a perfect candidate for storing workspace-specific shell functions that will be available to you once you enter the denv.

For example, I can help my C++ projects follow a more ergonomic build system by wrapping CMake calls.

# in <workspace>/.bash_aliases
build() {
  cmake -B build -S . $@ && cmake --build build
}

test() {
  build && cmake --build build --target test
}

run() {
  build && ./build/program $@
}

sh and .profile

Inside the denv, we use sh -lc to run the command provided to denv. The c flag is used to specify the command being run, but of more importance to us is the l flag which forces sh to be a login shell.

For our purposes, this means that various initialization files will be sourced while launching sh and before the command is run, specifically, one of the files that can be used is at ~/.profile (or <workspace>/.profile outside the denv). Updating this file with changes to *PATH variables (like PATH, LD_LIBRARY_PATH, and PYTHONPATH) can be helpful so that executables can be run interactively (i.e. denv then my-executable) and non-interactively (i.e. denv my-executable).

Extra Mounts

By default, denv only allows the container to view the files within the workspace¹. This works for many projects; however, sometimes software being developed requires input data from outside of its workspace, this could be input data files that need to be read when running the software or perhaps another source of software to run alongwith the code being developed. denv supports both of these workflows by allowing users to specify extra mounts to be connected to the denv when it is being run.

denv config mounts /path/to/extra/data/outside/workspace

denv requires any additional mounts to be specified by their full path and to already exist. This prevents user typos as well as ensures the user knows what path will be available within the denv (i.e. symlinks outside the denv may not map properly inside the denv).

Environment Variables

The default behavior of denv is to copy all of the host environment variables into the denv so that the environment within the denv is "familiar" to the user. Sometimes, this behavior is not desirable and so users can choose to disable it and seletively copy environment variables. In addition, users can choose to set specific values of environment variables within the denv that will stay that value regardless on what the value of that variable is on the host.

Some examples of using this behavior are provided in the EXAMPLES section of the denv config manual.

Cluster Computing

Note: In this section, to avoid typing, I'm going to refer to the Apptainer/SyLabs SingularityCE/Singularity group of runners as appatainer and the Docker/Podman group of runners as docker.

On many computing clusters, apptainer is the default container runner installed and used by denv and, in order to make the usage of apptainer (via denv) the same as docker, we reference images using the OCI image name ([registry/]owner/repo:tag). For example

denv init python:3.12

is the same on personal computers with docker and remote clusters with apptainer. We achieve this mimicry by piggy-backing on the apptainer cache directory where apptainer stores an intermediate SIF image it builds from the OCI image we provide it. This has two large effects for users of denv on computing clusters.

1. If the default cache location within users' home directories is too small to hold the images you want them to be using, they should update their shell configuration to define APPTAINER_CACHEDIR (or SINGULARITY_CACHEDIR for the singularity runners) to a different location with more space, preferably a place that supports atomic rename.
2. If you plan to run many parallel jobs using the same image, you should pre-build a SIF image using apptainer pull and provide this image to denv (in denv init or denv config image) so that denv uses a frozen image during parallel processing rather than referencing the cache which the user could change while the parallel jobs are running leading to potential differences.

These comments may apply to the docker family of runners especially if more clusters adopt a configuration where both Podman and Apptainer are installed. (For example, Containers on HPC proposes a configuration.) Personally, I have yet to gain access to a cluster where Podman is installed and configured in a way usable by denv. I have accessed a cluster where both Podman and Apptainer are installed but Podman is not given access to user namespaces and thus we are unable to pretend to be the correct user in a Podman-launched container.

Command Reference Manual

This section of the websites contains the HTML-rendered copies of the manpages for denv and its major sub-commands. Since these pages are mainly focused on being render-able by the man program, their appearance on this site is slightly odd. That being said, they are still a nice reference for anyone who prefers to view the website rather than use man denv on the command line.

DENV

NAME
SYNOPSIS
DESCRIPTION
COMMANDS
EXAMPLES
Basic Start-Up
SEE ALSO
ENVIRONMENT
SCRIPTING
Workspace Example
Workspace-Less Example (singularity or apptainer)
Workspace-Less Example (other runners)
RUNNER DEDUCTION
Automatic Deduction
FILES
config
skel-init
images

NAME

denv v1.1.3

SYNOPSIS

denv {help|-h|–help}

denv version

denv init [args]

denv config [args]

denv check [-h, –help] [-q, –quiet]

denv [COMMAND] [args...]

DESCRIPTION

denv is a light, POSIX-compliant wrapper around a few common container managers, allowing the user to efficiently interact with container-ized envorinments uniformly across systems with different installed managers. It has few commands, prioritizing simplicity so that users can easily and quickly pass their own commands to be run within the specialized and isolated environment.

COMMANDS

help prints a short help message and exits. The aliases -h and --help also exist for this command.

version prints the name and version of the currently installed denv

init initialize a new denv. See denv-init(1) for details.

config manipulate the configuration of the current denv. See denv-config(1) for details.

check check the installation of denv and look for supported container runners. See denv-check(1) for details.

COMMAND any other command not matching one of the options above is provided to the configured denv to run within the containerized environment. The rest of the command line is passed along with COMMAND so its args are seen as if they were run manually within the shell of the container.

EXAMPLES

denv is meant to be used after building a containerized developer environment. Look at the online manual for help getting started on developing the environment itself, but for these examples, we will assume that you already have an image built in which you wish to develop.

Basic Start-Up

First, we go into the directory that holds the code we wish to develop and tell denv that this workspace should be running a specific image for its developer environment.

denv init myuser/myrepo:mytag

Then we can open a shell in the denv.

denv

Now you can build and run programs from within the denv with its solidified set of software and tools while still editing the code files themselves with whatever text editor you wish outside of the denv. The init command produces a configuration file .denv/config which you can share between users and so it is excluded from the default .gitignore generated within .denv. All other files within .denv are internal to denv and can only be modified at your own risk.

SEE ALSO

denv-init(1), denv-config(1), denv-check(1)

ENVIRONMENT

denv tests the definition and reads the value of a few different environment variables - allowing the user to modify its behavior in an advanced way without having to provide many command line arguments.

DENV_DEBUG if set, enable xtrace in denv so the user can see exactly what commands are being run.

DENV_INFO if set, print progress information updates to terminal while denv is running

DENV_RUNNER set to the container manager command you wish denv to use. This should only be used in the case where multiple managers are installed and you wish to override the default denv behavior of using the first runner that it finds available.

DENV_NOPROMPT disable all user prompting. This makes the following decisions in the places where there would be prompts.

DENV_TAB_COMMANDS a space-separated list of commands to include in tab-completions of denv. This is helpful if there are a set of common commands you use within the denv.

SCRIPTING

denv has a shebang subcommand that can be used to construct a script to be run by a certain program within a constructed denv. denv’s shebang consists of several lines all beginning with the shebang signal characters #!. It begins with a normal Unix shebang. /usr/bin/env is used to avoid having to type the full path to denv and -S is used so the whitespace between denv and shebang is respected (i.e. split).

#!/usr/bin/env -S denv shebang

The following lines of the script file can then contain the configuration of the denv. This can be done in two ways. If you already have a denv workspace that you want to run inside of, you can just specify that

#!denv_workspace=/full/path/to/workspace

If you don’t have a workspace, then you will need to define the configuration of the denv. At minimum, you must inform denv which image it should be running.

#!denv_image=python:3

For singularity or apptainer runners, you need to pre-build this image since, without a workspace, denv doesn’t know where it should put the intermediary image file.

#!denv_image=/full/path/to/image.sif

Other denv configuration options can be specified in this running mode as well. The easiest way to see the options is to inspect the output of denv config print which will contain the options not related to environment variables. A full listing of available options is given by any config file written by denv into a .denv directory for a workspace. When running from a workspace (i.e. when providing denv_workspace within the shebang lines), the other options are ignored in favor of reading them from the workspace configuration.

The last line of the shebang lines is then the program that will be run with the file and the rest of the command line. This program is run within the denv so it does not need to reside on the host system. The path does not need to even be a full path like with the normal unix shebang. The following examples hope to give some more context for how to get started with denv’s shebang.

Workspace Example

#!/usr/bin/env -S denv shebang
#!denv_workspace=/full/path/to/workspace
#!program
script for program

Workspace-Less Example (singularity or apptainer)

#!/usr/bin/env -S denv shebang
#!denv_image=/full/path/to/image.sif
#!program
script for program

Workspace-Less Example (other runners)

#!/usr/bin/env -S denv shebang
#!denv_image=owner/repo:tag
#!program
script for program

RUNNER DEDUCTION

denv does not persist what runner is being used inside of its configuration for a specific workspace. This is done intentionally so that configurations could be shared across machines that may rely on different runners; however, this could lead to confusion if denv is being used on a machine that has multiple runners installed. In this case, it is highly suggested to use denv check and test-run the different runners to see which are capable of being used by denv.

# lists which runners it supports and which ones it has found
denv check

A simple test would be to make sure denv can open a shell in some ubuntu image. Check different runners by using the environment variable DENV_RUNNER.

denv init ubuntu:22.04
# run this for each of the runners "found" by denv check
DENV_RUNNER=<runner> denv

If any of the runners do not work (i.e. open an interactive bash terminal), please make a bug report by opening an issue for further investigation. However, there are some configurations of popular container runners that denv does not intend to support, so you may be forced to use a specific container runner out of the ones installed. In this case, it is highly recommended to define the DENV_RUNNER environment variable in your ~/.bashrc (or equivalent) to avoid complication.

Automatic Deduction

denv does make some attempts to avoid this complexity by having an automatic choosing behavior that prefers runners that are more likely to be configured properly. For this reason, denv chooses to prefer runners that act as emulators over the runners they are emulating (for example, podman is checked before docker and apptainer is checked before singularity). In addition, since the configuration of podman on some computing clusters is not supportive of denv and apptainer is installed on these clusters, apptainer is checked before podman. This leads to the following order of priority currently within denv when DENV_RUNNER is not defined.

FILES

This part of the manual is an attempt to list and explain the files within a .denv directory.

config

The file storing the configuration of the denv related to this workspace. While it is plain-text and you can edit it directly. Editing it with the denv config set of commands is helpful for doing basic typo- and existence- checking. The config file is a basic key=value shell file that will be sourced by denv. See the FILES section of denv-config(1) for more detail.

skel-init

This is an empty file that, if it exists, signals to the entrypoint executable that the files from /etc/skel have been copied into the denv home directory. This prevents accidental overwriting of files that the user may edit as well as saving time when starting up the container.

images

This is a directory that holds any image files that may be generated by the runner denv is using to run the container. For some runners, it is helpful to explicitly build an image outside of the cache directory and then run that image file. This directory holds those images. It can be deleted if the user wishes to reclaim some disk space; however, that means any image that are configured to be used by denv will then be re-downloaded and re-built.

DENV

NAME
SYNOPSIS
OPTIONS
ARGUMENTS
EXAMPLES
DEFAULT CONFIGURATION
SEE ALSO

NAME

denv init

SYNOPSIS

denv init [help|-h|--help] IMAGE [WORKSPACE] [--no-gitignore] [--clean-env|--no-copy-all] [--force] [--name NAME] [--no-mkdir|--mkdir] [--no-over|--over]

OPTIONS

--help, -h, or help print a short help message for denv or one of its sub commands

--no-gitignore do not generate a gitignore file when setting up a new denv configuration

--clean-env or --no-copy-all do not enable copying of all host environment variables within the new denv. Later activation (or deactivation) of copying all host environment variables can be done with denv config env all See denv-config(1) for details on denv config.

--force forces re-initialization of a denv even if the current workspace has one

--name sets the name for the denv workspace that is being initialized to NAME

--[no-]mkdir don’t prompt about if denv can create the workspace directory. Just do it (--mkdir) or not (--no-mkdir).

--[no-]over don’t prompt about if denv should overwrite a configuration within the workspace or override a configuration in a parent directory. Just do it (--over) or not (--no-over).

ARGUMENTS

IMAGE the name of a container image to use when starting a container to host the developer environment

WORKSPACE the directory where the environment should be stored and configured, used by default as the home directory within the developer environment so that the environment can also have its own shell configuration files and ~/.local paths. If not provided, we just use the current working directory. If provided, we make sure it exists, enter it and then continue. If WORKSPACE already has a denv (or one of its parent directories has a denv), then the user is prompted on if they wish to overwrite (in the case that WORKSPACE itself is a denv) or override (otherwise) the already-existing denv.

EXAMPLES

Print the command line help for denv init without making any edits to the filesystem or beginning the process of configuring a new denv.

denv init help

Create a new denv based off the python:3.11 container image within the current directory, allowing all host environment variables to be copied into the denv when running.

denv init python:3.11

Same as above, but do not allow the host environment variables to be copied into the denv.

denv init --clean-env python:3.11

Create a new denv based off the python:3.11 container image and set its name to “py311” rather than the workspace directory’s name.

denv init python:3.11 --name py311

Create a new denv in some other location besides the current directory. Since the directory has the same name as above, the denvs will appear similar even though their workspace directory (on the host) may be different names.

denv init python:3.11 py311

As a standalone program, denv can be used within other scripts and support programs. With this in mind, a common process is to have a denv configuration that can be ensured to exist for the rest of additional tasks. The following example uses the --no-mkdir and --no-over flags to silently ensure that the present working directory has the python:3.11 image configured.

denv init --no-mkdir --no-over python:3.11

WARNING: denv init does not ensure that the image is the same as the one passed, it just quielty refused to overwrite an existing configuration allowing it to be quickly bypassed if the configuration has already been made or initialize the configuration if no configuration is present.

DEFAULT CONFIGURATION

denv init makes the following choices on configuration that can later be edited by denv config if the user desires.

SEE ALSO

denv(1), denv-config(1), denv-check(1)

DENV

NAME
SYNOPSIS
DESCRIPTION
COMMANDS
ARGUMENTS
print
image
mounts
shell
env
EXAMPLES
Sharing Environment Variables
FILES
SEE ALSO

NAME

denv config

SYNOPSIS

denv config [help|-h|–help]

denv config print [env]

denv config image <pull | IMAGE>

denv config mounts DIR0 [DIR1 DIR2 ...]

denv config shell SHELL

denv config network {[yes|on|true]|[no|off|false]}

denv config env [help|-h|–help]

denv config env print

denv config env all [yes|no]

denv config env copy VAR0[=VAL0] [VAR1[=VAL1] ...]

DESCRIPTION

Manipulate the configuration of a denv that already exists. The commands here will all fail if a denv hasn’t been created, see denv-init(1) to create a new denv.

denv config is separated into a set of sub-commands that are focused on manipulating the different aspects of the denv configuration. These correspond to the different keywords specified after ‘config’.

COMMANDS

help print a help message for denv config. This is the command that is issued if no keywords are given. It also has the aliases -h and --help.

print print the loaded denv configuration. This is helpful for debugging purposes and inspecting the denv that you currently have configured.

image set (and potentially pull) the container image that should be used with the denv that you currently reside in.

mounts provide additional directories to mount into the denv during running.

shell set the shell that should be executed by denv if no other command is given by the user.

network enable network connection for the denv (passing yes, on, or true) or disable this network connection (passing no, off, or false).

env manipulate and view the environment variables that will be provided to the denv at run time.

ARGUMENTS

Below are different arguments that can be provided separated by which command they correspond to.

print

env If any argument is provided to print, then all of the environment variables that will be passed into the denv are printed after the deduced configuration. Without an argument, print will just show the deduced configuration and only print the environment variables if copy-all is false.

image

IMAGE The provided argument is the image tag that should be used for running with the denv. If this argument is the special key-word ‘pull’, then it won’t change the actual image tag and instead re-download the currently configured image from the registry.

For apptainer and singularity runners, this IMAGE can also be a filesystem path to a unpacked image that is already downloaded (and unpacked) somewhere else on the computer. In this case, we simply symlink the image to our image cache so denv can operate like normal. Work is on-going to investigate supporting this workflow for other container runners <https://github.com/tomeichlersmith/denv/issues/37>.

mounts

DIR Each of the space-separate arguments are interpreted as a directory that should be included in the list of mounts for the container that denv spawns. These are in addition to the mount of the denv workspace to the container home directory. They are mounted into the container at the same filesystem location that they have on the host. These directories are required to be full paths so that the user is cognizant of what paths will be available in the container and what arent. One can use realpath(1) to deduce a fullpath from a relative path in a POSIX-compliant way if desired.

shell

SHELL the program to use as the interactive shell within the containerized environment. No checks on what this program is or if it is even available within the container are done. As the name implies, denv expects it to be some shell that the user can interact with but technically it is just the default program that is run when the user does not provide any arguments to denv.

env

The env subcommand has its own sub-commands due to the variability of defining which environment variables should be copied into the containerized environment.

help print a help message for this subcommand. This has aliases -h and --help as well.

print print out the environment variables and their values that will be passed into the container.

all toggle the decision on if all possible environment variables from the host environment should be copied. yes, true, on all mean to copy all possible variables from the host environment, while their inverses no, false, off mean to disable this feature and only copy variables that are explicitly defined via the copy command below. Variables defined with a specific value overwrite any values that would be copied from the environment.

copy configure which environment variables to copy into the denv at runtime. Each of the space-separated arguments to this command are treated separated and are interpreted as a VAR with an optional VAL distinguished by a ‘=’ character.

EXAMPLES

Print out the current configuration of the denv.

denv config print

Change the image that the denv should use when running. Be careful. No cleaning or checking of compatibility is done. A drastic enough change in the image may require recompilations or even re-writes of code being written and developed within the denv.

denv config image my-repo/my-image:new-tag

Pull down the image that is currenlty configured again. This is helpful if the denv is using an image tag like “latest” and should be updated to the latest release again. Updating to the latest release is not done automatically because of the warnings above.

denv config image pull

Sharing Environment Variables

The syntax for sharing environment variables with the denv is a bit terse, so it is helpful to display some examples.

By default (without --no-copy-all or --clean-env when running denv init), denv will copy all possible environment variables from the host into the denv. This means one can

export foo=bar
printenv foo      # prints out "bar"
denv printenv foo # also prints "bar"

In some situations, this is over-sharing and you can disable this so that host environment variables are not copied into the denv anymore.

denv config env all no
export foo=bar
printenv foo      # prints out "bar"
denv printenv foo # does not print anything and returns the error code 1

Even with copying all environment variables disabled, one can still copy specific values from the host or set specific variables to have specific values for the denv.

denv config env copy baz myfoo=mybaz
denv printenv myfoo # prints "mybaz"
printenv myfoo      # does not print anything and returns error code 1
denv printenv baz   # not set in host yet so does not print anything
export baz="hooray"
denv printenv baz   # prints "hooray"

FILES

The denv config command is used to safely edit the .denv/config file so that the user does not accidentally break their configuration. Nevertheless, this file is a regular text file and so can be edited directly if the user wishes to do something more advanced that the basic commands described above can handle.

The config file is a basic key=value shell file that will be sourced by denv whenever the configuration is needed. denv assumes that this config file defines the following shell variables for it to use.

denv_name the name for this denv

denv_image the image to use when running the denv

denv_shell the program to run as a interactive shell if running denv without any arguments

denv_mounts a space separated list of extra mounts to mount into denv when running

denv_env_var_copy_all a boolean flag signalling if denv should copy all possible host environment variables into the denv ("true") or not ("false").

denv_env_var_copy a space-separated list of host environment variables to copy into the denv. This is ignored if denv_env_var_copy_all is "true". There are some restrictions on the names of variables that can be used and so editing this value directly is not recomended. Use denv config env copy which does this validation.

denv_env_var_set a space-separate list of key=value pairs that will be set as environment variables within the denv. These values override any values that could be copied from the host. There are restrictions on the names and values that can be kept here so editing this value directly is not recommended. Use denv config env copy to edit this value while validating that the rules are followed.

denv_network a boolean flag signalling if denv should connect the container to the host network ("true") or disable all network connection ("false").

Since denv v1, this configuration is considered stable. Any new configuration options that are desirable to introduce new features will be optional and thus are not required to reside within this file.

SEE ALSO

denv(1), denv-init(1), denv-check(1)

DENV

NAME
SYNOPSIS
OPTIONS
EXIT CODES
SEE ALSO

NAME

denv check

SYNOPSIS

denv check [-h|--help] [-q|--quiet] [-s|--silent] [--workspace]

OPTIONS

--help or -h print a short help message

--quiet or -q suppress non-error output

--silent or -s suppress all output include error messages (i.e. just return the exit code below)

--workspace check to see if denv can deduce a workspace from the current directory

EXIT CODES

denv check follows the POSIX convention of returning a non-zero exit code when a failure condition is encountered.

SEE ALSO

denv(1), denv-config(1), denv-init(1)

Developing the Environment

By "developing the environment" I mean changing the image denv runs in some way. Usually, this means adding a new dependency so that the code can continue to progress; however, it could also mean patching how a certain dependency is configured, where something is installed, or making quality-of-life updates to the environment.

It is important to emphasize once again that denv is not a tool for actually building the images that are used to create development environments. The reasons for this are (in rough order):

1. The separation of developing the environment from developing the code that requires the environment is helpful for isolating complicated dependency issues from the normal developer. The image build context could even be kept in a separate repository in order to enforce this isolation.
2. Avoid build repetition. Many projects I work on have dependencies that take hours to build even on fast multi-core machines. This means it can save many people time if the container image is built once for everyone and then distributed via a registry (like DockerHub).
3. Somes HPCs do not have apptainer/singularity installations that support building from a recipe file. While newer versions support building from a recipe file in a secure environment, I wanted a tool whose interface and user experince can be uniform including these runners and their common limitations.

Now that is out of the way, I have a few suggestions to make about how to develop these environments. Since my ability to build images from a recipe file with singularity or apptainer is restricted and I usually already have docker or podman installed, the images are built using docker (or podman) and then pushed to the registry from which denv can pull them.

Install Locations

Since the image being run by denv has its own root filesystem, it is usually helpful to install dependencies into system locations (like /usr/local) so that downstream projects (either more dependencies or the code that is being developed) can easily find these dependencies.

Environment Variables

Instead of expecting users to correctly write configurations into the .bashrc in their workspace, one can make heavy use of environment variables in the image definition.

In addition, all of the files in /etc/skel from the image are copied into the workspace when denv is run for the first time. This enables the image creators to update those files with any environment tuning that needs to happen at run time. For example, one could deduce where a project is installed and then add that directory to PATH if it isn't one of the standard locations.

apptainer support

Apptainer and singularity take a slightly different approach to running containers than docker or podman and so there are some restrictions on the images you should impose to ease this difference.

Take a look at Docker-Apptainer Compatibility in the apptainer docs to learn more.

distrobox

This is probably the biggest tip. Remember when I said that distrobox inspired denv and denv has a smaller feature set? Well, I think distrobox is a good companion for denv - especially for denv users who wish to develop the environment a bit.

I often open a distrobox with a specific image in order to try adding new dependencies and once that installation process is deduced, I can build the dependency into the image and use it later with denv.

Version Control

Similar to the code being developed, it is generally important to strictly version control the image used to create the denv. This helps keep track of the complicated process of aligning dependencies as well as help users know which dependencies (and the versions of them) they have access to.

How to Contribute

All contributing is helpful! Anything from correcting a spelling mistake in the documentation, adding a new example, patching bugs, or adding features is highly encouraged. Below, I've collected some notes on these various levels of contribution.

Documentation Updates

If you are writing more detailed explanation or adding in a new example, please git clone the repository and make sure the updated documentation can be built into a website by mdbook and has the format you expect.

This website actually has a helpful edit button in the top-right corner that will take you to the file on GitHub to edit and submit a pull request with any updates you wish to suggest. This is especially helpful for smaller updates that don't affect the format of the website pages.

New Examples

As far as I'm concerned, the more the merrier! If you are writing an example, please be detailed about which runner you are using, the version of denv, and how you've configured denv to aid in your workflow.

Patching Bugs or Adding Features

If you find a bug or think of a new feature to add, please open a GitHub Issue to start the discussion. This allows all collaborators to see what you plan to work on as well as potentially offer some insight on how to get going.

Code Contributions

When you work on developing denv make sure to install shellcheck add run the check and test scripts.

./ci/check # uses shellcheck to avoid common issues writing shell scripts
./ci/test <your-runner> # basic functionality checking

The GitHub workflows test all of the currently supported runners, so make sure to enable them in your fork of the repository so that they will test runners that you may not have installed on your system!

If the code being developed is anything larger than an extremely small one-line change, please open an issue and reference the issue number in your branch name. Generally, I like the format <issue_number>-short-title for example 19-connect-net was used when developing network-connection supported related to issue 19.

Version Control

When changing the denv version number, one must change it in three locations.

• denv itself at the top
• install so future pullers will get the latest version
• man/man1/denv.1 so the man page has the new version number

This is annoying to always have to remember to do, so there is a short shell script to do this for you.

./ci/set-version X.Y.Z

Writing tests

denv uses bats to run tests and pins its version (as well as the necessary plugins) using submodules. Look to these resources for the assert_* family of functions and how tests are structured and run.

• bats-core
• bats-assert
• bats-file

Adding a new Runner

I use "runner" and "manager" pretty much inter-changeably on this site because, for the purposes of denv, the difference between the two is not of great importance.

Requirements

There are two main features that a program needs to satisfy in order to be usable by denv as a container-interaction backend.

1. Download Images from a Registry
2. Check if an Image is already Downloaded
3. Run the Image as a Container

Usually container "runners" are specifically focused on doing the last task (and only that one) while "managers" can do all of these tasks and more (like building images, pushing images to a registry, tagging images, etc...).

The main reason I use manager and runner interchangeably here is because we do not use a lot of the features managers provide and so I could easily forsee denv supporting a runner whose download/check mechanism is some curl or wget shell scripting nonsense rather than a call to the runner program itself.

Running the Image

This section expands a bit on the requirements on the runner when running an image. For specifics, one will need to look at the denv source itself to see how the currently-supported runners run images as containers.

denv wishes to integrate the containerized environment with the host environment in several ways so that the user, while developing within a denv, can easily use programs within the denv as if they are installed within the host system. This leads to a few requirements on the program that runs the images.

• User Ownership: any files written when in the denv should be owned by the user once they exit the denv
• Launching GUI Programs: the user should be able to launch a GUI program from within the denv
• Network and Port Connection: the user should be able to use the host network and connect to ports on localhost within the denv
• Home Directory: the in-denv home directory should be set to the workspace outside of the denv
• Environment Variables: the user is able to configure which environment variables to pass into the denv, so the runner needs to be able to define environment variables for the spawned container at runtime

What to Test

If you are developing a new runner to be wrapped by denv, the natural next question to ask is how should I test that it is functional.

First and foremost, make sure your additions to denv still pass the non-interactive tests.

./ci/check
./ci/test <your-runner>

These can be enabled in your fork of denv so that they run automatically on GitHub when you push to a branch on your fork. In order for GitHub to be able to test your runner, you will need to install it into the GitHub runner during the testing workflow (.github/workflows/test.yml) and add it to the runners-to-test.json file in the include list.

Tracking New Releases in Test

I've written up a "dependabot" that will check the latest release of certain GitHub repositories and update the testing workflow with those releases if a new one is found. In order to follow new releases of the runner being added, you must make changes in two places of the CI infrastructure.

1. Add the runner repository (OWNER/REPO) under the track key in its entry in ci/runners-to-test.json.
2. Make sure your installation procedure will depend (and can handle) the version changing.

Right now, this is done for sylabs/singularity and apptainer/apptainer so look to those runners within the testing infrastructure as examples.

Besides the non-interactive tests, there are some additional, manual tests that I haven't figured out how to automate since they check interactions between the host and denv environments.

Make sure GUI Programs can be launched

There is a small image that can be used to test whether GUI programs can be run from within a denv. Launching from a in-denv shell, launching it directly from outside the denv, and launching without sharing environment variables should work properly.

I haven't found a quick and easy way to test this, but look at the test/gui directory for my notes on how I've tested this in the past. As a first pass, you can attempt to run the xeyes program from denv.

DENV_RUNNER=<your-runner> ./test/gui/run-gui-test

Make sure Network and Ports are Connected

My main reason for supporting this is to allow me to interact with a Jupyter Lab instance running from within the denv. The Jupyter Project has built some images with their software installed which can be used for testing.

mkdir net-test
cd net-test
denv init jupyter/datascience-notebook
denv jupyter lab --no-browser

The user should be able to access the localhost link displayed by jupyter. Note: Developers should know that these images have a very special user and entrypoint configuration specialized for jupyter lab which may cause extra complications.

The non-interactive tests do check for network connection within the container by attempting to connect a socket to a public Google DNS server. I do not expect this to be enough to guarantee network functionality especially for new runners that I am not familiar with. Any aid in more precisely testing external network connection as well as "localhost"-style connection would be an appreciated contribution.

Developing in a VM

A full virtual machine (VM) can be very helpful for developing and testing these applications. Oftentimes, developers don't want to have all of the container runners they wish to test on their bare metal system, so it is helpful to learn how to test these options within a VM.

These notes are developed for the Virtual Machine Manager that I use on Linux Mint. From casual browsing, it appears that other VM managers can do similar processes.

Mount Source Code

I don't want to install all of my code editing toolkits into a simple test VM, so I instead mount the directory my source code is in into the VM so that the VM can just see the current code I'm writing on the host.

Credit where credit is due, these notes are just parroting the tutorial given by Arindam on Debugpoint.

On the Host in the VM Manager

1. Make sure the "Enable shared memory" box is checked in the "Memory" section of the VM settings
2. Add a filesystem to the VM
   1. Select "Add Hardware" and choose "Filesystem"
   2. Driver: virtiofs (was the default for me)
   3. Source path: /full/path/to/denv/on/host (can use GUI file manager)
   4. Target path: denvsource (more of a name rather than a path as you'll see below)

Inside the VM

1. Make sure the mount point for the source code exists

mkdir -p denv

2. Mount the virtual filesystem to our mount point

sudo mount -t virtiofs denvsource denv/

3. Do a developer's install so that we can run denv like normal

cd denv/
./install -d