OPAM: A Package Management System for OCaml
Developer Manual (version 1.2.1)
 
 

Thomas GAZAGNAIRE
thomas@gazagnaire.org
Louis GESBERT
louis.gesbert@ocamlpro.com

Contents

Overview

Opam is a source-based package manager for OCaml. It supports multiple simultaneous compiler installations, flexible package constraints, and a Git-friendly development work-flow.
A package management system has typically two kinds of users: end-users who install and use packages for their own projects; and packagers, who create and upload packages. End-users want to install on their machine a consistent collection of packages – a package being a collection of OCaml libraries and/or programs. Packagers want to take a collection of their own libraries and programs and make them available to other developpers.
This document describes the design of Opam to answer both of these needs.

Conventions

In this document, $home, $opam, and $path are assumed to be defined as follows:

User variables are written in capital letters, prefixed by $. For instance package names will be written $NAME, package versions $VERSION, and the version of the ocaml compiler currently installed $SWITCH.
This document is organized as follows: Section 1 describes the core of Opam, e.g. the management of packages. Section 3 describes how repositories are handled, Section 2 focus on compiler switches and finally Section 4 explain how packages can define configuration variables (which can be later used by the build system).

1  Managing Packages

1.1  State

The client state is stored on the filesystem, under $opam. All the configurations files, libraries and binaries related to a specific instance of the OCaml compiler in $opam/$SWITCH, where $SWITCH is the name of that specific compiler instance. See Section 2 for more details about compiler switches.

1.2  Files

1.2.1  General Syntax of Structured Files

Most of the files in the client and server states share the same syntax defined in this section.

Comments
Two kinds of comments are available: the usual (* ... *) OCaml comment blocks and also # which discard everything until the end of the current line.
Base types
The base types for values are:
Compound types
Types can be composed together to build more complex values:

All structured files share the same syntax:

<file>  := <item>*

<item>  := IDENT : <value>
         | ?IDENT: <value>
         | IDENT STRING { <item>+ }

<value> := BOOL
         | INT
         | STRING
         | SYMBOL
         | IDENT
         | [ <value>+ ]
         | value { <value>+ }

1.2.2  Global Configuration File: config

$opam/config follows the syntax defined in §1.2.1 with the following restrictions:

<file> :=
    opam-version: "1.2"
    repositories: [ STRING+ ]
    switch: STRING
    ?jobs: INT
    ?solver: <single-command>
    ?solver-criteria: STRING
    ?solver-upgrade-criteria: STRING
    ?solver-fixup-criteria: STRING
    ?download-command: <single-command>
    ?download-jobs: INT

<single-command> := [ (<argument> ?{ <filter> })+ ]

1.2.3  Package Specification files: opam

$opam/packages/$NAME/$NAME.$VERSION/opam follows the syntax defined in §1.2.1 with the following restrictions:

<file> :=
    opam-version: "1.2"
    ?name:          STRING
    ?version:       STRING
    maintainer:     STRING
    ?authors:       [ STRING+ ]
    ?license:       STRING
    ?homepage:      STRING
    ?doc:           STRING
    ?bug-reports:   STRING
    ?dev-repo:      STRING
    ?tags:          [ STRING+ ]
    ?patches:       [ (STRING ?{ <filter> } )+ ]
    ?substs:        [ STRING+ ]
    ?build:         commands
    ?install:       commands
    ?build-doc:     commands
    ?build-test:    commands
    ?remove:        commands
    ?depends:       [ <and-formula(package-with-flags)> ]
    ?depopts:       [ (STRING ?{ <flags> })+ ]
    ?conflicts:     [ <package>+ ]
    ?depexts:       [ [[STRING+] [STRING+]]+ ]
    ?messages:      [ (STRING ?{ <filter> } )+ ]
    ?post-messages: [ (STRING ?{ <filter> } )+ ]
    ?available:     [ <filter> ]
    ?os:            [ <formula(os)>+ ]
    ?ocaml-version: [ <and-formula(constraint)>+ ]
    ?libraries:     [ STRING+ ]
    ?syntax:        [ STRING+ ]
    ?flags:         [ IDENT+ ]
    ?features:      [ (IDENT STRING <filter>)+ ]

<argument>       := STRING
                  | IDENT

<command>        := [ (<argument> ?{ <filter> })+ ] ?{ <filter> }

<commands>       := <command>
                  | [ <command>+ ]

<filter>         := <argument>
                  | !<argument>
                  | <argument> <comp> <argument>
                  | formula(<filter>)

<formula(x)>     := <formula(x)> '&' <formula(x)>
                  | <formula(x)> '|' <formula(x)>
                  | ( <formula(x)> )
                  | !<formula(x)>
                  | <x>

<package>        := STRING
                  | STRING { <and-formula(constraint)> }

<flags>          := IDENT
                  | IDENT '&' <flags>

<package-with-flags>
                 := STRING
                  | STRING { <flags> }
                  | STRING { ?(<flags> '&') <and-formula(constraint)> }

<constraint>     := <comp> STRING
<comp>           := '=' | '<' | '>' | '>=' | '<=' | '!='

<and-formula(x)> := <x> <and-formula(x)>
                  | <formula(x)>

<or-formula(x)>  := <x> <or-formula(x)>
                  | <package(x)>

<os>             := STRING
                  | '!' STRING

1.2.4  URL files: url

The syntax of url files follows the one described in §1.2.1 with the following restrictions:

<file> :=
    ?src:       STRING
    ?archive:   STRING
    ?http:      STRING
    ?local:     STRING
    ?git:       STRING
    ?darcs:     STRING
    ?hg:        STRING
    ?mirrors:   [ STRING+ ]
    ?checksum:  STRING

src, archive, http, local, git, hg, darcs are the location where the package upstream sources can be downloaded. It can be one of:

mirrors, if specified, is assumed to be a list of addresses of the same kind as the primary address. Mirrors will be tried in order in case the download from the primary upstream fails.

1.3  Commands

1.3.1  Creating a Fresh Client State

When an end-user starts Opam for the first time, he needs to initialize $opam/ in a consistent state. In order to do so, he should run:

    $ opam init [--kind $KIND] $REPO $ADDRESS [--comp $VERSION]

Where:

This command will:

  1. Create the file $opam/config (as specified in §1.2.2)
  2. Create an empty $opam/$SWITCH/installed file, $SWITCH is the version from the OCaml used to compile $opam. In particular, we will not fail now if there is no ocamlc in $path.
  3. Initialize $opam/repo/$REPO by running the appropriate operations (depending on the repository kind).
  4. Copy all Opam and description files (ie. copy every file in $opam/repo/$REPO/packages/ to $opam/packages/).
  5. Create $opam/repo/package-index and for each version $VERSION of package $NAME appearing in the repository, append the line '$REPO $NAME $VERSION' to the file.
  6. Create the empty directories $opam/archives, $opam/$SWITCH/lib/, $opam/$SWITCH/bin/ and $opam/$SWITCH/doc/.

1.3.2  Listing Packages

When an end-user wants to have information on all available packages, he should run:

    $ opam list

This command will parse $opam/$SWITCH/installed to know the installed packages, and $opam/packages/$NAME/$NAME.$VERSION/opam to get all the available packages. It will then build a summary of each packages. The description of each package will be read in $opam/packages/$NAME/$NAME.$VERSION/descr if it exists.

For instance, if batteries version 1.1.3 is installed, ounit version 2.3+dev is installed and camomille is not installed, then running the previous command should display:

    batteries   1.1.3  Batteries is a standard library replacement
    ounit     2.3+dev  Test framework
    camomile       --  Unicode support

1.3.3  Getting Package Info

In case the end-user wants a more details view of a specific package, he should run:

    $ opam info $NAME

This command will parse $opam/$SWITCH/installed to get the installed version of $NAME, will process $opam/repo/index to get the repository where the package comes from and will look for $opam/packages/$NAME/$NAME.$VERSION/opam to get available versions of $NAME. It can then display:

    package: $NAME
    version: $VERSION
    versions: $VERSION1, $VERSION2, ...
    libraries: $LIB1, $LIB2, ...
    syntax: $SYNTAX1, $SYNTAX2, ...
    repository: $REPO
    description:
      $SYNOPSIS

      $LINE1
      $LINE2
      $LINE3
      ...

1.3.4  Installing a Package

When an end-user wants to install a new package, he should run:

    $ opam install $NAME

This command will:

  1. Compute the transitive closure of dependencies and conflicts of packages using the dependency solver (see §1.3.8). If the dependency solver returns more than one answer, the tool will ask the user to pick one, otherwise it will proceed directly. The dependency solver should also mark the packages to recompile.
  2. The dependency solver sorts the collections of packages in topological order. Then, for each of them do:
    1. Check whether the package is already installed by looking for the line $NAME $VERSION in $opam/$SWITCH/installed. If not, then:
    2. Look into the archive cache to see whether it has already been downloaded. The cache location is: $opam/archives/$NAME.VERSION.tar.gz
    3. If not, process $opam/repo/index/ (see 3) to get the repository $REPO where the archive is available and then ask the repository to download the archive if necessary.

      Once this is done, the archive might become available in $opam/repo/$REPO/archives/. It it is, copy it in the global state (in $opam/archives). If it is not, download it upstream by looking at $opam/packages/$NAME/$NAME.$VERSION/url and add the optional overlay files located in $opam/packages/$NAME/$NAME.$VERSION/files. Note: this files can be overwritten by anything present in $opam/$SWITCH/overlay/$NAME/.

    4. Decompress the archive into $opam/$SWITCH/build/$NAME.$VERSION/.
    5. Substitute the required files.
    6. Run the list of commands to build the package with $opam/$SWITCH/bin in the path.
    7. Process $opam/$SWITCH/build/$NAME.$VERSION/$NAME.install to install the created files. The file format is described in §2.2.3.
    8. Install the installation file $opam/$SWITCH/build/$NAME.$VERSION/$NAME.install in $opam/$SWITCH/install/ and the configuration file $opam/$SWITCH/build/$NAME.$VERSION/$NAME.config in $opam/$SWITCH/config/.

1.3.5  Updating Index Files

When an end-user wants to know what are the latest packages available, he will write:

    $ opam update

This command will follow the following steps:

1.3.6  Upgrading Installed Packages

When an end-user wants to upgrade the packages installed on his host, he will write:

    $ opam upgrade

This command will:

1.3.7  Removing Packages

When the user wants to remove a package, he should write:

    $ opam remove $NAME

This command will check whether the package $NAME is installed, and if yes, it will display to the user the list packages that will be uninstalled (ie. the transitive closure of all forward-dependencies). If the user accepts the list, all the packages should be uninstalled, and the client state should be let in a consistent state.

1.3.8  Dependency Solver

Dependency solving is a hard problem and we do not plan to start from scratch implementing a new SAT solver. Thus our plan to integrate (as a library) the Debian depency solver for CUDF files, which is written in OCaml.

2  Managing Compiler Switches

Opam is able to manage concurrent compiler installations.

2.1  State

Compiler descriptions are stored in two files:

Switch-related meta-data are stored under $opam/$SWITCH/:

2.2  Files

2.2.1  Package List: installed and reinstall

The following configuration files: $opam/$SWITCH/installed, $opam/$SWITCH/reinstall, and $opam/repo/$REPO/updated follow a very simple syntax. The file is a list of lines which contains a space-separated name and a version. Each line $NAME $VERSION means that the version $VERSION of package $NAME has been compiled with the compiler instance $SWITCH and has been installed on the system in $opam/$SWITCH/lib/$NAME and $opam/$SWITCH/bin/.
For instance, if batteries version 1.0+beta and ocamlfind version 1.2 are installed, then $opam/$SWITCH/installed will contain:

batteries 1.0+beta
ocamlfind 1.2

2.2.2  Compiler Description Files: comp

The syntax of comp files follows the one described in §1.2.1 with the following restrictions:

<file> :=
    opam-version: "1.2"
    name:       STRING
    ?src:       STRING
    ?archive:   STRING
    ?http:      STRING
    ?local:     STRING
    ?git:       STRING
    ?darcs:     STRING
    ?hg:        STRING
    ?make:      STRING+ ]
    ?build:     [[STRING+]]
    ?patches:   [ STRING+ ]
    ?configure: [ STRING+ ]
    ?packages:  <package>+
    ?preinstalled: BOOL
    ?env:       [ <env>+ ]

<env>    := IDENT <eq> STRING
<eq>     := '=' | '+=' | '=+' | ':=' | '=:'

For instance the file, 3.12.1+memprof.comp describes OCaml, version 3.12.1 with the memory profiling patch enabled:

opam-version: "1.2"
name:         "3.12.1"
src:          "http://caml.inria.fr/pub/distrib/ocaml-3.12/ocaml-3.12.1.tar.gz"
make:         [ "world" "world.opt" ]
patches:      [ "http://bozman.cagdas.free.fr/documents/ocamlmemprof-3.12.0.patch" ]
env:          [ CAML_LD_LIBRARY_PATH = "%{lib}%/stublibs" ]

And the file trunk-g-notk-byte.comp describes OCaml from SVN trunk, with no tk support and only in bytecode, and all the libraries built with -g:

opam-version: "1.2"
name:         "trunk-g-notk-byte"
src:          "http://caml.inria.fr/pub/distrib/ocaml-3.12/ocaml-3.12.1.tar.gz"
configure:    [ "-no-tk" ]
make:         [ "world" ]
bytecomp:     [ "-g" ]
bytelink:     [ "-g" ]
env:          [ CAML_LD_LIBRARY_PATH = "%{lib}%/stublibs" ]

2.2.3  Package installation files: *.install

$opam/$SWITCH/install/NAME.install follows the syntax defined in §1.2.1 with the following restrictions:

<file> :=
    ?lib:        [ <mv>+ ]
    ?libexec:    [ <mv>+ ]
    ?bin:        [ <mv>+ ]
    ?sbin:       [ <mv>+ ]
    ?toplevel:   [ <mv>+ ]
    ?share:      [ <mv>+ ]
    ?share_root: [ <mv>+ ]
    ?etc:        [ <mv>+ ]
    ?doc:        [ <mv>+ ]
    ?misc:       [ <mv>+ ]
    ?stublibs:   [ <mv>+ ]
    ?man:        [ <mv>+ ]

<mv> := STRING
      | STRING { STRING }

General remarks:

2.2.4  Pinned Packages: pinned

$opam/$SWITCH/pinned contains a list of lines of the form:

<name> <kind> <path>

2.3  Commands

2.3.1  Switching Compiler Version

If the user wants to switch to another compiler version, he should run:

    $ opam switch [-alias-of $COMPILER] $SWITCH

This command will:

3  Managing Repositories

3.1  State

Configuration files for the remote repository REPO are stored in $opam/repo/$REPO. Repositories can be of different kinds (stored on the local filesystem, available via HTTP, stored under git, …); they all share the same filesystem hierachy, which is updated by different operations, depending on the repository kind.

3.2  Files

3.2.1  Index of packages

$opam/repo/index follows a very simple syntax: each line of the file contains a space separated list of words $NAME.$VERSION $REPO+ specifying that all the version $VERSION of package $NAME is available in the remote repositories $REPO+. The file contains information on all available packages (e.g. not only on the installed one).
For instance, if batteries version 1.0+beta is available in the testing repository and ocamlfind version 1.2 is available in the default and testing repositories (where default is one being used), then $opam/repo/index will contain:

batteries.1.0+beta testing
ocamlfind.1.2 default testing

3.3  Commands

3.3.1  Managing Remote Repository

An user can manage remote repositories, by writing:

    $ opam repository list # 'opam repository' works as well
    $ opam repository add [--kind $KIND] $REPO $ADRESS
    $ opam repository remove $REPO

4  Managing Configurations

4.1  State

4.2  Variables

Variables can appear in some fields of opam files (see §1.2.3), and in substition files (see below). There are three kinds. Note that this list is often out of date, you should check opam config list for a complete overview.

  1. Global variables, corresponding to the current configuration
  2. Package-local variables. Variable $VAR of package $NAME is normally accessed using %{$NAME:$VAR}%, which can be abbreviated as %{$VAR}% within $NAME’s own opam file (but global variables take precedence, in particular directory names. Check opam config list). Some are only available from the opam file or when the corresponding package is installed. Boolean variables can also be written as foo+bar:var as a shortcut to foo:var & bar:var.
  3. User-defined variables, as defined in *.config files (see §??). This can also be used to override values of the above variables.

As an experimental feature in Opam 1.2.1 (i.e. not to be used on the main package repository), converters of boolean variables to strings can be specified with the following syntax: "%{var?string-if-true:string-if-false-or-undefined}%". Either string may be empty.

4.3  Files

4.3.1  Substitution files: *.in

Any file can be processed using generated using a special mode of opam which can perform tests and substitutes variables (see §4 for the exact command to run). Substitution files contains some templates which will be replaced with some contents. The syntax of templates is the following:

Local and global variables

The definitions “IDENT: BOOL”, “IDENT: STRING” and “IDENT: [ STRING+ ]”, are used to defined variables associated to this package, and are used to substitute variables in template files (see §4.4.1):

4.4  Commands

4.4.1  Getting Package Configuration

Opam contains the minimal information to be able to use installed libraries. In order to do so, the end-user (or the packager) should run:

    $ opam config list
    $ opam config var $NAME:$VAR
    $ opam config var $NAME.$LIB:$VAR
    $ opam config subst $FILENAME+

This document was translated from LATEX by HEVEA.