Feedback on this post is welcomed on Discuss!
We are happy to announce the first beta release of opam 2.2.0. It contains some fixes and a new feature for Windows. You can view the full list in the release note.
This version is a beta, we invite users to test it to spot previously unnoticed bugs as we head towards the stable release.
Opam 2.2 is based on a Cygwin installation (previously installed, or managed
internally by opam). Cygwin's Git has three known usability issues when used
outside a Cygwin environment: it uses a different set of trusted certificate
authorities, has its own Credential Manager and potentially uses a separate Git
configuration. We therefore recommend using Git for
Windows either installed manually or via winget
.
At opam init
, opam checks for available Git(s), and asks the user to choose
one of the available, or to rerun opam init
after installing another Git.
libstdc++
instead of requiring the DLL to be distributed alongside it or present in the environmentopam env
containing carriage return on Cygwin - eval $(opam env)
now works from a Cygwin bash terminalpwsh
and cmd
opam env output./configure --enable-static
to compile the opam binary statically on Linuxopam tree
options --dev
and --no-switch
Various other improvements were made and bugs were fixed. API changes are denoted in the release note linked above. This release also includes PRs improving the documentation and improving and extending the tests.
The main opam-repository Windows compliance is still a work in progress, we recommend to use existing compatible repository (originally from @fdopen) and 32/64 bit mingw-w64 packages (by @dra27).
If you're feeling adventurous, you can use the experimental pre-built binary for Windows available here.
Otherwise you can compile opam yourself using the following steps:
This beta requires a preexisting Cygwin installation for compiling opam.
autoconf
, make
, patch
, curl
mingw64-x86_64-gcc-g++
, mingw64-i686-gcc-g++
make cold
cmd
or PowerShell
), and initialise opam with the Windows sunset repository:opam init git+https://github.com/ocaml-opam/opam-repository-mingw
From here, you can try to install the sunset repository packages. If you find any bugs, please submit an issue. It will help opam-repository maintainers to add Windows repository packages into the main repository.
In case you plan a possible rollback, you may want to first backup your
~/.opam
directory.
The upgrade instructions are unchanged:
Either from binaries: run
bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.2.0~beta1"
or download manually from the Github "Releases" page to your PATH.
Or from source, manually: see the instructions in the README.
You should then run:
opam init --reinit -ni
Please report any issues to the bug-tracker.
Thanks for trying this new release out, and we hope you will enjoy the new features!
Feedback on this post is welcomed on Discuss!
We are happy to announce the third alpha release of opam 2.2.0. It contains some fixes and a new feature for Windows. You can view the full list in the release note.
This version is an alpha, we invite users to test it to spot previously unnoticed bugs as we head towards the stable release.
opam files now support a new x-env-path-rewrite
field which
specifies rewriting rules for the environment variable updates defined in
the setenv
and build-env
fields. This field allows greater control over the
separator character used for PATH-like fields (i.e. ;
vs :
), conversion of
slashes to backslashes, and even conversion from Windows native path format
(C:\cygwin64\home\dra\.opam
) to Cygwin format (/home/dra/.opam
).
The rewriting rules allow opam directory-like variables (e.g. the %{lib}%
directory
of a switch) to be used in setenv
and build-env
fields in a portable way.
For example, given:
setenv: [
[PKG_CONFIG_PATH += "%{lib}%/pkgconfig"]
[PATH += "%{share}%/bin"]
]
x-env-path-rewrite: [
[ PKG_CONFIG_PATH ":" "host" ]
[ PATH (":" {os != "win32"} | ";" {os = "win32"}) ("target" {os != "win32"} | "target-quoted" {os = "win32"}) ]
]
the environment variable changes given by opam env
on Windows would be:
PKG_CONFIG_PATH='/cygdrive/c/Users/DRA/AppData/Local/opam/default/lib/pkgconfig[:<rest-of-PKG_CONFIG_PATH, if given>]'
PATH='C:\Users\DRA\AppData\Local\opam\default\share\bin;C:\Users\DRA\AppData\Local\opam\default\bin;<rest-of-PATH>'
with the following interesting parts for Windows users:
PKG_CONFIG_PATH
, which is consumed by a Cygwin-tool, has the directory given in Unix-like syntax, and opam's share
variable was automatically converted:
for PKG_CONFIG_PATH
, ;
for PATH
is used when adding entries)PATH
update, /bin
was converted to \bin
Note that the specification for PATH
is opam's default behaviour, so it's not actually necessary to have this formula for PATH
in the x-env-path-rewrite
field.
The full syntax is described in full in the manual.
Opam files carrying this new field are still compatible with the opam 2.0 syntax as it is an extension field, however its effect is only available with opam 2.2.0~alpha3 and above.
If you want to make sure users of the package containing it have a compatible opam, you can use the available
field to that end:
available: opam-version >= "2.2.0~alpha3"
or, if the change is Windows-specific:
available: opam-version >= "2.2.0~alpha3" | os != "win32"
/tmp
is now writable again, restoring POSIX complianceopam tree package.version
is now supported, displaying the dependency tree of a specific version of a package--recurse
and --subpath
are supported for directory argumentsadd-extrafiles
command to add/check/update the extra-files:
field according to the files present in the files/
directory-W @1..9
OPAMCURL
, OPAMFETCH
and OPAMVERBOSE
environment variables were fixed--assume-built
argument were fixedVarious other improvements were made and bugs were fixed. API changes are denoted in the release note linked above. This release also includes a handful of PRs improving the documentation and more than a dozen PRs improving and extending the tests.
The main opam-repository Windows compliance is still a work in progress, we recommend to use existing compatible repository (originally from @fdopen) and 32/64 bit mingw-w64 packages (by @dra27).
If you're feeling adventurous, you can use the experimental pre-built binary for Windows available here. It should work but since it's our first public pre-built binary for Windows please use at your own risk.
Otherwise you can compile opam yourself using the following steps:
This alpha requires a preexisting Cygwin installation for compiling opam.
autoconf
, make
, patch
, curl
mingw64-x86_64-gcc-g++
, mingw64-i686-gcc-g++
make cold
libc
dlls are in your PATH
and launch make cold
. For instance: export PATH='C:\cygwin64\usr\x86_64-w64-mingw32\sys-root\mingw\bin':$PATH && make cold
. Don’t forget to update PATH
accordingly or place the opam
binary in the same directory as the libc
dlls if you want to move the resulting opam binary.make cold CONFIGURE_ARGS=--with-private-runtime
. If you change the location of the resulting opam binary, don't forget to copy Opam.Runtime.amd64
directory (or Opam.Runtime.i386
) in the same directory.cmd
or PowerShell
), and initialise opam with the Windows sunset repository:opam init git+https://github.com/ocaml-opam/opam-repository-mingw
From here, you can try to install the sunset repository packages. If you find any bugs, please submit an issue. It will help opam-repository maintainers to add Windows repository packages into the main repository.
In case you plan a possible rollback, you may want to first backup your
~/.opam
directory.
The upgrade instructions are unchanged:
Either from binaries: run
bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.2.0~alpha3"
or download manually from the Github "Releases" page to your PATH.
Or from source, manually: see the instructions in the README.
You should then run:
opam init --reinit -ni
Please report any issues to the bug-tracker.
Thanks for trying this new release out, and we hope you will enjoy the new features!
Feedback on this post is welcomed on Discuss!
We are happy to announce the second alpha release of opam 2.2.0. It contains some fixes and a new feature for Windows. You can view the full list in the release note.
This version is an alpha, we invite users to test it to spot previously unnoticed bugs to head towards the stable release.
The first alpha came with native Windows compatibility. This second alpha comes with a more simple init for Windows: we no longer rely on an already present Cygwin UNIX-like environment for Windows as a compatibility layer. During initialisation, opam now proposes to embed its own fully managed Cygwin install.
The main opam-repository Windows compliance is still a work in progress, we recommend to use existing compatible repository (originally from @fdopen) and 32/64 bit mingw-w64 packages (by @dra27).
This alpha requires a preexisting Cygwin installation for compiling opam.
autoconf
, make
, patch
, curl
mingw64-x86_64-gcc-g++
, mingw64-i686-gcc-g++
make cold
cmd
or PowerShell
), and initialise opam with the Windows sunset repository:opam init git+https://github.com/ocaml-opam/opam-repository-mingw
From here, you can try to install the sunset repository packages. If you find any bugs, please submit an issue. It will help opam-repository maintainers to add Windows repository packages into the main repository.
Hint: if you use the MinGW compiler, don't forget to add to your
PATH
the path tolibc
dlls (usuallyC:\cygwin64\usr\x86_64-w64-mingw32\sys-root\mingw\bin
). Or compile opam withmake cold CONFIGURE_ARGS=--with-private-runtime
, and if you change opam location, don't forget to copyOpam.Runtime.amd64
(orOpam.Runtime.i386
) with it.
conflicts:
fieldIn case you plan a possible rollback, you may want to first backup your
~/.opam
directory.
The upgrade instructions are unchanged:
Either from binaries: run
bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.2.0~alpha2"
or download manually from the Github "Releases" page to your PATH.
Or from source, manually: see the instructions in the README.
You should then run:
opam init --reinit -ni
Please report any issues to the bug-tracker.
Thanks for trying this new release out, and we hope you will enjoy the new features!
Feedback on this post is welcomed on Discuss!
We are happy to announce the alpha release of opam 2.2.0. It contains numerous fixes, enhancements, and updates; including much-improved Windows support, addressing one of the most important pain points identified by the OCaml community. You can view the full list of changes in the release note.
This alpha release is a significant milestone, brought together by Raja Boujbel after years of work from the opam dev team (Raja Boujbel, David Allsopp, Kate Deplaix, Louis Gesbert, in a united OCamlPro/Tarides collaboration) with the help of many community contributors. We also thank Jane Street for their continued sponsorship.
This version is an alpha, so we invite users to test it to spot previously unnoticed bugs and work towards a stable release.
Opam 2.2 comes with native Windows compatibility. You can now use opam from your preferred Windows terminal! We rely on the Cygwin UNIX-like environment for Windows as a compatibility layer, but it is possible for a package to generate native executables.
The main opam repository is not Windows compatible at the moment, but existing
work on a compatible
repository (originally
from @fdopen) and 32/64 bit mingw-w64
packages (by
@dra27) is in the process of being merged. Before
the final release, we expect it to be possible to run opam init
and use the
main opam-repository for Windows.
This alpha requires a preexisting Cygwin installation. Support for full management of a local Cygwin environment inside of opam (so that it's as transparent as possible) is queued already and should be available in 2.2.0~alpha2 as the default option.
autoconf
, make
, patch
, curl
mingw64-x86_64-gcc-g++
, mingw64-i686-gcc-g++
make cold
opam init git+https://github.com/ocaml-opam/opam-repository-mingw
From here, you can try to install sunset repository packages. If any bug is found, please submit an issue. It will help opam repository maintainers to add Windows repository packages into the main repository.
Hint: if you use the MinGW compiler, don't forget to add to your
PATH
the path tolibc
dlls (usuallyC:\cygwin64\usr\x86_64-w64-mingw32\sys-root\mingw\bin
). Or compile opam withmake cold CONFIGURE_ARGS=--with-private-runtime
, and if you change opam location, don't forget to copyOpam.Runtime.amd64
(orOpam.Runtime.i386
) with it.
When installing or pinning a package using opam install
or opam pin
, opam
normally only looks for opam files at the root of the installed package. With
recursive pinning, you can now instruct opam to also look for .opam
files in
subdirectories, while maintaining the correct relationship between the .opam
files and the package root for versioning and build purposes.
Recursive pinning is used with the following options to opam pin
and opam install
:
--recursive
, opam will look for .opam
files recursively in all subdirectories.--subpath <path>
, opam will only look for .opam
files in the subdirectory <path>
.The two options can be combined: for instance, if your opam packages are stored
as a deep hierarchy in the mylib
subdirectory of your project, give opam pin
. --recursive --subpath mylib
a try!
You can use these options with opam pin
, opam install
, and opam remove
.
$ tree .
.
├── ba
│ └── z
│ └── z.opam
├── bar
│ └── bar.opam
└── foo.opam
$ opam pin . --subpath ba/z --no-action
Package z does not exist, create as a NEW package? [y/n] y
z is now subpath-pinned to directory /ba/z in git+file:///tmp/recpin#master (version 0.1)
$ opam pin --recursive . --no-action
This will pin the following packages: foo, z, bar. Continue? [y/n] y
foo is now pinned to git+file:///tmp/recpin#master (version 0.1)
Package z does not exist, create as a NEW package? [y/n] y
z is now subpath-pinned to directory /ba/z in git+file:///tmp/recpin#master (version 0.1)
Package bar does not exist, create as a NEW package? [y/n] y
bar is now subpath-pinned to directory /bar in file:///tmp/recpin (version 0.1)
$ opam pin
bar.0.1 (uninstalled) rsync directory /bar in file:///tmp/recpin
foo.0.1 (uninstalled) git git+file:///tmp/recpin#master
z.0.1 (uninstalled) git directory /ba/z in git+file:///tmp/recpin#master
$ opam pin . --recursive --subpath ba/ --no-action
Package z does not exist, create as a NEW package? [y/n] y
z is now subpath-pinned to directory /ba/z in git+file:///tmp/recpin#master (version 0.1)
opam tree
shows packages and their dependencies with a tree view. It is very
helpful to determine which packages bring which dependencies in your installed
switch.
$ opam tree cppo
cppo.1.6.9
├── base-unix.base
├── dune.3.8.2 (>= 1.10)
│ ├── base-threads.base
│ ├── base-unix.base [*]
│ └── ocaml.4.14.1 (>= 4.08)
│ ├── ocaml-base-compiler.4.14.1 (>= 4.14.1~ & < 4.14.2~)
│ └── ocaml-config.2 (>= 2)
│ └── ocaml-base-compiler.4.14.1 (>= 4.12.0~) [*]
└── ocaml.4.14.1 (>= 4.02.3) [*]
It can also display a reverse-dependency tree (through opam why
, which is an
alias to opam tree --rev-deps
). This is useful to examine how dependency
versions get constrained.
$ opam why cmdliner
cmdliner.1.2.0
├── (>= 1.1.0) b0.0.0.5
│ └── (= 0.0.5) odig.0.0.9
├── (>= 1.1.0) ocp-browser.1.3.4
├── (>= 1.0.0) ocp-indent.1.8.1
│ └── (>= 1.4.2) ocp-index.1.3.4
│ └── (= version) ocp-browser.1.3.4 [*]
├── (>= 1.1.0) ocp-index.1.3.4 [*]
├── (>= 1.1.0) odig.0.0.9 [*]
├── (>= 1.0.0) odoc.2.2.0
│ └── (>= 2.0.0) odig.0.0.9 [*]
├── (>= 1.1.0) opam-client.2.2.0~alpha
│ ├── (= version) opam.2.2.0~alpha
│ └── (= version) opam-devel.2.2.0~alpha
├── (>= 1.1.0) opam-devel.2.2.0~alpha [*]
├── (>= 0.9.8) opam-installer.2.2.0~alpha
└── user-setup.0.7
Special thanks to @cannorin for contributing this feature.
There is now a way for a project maintainer to share their project development
tools: the with-dev-setup
dependency flag. It is used in the same way as
with-doc
and with-test
: by adding a {with-dev-setup}
filter after a
dependency. It will be ignored when installing normally, but it's pulled in when the
package is explicitely installed with the --with-dev-setup
flag specified on
the command line. The variable is also resolved in the post-messages:
field
to allow maintainers to share more informations about that setup.
This is typically useful for tools that are required for bootstrapping or regenerating artifacts.
For example
opam-version: "2.0"
depends: [
"ocaml"
"dune"
"ocp-indent" {with-dev-setup}
]
build: [make]
install: [make "install"]
post-messages:
[ "Thanks for installing the package"
"and its tool dependencies too, it will help for your futur PRs" {with-dev-setup} ]
Software Heritage is a project that aims to archive all software source code in existence. This is done by collecting source code with a loader that uploads software source code to the Software Heritage distributed infrastructure. From there, any project/version is available via the search webpage and via a unique identifier called the SWHID. Some OCaml source code is already archived, and the main opam and Coq repository packages are continuously uploaded.
Opam now integrates a fallback to Software Heritage archive retrieval, based on SWHID. If an SWHID URL is present in an opam file, the fallback can be activated.
To keep backwards compatibility of opam files, we added a specific Software
Heritage URL syntax to the url.mirrors:
field, which is used to specify
mirrors of the main URL. Opam 2.2.+ understands this specific syntax as a
Software Heritage fallback URL: https://swhid.opam.ocaml.org/<SWHID>
.
url {
src: "https://faili.ng/url.tar.gz"
checksum: "sha512=e2146c1d7f53679fd22df66c9061b5ae4f8505b749513eedc67f3c304f297d92e54f5028f40fb5412d32c7d7db92592eacb183128d2b6b81d10ea716b7496eba"
mirrors: [
"https//failli.ng/mirror.tar.gz"
"https://swhid.opam.ocaml.org/swh:1:dir:9f2be900491e1dabfc027848204ae01aa88fc71d"
]
}
To add a Software Heritage fallback URL to your package, use the
swhid
library. Specifically the
Compute.directory_identifier_deep
function:
Compute.directory_identifier_deep
. You can use this oneliner in the directory:
ocaml -e '#use "topfind";; #require "digestif.ocaml";; #require "swhid";; Swhid_core.Object.pp Format.std_formatter (Result.get_ok (Swhid.Compute.directory_identifier_deep "."))'
Special thanks to @zapashcanon for collaborating on this feature.
It is now possible to leverage the full expressivity of package dependency formulas from the command line during switch creation and package operations.
It is possible to create a switch using a formula. For example, with
ocaml-variant
or ocaml-system
, excluding ocaml-base-compiler
:
opam switch create ocaml --formula '"ocaml-variants" {>= "4.14.1"} | "ocaml-system"'
This syntax is brought to install commands. For example, while installing a
package, let's say genet
, you can specify that you want to install either
conf-mariadb & mariadb
or conf-postgresql
:
opam install genet --formula '["mysql" ("conf-mariadb" & "mariadb" | "conf-postgresql")]'
Here are several of new options (possibly scripts breaking changes are marked with ✘):
opam pin --current
to fix a package to its current state (disabling pending
reinstallations or removals from the repository). The installed package will
be pinned with the opam file that is stored in opam internal state, the one
that is currently installed.
opam pin remove --all
to remove all the pinned packages from a switch.
opam pin remove pkg.version
now removes the pins on pinned pkg.version
.
opam exec --no-switch
to remove opam environment from launched command.
$ export FOOVAR=env
$ opam show foo --field setenv
FOOVAR = "package"
$ opam exec -- env | grep "OPAM_SWITCH\|FOO"
FOOVAR=package
OPAM_SWITCH_PREFIX=~/.opam/env
$ opam exec --no-switch -- env | grep "OPAM_SWITCH\|FOO"
FOOVAR=env
opam source --no-switch
to allow downloading package sources without having
an installed switch (instead of failing).
opam clean --untracked
to remove untracked files interactively remaining
from previous packages removal.
opam switch -
, inspired from git switch -
, that goes back to the previously
selected global switch.
opam admin add-constraint <cst> --packages pkg1,pkg2,pkg3
to select
a subset of packages to apply constraints.
✘ Change --base
into --invariant
. opam switch
compiler column now
contains installed packages that verifies invariant formula, and empty
synopsis shows switch invariant.
$ opam switch create inv --formula '["ocaml" {>= "4.14.1"} "dune"]'
$ opam switch invariant
["ocaml" {>= "4.14.1"} "dune"]
$ opam list --invariant
# Packages matching: invariant
# Name # Installed # Synopsis
dune 3.8.2 Fast, portable, and opinionated build system
ocaml 5.0.0 The OCaml compiler (virtual package)
$ opam switch list
# switch compiler description
→ inv ocaml-base-compiler.5.0.0,ocaml-options-vanilla.1 ocaml >= 4.14.1 & dune
In case you plan a possible rollback, you may want to first backup your
~/.opam
directory.
The upgrade instructions are unchanged:
From binaries: run
bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.2.0~alpha"
Or download manually from the Github "Releases" page to your PATH.
From source, manually: see the instructions in the README.
Then run:
opam init --reinit -ni
Please report any issues to the bug-tracker.
Thanks for trying this new release out, and we're hoping you will enjoy the new features!
Feedback on this post is welcomed on Discuss!
We are pleased to announce the patch release of [opam 2.1.5]((https://github.com/ocaml/opam/releases/tag/2.1.5).
This opam release consists of backported bug & security fixes:
archive-mirrors
field from init config file to config (#5315)From binaries: run
bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.1.5"
or download manually from the Github "Releases" page to your PATH. In this case, don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script.
From source, using opam:
opam update; opam install opam-devel
(then copy the opam binary to your PATH as explained, and don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update your sandbox script)
From source, manually: see the instructions in the README.
We hope you enjoy this new minor version, and remain open to bug reports and suggestions.
We are pleased that opam 2.1.5 has hit the road, since it contains a security fix since the local cache is considered trusted, and not all checksums are verified.
Opam uses a download cache since 2.0.0: if a source artifact is needed, first its hash is looked up in the local cache (~/.opam/download-cache/<hash-algorihm>/<hash>
). Opam supports multiple hash algorithms, a cache lookup goes through all hash algorithms present in the opam file to build, unpack, or install. Before 2.1.5, the hash algorithm that lead to the cache hit was not checked (but all others were).
If a package specifies only a single (non-weak) hash algorithm, this lead to the source artifact taken as is, any error while writing the artifact into the cache, or reading it from the cache, was not detected. Also, in certain setups, if the download cache is shared (writable) across containers (for example in some CI systems), this leads to the possibility of cache poisoning.
Thanks to Raja and Kate, the issue was fixed in PR 5538
The timeline of this issue is as follows:
Why we are interested in the security of opam? We plan to improve the supply chain security for opam, and develop conex. While developing conex, we validate the assumptions conex uses about opam. We did that in February 2023 with opam 2.1.2, and reported the missed assumptions to the opam development team.
As methodology we used a blackbox-testing, i.e. we used the opam binary for installing packages from a custom opam repository. We did not study the source code of opam extensively, neither did we exhaustively check opam: we only tested the assumptions that came to our mind while working on conex.
The collaborative work with the opam development team was great, they were open to our reporting and answered quickly. We have no knowledge of this issue being exploited in the wild - on some systems, the opam sandbox avoids writing to the download-cache from within the opam file.
Feedback on this post is welcomed on Discuss!
We are pleased to announce the minor release of opam 2.1.4, opam-publish 2.2.0 and opam-file-format 2.1.5.
This opam release consists of backported fixes:
dose3 >= 6.1
and base64 >= 3.1.0
are now required (#5357)The opam-publish release consists of the following breaking change:
and the following fixes:
The opam-file-format release consists of the following fixes:
From binaries: run
bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.1.4"
or download manually from the Github "Releases" page to your PATH. In this case, don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script.
From source, using opam:
opam update; opam install opam-devel
(then copy the opam binary to your PATH as explained, and don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update your sandbox script)
From source, manually: see the instructions in the README.
We hope you enjoy this new minor version, and remain open to bug reports and suggestions.
Feedback on this post is welcomed on Discuss!
We are pleased to announce the minor release of opam 2.1.3.
This opam release consists of backported fixes:
opam init
and opam init --reinit
when the jobs
variable has been set in the opamrc or the current config. (#5056)opam var
no longer fails if no switch is set (#5025)--switch <sw>
fails instead of writing an invalid switch-config
file (#5027)opam pin
(#5046)and improvements:
opam list --installable
queries combined with other filters (#4311)--available
, --installable
) (#4999)opam list --conflicts-with
when combined with other filters (#4999)opam show
by as much as 300% when the package to show is given explicitly or is unique (#4997)(#4172)You can also find API changes in the release note.
From binaries: run
bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.1.3"
or download manually from the Github "Releases" page to your PATH. In this case, don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script.
From source, using opam:
opam update; opam install opam-devel
(then copy the opam binary to your PATH as explained, and don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update your sandbox script)
From source, manually: see the instructions in the README.
We hope you enjoy this new minor version, and remain open to bug reports and suggestions.
Feedback on this post is welcomed on Discuss!
We are pleased to announce the minor release of opam 2.1.2.
This opam release consists of backported fixes, including:
dnf
if yum
does not exist on RHEL-based systems (#4825)--no-depexts
in CLI 2.0 mode. This further improves the use of opam 2.1 as a drop-in replacement for opam 2.0 in CI, for example with setup-ocaml in GitHub Actions. (#4908)From binaries: run
bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.1.2"
or download manually from the Github "Releases" page to your PATH. In this case, don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script.
From source, using opam:
opam update; opam install opam-devel
(then copy the opam binary to your PATH as explained, and don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update your sandbox script)
From source, manually: see the instructions in the README.
We hope you enjoy this new minor version, and remain open to bug reports and suggestions.
Feedback on this post is welcomed on Discuss!
We are pleased to announce several minor releases: opam 2.0.10, opam 2.1.1, and opam-depext 1.2.
The opam releases consist of backported fixes, while opam-depext
has been adapted to be compatible with opam 2.1, to allow for workflows which need to maintain compatibility with opam 2.0. With opam 2.1.1, if you export OPAMCLI=2.0
into your environment then workflows expecting opam 2.0 should now behave even more equivalently.
Previous versions of opam-depext were made unavailable when using opam 2.1, since depext handling is done directly by opam 2.1 and later. This is still the recommended way, but this left workflows which wanted to maintain compatibility with opam 2.0 without a single command to install depexts. You can now run OPAMCLI=2.0 opam depext -y package1 package2
and expect this to work correctly with any version of opam 2. If you don't specify OPAMCLI=2.0
then the plugin will remind you that you should be using the integrated depext support! Calling opam depext
this way with opam 2.1 and later still exposes the same double-solving problem that opam 2.0 has, but if for some reason the solver returns a different solution at opam install
then the correct depexts would be installed.
For opam 2.0, some useful depext handling changes are back-ported from opam 2.1.x to the plugin: With opam 2.0:
opam 2.1.1 includes both the fixes in opam 2.0.10.
General fixes:
--packages
option for CLI version 2.0, e.g. OPAMCLI=2.0 opam switch create . 4.12.0+options --packages=ocaml-option-flambda
. In opam 2.1 and later, this syntax remains an error (#4843)opam switch set-invariant
: default repositories were loaded instead of the switch's repositories selection (#4869)Integrated depext support has a few updates:
And finally two regressions have been dealt with:
Unix.environment
on load (as a toplevel expression). This regression affected opam's libraries, rather than the binary itself (#4789)A few issues with the compilation of opam from sources have been fixed as well (e.g. mingw-w64 with g++ 11.2 now works)
Two subtle fixes are included in opam 2.0.10. These actually affect the ocaml
package. Both of these are Heisenbugs - investigating what's going wrong on your system may well have fixed them, they were both found on Windows!
$(opam env --revert)
is the reverse of the more familiar $(opam env)
but it's effectively called by opam whenever you change switch. It has been wrong since 2.0.0 for the case where several values are added to an environment variable in one setenv
update. For example, if a package included a setenv
field of the form [PATH += "dir1:dir2"]
, then this would not be reverted, but [[PATH += "dir1"] [PATH += "dir2"]]
would be reverted. As it happens, this bug affects the ocaml
package, but it was masked by another setenv
update in the same package.
The other fix is also to do with setenv
. It can be seen immediately after creating a switch but before any additional packages are installed, as this Dockerfile
shows:
FROM ocaml/opam@sha256:244b948376767fe91e2cd5caca3b422b2f8d332f105ef2c8e14fcc9a20b66e25
RUN sudo apt-get install -y ocaml-nox
RUN opam --version
RUN opam switch create show-issue ocaml-system
RUN eval $(opam env) ; echo $CAML_LD_LIBRARY_PATH
RUN opam install conf-which
RUN eval $(opam env) ; echo $CAML_LD_LIBRARY_PATH
Immediately after switch creation, $CAML_LD_LIBRARY_PATH
was set to /home/opam/.opam/show-issue/lib/stublibs:
, rather than /home/opam/.opam/show-issue/lib/stublibs:/usr/local/lib/ocaml/4.08.1/stublibs:/usr/lib/ocaml/stublibs
From binaries: run
bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.1.1"
or download manually from the Github "Releases" page to your PATH. In this case, don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script.
From source, using opam:
opam update; opam install opam-devel
(then copy the opam binary to your PATH as explained, and don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update your sandbox script)
From source, manually: see the instructions in the README.
We hope you enjoy this new minor version, and remain open to bug reports and suggestions.
Feedback on this post is welcomed on Discuss!
We are happy to announce the release of opam 2.1.0.
Many new features made it in (see the pre-release changelogs or release notes for the details), but here are a few highlights.
option
and expanded var
sub-commands)opam has long included the ability to install system dependencies automatically via the depext plugin. This plugin has been promoted to a native feature of opam 2.1.0 onwards, giving the following benefits:
opam depext
, opam always checks
depexts (there are options to disable this or automate it for CI use).
Installation of an opam package in a CI system is now as easy as opam install
.
, without having to do the dance of opam pin add -n/depext/install
. Just
one command now for the common case!opam depext
stage and a
different solution for the opam install
stage, resulting in some depexts
missing.opam install mysql
will offer to install conf-mysql
and mysql
, but if you
have the MariaDB dev libraries installed, opam will offer to install
conf-mariadb
and mysql
.Hint: You can set OPAMCONFIRMLEVEL=unsafe-yes
or
--confirm-level=unsafe-yes
to launch non interactive system package commands.
When opam was first released, it had the mission of gathering together scattered OCaml source code to build a community repository. As time marches on, the size of the opam repository has grown tremendously, to over 3000 unique packages with over 19500 unique versions. opam looks at all these packages and is designed to solve for the best constraints for a given package, so that your project can keep up with releases of your dependencies.
While this works well for libraries, we need a different strategy for projects
that need to test and ship using a fixed set of dependencies. To satisfy this
use-case, opam 2.0.0 shipped with support for using project.opam.locked
files. These are normal opam files but with exact versions of dependencies. The
lock file can be used as simply as opam install . --locked
to have a
reproducible package installation.
With opam 2.1.0, the creation of lock files is also now integrated into the client:
opam lock
will create a .locked
file for your current switch and project,
that you can check into the repository.opam switch create . --locked
can be used by users to reproduce your
dependencies in a fresh switch.This lets a project simultaneously keep up with the latest dependencies (without lock files) while providing a stricter set for projects that need it (with lock files).
Hint: You can export the full configuration of a switch with opam switch
export
new options, --full
to have all packages metadata included, and
--freeze
to freeze all VCS to their current commit.
In opam 2.0, when a switch is created the packages selected are put into the “base” of the switch. These packages are not normally considered for upgrade, in order to ease pressure on opam's solver. This was a much bigger concern early on in opam 2.0's development, but is less of a problem with the default mccs solver.
However, it's a problem for system compilers. opam would detect that your
system compiler version had changed, but be unable to upgrade the ocaml-system
package unless you went through a slightly convoluted process with
--unlock-base
.
In opam 2.1, base packages have been replaced by switch invariants. The switch
invariant is a package formula which must be satisfied on every upgrade and
install. All existing switches' base packages could just be expressed as
package1 & package2 & package3
etc. but opam 2.1 recognises many existing
patterns and simplifies them, so in most cases the invariant will be
"ocaml-base-compiler" {= "4.11.1"}
, etc. This means that opam switch create
my_switch ocaml-system
now creates a switch invariant of "ocaml-system"
rather than a specific version of the ocaml-system
package. If your system
OCaml package is updated, opam upgrade
will seamlessly switch to the new
package.
This also allows you to have switches which automatically install new point releases of OCaml. For example:
opam switch create ocaml-4.11 --formula='"ocaml-base-compiler" {>= "4.11.0" & < "4.12.0~"}' --repos=old=git+https://github.com/ocaml/opam-repository#a11299d81591
opam install utop
Creates a switch with OCaml 4.11.0 (the --repos=
was just to select a version
of opam-repository from before 4.11.1 was released). Now issue:
opam repo set-url old git+https://github.com/ocaml/opam-repository
opam upgrade
and opam 2.1 will automatically offer to upgrade OCaml 4.11.1 along with a rebuild of the switch. There's not yet a clean CLI for specifying the formula, but we intend to iterate further on this with future opam releases so that there is an easier way of saying “install OCaml 4.11.x”.
Hint: You can set up a default invariant that will apply for all new switches,
via a specific opamrc
. The default one is ocaml >= 4.05.0
Configuring opam is not a simple task: you need to use an opamrc
at init
stage, or hack global/switch config file, or use opam config var
for
additional variables. To ease that step, and permit a more consistent opam
config tweaking, a new command was added : opam option
.
For example:
opam option download-jobs
gives the global download-jobs
value (as it
exists only in global configuration)opam option jobs=6 --global
will set the number of parallel build
jobs opam is allowed to run (along with the associated jobs
variable)opam option depext-run-commands=false
disables the use of sudo
for
handling system dependencies; it will be replaced by a prompt to run the
installation commandsopam option depext-bypass=m4 --global
bypass m4
system package check
globally, while opam option depext-bypass=m4 --switch myswitch
will only
bypass it in the selected switchThe command opam var
is extended with the same format, acting on switch and
global variables.
Hint: to revert your changes use opam option <field>=
, it will take its
default value.
A new --cli
switch was added to the first beta release, but it's only now
that it's being widely used. opam is a complex enough system that sometimes bug
fixes need to change the semantics of some commands. For example:
opam show --file
needed to change behaviouropam config
was becoming cluttered and some things want to move to opam var
opam switch install 4.11.1
still works in opam 2.0, but it's really an OPAM
1.2.2 syntax.Changing the CLI is exceptionally painful since it can break scripts and tools
which themselves need to drive opam
. CLI versioning is our attempt to solve
this. The feature is inspired by the (lang dune ...)
stanza in dune-project
files which has allowed the Dune project to rename variables and alter
semantics without requiring every single package using Dune to upgrade their
dune
files on each release.
Now you can specify which version of opam you expected the command to be run
against. In day-to-day use of opam at the terminal, you wouldn't specify it,
and you'll get the latest version of the CLI. For example: opam var --global
is the same as opam var --cli=2.1 --global
. However, if you issue opam var
--cli=2.0 --global
, you will told that --global
was added in 2.1 and so is
not available to you. You can see similar things with the renaming of opam
upgrade --unlock-base
to opam upgrade --update-invariant
.
The intention is that --cli
should be used in scripts, user guides (e.g. blog
posts), and in software which calls opam. The only decision you have to take is
the oldest version of opam which you need to support. If your script is using
a new opam 2.1 feature (for example opam switch create --formula=
) then you
simply don't support opam 2.0. If you need to support opam 2.0, then you can't
use --formula
and should use --packages
instead. opam 2.0 does not have the
--cli
option, so for opam 2.0 instead of --cli=2.0
you should set the
environment variable OPAMCLI
to 2.0
. As with all opam command line
switches, OPAMCLI
is simply the equivalent of --cli
which opam 2.1 will
pick-up but opam 2.0 will quietly ignore (and, as with other options, the
command line takes precedence over the environment).
Note that opam 2.1 sets OPAMCLI=2.0
when building packages, so on the rare
instances where you need to use the opam
command in a package build:
command (or in your build system), you must specify --cli=2.1
if you're
using new features.
Since 2.1.0~rc2, CLI versioning applies to opam environment variables. The previous behavior was to ignore unknown or wrongly set environment variable, while now you will have a warning to let you know that the environment variable won't be handled by this version of opam.
To ensure not breaking compatibility of some widely used deprecated options,
a default CLI is introduced: when no CLI is specified, those deprecated
options are accepted. It concerns opam exec
and opam var
subcommands.
There's even more detail on this feature in our wiki. We're hoping that this feature will make it much easier in future releases for opam to make required changes and improvements to the CLI without breaking existing set-ups and tools.
Note: For opam libraries users, since 2.1 environment variable are no more loaded by the libraries, only by opam client. You need to load them explicitly.
opam root format changes during opam life-cycle, new field are added or removed, new files are added ; an older opam version sometimes can no longer read an upgraded or newly created opam root. opam root format has been updated to allow new versions of opam to indicate that the root may still be read by older versions of the opam libraries. A plugin compiled against the 2.0.9 opam libraries will therefore be able to read information about an opam 2.1 root (plugins and tools compiled against 2.0.8 are unable to load opam 2.1.0 roots). It is a read-only best effort access, any attempt to modify the opam root fails.
Hint: for opam libraries users, you can safely load states with
OpamStateConfig
load functions.
Tremendous thanks to all involved people, who've developed, tested & retested, helped with issue reports, comments, feedback...
In case you plan a possible rollback, you may want to first backup your
~/.opam
directory.
The upgrade instructions are unchanged:
Either from binaries: run
bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.1.0"
or download manually from the Github "Releases" page to your PATH.
Or from source, manually: see the instructions in the README.
You should then run:
opam init --reinit -ni
Feedback on this post is welcomed on Discuss!
We are pleased to announce the minor release of opam 2.0.9.
This new version contains some back-ported fixes.
OPAM_USER_PATH_RO
for adding a custom read-only directory to the sandbox [#4589, #4609]OPAMROOT
and OPAMSWITCH
now reflect the --root
and --switch
parameters in the package build [#4668]$TMPDIR
read-only, then sets the sandbox $TMPDIR
to a new separate tmpfs. Hardcoded /tmp
access no longer works if TMPDIR
points to another directory [#4589]DUNE_CACHE
in the sandbox script [#4535, fixing ocaml/dune#4166]PWD
read-write on remove actions [#4589]conf
[#4549]--compiler
when creating local switches [#4718]^
syntax to fix support for Fish 3.3.0+ [#4736]From binaries: run
bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.0.9"
or download manually from the Github "Releases" page to your PATH. In this case, don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script.
From source, using opam:
opam update; opam install opam-devel
(then copy the opam binary to your PATH as explained, and don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update your sandbox script)
From source, manually: see the instructions in the README.
We hope you enjoy this new minor version, and remain open to bug reports and suggestions.
Feedback on this post is welcomed on Discuss!
The opam team has great pleasure in announcing opam 2.1.0~rc2!
The focus since beta4 has been preparing for a world with more than one released version of opam (i.e. 2.0.x and 2.1.x). The release candidate extends CLI versioning further and, under the hood, includes a big change to the opam root format which allows new versions of opam to indicate that the root may still be read by older versions of the opam libraries. A plugin compiled against the 2.0.9 opam libraries will therefore be able to read information about an opam 2.1 root (plugins and tools compiled against 2.0.8 are unable to load opam 2.1.0 roots).
Please do take this release candidate for a spin! It is available in the Docker images at ocaml/opam on Docker Hub as the opam-2.1 command (or you can sudo ln -f /usr/bin/opam-2.1 /usr/bin/opam
in your Dockerfile
to switch to it permanently). The release candidate can also be tested via our installation script (see the wiki for more information).
Thank you to anyone who noticed the unannounced first release candidate and tried it out. Between tagging and what would have been announcing it, we discovered an issue with upgrading local switches from earlier alpha/beta releases, and so fixed that for this second release candidate.
Assuming no showstoppers, we plan to release opam 2.1.0 next week. The improvements made in 2.1.0 will allow for a much faster release cycle, and we look forward to posting about the 2.2.0 plans soon!
In case you plan a possible rollback, you may want to first backup your
~/.opam
directory.
The upgrade instructions are unchanged:
Either from binaries: run
bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.1.0~rc2"
or download manually from the Github "Releases" page to your PATH.
Or from source, manually: see the instructions in the README.
You should then run:
opam init --reinit -ni
We hope there won't be any, but please report any issues to the bug-tracker. Thanks for trying it out, and hoping you enjoy!
Feedback on this post is welcomed on Discuss!
On behalf of the opam team, it gives me great pleasure to announce the third beta release of opam 2.1. Don’t worry, you didn’t miss beta3 - we had an issue with a configure script that caused beta2 to report as beta3 in some instances, so we skipped to beta4 to avoid any further confusion!
We encourage you to try out this new beta release: there are instructions for doing so in our wiki. The instructions include taking a backup of your ~/.opam
root as part of the process, which can be restored in order to wind back. Please note that local switches which are written to by opam 2.1 are upgraded and will need to be rebuilt if you go back to opam 2.0. This can either be done by removing _opam
and repeating whatever you use in your build process to create the switch, or you can use opam switch export switch.export
to backup the switch to a file before installing new packages. Note that opam 2.1 shouldn’t upgrade a local switch unless you upgrade the base packages (i.e. the compiler).
option
and expanded var
sub-commands)In opam 2.0, when a switch is created the packages selected are put into the “base” of the switch. These packages are not normally considered for upgrade, in order to ease pressure on opam’s solver. This was a much bigger concern early on in opam 2.0’s development, but is less of a problem with the default mccs solver.
However, it’s a problem for system compilers. opam would detect that your system compiler version had changed, but be unable to upgrade the ocaml-system package unless you went through a slightly convoluted process with --unlock-base
.
In opam 2.1, base packages have been replaced by switch invariants. The switch invariant is a package formula which must be satisfied on every upgrade and install. All existing switches’ base packages could just be expressed as package1 & package2 & package3
etc. but opam 2.1 recognises many existing patterns and simplifies them, so in most cases the invariant will be "ocaml-base-compiler" {= 4.11.1}
, etc. This means that opam switch create my_switch ocaml-system
now creates a switch invariant of "ocaml-system"
rather than a specific version of the ocaml-system
package. If your system OCaml package is updated, opam upgrade
will seamlessly switch to the new package.
This also allows you to have switches which automatically install new point releases of OCaml. For example:
opam switch create ocaml-4.11 --formula='"ocaml-base-compiler" {>= "4.11.0" & < "4.12.0~"}' --repos=old=git+https://github.com/ocaml/opam-repository#a11299d81591
opam install utop
Creates a switch with OCaml 4.11.0 (the --repos=
was just to select a version of opam-repository from before 4.11.1 was released). Now issue:
opam repo set-url old git+https://github.com/ocaml/opam-repository
opam upgrade
and opam 2.1 will automatically offer to upgrade OCaml 4.11.1 along with a rebuild of the switch. There’s not yet a clean CLI for specifying the formula, but we intend to iterate further on this with future opam releases so that there is an easier way of saying “install OCaml 4.11.x”.
opam has long included the ability to install system dependencies automatically via the depext plugin. This plugin has been promoted to a native feature of opam 2.1.0 onwards, giving the following benefits:
opam depext
, opam always checks depexts (there are options to disable this or automate it for CI use). Installation of an opam package in a CI system is now as easy as opam install .
, without having to do the dance of opam pin add -n/depext/install
. Just one command now for the common case!opam depext
stage and a different solution for the opam install
stage, resulting in some depexts missing.opam install mysql
will offer to install conf-mysql
and mysql
, but if you have the MariaDB dev libraries installed, opam will offer to install conf-mariadb
and mysql
.When opam was first released, it had the mission of gathering together scattered OCaml source code to build a community repository. As time marches on, the size of the opam repository has grown tremendously, to over 3000 unique packages with over 18000 unique versions. opam looks at all these packages and is designed to solve for the best constraints for a given package, so that your project can keep up with releases of your dependencies.
While this works well for libraries, we need a different strategy for projects that need to test and ship using a fixed set of dependencies. To satisfy this use-case, opam 2.0.0 shipped with support for using project.opam.locked
files. These are normal opam files but with exact versions of dependencies. The lock file can be used as simply as opam install . --locked
to have a reproducible package installation.
With opam 2.1.0, the creation of lock files is also now integrated into the client:
opam lock
will create a .locked
file for your current switch and project, that you can check into the repository.opam switch create . --locked
can be used by users to reproduce your dependencies in a fresh switch.This lets a project simultaneously keep up with the latest dependencies (without lock files) while providing a stricter set for projects that need it (with lock files).
A new --cli
switch was added to the first beta release, but it’s only now that it’s being widely used. opam is a complex enough system that sometimes bug fixes need to change the semantics of some commands. For example:
opam show --file
needed to change behaviouropam config
was becoming cluttered and some things want to move to opam var
opam switch install 4.11.1
still works in opam 2.0, but it’s really an OPAM 1.2.2 syntax.Changing the CLI is exceptionally painful since it can break scripts and tools which themselves need to drive opam
. CLI versioning is our attempt to solve this. The feature is inspired by the (lang dune ...)
stanza in dune-project
files which has allowed the Dune project to rename variables and alter semantics without requiring every single package using Dune to upgrade their dune
files on each release.
Now you can specify which version of opam you expected the command to be run against. In day-to-day use of opam at the terminal, you wouldn’t specify it, and you’ll get the latest version of the CLI. For example: opam var --global
is the same as opam var --cli=2.1 --global
. However, if you issue opam var --cli=2.0 --global
, you will told that --global
was added in 2.1 and so is not available to you. You can see similar things with the renaming of opam upgrade --unlock-base
to opam upgrade --update-invariant
.
The intention is that --cli
should be used in scripts, user guides (e.g. blog posts), and in software which calls opam. The only decision you have to take is the oldest version of opam which you need to support. If your script is using a new opam 2.1 feature (for example opam switch create --formula=
) then you simply don’t support opam 2.0. If you need to support opam 2.0, then you can’t use --formula
and should use --packages
instead. opam 2.0 does not have the --cli
option, so for opam 2.0 instead of --cli=2.0
you should set the environment variable OPAMCLI
to 2.0
. As with all opam command line switches, OPAMCLI
is simply the equivalent of --cli
which opam 2.1 will pick-up but opam 2.0 will quietly ignore (and, as with other options, the command line takes precedence over the environment).
Note that opam 2.1 sets OPAMCLI=2.0
when building packages, so on the rare instances where you need to use the opam
command in a package build:
command (or in your build system), you must specify --cli=2.1
if you’re using new features.
There’s even more detail on this feature in our wiki. We’re still finalising some details on exactly how opam
behaves when --cli
is not given, but we’re hoping that this feature will make it much easier in future releases for opam to make required changes and improvements to the CLI without breaking existing set-ups and tools.
opam install
now has a --download-only
flag (#4036), allowing opam’s caches to be primedopam init
now advises the correct shell-specific command for eval $(opam env)
(#4427)post-install
hooks are now allowed to modify or remove installed files (#4388)opamfile-loc
with the location of the installed package opam file (#4402)opam update
now has --depexts
flag (#4355), allowing the system package manager to update tooos-family=ubuntu
is now treated as os-family=debian
(#4441)opam lint
now checks that strings in filtered package formulae are booleans or variables (#4439)and many other bug fixes as listed on the release page.
Several features that were formerly plugins have been integrated into opam 2.1.0. We have also developed some new plugins that satisfy emerging workflows from the community and the core OCaml team. They are available for use with the opam 2.1 beta as well, and feedback on them should be directed to the respective GitHub trackers for those plugins.
The opam compiler
plugin can be used to create switches from various sources such as the main opam repository, the ocaml-multicore fork, or a local development directory. It can use Git tag names, branch names, or PR numbers to specify what to install.
Once installed, these are normal opam switches, and one can install packages in them. To iterate on a compiler feature and try opam packages at the same time, it supports two ways to reinstall the compiler: either a safe and slow technique that will reinstall all packages, or a quick way that will just overwrite the compiler in place.
The opam monorepo
plugin lets you assemble standalone dune workspaces with your projects and all of their opam dependencies, letting you build it all from scratch using only Dune and OCaml. This satisfies the “monorepo” workflow which is commonly requested by large projects that need all of their dependencies in one place. It is also being used by projects that need global cross-compilation for all aspects of a codebase (including C stubs in packages), such as the MirageOS unikernel framework.
This is anticipated to be the final beta in the 2.1 series, and we will be moving to release candidate status after this. We could really use your help with testing this release in your infrastructure and projects and let us know if you run into any blockers. If you have feature requests, please also report them on our issue tracker -- we will be planning the next release cycle once we ship opam 2.1.0 shortly.
We are pleased to announce the minor release of opam 2.0.8.
This new version contains some backported fixes:
.
to PATH
. [#4078]ccache
versions. [#4079 and #4087]~/.cache
is a symlink. [#4068]opam init
. [#4020 & #4092]/tmp
read-write, regardless of TMPDIR
[#3742, addressing ocaml/opam-repository#13339]pre-
and post-session
hooks can now print to the console [#4359]opam-installer
now correctly builds from sources [#4173]arch
variable detection when using 32bit mode on ARM64 and i486 [#4462]A more complete release note is available.
From binaries: run
bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.0.8"
or download manually from the Github "Releases" page to your PATH. In this case, don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script.
From source, using opam:
opam update; opam install opam-devel
(then copy the opam binary to your PATH as explained, and don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script)
From source, manually: see the instructions in the README.
We hope you enjoy this new minor version, and remain open to bug reports and suggestions.
NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com, and published in discuss.ocaml.org. Please head to the latter for the comments!
We are happy to announce a alpha for opam 2.1.0, one year and a half in the making after the release of 2.0.0.
Many new features made it in (see the complete changelog or release note for the details), but here are a few highlights of this release.
The two following features have been around for a while as plugins and are now completely integrated in the core of opam. No extra installs needed anymore, and a more smooth experience.
A number of opam packages depend on tools or libraries installed on the system, which are out of the scope of opam itself. Previous versions of opam added a specification format, and opam 2.0 already handled checking the OS and extracting the required system package names.
However, the workflow generally involved letting opam fail once, then installing the dependencies and retrying, or explicitely using the opam-depext plugin, which was invaluable for CI but still incurred extra steps.
With opam 2.1.0, depexts are seamlessly integrated, and you basically won't have to worry about them ahead of time:
sudo
, or just run the provided commands yourself.This is all fully configurable, and can be bypassed without tricky commands when you need it (e.g. when you compiled a dependency yourself).
To share a project for development, it is often necessary to be able to reproduce the exact same environment and dependencies setting — as opposed to allowing a range of versions as opam encourages you to do for releases.
For some reason, most other package managers call this feature "lock files".
Opam can handle those in the form of [foo.]opam.locked
files, and the
--locked
option.
With 2.1.0, you no longer need a plugin to generate these files: just running
opam lock
will create them for existing opam
files, enforcing the exact
version of all dependencies (including locally pinned packages).
If you check-in these files, new users would just have run
opam switch create . --locked
on a fresh clone to get a local switch ready to
build the project.
This one is completely new: fans of the Monorepo rejoice, opam is now able to handle projects in subtrees of a repository.
opam pin PROJECT_ROOT --subpath SUB_PROJECT
, opam will look for
PROJECT_ROOT/SUB_PROJECT/foo.opam
. This will behave as a pinning to
PROJECT_ROOT/SUB_PROJECT
, except that the version-control handling is done
in PROJECT_ROOT
.opam pin PROJECT_ROOT --recursive
to automatically lookup all sub-trees
with opam files and pin them.Previous versions of opam defined switches based on base packages, which typically included a compiler, and were immutable. Opam 2.1.0 instead defines them in terms of an invariant, which is a generic dependency formula.
This removes a lot of the rigidity opam switch
commands had, with little
changes on the existing commands. For example, opam upgrade ocaml
commands are
now possible; you could also define the invariant as ocaml-system
and have
its version change along with the version of the OCaml compiler installed
system-wide.
The new opam option
command allows to configure several options,
without requiring manual edition of the configuration files.
For example:
opam option jobs=6 --global
will set the number of parallel build
jobs opam is allowed to run (along with the associated jobs
variable)opam option depext-run-commands=false
disables the use of sudo
for
handling system dependencies; it will be replaced by a prompt to run the
installation commands.The command opam var
is extended with the same format, acting on switch and
global variables.
In case you plan a possible rollback, you may want to first backup your
~/.opam
directory.
The upgrade instructions are unchanged:
Either from binaries: run
bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.1.0~alpha"
or download manually from the Github "Releases" page to your PATH.
Or from source, manually: see the instructions in the README.
You should then run:
opam init --reinit -ni
This is still a alpha, so a few glitches or regressions are to be expected. Please report them to the bug-tracker. Thanks for trying it out, and hoping you enjoy!
NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com. Please head to the latter for the comments!
We are pleased to announce the minor release of opam 2.0.7.
This new version contains backported small fixes:
From binaries: run
bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.0.7"
or download manually from the Github "Releases" page to your PATH. In this case, don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script.
From source, using opam:
opam update; opam install opam-devel
(then copy the opam binary to your PATH as explained, and don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script)
From source, manually: see the instructions in the README.
We hope you enjoy this new minor version, and remain open to bug reports and suggestions.
NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com. Please head to the latter for the comments!
We are pleased to announce the minor release of opam 2.0.6.
This new version contains some small backported fixes and build update:
As sandbox scripts have been updated, don't forget to run opam init --reinit -ni
to update yours.
Note: To homogenise macOS name on system detection, we decided to keep
macos
, and convertdarwin
tomacos
in opam. For the moment, to not break jobs & CIs, we keep uploadingdarwin
&macos
binaries, but from the 2.1.0 release, onlymacos
ones will be kept.
From binaries: run
bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.0.6"
or download manually from the Github "Releases" page to your PATH. In this case, don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script.
From source, using opam:
opam update; opam install opam-devel
(then copy the opam binary to your PATH as explained, and don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script)
From source, manually: see the instructions in the README.
We hope you enjoy this new minor version, and remain open to bug reports and suggestions.
NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com. Please head to the latter for the comments!
We are pleased to announce the minor release of opam 2.0.5.
This new version contains build update and small fixes:
Note: To homogenise macOS name on system detection, we decided to keep
macos
, and convertdarwin
tomacos
in opam. For the moment, to not break jobs & CIs, we keep uploadingdarwin
&macos
binaries, but from the 2.1.0 release, onlymacos
ones will be kept.
From binaries: run
sh <(curl -sL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)
or download manually from the Github "Releases" page to your PATH. In this case, don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script.
From source, using opam:
opam update; opam install opam-devel
(then copy the opam binary to your PATH as explained, and don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script)
From source, manually: see the instructions in the README.
We hope you enjoy this new minor version, and remain open to bug reports and suggestions.
NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com. Please head to the latter for the comments!
We are pleased to announce the release of opam 2.0.4.
This new version contains some backported fixes:
opam config var
display, aligned on opam config list
[#3723 @rjbou - rel. #3717]synopsis
field internally inferred from descr
[#3753 @rjbou - fix #3738]diff.noprefix=false
config argument to overwrite user defined configuration [#3788 @rjbou, #3628 @Blaisorblade - fix #3627]Note: To homogenise macOS name on system detection, we decided to keep
macos
, and convertdarwin
tomacos
in opam. For the moment, to not break jobs & CIs, we keep uploadingdarwin
&macos
binaries, but from the 2.1.0 release, onlymacos
ones will be kept.
From binaries: run
sh <(curl -sL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)
or download manually from the Github "Releases" page to your PATH. In this case, don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script.
From source, using opam:
opam update; opam install opam-devel
(then copy the opam binary to your PATH as explained, and don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script)
From source, manually: see the instructions in the README.
We hope you enjoy this new minor version, and remain open to bug reports and suggestions.
NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com. Please head to the latter for the comments!
This blog post looks back on some of the improvements in opam 2.0, and gives tips on the new workflows available.
Opam 2.0 has been vastly improved to handle locally defined packages. Assuming
you have a project ~/projects/foo
, defining two packages foo-lib
and
foo-bin
, you would have:
~/projects/foo
|-- foo-lib.opam
|-- foo-bin.opam
`-- src/ ...
(See also about computed dependency constraints for handling multiple package definitions with mutual constraints)
The underlying mechanism is the same, but this is an interface improvement that
replaces most of the opam 1.2 workflows based on opam pin
.
The usual commands (install
, upgrade
, remove
, etc.) have been extended to
support specifying a directory as argument. So when working on project foo
,
just write:
cd ~/projects/foo
opam install .
and both foo-lib
and foo-bin
will get automatically pinned to the current
directory (using git if your project is versioned), and installed. You may
prefer to use:
opam install . --deps-only
to just get the package dependencies ready before you start hacking on it.
See below for details on how to reproduce a
build environment more precisely. Note that opam depext .
will not work at the
moment, which will be fixed in the next release when the external dependency
handling is integrated (opam will still list you the proper packages to install
for your OS upon failure).
If your project is versioned and you made changes, remember to either commit, or
add --working-dir
so that your uncommitted changes are taken into account.
Opam 2.0 introduced a new feature called "local switches". This section explains what it is about, why, when and how to use them.
Opam switches allow to maintain several separate development environments, each with its own set of packages installed. This is particularly useful when you need different OCaml versions, or for working on projects with different dependency sets.
It can sometimes become tedious, though, to manage, or remember what switch to use with what project. Here is where "local switches" come in handy.
A local switch is simply stored inside a _opam/
directory, and will be
selected automatically by opam whenever your current directory is below its
parent directory.
NOTE: it's highly recommended that you enable the new shell hooks when using local switches. Just run
opam init --enable-shell-hook
: this will make sure your PATH is always set for the proper switch.You will otherwise need to keep remembering to run
eval $(opam env)
every time youcd
to a directory containing a local switch. See also how to display the current switch in your prompt
For example, if you have ~/projects/foo/_opam
, the switch will be selected
whenever in project foo
, allowing you to tailor what it has installed for the
needs of your project.
If you remove the switch dir, or your whole project, opam will forget about it transparently. Be careful not to move it around, though, as some packages still contain hardcoded paths and don't handle relocation well (we're working on that).
This can generally start with:
cd ~/projects/foo
opam switch create . --deps-only
Local switch handles are just their path, instead of a raw name. Additionally,
the above will detect package definitions present in ~/projects/foo
, pick a
compatible version of OCaml (if you didn't explicitely mention any), and
automatically install all the local package dependencies.
Without --deps-only
, the packages themselves would also get installed in the
local switch.
If you just want an already existing switch to be selected automatically,
without recompiling one for each project, you can use opam switch link
:
cd ~/projects/bar
opam switch link 4.07.1
will make sure that switch 4.07.1
is chosen whenever you are in project bar
.
You could even link to ../foo
here, to share foo
's local switch between the
two projects.
If your package depends on development versions of some dependencies (e.g. you had to push a fix upstream), add to your opam file:
depends: [ "some-package" ] # Remember that pin-depends are depends too
pin-depends: [
[ "some-package.version" "git+https://gitfoo.com/blob.git#mybranch" ]
]
This will have no effect when your package is published in a repository, but
when it gets pinned to its dev version, opam will first make sure to pin
some-package
to the given URL.
Dependency contraints are sometimes too wide, and you don't want to explore all the versions of your dependencies while developing. For this reason, you may want to reproduce a known-working set of dependencies. If you use:
opam lock .
opam will check what version of the dependencies are installed in your current
switch, and explicit them in *.opam.locked
files. opam lock
is a plugin at
the moment, but will get automatically installed when needed.
Then, assuming you checked these files into version control, any user can do
opam install . --deps-only --locked
to instruct opam to reproduce the same build environment (the --locked
option
is also available to opam switch create
, to make things easier).
The generated lock-files will also contain added constraints to reproduce the
presence/absence of optional dependencies, and reproduce the appropriate
dependency pins using pin-depends
. Add the --direct-only
option if you don't
want to enforce the versions of all recursive dependencies, but only direct
ones.
We are pleased to announce the release of opam 2.0.3.
This new version contains some backported fixes:
From binaries: run
sh <(curl -sL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)
or download manually from the Github "Releases" page to your PATH. In this case, don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script.
From source, using opam:
opam update; opam install opam-devel
(then copy the opam binary to your PATH as explained, and don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script)
From source, manually: see the instructions in the README.
We hope you enjoy this new major version, and remain open to bug reports and suggestions.
NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com. Please head to the latter for the comments!
We are pleased to announce the release of opam 2.0.2.
As sandbox scripts have been updated, don't forget to run opam init --reinit -ni
to update yours.
This new version contains mainly backported fixes:
--alias-of
)no-aspcud
pin-depends
packages reliably/bin
or /lib
(#3661). Fixes sandboxing on some distributions such as CentOS 7 and Arch Linux.ocaml-base-compiler
⇄ ocaml
) From binaries: run
sh <(curl -sL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)
or download manually from the Github "Releases" page to your PATH. In this case, don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update your sandbox script.
From source, using opam:
opam update; opam install opam-devel
(then copy the opam binary to your PATH as explained, and don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script)
From source, manually: see the instructions in the README.
We hope you enjoy this new minor version, and remain open to bug reports and suggestions.
NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com. Please head to the latter for the comments!
We are pleased to announce the release of opam 2.0.1.
This new version contains mainly backported fixes, some platform-specific:
gtar
And some opam specific:
test
variable warning and empty description errorlist
doesn't return non-zero code if list is empty, add --silent
option for a silent output and returns 1 if list is emptyFrom binaries: run
sh <(curl -sL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)
or download manually from the Github "Releases" page to your PATH. In this case, don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed.
From source, using opam:
opam update; opam install opam-devel
(then copy the opam binary to your PATH as explained, and don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed)
From source, manually: see the instructions in the README.
We hope you enjoy this new major version, and remain open to bug reports and suggestions.
NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com. Please head to the latter for the comments!
We are happy to announce the final release of opam 2.0.0.
A few weeks ago, we released a last release candidate to be later promoted to 2.0.0, synchronised with the opam package repository upgrade.
You are encouraged to update as soon as you see fit, to continue to get package updates: opam 2.0.0 supports the older formats, and 1.2.2 will no longer get regular updates. See the Upgrade Guide for details about the new features and changes.
The website opam.ocaml.org has been updated, with the full 2.0.0 documentation pages. You can still find the documentation for the previous versions in the corresponding menu.
Package maintainers should be aware of the following:
opam-publish
(2.0.0)For custom repositories, the advice remains the same.
From binaries: run
sh <(curl -sL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)
or download manually from the Github "Releases" page to your PATH. In this case, don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed.
From source, using opam:
opam update; opam install opam-devel
(then copy the opam binary to your PATH as explained, and don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed)
From source, manually: see the instructions in the README.
We hope you enjoy this new major version, and remain open to bug reports and suggestions.
NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com. Please head to the latter for the comments!
A few days ago, we released opam 2.0.0~rc4, and explained that this final release candidate was expected be promoted to 2.0.0, in sync with an upgrade to the opam package repository. So here are the details about this!
You are encouraged to upgrade) as soon as comfortable, and get used to the changes and new features
All packages installing in opam 1.2.2 should exist and install fine on 2.0.0~rc4 (if you find one that doesn't, please report!)
If you haven't updated by September 17th, the amount of updates and new packages you receive may become limited¹.
Opam 2.0.0~rc4 gets officially released as 2.0.0
On the ocaml/opam-repository
Github repository, a 1.2 branch is forked, and the 2.0.0 branch is merged into the master branch
From then on, pull-requests to ocaml/opam-repository
need to be in 2.0.0 format. Fixes to the 1.2 repository can be merged if important: pulls need to be requested against the 1.2 branch in that case.
The opam website shows the 2.0.0 repository by default (https://opam.ocaml.org/2.0-preview/ becomes https://opam.ocaml.org/)
The http repositories for 1.2 and 2.0 (as used by opam update
) are accordingly moved, with proper redirections put in place
Until September 17th, pull-requests filed to the master branch of ocaml/opam-repository
need to be in 1.2.2 format
The CI checks for all PRs ensure that the package passes on both 1.2.2 and 2.0.0. After the 17th of september, only 2.0.0 will be checked (and 1.2.2 only if relevant fixes are required).
The 2.0.0 branch of the repository will contain the automatically updated 2.0.0 version of your package definitions
You can publish 1.2 packages while using opam 2.0.0 by installing opam-publish.0.3.5
(running opam pin opam-publish 0.3.5
is recommended)
You should only need to keep an opam 1.2 installation for more complex setups (multiple packages, or if you need to be able to test the 1.2 package installations locally). In this case you might want to use an alias, e.g. alias opam.1.2="OPAMROOT=$HOME/.opam.1.2 ~/local/bin/opam.1.2
. You should also probably disable opam 2.0.0's automatic environment update in that case (opam init --disable-shell-hook
)
opam-publish.2.0.0~beta
has a fully revamped interface, and many new features, like filing a single PR for multiple packages. It files pull-request in 2.0 format only, however. At the moment, it will file PR only to the 2.0.0 branch of the repository, but pushing 1.2 format packages to master is still preferred until September 17th.
It is also advised to keep in-source opam files in 1.2 format until that date, so as not to break uses of opam pin add --dev-repo
by opam 1.2 users. The small opam-package-upgrade
plugin can be used to upgrade single 1.2 opam
files to 2.0 format.
ocaml-ci-script
already switched to opam 2.0.0. To keep testing opam 1.2.2, you can set the variable OPAM_VERSION=1.2.2
in the .travis.yml
file.
The opam admin upgrade
command can be used to upgrade your repository to 2.0.0 format. We recommand using it, as otherwise clients using opam 2.0.0 will do the upgrade locally every time. Add the option --mirror
to continue serving both versions, with automatic redirects.
It's your place to decide when/if you want to switch your base repository to 2.0.0 format. You'll benefit from many new possibilities and safety features, but that will exclude users of earlier opam versions, as there is no backwards conversion tool.
¹ Sorry for the inconvenience. We'd be happy if we could keep maintaining the 1.2.2 repository for more time; repository maintainers are doing an awesome job, but just don't have the resources to maintain both versions in parallel.
We are happy to announce the opam 2.0.0 final release candidate! 🍾
This release features a few bugfixes over Release Candidate 3. It will be promoted to 2.0.0 proper within a few weeks, when the official repository format switches from 1.2.0 to 2.0.0. After that date, updates to the 1.2.0 repository may become limited, as new features are getting used in packages.
It is safe to update as soon as you see fit, since opam 2.0.0 supports the older formats. See the Upgrade Guide for details about the new features and changes. If you are a package maintainer, you should keep publishing as before for now: the roadmap for the repository upgrade will be detailed shortly.
The opam.ocaml.org pages have also been refreshed a bit, and the new version showing the 2.0.0 branch of the repository is already online at http://opam.ocaml.org/2.0-preview/ (report any issues here).
From binaries: run
sh <(curl -sL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)
or download manually from the Github "Releases" page to your PATH. In this case, don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed.
From source, using opam:
opam update; opam install opam-devel
(then copy the opam binary to your PATH as explained, and don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed)
From source, manually: see the instructions in the README.
We hope you enjoy this new version, and remain open to bug reports and suggestions.
NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com. Please head to the latter for the comments!
We are pleased to announce the release of a third release candidate for opam 2.0.0. This one is expected to be the last before 2.0.0 comes out.
Changes since the 2.0.0~rc2 are, as expected, mostly fixes. We deemed it useful, however, to bring in the following:
opam switch link
that allows to select a switch to be used in a given directory (particularly convenient if you use the shell hook for automatic opam environment update)opam install --assume-built
, that allows to install a package using its normal opam procedure, but for a source repository that has been built by hand. This fills a gap that remained in the local development workflows.The preview of the opam 2 webpages can be browsed at http://opam.ocaml.org/2.0-preview/ (please report issues here).
Installation instructions (unchanged):
From binaries: run
sh <(curl -sL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)
or download manually from the Github "Releases" page to your PATH. In this case, don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed.
From source, using opam:
opam update; opam install opam-devel
(then copy the opam binary to your PATH as explained, and don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed)
From source, manually: see the instructions in the README.
Thanks a lot for testing out this new RC and reporting any issues you may find.
We are pleased to announce the release of a second release candidate for opam 2.0.0.
This new version brings us very close to a final 2.0.0 release, and in addition to many fixes, features big performance enhancements over the RC1.
Among the new features, we have squeezed in full sandboxing of package commands for both Linux and macOS, to protect our users from any misbehaving scripts.
NOTE: if upgrading manually from 2.0.0~rc, you need to run
opam init --reinit -ni
to enable sandboxing.
The new release candidate also offers the possibility to setup a hook in your shell, so that you won't need to run eval $(opam env)
anymore. This is specially useful in combination with local switches, because with it enabled, you are guaranteed that running make
from a project directory containing a local switch will use it.
The documentation has also been updated, and a preview of the opam 2 webpages can be browsed at http://opam.ocaml.org/2.0-preview/ (please report issues here). This provides the list of packages available for opam 2 (the 2.0
branch of opam-repository), including the compiler packages.
Installation instructions:
From binaries: run
sh <(curl -sL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)
or download manually from the Github "Releases" page to your PATH. In this case, don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed.
From source, using opam:
opam update; opam install opam-devel
(then copy the opam binary to your PATH as explained, and don't forget to run opam init --reinit -ni
to enable sandboxing if you had version 2.0.0~rc manually installed)
From source, manually: see the instructions in the README.
Thanks a lot for testing out this new RC and reporting any issues you may find.
NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com. Please head to the latter for the comments!
Between 26 Oct 2017 and 17 Feb 2018, the OPAM package for camlp5 7.03 in opam-repository was under certain circumstances able to trigger rm -rf /
on macOS and other systems which don't by default prevent recursive root deletion. This article contains advice on how to identify if your OPAM installation is affected and what you can do to fix it.
TL;DR If rm --preserve-root
gives a message along the lines of unrecognised option
rather than missing operand
and you are running OPAM 1.2.2, ensure you run opam update
before upgrading your system compiler to OCaml 4.06.1. If you have already upgraded your system compiler to OCaml 4.06.1 (e.g. with Homebrew) then please read on.
You are at serious risk of erasing all your files if the following three things are true:
rm
command does not support the --preserve-root
default (you can identify this by running rm --preserve-root
and noting whether the error message refers to an ‘unrecognised option’ rather than a ‘missing operand’)If your system is affected, most OPAM commands cannot be run. In particular, if OPAM asks:
dra@bionic:~$ opam update
Your system compiler has been changed. Do you want to upgrade your OPAM installation ? [Y/n] n
YOU MUST ANSWER NO TO THIS QUESTION.
I have written a script which can safely identify if your system is affected, which can be reviewed on GitHub or run directly by executing:
$ curl -L https://raw.githubusercontent.com/dra27/opam/camlp5-detection/shell/opam-detect.sh | sh -
This script scans the directory identified by $HOME
for anything which looks like an OPAM root. Virtually all users will have one OPAM root in ~/.opam
and if you don't know how to run OPAM with multiple roots, then you probably don't have any others to worry about!
The script may display a variety of messages. If your system contains at least one affected OPAM 1.2 root, you will see output like this:
dra@bionic:~/opam$ shell/opam-detect.sh
opam 1.2.2 found
Scanning /home/dra for opam roots...
opam 1.2 root found in /home/dra/.opam
camlp5 is faulty AND installed AND the system compiler is OCaml 4.06.1
THIS ROOT CANNOT BE UPDATED OR UPGRADED. DO NOT ALLOW OPAM TO UPGRADE THE SYSTEM
COMPILER. DOING SO WILL ATTEMPT TO ERASE YOUR MACHINE
Please see https://github.com/ocaml/opam/issues/3322 for more information
In all cases, one fix is to install the latest release candidate for opam 2, and upgrade your OPAM 1.2 root to opam 2 format. The upgrade prevents OPAM 1.2.2 from being able to read the root. If you received the message above and choose to upgrade to opam 2 (the easiest way to upgrade a root is to run opam list
after installing opam 2) and then run the opam-detect.sh
script again. As before, DO NOT ANSWER YES TO THE Your system compiler has changed. QUESTION IF YOU ARE ASKED.
If you do not wish to upgrade to opam 2, and there are many good reasons for not wanting to do this, there are two other possibilities. The easiest is to downgrade your system compiler back to 4.06.0 (or an earlier release). You can then run opam-detect.sh
again and check the error message. As long as the message is no longer the one above, you can then run opam update
to update the repository metadata on the switch. You can then upgrade your system compiler back to OCaml 4.06.1 again. To be absolutely sure, you can then run the opam-detect.sh
script again and, assuming the message is still not the one above, you can then allow OPAM 1.2.2 to upgrade your system switch. This is the recommended course of action.
The final option is that you can edit the opam root by hand and trick opam into believing that the camlp5 package is no longer installed. This is done by editing the file system/installed
within the root and removing both the camlp5
line and any package which depends on camlp5 (for example, coq
). You cannot use the opam
to determine dependencies at this stage, so you will need to use the online index to check for dependent packages. If you fail to remove all the packages which depend on camlp5, OPAM will display an installation prompt like this:
dra@bionic:~$ opam update
Your system compiler has been changed. Do you want to upgrade your OPAM installation ? [Y/n] y
=-=- Upgrading system -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
The following actions will be performed:
∗ install camlp5 7.03 [required by coq]
∗ install conf-m4 1
∗ install base-threads base
∗ install base-unix base
∗ install base-bigarray base
∗ install ocamlfind 1.7.3
∗ install num 1.1
∗ install coq 8.7.0
===== ∗ 8 =====
Do you want to continue ? [Y/n]
If this happens, answer no, but at this stage your system switch will have been emptied of all packages (you can now safely run opam update
of course). Of course, you can back-up the OPAM root prior to trying this. Once upgraded, you can then run opam update
and reinstall the missing packages. This course of action is not recommended as the opam-detect.sh
script will no longer help. You are strongly advised to back up your files before attempting this solution.
On 26 Oct 2017, PR#10523 was merged which packaged camlp5 7.03. This was the first version of camlp5 released to opam which supported OCaml 4.06.0.
Unfortunately, it was also the first version of the opam package to include a remove
section which executed make uninstall
. The package also contained an incorrect available
constraint - it should have permitted only OCaml 4.06.0 from the 4.06 branch, but the constraint given permitted all versions.
camlp5's configure
script is responsible for writing config/Makefile
with all the usual configuration settings, including PREFIX
and so forth. This script includes a version check for OCaml and fails if the version is not supported. Unfortunately, even when it fails, it writes a partial config/Makefile
to enable some development targets. Sadly this left the command rm -rf "$(DESTDIR)$(LIBDIR)/$(CAMLP5N)"
in the uninstall
target with all three variables undefined, leaving the certainly unwanted rm -rf /
.
Users of GNU coreutils have, since November 2003 (in release 5.1.0) had the --preserve-root
option set by default, which causes rm -rf /
to raise an error. Unfortunately, macOS does not use GNU coreutils by default.
Prior to OPAM 1.2, the build
and install
sections of an opam
file were combined. For this reason, if the build
failed, OPAM would silently execute the remove
commands in order to clean-up any partial installation which may have taken place. Although OPAM 1.2 recommended separating the build
and install
commands, this was not mandatory and it therefore retains the “silent remove” behaviour. opam 2 mandates the separation (and, if sandboxing is available, now enforces it). opam 2 also expects remove
commands to be run in a clean source tree which, for this camlp5 case, means opam 2 users are UNAFFECTED by this issue.
OCaml 4.06.1 was added to opam-repository on 16 Feb 2018 in PR#11433. During the following 48 hours, it was noticed that the camlp5 package was attempting to run rm -rf /
(see Issue #11440) and the package was patched on 18 Feb 2018 in PR#11443. Unfortunately, the signifance of the GNU coreutils protection was not realised at this point and it was also assumed that the problem would only be hit if one were unlucky enough to have updated OPAM between the release of OCaml 4.06.1 to opam-repository and the patching of camlp5 7.03 in opam-repository (so 16–18 February 2018) and it was on this basis that OPAM PR#3231 was deemed to have been very unlucky.
However, the real problem is the upgrading of the system compiler to 4.06.1, which wasn't noticed in that Issue but was correctly identified in Issue #3316. This unfortunately gives a much wider window for the problem - if you ran opam update
between 26 Oct 2017 and 18 Feb 2018 and haven't run it since, then your system will be at risk if you update your system compiler to OCaml 4.06.1 without first running opam update
.
If the system compiler alters, OPAM 1.2.2 on virtually all commands (including opam update
) first asks to upgrade the system
switch. This step is mandatory, preventing further safe use of OPAM 1.2.2.
Owing to the changes made to how opam 2 processes package installations, opam 2 has been unaffected by this situation but opam 2's lead developer @AltGr freely admits that this is more by luck than judgement. However, the second release candidate for opam 2 includes mandatory support for sandboxing on Linux and macOS. Sandboxing package building and installation will protect opam 2 against future issues of this kind, as a malfunctioning build system will be unable to operate on files outside its build directory or, during installation, switch root.
We are pleased to announce a first release candidate for the long-awaited opam 2.0.0.
A lot of polishing has been done since the last beta, including tweaks to the built-in solver, allowing in-source package definitions to be gathered in an opam/
directory, and much more.
With all of the 2.0.0 features getting pretty solid, we are now focusing on bringing all the guides up-to-date¹, updating the tools and infrastructure, making sure there are no usability issues with the new workflows, and being future-proof so that further updates break as little as possible.
You are invited to read the beta5 announcement for details on the 2.0.0 features. Installation instructions haven't changed:
From binaries: run
sh <(curl -sL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)
or download manually from the Github "Releases" page to your PATH.
From source, using opam:
opam update; opam install opam-devel
(then copy the opam binary to your PATH as explained)
From source, manually: see the instructions in the README.
Thanks a lot for testing out the RC and reporting any issues you may find. See what we need tested for more detail.
After a few more months brewing, we are pleased to announce a new beta release of opam. With this new milestone, opam is reaching feature-freeze, with an expected 2.0.0 by the beginning of next year.
This version brings many new features, stability fixes, and big improvements to the local development workflows.
The features presented in past announcements: local switches, in-source package definition handling, extended dependencies are of course all present. But now, all the glue to make them interact nicely together is here to provide new smooth workflows. For example, the following command, if run from the source tree of a given project, creates a local switch where it will restore a precise installation, including explicit versions of all packages and pinnings:
opam switch create ./ --locked
this leverages the presence of opam.locked
or <name>.opam.locked
files,
which are valid package definitions that contain additional details of the build
environment, and can be generated with the
opam-lock
plugin (the lock
command may
be merged into opam once finalised).
But this new beta also provides a large amount of quality of life improvements,
and other features. A big one, for example, is the integration of a built-in
solver (derived from mccs
and
glpk
). This means that the opam
binary
works out-of-the box, without requiring the external
aspcud
solver, and on all
platforms. It is also faster.
Another big change is that detection of architecture and OS details is now done
in opam, and can be used to select the external dependencies with the new format
of the depexts:
field, but also to affect dependencies or build flags.
There is much more to it. Please see the changelog, and the updated manual.
Our warm thanks for trying the new beta and reporting any issues you may hit.
There are three main ways to get the update:
The easiest is to use our pre-compiled binaries. This script will also make backups if you migrate from 1.x, and has an option to revert back:
sh <(curl -sL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)
This uses the binaries from https://github.com/ocaml/opam/releases/tag/2.0.0-beta5
Another option is to compile from source, using an existing opam installation. Simply run:
opam update; opam install opam-devel
and follow the instructions (you will need to copy the compiled binary to your PATH).
Compiling by hand from the
inclusive source archive,
or from the git repo. Use
./configure && make lib-ext && make
if you have OCaml >= 4.02.3 already
available; make cold
otherwise.
If the build fails after updating a git repo from a previous version, try
git clean -fdx src/
to remove any stale artefacts.
Note that the repository format is different from that of opam 1.2. Opam 2 will be automatically redirected from the opam-repository to an automatically rewritten 2.0 mirror, and is otherwise able to do the conversion on the fly (both for package definitions when pinning, and for whole repositories). You may not yet contribute packages in 2.0 format to opam-repository, though.
We are interested in all opinions and reports, but here are a few areas where your feedback would be specially useful to us:
--working-dir
option.opam install DIR
).
Give us feedback on the workflow. Use opam lock
and share development
environments.opam admin upgrade --mirror
on them, and use the generated mirror.--json
output).Opam 1.2.0 will be actively deprecated in favour of opam 1.2.2, which now becomes the only supported stable release.
OPAM 1.2.0 was released in October 2014, and saw rapid uptake from the community. We did some rapid bugfixing to solve common problems, and OPAM 1.2.2 was released in April 2015. Since then, 1.2.2 has been a very solid release and has been the stable version in use to date.
Unfortunately, part of the bugfixes in the 1.2.2 series resulted in an opam
file format that is not fully backwards compatible with the 1.2.0 syntax, and
the net effect is that users of 1.2.0 now see a broken package repository. Our
CI tests for new packages regularly fail on 1.2.0, even if they succeed on 1.2.2
and higher.
As we prepare the plan for 1.2.2 -> 2.0 migration, it is clear that we need a "one-in one-out" policy on releases in order to preserve the overall health of the package repository -- maintaining three separate releases and formats of the repository is not practical. Therefore the 1.2.0 release needs to be actively deprecated, and we could use some help from the community to make this happen.
I found that the Debian Jessie (stable) release includes 1.2.0, and this is probably the last major distribution including it. The Debian Stretch is due to become the stable release on the 17th June 2017, and so at that point there will hopefully be no distributions actively sending opam 1.2.0 out.
The format changes, although small, would cause errors on 1.2.0 users with the main repository. To avoid those, as was done for 1.1.0, we are going to redirect users of 1.2.0 to a frozen mirror of the repository, making new package updates unavailable to them.
If there are any remaining users of opam 1.2.0, particularly industrial ones, please reach out (e.g. on Github). By performing an active deprecation of an older release, we hope we can focus our efforts on ensuring the opam users have a good out-of-the-box experience with opam 1.2.2 and the forthcoming opam 2.0.
Please also see the discussion thread regarding the deprecation on the OCaml Discourse forums.
This blog will cover yet another aspect of the improvements opam 2.0 has over opam 1.2. I may be a little more technical than previous issues, as it covers a feature directed specifically at packagers and repository maintainers, and regarding the package definition format.
Opam 1.2 already has an advanced way of specifying package dependencies, using formulas on packages and versions, with the following syntax:
depends: [
"foo" {>= "3.0" & < "4.0~"}
( "bar" | "baz" {>= "1.0"} )
]
meaning that the package being defined depends on both package foo
, within the 3.x
series, and one of bar
or baz
, the latter with version at least 1.0
. See here for a complete documentation.
This only allows, however, dependencies that are static for a given package.
Opam 1.2 introduced build
, test
and doc
"dependency flags" that could provide some specifics for dependencies (e.g. test
dependencies would only be needed when tests were requested for the package). These were constrained to appear before the version constraints, e.g. "foo" {build & doc & >= "3.0"}
.
Opam 2.0 generalises the dependency flags, and makes the dependencies specification more expressive by allowing to mix filters, i.e. formulas based on opam variables, with the version constraints. If that formula holds, the dependency is enforced, if not, it is discarded.
This is documented in more detail in the opam 2.0 manual.
Note also that, since the compilers are now packages, the required OCaml version is now expressed using this mechanism as well, through a dependency to the (virtual) package ocaml
, e.g. depends: [ "ocaml" {>= "4.03.0"} ]
. This replaces uses of the available:
field and ocaml-version
switch variable.
This makes it trivial to add, for example, a condition on the OS to a given dependency, using the built-in variable os
:
depends: [ "foo" {>= "3.0" & < "4.0~" & os = "linux"} ]
here, foo
is simply not needed if the OS isn't Linux. We could also be more specific about other OSes using more complex formulas:
depends: [
"foo" { "1.0+linux" & os = "linux" |
"1.0+osx" & os = "darwin" }
"bar" { os != "osx" & os != "darwin" }
]
Meaning that Linux and OSX require foo
, respectively versions 1.0+linux
and 1.0+osx
, while other systems require bar
, any version.
Dependency flags, as used in 1.2, are no longer needed, and are replaced by variables that can appear anywhere in the version specification. The following variables are typically useful there:
with-test
, with-doc
: replace the test
and doc
dependency flags, and are true
when the package's tests or documentation have been requestedbuild
behaves similarly as before, limiting the dependency to a "build-dependency", implying that the package won't need to be rebuilt if the dependency changesdev
: this boolean variable holds true
on "development" packages, that is, packages that are bound to a non-stable source (a version control system, or if the package is pinned to an archive without known checksum). dev
sources often happen to need an additional preliminary step (e.g. autoconf
), which may have its own dependencies.Use opam config list
for a list of pre-defined variables. Note that the with-test
, with-doc
and build
variables are not available everywhere: the first two are allowed only in the depends:
, depopts:
, build:
and install:
fields, and the latter is specific to the depends:
and depopts:
fields.
For example, the datakit.0.9.0
package has:
depends: [
...
"datakit-server" {>= "0.9.0"}
"datakit-client" {with-test & >= "0.9.0"}
"datakit-github" {with-test & >= "0.9.0"}
"alcotest" {with-test & >= "0.7.0"}
]
When running opam install datakit.0.9.0
, the with-test
variable is set to false
, and the datakit-client
, datakit-github
and alcotest
dependencies are filtered out: they won't be required. With opam install datakit.0.9.0 --with-test
, the with-test
variable is true (for that package only, tests on packages not listed on the command-line are not enabled!). In this case, the dependencies resolve to:
depends: [
...
"datakit-server" {>= "0.9.0"}
"datakit-client" {>= "0.9.0"}
"datakit-github" {>= "0.9.0"}
"alcotest" {>= "0.7.0"}
]
which is treated normally.
It is also possible to use variables, not only as conditions, but to compute the version values: "foo" {= var}
is allowed and will require the version of package foo
corresponding to the value of variable var
.
This is useful, for example, to define a family of packages, which are released together with the same version number: instead of having to update the dependencies of each package to match the common version at each release, you can leverage the version
package-variable to mean "that other package, at the same version as current package". For example, foo-client
could have the following:
depends: [ "foo-core" {= version} ]
It is even possible to use variable interpolations within versions, e.g. specifying an os-specific version differently than above:
depends: [ "foo" {= "1.0+%{os}%"} ]
this will expand the os
variable, resolving to 1.0+linux
, 1.0+darwin
, etc.
Getting back to our datakit
example, we could leverage this and rewrite it to the more generic:
depends: [
...
"datakit-server" {>= version}
"datakit-client" {with-test & >= version}
"datakit-github" {with-test & >= version}
"alcotest" {with-test & >= "0.7.0"}
]
Since the datakit-*
packages follow the same versioning, this avoids having to rewrite the opam file on every new version, with a risk of error each time.
As a side note, these variables are consistent with what is now used in the build:
field, and the build-test:
field is now deprecated. So this other part of the same datakit
opam file:
build:
["ocaml" "pkg/pkg.ml" "build" "--pinned" "%{pinned}%" "--tests" "false"]
build-test: [
["ocaml" "pkg/pkg.ml" "build" "--pinned" "%{pinned}%" "--tests" "true"]
["ocaml" "pkg/pkg.ml" "test"]
]
would now be preferably written as:
build: ["ocaml" "pkg/pkg.ml" "build" "--pinned" "%{pinned}%" "--tests" "%{with-test}%"]
run-test: ["ocaml" "pkg/pkg.ml" "test"]
which avoids building twice just to change the options.
Hopefully this extension to expressivity in dependencies will make the life of packagers easier; feedback is welcome on your personal use-cases.
Note that the official repository is still in 1.2 format (served as 2.0 at https://opam.ocaml.org/2.0
, through automatic conversion), and will only be migrated a little while after opam 2.0 is finally released. You are welcome to experiment on custom repositories or pinned packages already, but will need a little more patience before you can contribute package definitions making use of the above to the official repository.
NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com. Please head to the latter for the comments!
After the opam build feature was announced followed a lot of discussions, mainly having to do with its interface, and misleading name. The base features it offered, though, were still widely asked for:
destdir
opam build
opam build
, as described in a previous post has been dropped. It will be absent from the next Beta, where the following replaces it.
Consistently with what was done with local switches, it was decided, where meaningful, to overload the <packages>
arguments of the commands, allowing directory names instead, and meaning "all packages defined there", with some side-effects.
For example, the following command is now allowed, and I believe it will be extra convenient to many:
opam install . --deps-only
What this does is find opam
(or <pkgname>.opam
) files in the current directory (.
), resolve their installations, and install all required packages. That should be the single step before running the source build by hand.
The following is a little bit more complex:
opam install .
This also retrieves the packages defined at .
, pins them to the current source (using version-control if present), and installs them. Note that subsequent runs actually synchronise the pinnings, so that packages removed or renamed in the source tree are tracked properly (i.e. removed ones are unpinned, new ones pinned, the other ones upgraded as necessary).
opam upgrade
, opam reinstall
, and opam remove
have also been updated to handle directories as arguments, and will work on "all packages pinned to that target", i.e. the packages pinned by the previous call to opam install <dir>
. In addition, opam remove <dir>
unpins the packages, consistently reverting the converse install
operation.
opam show
already had a --file
option, but has also been extended in the same way, for consistency and convenience.
This all, of course, works well with a local switch at ./
, but the two features can be used completely independently. Note also that the directory name must be made unambiguous with a possible package name, so make sure to use ./foo
rather than just foo
for a local project in subdirectory foo
.
This relies on installed files tracking, but was actually independent from the other opam build
features. It is now simply a new option to opam install
:
opam install foo --destdir ~/local/
will install foo
normally (if it isn't installed already) and copy all its installed files, following the same hierarchy, into ~/local
. opam remove --destdir
is also supported, to remove these files.
Automatic initialisation has been dropped for the moment. It was only saving one command (opam init
, that opam will kindly print out for you if you forget it), and had two drawbacks:
opam init
anyway. The other possibility being to duplicate init
options to all commands, adding lots of noise. Keeping things separate has its merits.Granted, another command, opam switch create .
, was made implicit. But using a local switch is a user choice, and worse, in contradiction with the previous de facto opam default, so not creating one automatically seems safer: having to specify --no-autoinit
to opam build
in order to get the more simple behaviour was inconvenient and error-prone.
One thing is provided to help with initialisation, though: opam switch create <dir>
has been improved to handle package definitions at <dir>
, and will use them to choose a compatible compiler, as opam build
did. This avoids the frustration of creating a switch, then finding out that the package wasn't compatible with the chosen compiler version, and having to start over with an explicit choice of a different compiler.
If you would really like automatic initialisation, and have a better interface to propose, your feedback is welcome!
A few other new options have been added to opam install
and related commands, to improve the project-local workflows:
opam install --keep-build-dir
is now complemented with --reuse-build-dir
, for incremental builds within opam (assuming your build-system supports it correctly). At the moment, you should specify both on every upgrade of the concerned packages, or you could set the OPAMKEEPBUILDDIR
and OPAMREUSEBUILDDIR
environment variables.opam install --inplace-build
runs the scripts directly within the source dir instead of a dedicated copy. If multiple packages are pinned to the same directory, this disables parallel builds of these packages.opam install --working-dir
uses the working directory state of your project, instead of the state registered in the version control system. Don't worry, opam will warn you if you have uncommitted changes and forgot to specify --working-dir
.NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com. Please head to the latter for the comments!
Among the areas we wanted to improve on for opam 2.0 was the handling of
switches. In opam 1.2, they are simply accessed by a name (the OCaml version
by default), and are always stored into ~/.opam/<name>
. This is fine, but can
get a bit cumbersome when many switches are in presence, as there is no way to
sort them or associate them with a given project.
A reminder about switches
For those unfamiliar with it, switches, in opam, are independent prefixes with their own compiler and set of installed packages. The
opam switch
command allows to create and remove switches, as well as select the currently active one, where operations likeopam install
will operate.Their uses include easily juggling between versions of OCaml, or of a library, having incompatible packages installed separately but at the same time, running tests without damaging your "main" environment, and, quite often, separation of environment for working on different projects.
You can also select a specific switch for a single command, with
opam install foo --switch other
or even for a single shell session, with
eval $(opam env --switch other)
What opam 2.0 adds to this is the possibility to create so-called local switches, stored below a directory of your choice. This gets users back in control of how switches are organised, and wiping the directory is a safe way to get rid of the switch.
This is the main intended use: the user can define a switch within the source of
a project, for use specifically in that project. One nice side-effect to help
with this is that, if a "local switch" is detected in the current directory or a
parent, opam will select it automatically. Just don't forget to run eval $(opam
env)
to make the environment up-to-date before running make
.
The interface simply overloads the switch-name
arguments, wherever they were
present, allowing directory names instead. So for example:
cd ~/src/project
opam switch create ./
will create a local switch in the directory ~/src/project
. Then, it is for
example equivalent to run opam list
from that directory, or opam list
--switch=~/src/project
from anywhere.
Note that you can bypass the automatic local-switch selection if needed by using
the --switch
argument, by defining the variable OPAMSWITCH
or by using eval
$(opam env --switch <name>)
In practice, the switch contents are placed in a _opam/
subdirectory. So if
you create the switch ~/src/project
, you can browse its contents at
~/src/project/_opam
. This is the direct prefix for the switch, so e.g.
binaries can be found directly at _opam/bin/
: easier than searching the opam
root! The opam metadata is placed below that directory, in a .opam-switch/
subdirectory.
Local switches still share the opam root, and in particular depend on the repositories defined and cached there. It is now possible, however, to select different repositories for different switches, but that is a subject for another post.
Finally, note that removing that _opam
directory is handled transparently by
opam, and that if you want to share a local switch between projects, symlinking
the _opam
directory is allowed.
This feature has been present in our dev builds for a while, and you can already use it in the current beta.
It is not, at the moment, possible to move a local switch directory around, mainly due to issues related to relocating the OCaml compiler.
Creating a new switch still implies to recompile all the packages, and even the compiler itself (unless you rely on a system installation). The projected solution is to add a build cache, avoiding the need to recompile the same package with the same dependencies. This should actually be possible with the current opam 2.0 code, by leveraging the new hooks that are made available. Note that relocation of OCaml is also an issue for this, though.
Editing tools like ocp-indent
or merlin
can also become an annoyance with
the multiplication of switches, because they are not automatically found if not
installed in the current switch. But the user-setup
plugin (run opam
user-setup install
) already handles this well, and will access ocp-indent
or
tuareg
from their initial switch, if not found in the current one. You will
still need to install tools that are tightly bound to a compiler version, like
merlin
and ocp-index
, in the switches where you need them, though.
NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com. Please head to the latter for the comments!
UPDATE: after discussions following this post, this feature was abandoned with the interface presented below. See this post for the details and the new interface!
The new opam 2.0 release, currently in beta, introduces several new features.
This post gets into some detail on the new opam build
command, its purpose,
its use, and some implementation aspects.
opam build
is run from the source tree of a project, and does not rely on a
pre-existing opam installation. As such, it adds a new option besides the
existing workflows based on managing shared OCaml installations in the form of
switches.
Typically, this is used in a fresh git clone of some OCaml project. Like when
pinning the package, opam will find and leverage package definitions found in
the source, in the form of opam
files.
~/.opam
), this is taken care of../_opam/
)This is particularly useful for distributing projects to people not used to
opam and the OCaml ecosystem: the setup steps are automatically taken care of,
and a single opam build
invocation can take care of resolving the dependency
chains for your package.
If building the project directly is preferred, adding --deps-only
is a good
way to get the dependencies ready for the project:
opam build --deps-only
eval $(opam config env)
./configure; make; etc.
Note that if you just want to handle project-local opam files, opam build
can
also be used in your existing switches: just specify --no-autoinit
, --switch
or make sure the OPAMSWITCH
variable is set. E.g. opam build --no-autoinit
--deps-only
is a convenient way to get the dependencies for the local project
ready in your current switch.
The installation of the packages happens as usual to the prefix corresponding to
the switch used (<project-root>/_opam/
for a local switch). But it is
possible, with --install-prefix
, to further install the package to the system:
opam build --install-prefix ~/local
will install the results of the package found in the current directory below ~/local.
The dependencies of the package won't be installed, so this is intended for programs, assuming they are relocatable, and not for libraries.
The user can pre-select the repositories to use on the creation of the local switch with:
opam build --repositories <repos>
where <repos>
is a comma-separated list of repositories, specified either as
name=URL
, or name
if already configured on the system.
Multiple packages are commonly found to share a single repository. In this case,
opam build
registers and builds all of them, respecting cross-dependencies.
The opam files to use can also be explicitely selected on the command-line.
In this case, specific opam files must be named <package-name>.opam
.
The choice of the compiler, on automatic initialisation, is either explicit,
using the --compiler
option, or automatic. In the latter case, the default
selection is used (see opam init --help
, section "CONFIGURATION FILE" for
details), but a compiler compatible with the local packages found is searched
from that. This allows, for example, to choose a system compiler when available
and compatible, avoiding a recompilation of OCaml.
When using --install-prefix
, the normal installation is done, then the
tracking of package-installed files, introduced in opam 2.0, is used to extract
the installed files from the switch and copy them to the prefix.
The packages installed through opam build
are not registered in any
repository, and this is not an implicit use of opam pin
: the rationale is that
packages installed this way will also be updated by repeating opam build
. This
means that when using other commands, e.g. opam upgrade
, opam won't try to
keep the packages to their local, source version, and will either revert them to
their repository definition, or remove them, if they need recompilation.
This is still in beta: there are still rough edges, please experiment and give feedback! It is still possible that the command syntax and semantics change significantly before release.
Another use-case that we are striving to improve is sharing of development
setups (share sets of pinned packages, depend on specific remotes or git hashes,
etc.). We have many
ideas to
improve on this, but opam build
is not, as of today, a direct solution to this. In particular, installing this
way still relies on the default opam repository; a way to define specific
options for the switch that is implicitely created on opam build
is in the
works.
NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com. Please head to the latter for the comments!
UPDATE (2017-02-14): A beta2 is online, which fixes issues and performance of the
opam build
command. Get the new binaries, or recompile the opam-devel package and replace the previous binary.
We are pleased to announce that the beta release of opam 2.0 is now live! You can try it already, bootstrapping from a working 1.2 opam installation, with:
opam update; opam install opam-devel
With about a thousand patches since the last stable release, we took the time to gather feedback after our last announcement and implemented a couple of additional, most-wanted features:
opam build
command that, from the root of a source tree containing one
or more package definitions, can automatically handle initialisation and
building of the sources in a local switch.There are many more features, like the new opam clean
and opam admin
commands, a new archive caching system, etc., but we'll let you check the full
changelog.
We also improved still on the already announced features, including compilers as packages, local switches, per-switch repository configuration, package file tracking, etc.
The updated documentation is at http://opam.ocaml.org/doc/2.0/. If you are developing in opam-related tools, you may also want to browse the new APIs.
Please try out the beta, and report any issues or missing features. You can:
opam install opam-devel
)./configure && make lib-ext && make
if you have OCaml >=
4.01 already available; make cold
otherwisegit clean -dx
).Some users have been using the alpha for the past months without problems, but you may want to keep your opam 1.2 installation intact until the release is out. An easy way to do this is with an alias:
alias opam2="OPAMROOT=~/.opam2 path/to/opam-2-binary"
opam switch create
is now needed to create new switches, and opam switch
is now much more expressiveopam list
is also much more expressive, but be aware that the output may
have changed if you used it in scriptsopam build
: setup and build a local source treeopam clean
: various cleanup operations (wiping caches, etc.)opam admin
: manage software repositories, including upgrading them to
opam 2.0 format (replaces the opam-admin
tool)opam env
, opam exec
, opam var
: shortcuts for the opam config
subcommandsopam repository add
will now setup the new repository for the current switch
only, unless you specify --all
--test
, now apply to the packages listed on the
command-line only. For example, opam install lwt --test
will build and
install lwt and all its dependencies, but only build/run the tests of the
lwt
package. Test-dependencies of its dependencies are also ignoredopam install --soft-request
is useful for batch runs, it will
maximise the installed packages among the requested ones, but won't fail if
all can't be installedAs before, opam is self-documenting, so be sure to check opam COMMAND --help
first when in doubt. The bash completion scripts have also been thoroughly
improved, and may help navigating the new options.
There are both a few changes (extensions, mostly) to the package description format, and more drastic changes to the repository format, mainly related to translating the old compiler definitions into packages.
opam admin upgrade
command on your repositories. The
--mirror
option will create a 2.0 mirror and put in place proper
redirections, allowing your original repository to retain the old formatThe official opam repository at https://opam.ocaml.org remains in 1.2 format for now, but has a live-updated 2.0 mirror to which you should be automatically redirected. It cannot yet accept package definitions in 2.0 format.
available:
constraints based on the OCaml compiler version should be
rewritten into dependencies to the ocaml
packagebuild:
and install:
instructions are now requiredurl
and descr
files (containing the
archive URL and package description) in the opam
file itself: (see the new
synopsis:
and
description:
fields, and the
url {} file
section)build:
instructions, using the {test}
and {doc}
filters. The build-test:
and
build-doc:
fields are still supported.depends: [ "foo" {= version} ]
, for a dependency to package foo
at the
same version as the package being defined, or depends:
[ "bar" {os = "linux"} ]
for a dependency that only applies on Linux.conflict-class:
field allows mutual conflicts among a set of
packages to be declared. Useful, for example, when there are many concurrent,
incompatible implementations.ocaml-version:
field has been deprecated for a long time and is no
longer accepted. This should now be a dependency on the ocaml
packagemd5=<hex-value>
,
sha256=<hex-value>
or sha512=<hex-value>
. We'll be gradually deprecating
md5 in favour of the more secure algorithms; multiple checksums are allowedpatches:
field must apply with patch -p1
setenv:
field allows packages to export updates to environment
variables;x-foo:
can be used for extensions and external tools"""
delimiters allow unescaped strings&
has now the customary higher precedence than |
in formulasremove:
field is usually no longer required.The full, up-to-date specification of the format can be browsed in the manual.
In the official, default repository, and also when migrating repositories from older format versions, there are:
ocaml
package, that depends on any implementation of the OCaml
compiler. This is what packages should depend on, and the version is the
corresponding base OCaml version (e.g. 4.04.0
for the 4.04.0+fp
compiler).
It also defines various configuration variables, see opam config list ocaml
.ocaml-base-compiler
is the official releasesocaml-variants.<base-version>+<variant-name>
contains all the other
variantsocaml-system-compiler
maps to a compiler installed on the system
outside of opamThe layout is otherwise the same, apart from:
compilers/
directory is ignoredrepo
file should be present, containing at least the line opam-version: "2.0"
urls.txt
is no
longer needed. See opam admin index --help
archives/
directory is no longer used. The cache now uses a different
format and is configured through the repo
file, defaulting to cache/
on
the same server. See opam admin cache --help
Thanks for trying out the beta! Please let us have feedback, preferably to the opam tracker; other options include the opam-devel list and #opam IRC channel on Freenode.
The package for opam-lib version 1.3 has just been released in the official opam repository. There is no release of opam with version 1.3, but this is an intermediate version of the library that retains compatibility of the file formats with 1.2.2.
The purpose of this release is twofold:
lint
functionThis version is compatible with the current stable release of opam (1.2.2), but dependencies have been updated so that you are not (e.g.) stuck on an old version of ocamlgraph.
Therefore, I encourage all maintainers of tools based on opam-lib to migrate to 1.3.
The respective APIs are available in html for 1.2 and 1.3.
A note on plugins: when you write opam-related tools, remember that by setting
flags: plugin
in their definition and installing a binary namedopam-toolname
, you will enable the users to install packagetoolname
and run your tool with a singleopam toolname
command.
If you need to migrate from 1.2 to 1.3, these tips may help:
there are now 6 different ocamlfind sub-libraries instead of just 4: format
contains the handlers for opam types and file formats, has been split out from
the core library, while state
handles the state of a given opam root and
switch and has been split from the client
library.
OpamMisc
is gone and moved into the better organised OpamStd
, with
submodules for String
, List
, etc.
OpamGlobals
is gone too, and its contents have been moved to:
OpamConsole
for the printing, logging, and shell interface handling part
OpamXxxConfig
modules for each of the libraries for handling the global
configuration variables. You should call the respective init
functions,
with the options you want to set, for proper initialisation of the lib
options (and handling the OPAMXXX
environment variables)
OpamPath.Repository
is now OpamRepositoryPath
, and part of the
repository
sub-library.
The development version of the opam-lib (2.0~alpha5
as of writing) is already
available on opam. The name has been changed to provide a finer granularity, so
it can actually be installed concurrently -- but be careful not to confuse the
ocamlfind package names (opam-lib.format
for 1.3 vs opam-format
for 2.0).
The provided packages are:
opam-file-format
: now
separated from the opam source tree, this has no dependencies and can be used
to parse and print the raw opam syntax.opam-core
: the basic toolbox
used by opam, which actually doesn't include the opam specific part. Includes
a tiny extra stdlib, the engine for running a graph of processes in parallel,
some system handling functions, etc. Depends on ocamlgraph and re only.opam-format
: defines opam
data types and their file i/o functions. Depends just on the two above.opam-solver
: opam's interface
with the dose3 library and external
solvers.opam-repository
: fetching
repositories and package sources from all handled remote types.opam-state
: handling of the
opam states, at the global, repository and switch levels.opam-client
: the client
library, providing the top-level operations (installing packages...), and CLI.opam-devel
: this packages the
development version of the opam tool itself, for bootstrapping. You can
install it safely as it doesn't install the new opam
in the PATH.The new API can be also be browsed ; please get in touch if you have trouble migrating.
We are pleased to announce a preview release for opam 2.0, with over 700 patches since 1.2.2. Version 2.0~alpha4 has just been released, and is ready to be more widely tested.
This version brings many new features and changes, the most notable one being that OCaml compiler packages are no longer special entities, and are replaced by standard package definition files. This in turn means that opam users have more flexibility in how switches are managed, including for managing non-OCaml environments such as Coq using the same familiar tools.
This is just a sample, see the full changelog for more:
Sandboxed builds: Command wrappers can be configured to, for example, restrict permissions of the build and install processes using Linux namespaces, or run the builds within Docker containers.
Compilers as packages: This brings many advantages for opam workflows,
such as being able to upgrade the compiler in a given switch, better tooling for
local compilers, and the possibility to define coq
as a compiler or even
use opam as a generic shell scripting engine with dependency tracking.
Local switches: Create switches within your projects for easier
management. Simply run opam switch create <directory> <compiler>
to get
started.
Inplace build: Use opam to build directly from
your source directory. Ensure the package is pinned locally then run opam
install --inplace-build
.
Automatic file tracking:: opam now tracks the files installed by packages
and is able to cleanly remove them when no existing files were modified.
The remove:
field is now optional as a result.
Configuration file: This can be used to direct choices at opam init
automatically (e.g. specific repositories, wrappers, variables, fetch
commands, or the external solver). This can be used to override all of opam's
OCaml-related settings.
Simpler library: the OCaml API is completely rewritten and should make it much easier to write external tools and plugins. Existing tools will need to be ported.
Better error mitigation: Through clever ordering of the shell actions and
separation of build
and install
, most build failures can keep your current
installation intact, not resulting in removed packages anymore.
You are very welcome to try out the alpha, and report any issues. The repository
at opam.ocaml.org
will remain in 1.2 format (with a 2.0 mirror at
opam.ocaml.org/2.0~dev
in sync) until after the release is out, which means
the extensions can not be used there yet, but you are welcome to test on local
or custom repositories, or package pinnings. The reverse translation (2.0 to
1.2) is planned, to keep supporting 1.2 installations after that date.
The documentation for the new version is available at http://opam.ocaml.org/doc/2.0/. This is still work in progress, so please do ask if anything is unclear.
Commands opam switch
and opam list
have been rehauled for more consistency
and flexibility: the former won't implicitly create new switches unless called
with the create
subcommand, and opam list
now allows to combine filters and
finely specify the output format. They may not be fully backwards compatible, so
please check your scripts.
Most other commands have also seen fixes or improvements. For example, opam
doesn't forget about your set of installed packages on the first error, and the
new opam install --restore
can be used to reinstall your selection after a
failed upgrade.
While users of opam 1.2 should feel at home with the changes, the 2.0 repository and package formats are not compatible. Indeed, the move of the compilers to standard packages implies some conversions, and updates to the relationships between packages and their compiler. For example, package constraints like
available: [ ocaml-version >= "4.02" ]
are now written as normal package dependencies:
depends: [ "ocaml" {>= "4.02"} ]
To make the transition easier,
opam-admin
upgrade-format
at its root;opam.ocaml.org
already has a 2.0 mirror, to which
you will be automatically redirected;Note that the ocaml
package on the official repository is actually a wrapper
that depends on one of ocaml-base-compiler
, ocaml-system
or
ocaml-variants
, which contain the different flavours of the actual compiler.
It is expected that it may only get picked up when requested by package
dependencies.
The opam package definition format is very similar to before, but there are quite a few extensions and some changes:
build:
and install:
steps (this allows
tracking of installed files, better error recovery, and some optional security
features);opam
file
using the section url {}
and fields synopsis:
and description:
;setenv:
field allows packages to export updates to environment
variables;x-foo:
can be used for extensions and external tools;"""
delimiters around unescaped strings&
is now parsed with higher priority than |
ocaml-version:
can no longer be usedremove:
field should not be used anymore for simple cases (just removing
files)First, be aware that you'll be prompted to update your ~/.opam
to 2.0 format
before anything else, so if you value it, make a backup. Or just export
OPAMROOT
to test the alpha on a temporary opam root.
Packages for opam 2.0 are already in the opam repository, so if you have a working opam installation of opam (at least 1.2.1), you can bootstrap as easily as:
opam install opam-devel
This doesn't install the new opam to your PATH within the current opam root for obvious reasons, so you can manually install it as e.g. "opam2" using:
sudo cp $(opam config var "opam-devel:lib")/opam /usr/local/bin/opam2
You can otherwise install as usual:
Using pre-built binaries (available for OSX and Linux x86, x86_64, armhf) and our install script:
wget https://raw.github.com/ocaml/opam/2.0-alpha4-devel/shell/opam_installer.sh -O - | sh -s /usr/local/bin
Equivalently, pick your version and download it to your PATH;
Building from our inclusive source tarball:
download here
and build using ./configure && make lib-ext && make && make install
if you
have OCaml >= 4.01 already available, make cold && make install
otherwise;
Or from source, following the
included instructions from the README. Some files have been moved around, so
if your build fails after you updated an existing git clone, try to clean it
up (git clean -fdx
).
NOTE (September 2016): updated proposal from OCaml 2016 workshop is available, including links to prototype implementation.
This is an initial proposal on signing the OPAM repository. Comments and discussion are expected on the platform mailing-list.
The purpose of this proposal is to enable a secure distribution of OCaml packages. The package repository does not have to be trusted if package developers sign their releases.
Like Python's pip, Ruby's gems or more recently Haskell's hackage, we are going to implement a flavour of The Upgrade Framework (TUF). This is good because:
Importantly, it doesn't enforce any specific cryptography, allowing us to go with what we have at the moment in native OCaml, and evolve later, e.g. by allowing ed25519.
There are several differences between the goal of TUF and opam, namely TUF distributes a directory structure containing the code archive, whereas opam distributes metadata about OCaml packages. Opam uses git (and GitHub at the moment) as a first class citizen: new packages are submitted as pull requests by developers who already have a GitHub account.
Note that TUF specifies the signing hierarchy and the format to deliver and check signatures, but allows a lot of flexibility in how the original files are signed: we can have packages automatically signed on the official repository, or individually signed by developers. Or indeed allow both, depending on the package.
Below, we tried to explain the specifics of our implementation, and mostly the user and developer-visible changes. It should be understandable without prior knowledge of TUF.
We are inspired by Haskell's adjustments (and e2e) to TUF using a git repository for packages. A signed repository and signed packages are orthogonal. In this proposal, we aim for both, but will describe them independently.
An attacker can compromise at least one of the package distribution system's online trusted keys.
An attacker compromising multiple keys may do so at once or over a period of time.
An attacker can respond to client requests (MITM or server compromise) during downloading of the repository, a package, and also while uploading a new package release.
An attacker knows of vulnerabilities in historical versions of one or more packages, but not in any current version (protecting against zero-day exploits is emphatically out-of-scope).
Offline keys are safe and securely stored.
An attacker is considered successful if they can cause a client to build and install (or leave installed) something other than the most up-to-date version of the software the client is updating. If the attacker is preventing the installation of updates, they want clients to not realize there is anything wrong.
Arbitrary package: an attacker should not be able to provide a package they created in place of a package a user wants to install (via MITM during package upload, package download, or server compromise).
Rollback attacks: an attacker should not be able to trick clients into installing software that is older than that which the client previously knew to be available.
Indefinite freeze attacks: an attacker should not be able to respond to client requests with the same, outdated metadata without the client being aware of the problem.
Endless data attacks: an attacker should not be able to respond to client requests with huge amounts of data (extremely large files) that interfere with the client's system.
Slow retrieval attacks: an attacker should not be able to prevent clients from being aware of interference with receiving updates by responding to client requests so slowly that automated updates never complete.
Extraneous dependencies attacks: an attacker should not be able to cause clients to download or install software dependencies that are not the intended dependencies.
Mix-and-match attacks: an attacker should not be able to trick clients into using a combination of metadata that never existed together on the repository at the same time.
Malicious repository mirrors: should not be able to prevent updates from good mirrors.
Wrong developer attack: an attacker should not be able to upload a new version of a package for which they are not the real developer.
A difficult problem in a cryptosystem is key distribution. In TUF and this proposal, a set of root keys are distributed with opam. A threshold of these root keys needs to sign (transitively) all keys which are used to verify opam repository and its packages.
The root of trust is stored in a set of root keys. In the case of the official opam OCaml repository, the public keys are to be stored in the opam source, allowing it to validate the whole trust chain. The private keys will be held by the opam and repository maintainers, and stored password-encrypted, securely offline, preferably on unplugged storage.
They are used to sign all the top-level keys, using a quorum. The quorum has several benefits:
The added cost is more maintenance burden, but this remains small since these keys are not often used (only when keys are going to expire, were compromised or in the event new top-level keys need to be added).
The initial root keys could be distributed as such:
Keys will be set with an expiry date so that one expires each year in turn, leaving room for smooth rollover.
For other repositories, there will be three options:
This requires the end-user to be able to validate a signature made by the original developer. There are two trust paths for the chain of trust (where "→" stands for "signs for"):
It is intended that packages may initially follow the low trust path, adding as little burden and delay as possible when adding new packages, and may then be promoted to the high path with manual intervention, after verification, from repository maintainers. This way, most well-known and widely used packages will be provided with higher trust, and the scope of an attack on the low trust path would be reduced to new, experimental or little-used packages.
This provides consistent, up-to-date snapshots of the repository, and protects against a whole different class of attacks than end-to-end signing (e.g. rollbacks, mix-and-match, freeze, etc.)
This is done automatically by a snapshot bot (might run on the repository server), using the snapshot key, which is signed directly by the root keys, hence the chain of trust:
Where "commit-hash" is the head of the repository's git repository (and thus a valid cryptographic hash of the full repository state, as well as its history)
Repository maintainers hold the central role in monitoring the repository and warranting its security, with or without signing. Their keys (called targets keys in the TUF framework) are signed directly by the root keys. As they have a high security potential, in order to reduce the consequences of a compromise, we will be requiring a quorum for signing sensitive operations
These keys are stored password-encrypted on the RM computers.
This key is held by the snapshot bot and signed directly by the root keys. It is used to guarantee consistency and freshness of repository snapshots, and does so by signing a git commit-hash and a time-stamp.
It is held online and used by the snapshot bot for automatic signing: it has lower security than the RM keys, but also a lower potential: it can not be used directly to inject malicious code or metadata in any existing package.
These keys are used by the package developers for end-to-end signing. They can
be generated locally as needed by new packagers (e.g. by the opam-publish
tool), and should be stored password-encrypted. They can be added to the
repository through pull-requests, waiting to be signed (i) as part of snapshots
(which also prevents them to be modified later, but we'll get to it) and (ii)
directly by RMs.
We'll need to start somewhere, and the current repository isn't signed. An additional key, initial-bootstrap, will be used for guaranteeing integrity of existing, but yet unverified packages.
This is a one-go key, signed by the root keys, and that will then be destroyed. It is allowed to sign for packages without delegation.
In order to build the trust chain, the opam client downloads a keys/root
key
file initially and before every update operation. This file is signed by the
root keys, and can be verified by the client using its built-in keys (or one of
the ways mentioned above for unofficial repositories). It must be signed by a
quorum of known root keys, and contains the comprehensive set of root, RM,
snapshot and initial bootstrap keys: any missing keys are implicitly revoked.
The new set of root keys is stored by the opam client and used instead of the
built-in ones on subsequent runs.
Developer keys are stored in files keys/dev/<id>
, self-signed, possibly RM
signed (and, obviously, snapshot-signed). The conditions of their verification,
removal or replacement are included in our validation of metadata update (see
below).
The files follow the opam syntax: a list of fields fieldname:
followed by
contents. The format is detailed in opam's documentation.
The signature of files in opam is done on the canonical textual representation, following these rules:
signature:
field is removedfieldname:
, between elements in lists, before braced options, between
operators and their operandsThe signature:
field is a list with elements in the format of string triplets
[ "<keyid>" "<algorithm>" "<signature>" ]
. For example:
opam-version: "1.2"
name: "opam"
signature: [
[ "louis.gesbert@ocamlpro.com" "RSASSA-PSS" "048b6fb4394148267df..." ]
]
Signed tags are git annotated tags, and their contents follow the same rules. In
this case, the format should contain the field commit:
, pointing to the
commit-hash that is being signed and tagged.
The repository format is changed by the addition of:
keys/
at the rootpackages/<pkgname>/delegate
and
compilers/<patchname>.delegate
packages/<pkgname>/<pkgname>.<version>/signature
Here is an example:
repository root /
|--packages/
| |--pkgname/
| | |--delegation - signed by developer, repo maintainer
| | |--pkgname.version1/
| | | |--opam
| | | |--descr
| | | |--url
| | | `--signature - signed by developer1
| | `--pkgname.version2/ ...
| `--pkgname2/ ...
|--compilers/
| |--version/
| | |--version+patch/
| | | |--version+patch.comp
| | | |--version+patch.descr
| | | `--version+patch.signature
| | `--version+patch2/ ...
| |--patch.delegate
| |--patch2.delegate
| `--version2/ ...
`--keys/
|--root
`--dev/
|--developer1-email - signed by developer1,
`--developer2-email ... and repo maint. once verified
Keys are provided in different files as string triplets
[ [ "keyid" "algo" "key" ] ]
. keyid
must not conflict with any
previously-defined keys, and algo
may be "rsa" and keys encoded in PEM format,
with further options available later.
For example, the keys/root
file will have the format:
date: "2015-06-04T13:53:00Z"
root-keys: [ [ "keyid" "{expire-date}" "algo" "key" ] ]
snapshot-keys: [ [ "keyid" "algo" "key" ] ]
repository-maintainer-keys: [ [ "keyid" "algo" "key" ] ]
This file is signed by current and past root keys -- to allow clients to
update. The date:
field provides further protection against rollback attacks:
no clients may accept a file with a date older than what they currently have.
Date is in the ISO 8601 standard with 0 UTC offset, as suggested in TUF.
/packages/pkgname/delegation
delegates ownership on versions of package
pkgname
. The file contains version constraints associated with keyids, e.g.:
name: pkgname
delegates: [
"thomas@gazagnaire.org"
"louis.gesbert@ocamlpro.com" {>= "1.0"}
]
The file is signed:
Every key a developer delegates trust to must also be signed by the developer.
compilers/patch.delegate
files follow a similar format (we are considering
changing the hierarchy of compilers to match that of packages, to make things
simpler).
The delegates:
field may be empty: in this case, no packages by this name are
allowed on the repository. This may be useful to mark deletion of obsolete
packages, and make sure a new, different package doesn't take the same name by
mistake or malice.
These guarantee the integrity of a package: this includes metadata and the package archive itself (which may, or may not, be mirrored on the the opam repository server).
The file, besides the package name and version, has a field package-files:
containing a list of files below packages/<pkgname>/<pkgname>.<version>
together with their file sizes in bytes and one or more hashes, prefixed by their
kind, and a field archive:
containing the same details for the upstream
archive. For example:
name: pkgname
version: pkgversion
package-files: [
"opam" {901 [ sha1 "7f9bc3cc8a43bd8047656975bec20b578eb7eed9" md5 "1234567890" ]}
"descr" {448 [ sha1 "8541f98524d22eeb6dd669f1e9cddef302182333" ]}
"url" {112 [ sha1 "0a07dd3208baf4726015d656bc916e00cd33732c" ]}
"files/ocaml.4.02.patch" {17243 [ sha1 "b3995688b9fd6f5ebd0dc4669fc113c631340fde" ]}
]
archive: [ 908460 [ sha1 "ec5642fd2faf3ebd9a28f9de85acce0743e53cc2" ] ]
This file is signed either:
initial-bootstrap
key, only initiallyThe latter is needed to hot-fix packages on the repository: repository maintainers often need to do so. A quorum is still required, to prevent a single RM key compromise from allowing arbitrary changes to every package. The quorum is not initially required to sign a delegation, but is, consistently, required for any change to an existing, signed delegation.
Compiler signature files <version>+<patch>.signature
are similar, with fields
compiler-files
containing checksums for <version>+<patch>.*
, the same field
archive:
and an additional optional field patches:
, containing the sizes and
hashes of upstream patches used by this compiler.
If the delegation or signature can't be validated, the package or compiler is ignored. If any file doesn't correspond to its size or hashes, it is ignored as well. Any file not mentioned in the signature file is ignored.
The snapshot key automatically adds a signed
annotated tag to the top of the
served branch of the repository. This tag contains the commit-hash and the
current timestamp, effectively ensuring freshness and consistency of the full
repository. This protects against mix-and-match, rollback and freeze attacks.
The signed
annotated tag is deleted and recreated by the snapshot bot, after
checking the validity of the update, periodically and after each change.
The repository is served using git: this means, not only the latest version, but the full history of changes are known. This as several benefits, among them, incremental downloads "for free"; and a very easy way to sign snapshots. Another good point is that we have a working full OCaml implementation.
We mentioned above that we use the snapshot signatures not only for repository signing, but also as an initial guarantee for submitted developer's keys and delegations. One may also have noticed, in the above, that we sign for delegations, keys etc. individually, but without a bundle file that would ensure no signed files have been maliciously removed.
These concerns are all addressed by a linearity condition on the repository's git: the snapshot bot does not only check and sign for a given state of the repository, it checks every individual change to the repository since the last well-known, signed state: patches have to follow from that git commit (descendants on the same branch), and are validated to respect certain conditions: no signed files are removed or altered without signature, etc.
Moreover, this check is also done on clients, every time they update: it is slightly weaker, as the client don't update continuously (an attacker may have rewritten the commits since last update), but still gives very good guarantees.
A key and delegation that have been submitted by a developer and merged, even without RM signature, are signed as part of a snapshot: git and the linearity conditions allow us to guarantee that this delegation won't be altered or removed afterwards, even without an individual signature. Even if the repository is compromised, an attacker won't be able to roll out malicious updates breaking these conditions to clients.
The linearity invariants are:
It is sometimes needed to do operations, like key revocation, that are not allowed by the above rules. These are enabled by the following additional rules, that require the commit including the changes to be signed by a quorum of repository maintainers using an annotated tag:
Changes to the keys/root
file, which may add, modify or revoke keys for root,
RMs and snapshot keys is verified in the normal way, but needs to be handled for
checking linearity since it decides the validity of RM signatures. Since this
file may be needed before we can check the signed
tag, it has its own
timestamp to prevent rollback attacks.
In case the linearity invariant check fail:
signed
tag and master) need to be amended.Write modules for key handling ; signing and verification of opam files.
Write the git synchronisation module with linearity checks.
Rewrite the default HTTP repository synchronisation module to use git fetch, verify, and git pull. This should be fully transparent, except:
Include the public root keys for the default repository, and implement
management of updated keys in ~/.opam/repo/name
.
Handle the new formats for checksums and non-repackaged archives.
Allow a per-repository security threshold (e.g. allow all, allow only signed packages, allow only packages signed by a verified key, allow only packages signed by their verified developer). It should be easy but explicit to add a local network, unsigned repository. Backends other than git won't be signed anyway (local, rsync...).
Generate keys, handle locally stored keys, generate signature
files, handle
signing, submit signatures, check delegation, submit new delegation, request
delegation change (might require repository maintainer intervention if an RM
signed the delegation), delete developer, delete package.
Manage local keys. Probably including re-generating, and asking for revocation.
Most operations on signatures and keys will be implemented as independent
modules (as to be usable from e.g. unikernels working on the repository). We
should also make them available from opam-admin
, for testing and manual
management. Special tooling will also be needed by RMs.
pkg+opam.tar.gz
anymore)If we don't want to have this processed on the publicly visible host serving the repository, we'll need a mechanism to fetch the repository, and submit the signed tag back to the repository server.
Doing this through mirage unikernels would be cool, and provide good isolation. We could imagine running this process regularly:
signed
tagAll security information and check results should be available to RMs before they make the decision to merge a commit to the repository. This means including signature and linearity checks in a process running on Travis, or similarly on every pull-request to the repository, and displaying the results in the GitHub tracker.
This should avoid most cases where the snapshot bot fails the validation, leaving it stuck (as well as any repository updates) until the bad commits are rewritten.
opam init
and update
scenarioOn init
, the client would clone the repository and get to the signed
tag,
get and check the associated keys/root
file, and validate the signed
tag
according to the new keyset. If all goes well, the new set of root, RM and
snapshot keys is registered.
Then all files' signatures are checked following the trust chains, and copied to
the internal repository mirror opam will be using (~/.opam/repo/<name>
). When
a package archive is needed, the download is done either from the repository, if
the file is mirrored, or from its upstream, in both cases with known file size
and upper limit: the download is stopped if going above the expected size, and
the file removed if it doesn't match both.
On subsequent updates, the process is the same except that a fetch operation is
done on the existing clone, and that the repository is forwarded to the new
signed
tag only if linearity checks passed (and the update is aborted
otherwise).
opam-publish
scenarioopam-publish submit
, a developer key is
generated, and stored locally.opam-publish submit
, the package is signed using the key, and the
signature is included in the submission./keys/dev/
within the
pull-request, self-signed./packages/pkgname/delegation
file is added, delegating to the developer key and signed by it.We claim that the above measures give protection against:
Arbitrary packages: if an existing package is not signed, it is not installed (or even visible) to the user. Anybody can submit new unclaimed packages (but, in the current setting, still need GitHub write access to the repository, or to bypass GitHub's security).
Rollback attacks: git updates must follow the currently known signed
tag. if
the snapshot bot detects deletions of packages, it refuses to sign, and
clients double-check this. The keys/root
file contains a timestamp.
Indefinite freeze attacks: the snapshot bot periodically signs the signed
tag with a timestamp, if a client receives a tag older than the expected age
it will notice.
Endless data attacks: we rely on the git protocol and this does not defend against endless data. Downloading of package archive (of which the origin may be any mirror), though, is protected. The scope of the attack is mitigated in our setting, because there are no unattended updates: the program is run manually, and interactively, so the user is most likely to notice.
Slow retrieval attacks: same as above.
Extraneous dependencies attacks: metadata is signed, and if the signature does not match, it is not accepted.
NOTE: the
provides
field -- yet unimplemented, see the document inopam/doc/design
-- could provide a vector in this case, by advertising a replacement for a popular package. Additional measures will be taken when implementing the feature, like requiring a signature for the provided package.
Mix-and-match attacks: the repository has a linearity condition, and partial repositories are not possible.
Malicious repository mirrors: if the signature does not match, reject.
Wrong developer attack: if the developer is not in the delegation, reject.
Is the link between GitHub (opam-repository) and the signing bot special? If there is a MITM on this link, they can add arbitrary new packages, but due to missing signatures only custom universes. No existing package can be altered or deleted, otherwise consistency condition above does not hold anymore and the signing bot will not sign.
Certainly, the access can be frozen, thus the signing bot does not receive updates, but continues to sign the old repository version.
If the snapshot key is compromised, an attacker is able to:
Add arbitrary (non already existing) packages, as above.
Freeze, by forever re-signing the signed
tag with an updated timestamp.
Most importantly, the attacker won't be able to tamper with existing packages. This hudgely reduces the potential of an attack, even with a compromised snapshot key.
The attacks above would also require either a MITM between the repository and the client, or a compromise of the opam repository: in the latter case, since the linearity check is reproduces even from the clients:
The repository would then just have to be reset to before the attack, which git
makes as easy as it can get, and the holders of the root keys would sign a new
/auth/root
, revoking the compromised snapshot key and introducing a new one.
In the time before the signing bot can be put back online with the new snapshot key -- i.e. the breach has been found and fixed -- a developer could manually sign time-stamped tags before they expire (e.g. once a day) so as not to hold back updates.
Repository maintainers are powerful, they can modify existing opam files and sign them (as hotfix), introduce new delegations for packages, etc.).
However, by requiring a quorum for sensitive operations, we limit the scope of a single RM key compromise to the validation of new developer keys or delegations (which should be the most common operation done by RMs): this enables to raise the level of security of the new, malicious packages but otherwise doesn't change much from what can be done with just access to the git repository.
A further compromise of a quorum of RM keys would allow to remove or tamper with any developer key, delegation or package: any of these amounts to being able to replace any package with a compromised version. Cleaning up would require replacing all but the root keys, and resetting the repository to before any malicious commit.
OPAM 1.2.2 has just been released. This fixes a few issues over 1.2.1 and brings a couple of improvements, in particular better use of the solver to keep the installation as up-to-date as possible even when the latest version of a package can not be installed.
See the normal installation instructions: you should generally pick up the packages from the same origin as you did for the last version -- possibly switching from the official repository packages to the ones we provide for your distribution, in case the former are lagging behind.
There are no changes in repository format, and you can roll back to earlier versions in the 1.2 branch if needed.
opam lint
checks, opam lint
now numbers its warnings and may
provide script-friendly outputopam depext
will prompt
to install depext
if available and not already installedopam list --resolve
to list a consistent installation scenarioopam lint
to check them.opam config report
has been fixed to report the external solver properly--dry-run --verbose
properly outputs all commands that would be run againopam list
now returns 0 when no packages match but no pattern was supplied,
which is more helpful in scripts relying on it to check dependencies.OPAM 1.2.1 has just been released. This patch version brings a number of fixes and improvements over 1.2.0, without breaking compatibility.
See the normal installation instructions: you should generally pick up the packages from the same origin as you did for the last version -- possibly switching from the official repository packages to the ones we provide for your distribution, in case the former are lagging behind.
No huge new features in this point release -- which means you can roll back to 1.2.0 in case of problems -- but lots going on under the hood, and quite a few visible changes nonetheless:
jobs:
to a value greater than 1 in
~/.opam/config
in case you updated from an older version.<name>.opam
files for package
pinning. URLs of the form git+ssh://
or hg+https://
are now allowed.opam lint
has been vastly improved.... and much more
There is also a new manual documenting the file and repository formats.
See the changelog for a summary or closed issues in the bug-tracker for an overview.
These are mostly improvements to the file formats. You are welcome to use them, but they won't be accepted into the official repository until the next release.
features:
in opam files, to help with ./configure
scripts and
documenting the specific features enabled in a given build. See the
original proposal
and the section in the new manuallibexec:
in <name>.install
files, to install into the package's
lib dir with the execution bit set.packages:
field are now resolved and then
locked. In practice, this means that repository maintainers can move the
compiler itself to a package, giving a lot more flexibility.Last week, we published an alpha version of a new OCaml documentation generator, codoc 0.2.0. In the 2014 OCaml workshop presentation (abstract, slides, video), we mentioned the 'module wall' for documentation and this attempts to fix it. To try it out, simply follow the directions in the README on that repository, or browse some samples of the current, default output of the tool. Please do bear in mind codoc and its constituent libraries are still under heavy development and are not feature complete.
codoc
's aim is to provide a widely useful set of tools for generating OCaml
documentation. In particular, we are striving to:
codoc
We haven't yet achieved all of these at all levels of our tool stack but are
getting close. codoc
0.2.0 is usable today (if a little rough in some areas
like default CSS). This post outlines the architecture of the new system to
make it easier to understand the design decisions that went into it.
There are five stages in generating documentation from OCaml source code. Here we describe how each was handled in the past (using OCamldoc), the present (using our current prototype), and the future (using the final version of the tools we are developing).
The first stage is to associate the various documentation comments in
an .ml
or .mli
file with the definitions that they correspond to.
Associating comments with definitions is handled by the OCamldoc tool, which does this in two steps. First it parses the file using the regular OCaml parser or camlp4, just as in normal compilation. It uses the syntax tree from the first step and then re-parses the file looking for comments. This second parse is guided by the location information in the syntax tree; for example if there is a definition which ends on line 5 then OCamldoc will look for comments to attach to that definition starting at line 6.
The rules used for attaching comments are quite intricate and whitespace dependent. This makes it difficult to parse the file and attach comments using a single parser. In particular, it would be difficult to do so in a way that doesn't cause a lot of problems for camlp4 extensions. This is why OCamldoc does the process in two steps.
A disadvantage of this two-step approach is that it assumes that the input to any preprocessor is something which could reasonably be read by the compiler/tool creating documentation, which may not always be the case.
Our current prototype associates comments with definitions within the
compiler itself. This relies on a patch to the OCaml compiler
(pull request #51 on GitHub).
Comment association is activated by the -doc
command-line flag. It
uses (a rewritten version of) the same two-step algorithm currently
used by OCamldoc. The comments are then attached to the appropriate node
in the syntax tree as an attribute. These attributes are passed through
the type-checker and appear in .cmt
/.cmti
files, where they can be
read by other tools.
We intend to move away from the two-step approach taken by OCamldoc. To do this we will need to simplify the rules for associating comments with definitions. One suggestion was to use the same rules as attributes, however that seems to be overly restrictive. So the approach we hope to take is to keep quite close to what OCamldoc currently supports, but disallow some of the more ambiguous cases. For example,
val x : int
(** Is this for x or y? *)
val y : float
may well not be supported in our final version.
We will take care to understand the impact of such design decisions and we
hope to arrive at a robust solution for the future.
By avoiding the two-step
approach, it should be safe to always turn on comment association rather
than requiring a -doc
command-line flag.
Once you have associated documentation comments with definitions, you must parse the contents of these comments.
OCamldoc parses the contents of comments.
In our current prototype, the contents of comments are parsed in the
compiler, so that the documentation attributes available in
.cmt
/.cmti
files contain a structured representation of the
documentation.
We intend to separate parsing the contents of documentation comments
from the compiler. This means that the documentation will be stored as
strings within the .cmt
/.cmti
files and parsed by external
tools. This will allow the documentation language (and its parser) to
evolve faster than the distribution cycle of the compiler.
The typed syntax tree stored in .cmt
/.cmti
files is not a convenient
representation for generating documentation from, so the next stage is
to convert the syntax tree and comments into some suitable intermediate
form. In particular, this allows .cmt
files and .cmti
files to be
treated uniformly.
OCamldoc generates an intermediate form from a syntax tree, a typed syntax tree, and the comments that it found and parsed in the earlier stages. The need for both an untyped and typed syntax tree is a historical artefact that is no longer necessary.
Our current prototype creates an intermediate form in the
doc-ock library. This form can be
currently be created from .cmti
files or .cmi
files. .cmi
files do
not contain enough information for complete documentation, but you can
use them to produce partial documentation if the .cmti
files are not
available to you.
This intermediate form can be serialised to XML using doc-ock-xml.
In the final version, doc-ock will also support reading .cmt
files.
Once you have a representation for documentation, you need to resolve all the paths and references so that links can point to the correct locations. For example,
(* This type is used by {!Foo} *) type t = Bar.t
The path Bar.t
and the reference Foo
must be resolved so that the
documentation can include links to the corresponding definitions.
If you are generating documentation for a large collection of packages, there
may be more than one module called Foo
. So it is important to be able
to work out which one of these Foo
s the reference is referring to.
Unlike most languages, resolving paths can be very difficult in OCaml due to the powerful module system. For example, consider the following code:
module Dep1 : sig
module type S = sig
class c : object
method m : int
end
end
module X : sig
module Y : S
end
end
module Dep2 :
functor (Arg : sig module type S module X : sig module Y : S end end) ->
sig
module A : sig
module Y : Arg.S
end
module B = A.Y
end
type dep1 = Dep2(Dep1).B.c;;
Here it looks like, Dep2(Dep1).B.c
would be defined by a type
definition c
within the submodule B
of the functor Dep2
. However,
Dep2.B
's type is actually dependent on the type of Dep2
's Arg
parameter, so the actual definition is the class definition within the
module type S
of the Dep1
module.
OCamldoc does resolution using a very simple string based lookup. This
is not designed to handle collections of projects, where module names
are not unique. It is also not sophisticated enough to handle advanced
uses of OCaml's module system (e.g. it fails to resolve the path
Dep2(Dep1).B.c
in the above example).
In our current prototype, path and reference resolution are performed by
the doc-ock library. The implementation
amounts to a reimplementation of OCaml's module system that tracks
additional information required to produce accurate paths and references
(it is also lazy to improve performance). The system uses the digests
provided by .cmti
/.cmi
files to resolve references to other modules,
rather than just relying on the module's name.
There are still some paths handled incorrectly by doc-ock-lib, which will be fixed, but mostly the final version will be the same as the current prototype.
Finally, you are ready to produce some output from the tools.
OCamldoc supports a variety of output formats, including HTML and LaTeX. It also includes support for plugins called "custom generators" which allow users to add support for additional formats.
codoc
only supports HTML and XML output at present, although extra output
formats such as JSON should be very easy to add once the interfaces settle
down. codoc
defines a documentation index XML format for tracking package
hierarchies, documentation issues, and hierarchically localized configuration.
codoc
also defines a scriptable command-line interface giving users access
to its internal documentation phases: extraction, linking, and rendering. The
latest instructions on how to use the CLI can be found in the
README. We provide an OPAM remote with
all the working versions of the new libraries and compiler patches required to
drive the new documentation engine.
As previously mentioned, codoc and its constituent libraries doc-ock-lib and doc-ock-xml are still under heavy development and are not yet feature complete. Notably, there are some important outstanding issues:
--package
. (issue codoc#42)We are very happy to take bug reports and patches at https://github.com/dsheets/codoc/issues. For wider suggestions, comments, complaints and discussions, please join us on the Platform mailing list. We do hope that you'll let us know what you think and help us build a next generation documentation tool which will serve our community admirably.
This is a guest post from an OPAM user about how they use it. If you would like to post about your own use, please let us know.
XenServer uses the Xen project's "Xapi toolstack": a suite of tools written mostly in OCaml which
The Xapi toolstack is built from a large set of libraries and components which are developed independently and versioned separately. It's easy for us to share code with other open-source projects like Mirage, however this flexibility comes with a cost: when one binary such as "xapi" (the cluster manager) depends on 45 separate libraries, how do we quickly set up a build environment? Exactly which libraries do we need? How do we apply updates? If we change one of these libraries (e.g. to make a bugfix), exactly which bits should we rebuild? This is where OPAM, the source package manager, makes everything easy.
Installing a build environment with OPAM is particularly easy. For example in a CentOS 6.5 VM, first install OPAM:
and then:
$ opam init --comp=4.01.0
$ eval `opam config env`
Next install the necessary C libraries and development tools for xapi using a command like
$ sudo yum install `opam install xapi -e centos`
Finally to build xapi itself:
$ opam install xapi
∗ install obuild 0.1.1 [required by cdrom, nbd]
∗ install base-no-ppx base [required by lwt]
∗ install cmdliner 0.9.7 [required by nbd, tar-format]
∗ install camlp4 4.01.0 [required by nbd]
∗ install ocamlfind 1.5.5 [required by xapi]
∗ install xmlm 1.2.0 [required by xapi-libs-transitional, rpc, xen-api-client]
∗ install uuidm 0.9.5 [required by xapi-forkexecd]
∗ install type_conv 111.13.00 [required by rpc]
∗ install syslog 1.4 [required by xapi-forkexecd]
∗ install ssl 0.4.7 [required by xapi]
∗ install ounit 2.0.0 [required by xapi]
∗ install omake 0.9.8.6-0.rc1 [required by xapi]
∗ install oclock 0.4.0 [required by xapi]
∗ install libvhd 0.9.0 [required by xapi]
∗ install fd-send-recv 1.0.1 [required by xapi]
∗ install cppo 1.1.2 [required by ocplib-endian]
∗ install cdrom 0.9.1 [required by xapi]
∗ install base-bytes legacy [required by ctypes, re]
∗ install sexplib 111.17.00 [required by cstruct]
∗ install fieldslib 109.20.03 [required by cohttp]
∗ install lwt 2.4.7 [required by tar-format, nbd, rpc, xen-api-client]
∗ install xapi-stdext 0.12.0 [required by xapi]
∗ install stringext 1.2.0 [required by uri]
∗ install re 1.3.0 [required by xapi-forkexecd, tar-format, xen-api-client]
∗ install ocplib-endian 0.8 [required by cstruct]
∗ install ctypes 0.3.4 [required by opasswd]
∗ install xenctrl 0.9.26 [required by xapi]
∗ install rpc 1.5.1 [required by xapi]
∗ install xapi-inventory 0.9.1 [required by xapi]
∗ install uri 1.7.2 [required by xen-api-client]
∗ install cstruct 1.5.0 [required by tar-format, nbd, xen-api-client]
∗ install opasswd 0.9.3 [required by xapi]
∗ install xapi-rrd 0.9.1 [required by xapi-idl, xapi-rrd-transport]
∗ install cohttp 0.10.1 [required by xen-api-client]
∗ install xenstore 1.2.5 [required by xapi]
∗ install tar-format 0.2.1 [required by xapi]
∗ install nbd 1.0.2 [required by xapi]
∗ install io-page 1.2.0 [required by xapi-rrd-transport]
∗ install crc 0.9.0 [required by xapi-rrd-transport]
∗ install xen-api-client 0.9.7 [required by xapi]
∗ install message-switch 0.10.4 [required by xapi-idl]
∗ install xenstore_transport 0.9.4 [required by xapi-libs-transitional]
∗ install mirage-profile 0.4 [required by xen-gnt]
∗ install xapi-idl 0.9.19 [required by xapi]
∗ install xen-gnt 2.2.0 [required by xapi-rrd-transport]
∗ install xapi-forkexecd 0.9.2 [required by xapi]
∗ install xapi-rrd-transport 0.7.2 [required by xapi-rrdd-plugin]
∗ install xapi-tapctl 0.9.2 [required by xapi]
∗ install xapi-netdev 0.9.1 [required by xapi]
∗ install xapi-libs-transitional 0.9.6 [required by xapi]
∗ install xapi-rrdd-plugin 0.6.0 [required by xapi]
∗ install xapi 1.9.56
===== ∗ 52 =====
Do you want to continue ? [Y/n] y
Obviously it's extremely tedious to do all that by hand!
OPAM also makes iterative development very easy. Consider a scenario where a common interface has to be changed. Without OPAM we have to figure out which components to rebuild manually-- this is both time-consuming and error-prone. When we want to make some local changes we simply clone the repo and tell OPAM to "pin" the package to the local checkout. OPAM will take care of rebuilding only the dependent packages:
$ git clone git://github.com/xapi-project/xcp-idl
... make some local changes ...
$ opam pin add xapi-idl ./xcp-idl
$ opam install xapi
...
xapi-idl needs to be reinstalled.
The following actions will be performed:
↻ recompile xapi-idl 0.9.19*
↻ recompile xapi-rrd-transport 0.7.2 [uses xapi-idl]
↻ recompile xapi-forkexecd 0.9.2 [uses xapi-idl]
↻ recompile xapi-tapctl 0.9.2 [uses xapi-forkexecd]
↻ recompile xapi-netdev 0.9.1 [uses xapi-forkexecd]
↻ recompile xapi-libs-transitional 0.9.6 [uses xapi-forkexecd]
↻ recompile xapi-rrdd-plugin 0.6.0 [uses xapi-idl]
↻ recompile xapi 1.9.56 [uses xapi-idl]
===== ↻ 8 =====
Do you want to continue ? [Y/n]
It's even easier if you just want to pin to a branch, such as master:
$ opam pin add xapi-idl git://github.com/xapi-project/xcp-idl
It's important to be able to iterate quickly when testing a bugfix-- OPAM makes this easy too. After making a change to a "pinned" repository the user just has to type
$ opam update -u
and only the affected components will be rebuilt.
OPAM allows us to create our own 'development remotes' containing the latest, bleeding-edge versions of our libraries. To install these unstable versions we only need to type:
$ opam remote add xapi-project git://github.com/xapi-project/opam-repo-dev
$ opam update -u
=-=- Updating package repositories =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
[xapi-project] git://github.com/xapi-project/opam-repo-dev already up-to-date
[default] /home/djs/ocaml/opam-repository synchronized
Updates available for 4.01.0, apply them with 'opam upgrade':
=== ∗ 1 ↻ 6 ↗ 6 ===
The following actions will be performed:
∗ install xapi-backtrace 0.2 [required by xapi-idl, xapi-stdext]
↗ upgrade xenctrl 0.9.26 to 0.9.28
↗ upgrade xapi-stdext 0.12.0 to 0.13.0
↻ recompile xapi-rrd 0.9.1 [uses xapi-stdext]
↻ recompile xapi-inventory 0.9.1 [uses xapi-stdext]
↗ upgrade xapi-idl 0.9.19 to 0.9.21
↻ recompile xapi-rrd-transport 0.7.2 [uses xapi-idl]
↻ recompile xapi-forkexecd 0.9.2 [uses xapi-idl, xapi-stdext]
↗ upgrade xapi-libs-transitional 0.9.6 to 0.9.7
↻ recompile xapi-tapctl 0.9.2 [uses xapi-stdext]
↻ recompile xapi-netdev 0.9.1 [uses xapi-stdext]
↗ upgrade xapi-rrdd-plugin 0.6.0 to 0.6.1
↗ upgrade xapi 1.9.56 to 1.9.58
===== ∗ 1 ↻ 6 ↗ 6 =====
Do you want to continue ? [Y/n]
When a linked set of changes are ready to be pushed, we can make a single pull request updating a set of components, which triggers the travis integration tests.
The Xapi toolstack is built from a large set of libraries, independently versioned and released, many of them shared with other projects (such as Mirage). The libraries are easy to build and test separately, but the sheer number of dependencies makes it difficult to build the whole project -- this is where opam really shines. OPAM simplifies our day-to-day lives by
git push
and then trigger integration tests via travisIf you have a lot of OCaml code to build, try OPAM!
The new pinning feature of OPAM 1.2 enables new interesting workflows for your day-to-day development in OCaml projects. I will briefly describe one of them here: simplifying continuous testing with Travis CI and GitHub.
opam
fileAs explained in the previous post, adding an opam
file at the
root of your project now lets you pin development versions of your
project directly. It's very easy to create a default template with OPAM 1.2:
$ opam pin add <my-project-name> . --edit
[... follow the instructions ...]
That command should create a fresh opam
file; if not, you might
need to fix the warnings in the file by re-running the command. Once
the file is created, you can edit it directly and use opam lint
to
check that is is well-formed.
If you want to run tests, you can also mark test-only dependencies with the
{test}
constraint, and add a build-test
field. For instance, if you use
oasis
and ounit
, you can use something like:
build: [
["./configure" "--prefix=%{prefix}%" "--%{ounit:enable}%-tests"]
[make]
]
build-test: [make "test"]
depends: [
...
"ounit" {test}
...
]
Without the build-test
field, the continuous integration scripts
will just test the compilation of your project for various OCaml
compilers.
OPAM doesn't run tests by default, but you can make it do so by
using opam install -t
or setting the OPAMBUILDTEST
environment variable in your local setup.
Travis CI is a free service that enables continuous testing on your GitHub projects. It uses Ubuntu containers and runs the tests for at most 50 minutes per test run.
To use Travis CI with your OCaml project, you can follow the instructions on https://github.com/ocaml/ocaml-travisci-skeleton. Basically, this involves:
<VERSION>
are 3.12
, 4.00
,
4.01
and 4.02
) add the line:env:
- OCAML_VERSION=<VERSION>
+
button on the
left pane).And that's it, your project now has continuous integration, using the OPAM 1.2 pinning feature and Travis CI scripts.
By default, the script will not try to install the optional
dependencies specified in your opam
file. To do so, you
need to manually specify which combination of optional dependencies
you want to tests using the DEPOPTS
environment variable. For
instance, to test cohttp
first with lwt
, then with async
and
finally with both lwt
and async
(but only on the 4.01
compiler)
you should write:
env:
- OCAML_VERSION=latest DEPOPTS=lwt
- OCAML_VERSION=latest DEPOPTS=async
- OCAML_VERSION=4.01 DEPOPTS="lwt async"
As usual, your contributions and feedback on this new feature are gladly welcome.
After a few months of development, we are pleased to announce the
stable release of
Merlin 2.0.
Supported OCaml versions range from 4.00.1 to 4.02.1.
Merlin is a tool focused on helping you code in OCaml by providing features such as:
We provide integration into Vim and Emacs. An external plugin is also available for Sublime Text.
This release provides great improvements in robustness and quality of analysis. Files that changed on disk are now automatically reloaded. The parsing process is finer grained to provide more accurate recovery and error messages. Integration with Jane Street Core and js_of_ocaml has also improved.
Vim & Emacs are still the main targeted editors. Thanks to Luc Rocher, preliminary support for Sublime Text is also available, see Sublime-text-merlin. Help is welcome to improve and extend supported editing environments.
Windows support also received some fixes. Merlin is now distributed in WODI. Integration in OCaml-on-windows is planned.
This new version of Merlin is already available with opam using opam install
merlin
, and can also be built from the sources which are available at
the-lambda-church/merlin.
This is a major release which we worked on for several months, rewriting many parts of the codebase. An exhaustive list of changes is therefore impossible to give, but here are some key points (from an user perspective):
This release also contains contributions from: Yotam Barnoy, Jacques-Pascal Deplaix, Geoff Gole, Rudi Grinberg, Steve Purcell and Jan Rehders.
We also thank Gabriel Scherer and Jane Street for their continued support.
We are very proud to announce the availability of OPAM 1.2.0.
Simply follow the usual instructions, using your preferred method (package from your distribution, binary, source, etc.) as documented on the homepage.
NOTE: There are small changes to the internal repository format (~/.opam). It will be transparently updated on first run, but in case you might want to go back and have anything precious there, you're advised to back it up.
Lot of work has been put into providing a cleaner interface, with helpful behaviour and messages in case of errors.
The documentation pages also have been largely rewritten for consistency and clarity.
This is just the top of the list:
opam pin
command. See the
Simplified packaging workflowopam source
opam lint
command to check the quality of packagesFor more detail, see the announcement for the beta, the full changelog, and the bug-tracker.
The package format has been extended to the benefit of both packagers and users. The repository already accepts packages in the 1.2 format, and this won't affect 1.1 users as a rewrite is done on the server for compatibility with 1.1.
If you are hosting a repository, you may be interested in these administration scripts to quickly take advantage of the new features or retain compatibility.
0install provides an easy way to distribute binaries to users, complementing OPAM's support for source distribution.
The traditional way to distribute binaries is to make separate packages for recent versions of the more popular Linux distributions (RPMs, Debs, PKGBUILDs, etc), a .pkg
for OS X, a setup.exe
Windows, etc, plus some generic binaries for users of other systems or who lack root privileges.
This requires a considerable amount of work, and expertise with many different package formats.
0install provides a single format that supports all platforms, meaning that you only need to do the packaging work once (though you should still test on multiple platforms).
You can install a binary release of OPAM on most systems by first installing the "0install" package from your distribution ("zeroinstall-injector" on older distributions) and then adding OPAM.
Get 0install. For the major Linux distributions:
$ yaourt -S zeroinstall-injector # Arch Linux
$ sudo apt-get install zeroinstall-injector # Debian, Ubuntu, etc
$ sudo yum install 0install # Fedora
$ sudo yum install zeroinstall-injector # Red Hat
$ sudo zypper install zeroinstall-injector # OpenSUSE
For OS X: ZeroInstall.pkg
0install also has Windows support, but there is currently no Windows binary of OPAM so I didn't publish one with 0install either.
Install OPAM:
$ 0install add opam http://tools.ocaml.org/opam.xml
$ opam --version
1.1.1
If you already have an opam
command but want to try the 0install version anyway,
just give it a different name (e.g. 0install add 0opam http://tools.ocaml.org/opam.xml
to create a 0opam
command).
0install add
will open a window if a display is available, or show progress on the console if not:
If you want to use the console in all cases, use --console
.
0install identifies each package with a full URI rather than with a short name.
Here, we are telling 0install to create a local command called opam
that runs the program http://tools.ocaml.org/opam.xml.
0install keeps each package it installs in a separate directory where it won't conflict with anything. You can see where it put OPAM with the "show" command:
$ 0install show opam
- URI: http://tools.ocaml.org/opam.xml
Version: 1.1.1
Path: /home/test/.cache/0install.net/implementations/sha256new_RUOX6PWGDCHH5TDNEDRHQJ54YZZ4TSAGBB5AEBRNOKSHM3N7XORA
- URI: http://repo.roscidus.com/utils/aspcud
Version: 1.9.0
Path: /home/test/.cache/0install.net/implementations/sha1new=5f838f78e489dabc2e2965ba100f14ae8350cbce
- URI: http://repo.roscidus.com/utils/curl
Version: 7.32.0-10.20
Path: (package:rpm:curl:7.32.0-10.20:x86_64)
OPAM depends on two other programs: aspcud
provides a better solver than the internal one and curl
is used to download OPAM packages (it also generally needs gcc
, m4
and patch
, but I started with just the ones people are likely to be missing).
In this case, 0install has satisfied the curl dependency using an official Fedora package, but needed to install aspcud using 0install. On Arch Linux, it can use distribution packages for both.
0install will install any required distribution packages using PackageKit, which will prompt the user for permission according to its policy.
You can upgrade (or downgrade) the package by adding a version constraint. By default, 0install prefers the "stable" version of a program:
$ 0install update opam
A later version (http://tools.ocaml.org/opam.xml 1.2.0-pre4) exists but was not selected.
Using 1.1.1 instead.
To select "testing" versions, use:
0install config help_with_testing True
You could do as it suggests and tell it to prefer testing versions globally, or you can add a version constraint if you just want to affect this one program:
$ 0install update opam --not-before=1.2.0-pre4
http://tools.ocaml.org/opam.xml: 1.1.1 -> 1.2.0-pre4
You can also specify an upper bound (--before
) or a fixed version (--version
) if you prefer.
You can control the versions of dependencies with --version-for
.
By the way, 0install supports tab-completion everywhere: it can do completion on the sub-command (update
), the application name (opam
), the option (--not-before
) and even the list of available versions!
Finally, if an upgrade stops a program from working then you can use whatchanged
to see the latest changes:
$ 0install whatchanged opam
Last checked : 2014-08-26 11:00
Last update : 2014-08-26
Previous update : 2014-07-03
http://tools.ocaml.org/opam.xml: 1.1.1 -> 1.2.0-pre4
To run using the previous selections, use:
0install run /home/tal/.config/0install.net/apps/opam/selections-2014-07-03.xml
Note: this has a granularity of a day, so you won't see any changes if you're following along, since you didn't have it installed yesterday.
If you visit http://tools.ocaml.org/opam.xml you should see a normal-looking web-page describing the package. If you view the source in your browser, you'll see that it's actually an XML document with a stylesheet providing the formatting. Here's an extract:
<group license="OSI Approved :: GNU Lesser General Public License (LGPL)">
<group>
<requires interface="http://repo.roscidus.com/utils/aspcud"
importance="recommended">
<executable-in-var name="OPAMEXTERNALSOLVER"/>
</requires>
<requires interface="http://repo.roscidus.com/utils/curl"
importance="recommended">
<executable-in-var name="OPAMCURL"/>
</requires>
<command name="run" path="opam"/>
<implementation arch="Linux-x86_64"
id="sha1new=6e16ff6ee58e39c9ebbed2fb6c6b6cc437b624a4"
released="2014-04-17"
stability="stable"
version="1.1.1">
<manifest-digest sha256new="RUOX6PWGDCHH5TDNEDRHQJ54YZZ4TSAGBB5AEBRNOKSHM3N7XORA"/>
<archive href="http://test.roscidus.com/archives/opam-Linux-x86_64-1.1.1.tgz"
size="1476315"/>
</implementation>
This says that the 64-bit Linux binary for OPAM 1.1.1 is available at the given URL and, when unpacked, has the given SHA256 digest.
It can be run by executing the opam
binary within the archive.
It depends on aspcud
and curl
, which live in other repositories (ideally, these would be the official project sites for these programs, but currently they are provided by a third party).
In both cases, we tell OPAM about the chosen version by setting an environment variable.
The <group>
elements avoid repeating the same information for multiple versions.
curl
is marked as recommended
because, while 0install supports most distribution package managers, if it can't find a curl package then it's more likely that it failed to find the curl package than that the platform doesn't have curl.
Lower down there is a more complex entry saying how to build from source, which provides a way to generate more binaries, and the XML is followed by a GPG signature block (formatted as an XML comment so that the document is still valid XML).
When you use a program for the first time, 0install downloads the signing GPG key and checks it with the key information service.
If this service knows the key, it saves it as a trusted key for that site.
If not, it prompts you to confirm.
In future, it will check that all updates from that site are signed with the same key, prompting you if not (much like ssh
does).
If you would like to see the key information hints rather than having them approved automatically, use 0install config auto_approve_keys false
or turn off "Automatic approval for new feeds" in the GUI, and untrust the key (right-click on it for a menu):
You will then see prompts like this when using a new site for the first time:
Ideally, OPAM's own Git repository would contain an XML file describing its build and runtime dependencies (curl
and aspcud
in this case) and
how to build binaries from it.
We would then generate the XML for releases from it automatically using tools such as 0release.
However, when trying out 0install you may prefer to package up an existing binary release, and this is what I did for OPAM.
The simplest case is that the binary is in the current directory. In this case, the XML just describes its dependencies and how to run it, but not how to download the program. You can create a template XML file using 0template (or just write it yourself):
$ 0install add 0template http://0install.net/tools/0template.xml
$ 0template opam.xml
'opam.xml' does not exist; creating new template.
As it ends with .xml, not .xml.template, I assume you want a feed for
a local project (e.g. a Git checkout). If you want a template for
publishing existing releases, use opam.xml.template instead.
Does your program need to be compiled before it can be used?
1) Generate a source template (e.g. for compiling C source code)
2) Generate a binary template (e.g. for a pre-compiled binary or script)
> 2
Writing opam.xml
Filling in the blanks, we get:
<?xml version="1.0"?>
<interface xmlns="http://zero-install.sourceforge.net/2004/injector/interface">
<name>OPAM</name>
<summary>OCaml package manager</summary>
<description>
OPAM is an open-source package manager edited by OCamlPro.
It supports multiple simultaneous compiler installations, flexible
package constraints, and a Git-friendly development workflow.
</description>
<homepage>https://opam.ocaml.org/</homepage>
<feed-for interface="http://tools.ocaml.org/opam.xml"/>
<group license="OSI Approved :: GNU Lesser General Public License (LGPL)">
<requires importance="recommended" interface="http://repo.roscidus.com/utils/aspcud">
<executable-in-var name="OPAMEXTERNALSOLVER"/>
</requires>
<requires importance="recommended" interface="http://repo.roscidus.com/utils/curl">
<executable-in-var name="OPAMCURL"/>
</requires>
<command name="run" path="opam"/>
<implementation arch="Linux-x86_64" id="." local-path="." version="1.1.1"/>
</group>
</interface>
This is almost the same as the XML above, except that we specify local-path="."
rather than giving a URL and digest
(see the Feed file format specification for details).
The <feed-for>
says where we will eventually host the public list of all versions.
You can test your XML locally like this:
$ 0install run opam.xml --version
1.1.1
$ 0install select opam.xml
- URI: /tmp/opam/opam.xml
Version: 1.1.1
Path: /tmp/opam
- URI: http://repo.roscidus.com/utils/aspcud
Version: 1.9.0-1
Path: (package:arch:aspcud:1.9.0-1:x86_64)
- URI: http://repo.roscidus.com/utils/curl
Version: 7.37.1-1
Path: (package:arch:curl:7.37.1-1:x86_64)
Now we need a way to generate similar XML for released archives on the web.
Rename opam.xml
to opam.xml.template
and change the <implementation>
to:
<implementation arch="{arch}" version="{version}">
<manifest-digest/>
<archive href="http://example.com/archives/opam-{arch}-{version}.tgz"/>
</implementation>
If the archive is already published somewhere, you can use the full URL in the <archive>
.
If you're making a new release locally, just put the archive in the same directory as the XML and give the leaf only (href="opam-{arch}-{version}.tgz"
).
You can now run 0template
on the template XML to generate the XML with the hashes, sizes, etc, filled in:
$ 0template opam.xml.template version=1.1.1 arch=Linux-x86_64
Writing opam-1.1.1.xml
This generates:
<group license="OSI Approved :: GNU Lesser General Public License (LGPL)">
<requires importance="recommended" interface="http://repo.roscidus.com/utils/aspcud">
<executable-in-var name="OPAMEXTERNALSOLVER"/>
</requires>
<requires importance="recommended" interface="http://repo.roscidus.com/utils/curl">
<executable-in-var name="OPAMCURL"/>
</requires>
<command name="run" path="opam"/>
<implementation arch="Linux-x86_64"
id="sha1new=6e16ff6ee58e39c9ebbed2fb6c6b6cc437b624a4"
released="2014-08-26"
version="1.1.1">
<manifest-digest sha256new="RUOX6PWGDCHH5TDNEDRHQJ54YZZ4TSAGBB5AEBRNOKSHM3N7XORA"/>
<archive href="http://example.com/archives/opam-Linux-x86_64-1.1.1.tgz" size="1476315"/>
</implementation>
</group>
You can test this as before:
$ 0install run opam-1.1.1.xml --version
1.1.1
Finally, you can submit the XML to a repository (which is easy to host yourself) using the 0repo tool:
$ 0install add 0repo http://0install.net/tools/0repo.xml
[ ... configure your repository ... ]
$ 0repo add opam-1.1.1.xml
0repo will merge the XML into the repository's master list of versions, upload the archive (or test the URL, if already uploaded), commit the final XML to Git, and push the XML to your web server. There are simpler ways to get the signed XML, e.g. using 0publish-gui for a graphical UI, but if you're going to release more than one version of one program then the automation (and sanity checking) you get from 0repo is usually worth it. In my case, I configured 0repo to push the signed XML to a GitHub Pages repository.
There are plenty of ways to extend this. For the OPAM 1.2 release, instead of adding the official binaries one by one, I used 0template to make a template for the source code, added the 1.2 source release, and used that to generate the binaries (mainly because I wanted to build in an older docker container so the binaries would work on more systems). For my own software, I commit an XML file saying how to build it to my Git repository and let 0release handle the whole release process (from tagging the Git repository, to building the binaries in various VMs, to publishing the archives and the final signed XML). In the future, we hope to integrate this with OPAM so that source and binary releases can happen together.
0install is easy to script. It has a stable command line interface (and a new JSON API too). There is also an OCaml API, but this is not yet stable (it's still rather Pythonic, as the software was recently ported from Python).
For example, I have used 0install to manage Xen images of Mirage unikernels.
This command shows the latest binary of the mir-hello
unikernel for Xen (downloading it first if needed):
$ 0install download --os=Xen http://test.roscidus.com/mir-console.xml --show
- URI: http://test.roscidus.com/mir-console.xml
Version: 0.2
Path: /home/tal/.cache/0install.net/implementations/sha256new_EY2FDE4ECMNCXSRUZ3BSGTJQMFXE2U6C634PBDJKOBUU3SWD5GDA
0install is a mature cross-platform system for managing software that is included in the repositories of most Linux distributions and is also available for Windows, OS X and Unix. It is useful to distribute binary executables in cases where users shouldn't have to compile from source. It supports GPG signature checking, automatic updates, pinned versions and parallel installations of multiple versions. A single package format is sufficient for all platforms (you still need to create separate binary archives for e.g. OS X and Linux, of course).
0install is a decentralised system, meaning that there is no central repository. Packages are named by URI and the metadata is downloaded directly from the named repository. There are some extra services (such as the default mirror service, the search service and the key information service), but these are all optional.
Using 0install to get OPAM means that all platforms can be supported without the need to package separately for each one, and users who don't wish to install as root still get signature checking, dependency handling and automatic updates. We hope that 0install will make it easier for you to distribute binaries of your own applications.
My talk at OCaml 2014 (video, slides) gives more information about 0install and its conversion to OCaml.
This is a post about the utop
toplevel provided in the OPAM
repository as an alternative to the standard OCaml one.
OCaml comes with an interactive toplevel. If you type ocaml
in a
shell you will get a prompt where you can type OCaml code that is
compiled and executed on the fly.
$ ocaml
OCaml version 4.02.0+dev12-2014-07-30
# 1 + 1;;
- : int = 2
You can load libraries and your own modules in the toplevel, and it is great for playing with your code. You'll quickly notice that the user experience is not ideal, as there is no editing support: you cannot conveniently change what you type nor can you rewind to previously typed phrases.
This can be improved by using tools such as
ledit or
rlwrap which adds line editing
support for any program: rlwrap ocaml
. This is better but still
doesn't provide fancy features such as context sensitive completion.
That's why UTop was started. UTop is a shiny frontend to the OCaml interactive toplevel, which tries to focus on the user experience and features:
And many other things which make life easier for users that have been added over time.
UTop stands for Universal Toplevel
. Universal because it can be used
in a terminal or in Emacs (I originally planned to add a windowed
version using GTK but unfortunately never completed it).
The utop prompt looks much more 'blinky' than the one of the default toplevel. Install it using OPAM very simply:
opam install utop
eval `opam config env` # may not be needed
utop
This is typically what you see when you start utop:
─( 16:36:52 )─< command 0 >───────────────────────{ counter: 0 }─
utop #
┌───┬────────────┬─────┬───────────┬──────────────┬───────┬─────┐
│Arg│Arith_status│Array│ArrayLabels│Assert_failure│Big_int│Bigar│
└───┴────────────┴─────┴───────────┴──────────────┴───────┴─────┘
It displays:
The box at the bottom is for completion, which is described in the next section.
If the colors seem too bright you can type #utop_prompt_fancy_light
,
which is better for light backgrounds. This can be set permanently by
adding the line to ~/.ocamlinit
or by adding profile: light
to
~/.utoprc
.
The prompt can be customized by the user, by setting the reference
UTop.prompt
:
utop # UTop.prompt;;
- : LTerm_text.t React.signal ref = {contents = <abstr>}
LTerm_text.t
is for styled text while React.signal
means that it
is a reactive signal, from the
react library. This makes it very
easy to create a prompt where the time is updated every second for
instance.
This is the main feature that motivated the creation of UTop. UTop makes use of the compiler internals to find possible completions on:
Instead of the classic way of displaying a list of words when the user press TAB, I chose to dynamically display the different possibilities as the user types. This idea comes from the dmenu tool from the dwm window manager.
The possible completions are displayed in the completion bar below the
prompt. It is possible to navigate in the list by using the meta key
(Alt
by default most of the time) and the left and right arrows. A
word can be selected by pressing the meta key and TAB
. Also pressing
just TAB
will insert the longest common prefix of all possibilities.
UTop can do basic syntax highlighting. This is disabled by default but
can be enabled by writing a ~/.utoprc
file. You can copy one from
the repository, either for
dark background
or
light background.
As said earlier UTop can be run in Emacs. Instructions to set this up can be found in UTop's readme. The default toplevel can also be run this way but UTop is better in the following respects:
They are several Emacs libraries for writing shell-like modes but I wrote my own because with all of the ones I found it is possible to insert or remove characters from the prompt, which I found frustrating Even with the mode used by the Emacs Shell mode it is possible. AFAIK at the time I wrote it the UTop mode was the only one where it was really impossible to edit the something in the frozen part of the buffer.
This is a non-exhaustive list of features that have been added over time to enhance the user experience. Some of them might be controversial, so I tried to choose what was the most requested most of the time.
-short-paths
the default. This option allow to display
shorter types when using packed libraries such as
core_
to the user. This is for hiding
the churn generated by syntax extensions. This can be disabled with
UTop.set_hide_reserved
or with the command line argument
-show-reserved
.camlp4
when the user requests a syntax
extension. In the default toplevel one has to type #camlp4
first.findlib
library manager.typeof
directive to show the type of modules and values.$OCAML_TOPLEVEL_PATH/autoload
at
startup.For the needs of UTop I wrote lambda-term, which is kind of an equivalent of ncurses+readline, but written in OCaml. It was written because I wasn't happy with the ncurses API and I wanted something more fancy than readline, especially for completion. In the end I believe that it is much more fun to write terminal applications in OCaml using lambda-term.
The pure editing part is managed by the zed library, which is independent from the user interface.
Utop is fairly feature-complete, and so I don't spend much time on it these days. It became the recommended toplevel to use with the Real World OCaml book, and most users are happier with the interactive interface than using the traditional toplevel.
Many thanks to Peter Zotov who recently joined the project to keep it up-to-date and add new features such as extension point support. Contributions from others (particularly around editor integration) are very welcome, so if you are interested on hacking on it get in touch via the GitHub issue tracker or via the OCaml Platform mailing list.
This post is a short presentation of a couple of tools you can use with your editor to have a smoother experience while developing in OCaml. We are working towards making these tools work out-of-the-box with OPAM, and hence will be blogging about them here along with the OPAM tool itself.
At the time of writing, interfaces to these tools are available for Emacs and Vim. Efforts are underway to add support for other editors, including Acme and Sublime Text 3.
The first tool, ocp-indent, handles the task of indenting your OCaml files. It is an OCaml executable that can be used from the command line or directly from your editor.
The second tool, merlin performs "static analysis" of your source files. The analysis is then used to provide error reporting, source browsing, auto-completion and more.
Most editors provide some kind of indentation "out of the box". However recently a good chunk of the OCaml community has moved to using ocp-indent for fine-tuned indentation:
Indeed the indentation behaviour of ocp-indent can be configured through several options, directing how it will behave when encountering different OCaml language constructs. These options can either be set in a configuration file (such as this example configuration) or passed directly as parameters when ocp-indent is invoked from the command line.
Finally, ocp-indent will also recognize a number of common syntax extensions of the OCaml ecosystem and indent them meaningfully, while your editor probably will not.
Merlin enhances your experience editing OCaml code by providing interactive feedback about your code.
Under the hood, it maintains a "code model" of the file you are editing. For other files in your project, it will use the output produced by the compiler; rebuild regularly after editing to keep the model synchronized with your code.
From this code model, it provides a lot of useful features:
Assuming opam is already installed on your system, you just need to invoke
$ opam install ocp-indent merlin
to install these two tools.
Emacs. You will have to add opam/share
to 'load-path
and then load the plugin-specific
file. This can be done by adding the following lines to your .emacs
:
(setq opam-share (substring (shell-command-to-string "opam config var share 2> /dev/null") 0 -1))
(add-to-list 'load-path (concat opam-share "/emacs/site-lisp"))
(require 'ocp-indent)
(require 'merlin)
For more information about merlin setup, you can look at the dedicated merlin wiki.
Vim & ocp-indent. We recommend using the ocp-indent-vim plugin instead of the default one. It provides interactive indentation "as you type", while the official mode only provides an indenting function to call manually but no passive indentation.
This mode does require vim to be compiled with Python support, while the official one doesn't.
Installing is as simple as cloning ocp-indent-vim and adding the directory to your runtime-path.
Assuming your clone is in ~/my-clone-of/ocp-indent-vim
, add this to .vimrc
:
set rtp+=~/my-clone-of/ocp-indent-vim
Vim & merlin. A comprehensive guide to the installation procedure for merlin is available on the dedicated merlin wiki. Once again, if you just want to get started the following lines contain everything you need.
Loading merlin in vim boils down to adding the plugin directory to the runtime path. However as merlin depends on your current opam switch, a more flexible way is to find the current switch and use it as the base directory.
This code does exactly that: it finds the current opam share directory, then adds
the merlin plugin subdirectory to the current runtime path. Add it to your .vimrc
:
let g:opamshare = substitute(system('opam config var share'),'\n$','','''')
execute "set rtp+=" . g:opamshare . "/merlin/vim"
To maintain synchronization with the compiler, merlin needs some information
about the structure of your project: build and source directories, package
dependencies, syntax extensions. This structure can be described in a .merlin
file in the root directory of your project.
The .merlin
file for the merlin project illustrates the syntax.
The .merlin
file will be loaded the next time you open an OCaml file in the editor.
To benefit from code navigation across files you'll also need to turn on
generation of "cmt" files by passing the -bin-annot
flag to the OCaml
compiler. You can do this in ocamlbuild
by adding the bin_annot
tag
into the _tags
file with OCaml 4.01 and higher.
Most package managers support some pin functionality to ensure that a given package remains at a particular version without being upgraded. The stable OPAM 1.1 already supported this by allowing any existing package to be pinned to a target, which could be a specific released version, a local filesystem path, or a remote version-controlled repository.
However, the OPAM 1.1 pinning workflow only lets you pin packages that already exist in your OPAM repositories. To declare a new package, you had to go through creating a local repository, registering it in OPAM, and adding your package definition there. That workflow, while reasonably clear, required the user to know about the repository format and the configuration of an internal repository in OPAM before actually getting to writing a package. Besides, you were on your own for writing the package definition, and the edit-test loop wasn't as friendly as it could have been.
A natural, simpler workflow emerged from allowing users to pin new package names that don't yet exist in an OPAM repository:
opam pin add
in the development source treeTo make it even easier, OPAM can now interactively help you write the package definition, and you can test your updates with a single command. This blog post explains this new OPAM 1.2 functionality in more detail; you may also want to check out the new Packaging tutorial relying on this workflow.
For illustration purposes in this post I'll use a tiny tool that I wrote some time ago and never released: ocp-reloc. It's a simple binary that fixes up the headers of OCaml bytecode files to make them relocatable, which I'd like to release into the public OPAM repository.
The command opam pin add <name> <target>
pins package <name>
to
<target>
. We're interested in pinning the ocp-reloc
package
name to the project's source directory.
cd ocp-reloc
opam pin add ocp-reloc .
If ocp-reloc
were an existing package, the metadata would be fetched from
the package description in the OPAM repositories. Since the package doesn't yet exist,
OPAM 1.2 will instead prompt for on-the-fly creation:
Package ocp-reloc does not exist, create as a NEW package ? [Y/n] y
ocp-reloc is now path-pinned to ~/src/ocp-reloc
NOTE: if you are using beta4, you may get a version-control-pin instead, because we added auto-detection of version-controlled repos. This turned out to be confusing (issue #1582), because your changes wouldn't be reflected until you commit, so this has been reverted in favor of a warning. Add the
--kind path
option to make sure that you get a path-pin.
Now your package still needs some kind of definition for OPAM to acknowledge it;
that's where templates kick in, the above triggering an editor with a pre-filled
opam
file that you just have to complete. This not only saves time in
looking up the documentation, it also helps getting consistent package
definitions, reduces errors, and promotes filling in optional but recommended
fields (homepage, etc.).
opam-version: "1.2"
name: "ocp-reloc"
version: "0.1"
maintainer: "Louis Gesbert <louis.gesbert@ocamlpro.com>"
authors: "Louis Gesbert <louis.gesbert@ocamlpro.com>"
homepage: ""
bug-reports: ""
license: ""
build: [
["./configure" "--prefix=%{prefix}%"]
[make]
]
install: [make "install"]
remove: ["ocamlfind" "remove" "ocp-reloc"]
depends: "ocamlfind" {build}
After adding some details (most importantly the dependencies and
build instructions), I can just save and exit. Much like other system tools
such as visudo
, it checks for syntax errors immediately:
[ERROR] File "/home/lg/.opam/4.01.0/overlay/ocp-reloc/opam", line 13, character 35-36: '.' is not a valid token.
Errors in /home/lg/.opam/4.01.0/overlay/ocp-reloc/opam, retry editing ? [Y/n]
You probably want to try your brand new package right away, so
OPAM's default action is to try and install it (unless you specified -n
):
ocp-reloc needs to be installed.
The following actions will be performed:
- install cmdliner.0.9.5 [required by ocp-reloc]
- install ocp-reloc.0.1*
=== 1 to install ===
Do you want to continue ? [Y/n]
I usually don't get it working the first time around, but opam pin edit
ocp-reloc
and opam install ocp-reloc -v
can be used to edit and retry until
it does.
How do you keep working on your project as you edit the source code, now that you are installing through OPAM? This is as simple as:
opam upgrade ocp-reloc
This will pick up changes from your source repository and reinstall any packages
that are dependent on ocp-reloc
as well, if any.
So far, we've been dealing with the metadata locally used by your OPAM
installation, but you'll probably want to share this among developers of your
project even if you're not releasing anything yet. OPAM takes care of this
by prompting you to save the opam
file back to your source tree, where
you can commit it directly into your code repository.
cd ocp-reloc
git add opam
git commit -m 'Add OPAM metadata'
git push
The above information is sufficient to use OPAM locally to integrate new code into an OPAM installation. Let's look at how other developers can share this metadata.
If another developer wants to pick up ocp-reloc
, they can directly use
your existing metadata by cloning a copy of your repository and issuing their
own pin.
git clone git://github.com/OCamlPro/ocp-reloc.git
opam pin add ocp-reloc/
Even specifying the package name is optional since this is documented in
ocp-reloc/opam
. They can start hacking, and if needed use opam pin edit
to
amend the opam file too. No need for a repository, no need to share anything more than a
versioned opam
file within your project.
We have been focusing on an unreleased package, but the same functionality is also of great help in handling existing packages, whether you need to quickly hack into them or are just curious. Let's consider how to modify the `omd` Markdown library.
opam source omd --pin
cd omd.0.9.7
...patch...
opam upgrade omd
The new opam source
command will clone the source code of the library you
specify, and the --pin
option will also pin it locally to ensure it is used
in preference to all other versions. This will also take care of recompiling
any installed packages that are dependent on omd
using your patched version
so that you notice any issues right away.
There's a new OPAM field available in 1.2 called
dev-repo
. If you specify this in your metadata, you can directly pin to the upstream repository viaopam source --dev-repo --pin
.
If the upstream repository for the package contains an opam
file, that file will be picked up
in preference to the one from the OPAM repository as soon as you pin the package.
The idea is to have:
opam
file that is versioned along with your source code
(and thus accurately tracks the latest dependencies for your package).opam
file that is published on the OPAM repository and can
be updated independently without making a new release of the source code.How to get from the former to the latter will be the subject of another post! In the meantime, all users of the beta are welcome to share their experience and thoughts on the new workflow on the bug tracker.
It has only been 18 months since the first release of OPAM, but it is already difficult to remember a time when we did OCaml development without it. OPAM has helped bring together much of the open-source code in the OCaml community under a single umbrella, making it easier to discover, depend on, and maintain OCaml applications and libraries. We have seen steady growth in the number of new packages, updates to existing code, and a diverse group of contributors.
OPAM has turned out to be more than just another package manager. It is also increasingly central to the demanding workflow of industrial OCaml development, since it supports multiple simultaneous (patched) compiler installations, sophisticated package version constraints that ensure statically-typed code can be recompiled without conflict, and a distributed workflow that integrates seamlessly with Git, Mercurial or Darcs version control. OPAM tracks multiple revisions of a single package, thereby letting packages rely on older interfaces if they need to for long-term support. It also supports multiple package repositories, letting users blend the global stable package set with their internal revisions, or building completely isolated package universes for closed-source products.
Since its initial release, we have been learning from the extensive feedback from our users about how they use these features as part of their day-to-day workflows. Larger projects like XenAPI, the Ocsigen web suite, and the Mirage OS publish OPAM remotes that build their particular software suites. Complex applications such as the Pfff static analysis tool and Hack language from Facebook, the Frenetic SDN language and the Arakoon distributed key store have all appeared alongside these libraries. Jane Street pushes regular releases of their production Core/Async suite every couple of weeks.
One pleasant side-effect of the growing package database has been the contribution of tools from the community that make the day-to-day use of OCaml easier. These include the utop interactive toplevel, the IOCaml browser notebook, and the Merlin IDE extension. While these tools are an essential first step, there's still some distance to go to make the OCaml development experience feel fully integrated and polished.
Today, we are kicking off the next phase of evolution of OPAM and starting the journey towards building an OCaml Platform that combines the OCaml compiler toolchain with a coherent workflow for build, documentation, testing and IDE integration. As always with OPAM, this effort has been a collaborative effort, coordinated by the OCaml Labs group in Cambridge and OCamlPro in France. The OCaml Platform builds heavily on OPAM, since it forms the substrate that pulls together the tools and facilitates a consistent development workflow. We've therefore created this blog on opam.ocaml.org to chart its progress, announce major milestones, and eventually become a community repository of all significant activity.
Major points:
OPAM 1.2 beta available: Firstly, we're announcing the availability of the OPAM 1.2 beta, which includes a number of new features, hundreds of bug fixes, and pretty new colours in the CLI. We really need your feedback to ensure a polished release, so please do read the release notes below.
In the coming weeks, we will provide an overview of what the OCaml Platform is (and is not), and describe an example workflow that the Platform can enable.
Feedback: If you have questions or comments as you read these posts, then please do join the platform@lists.ocaml.org and make them known to us.
We are proud to announce the latest beta of OPAM 1.2. It comes packed with new features, stability and usability improvements. Here the highlights.
We now have binary packages available for Fedora 19/20, CentOS 6/7, RHEL7, Debian Wheezy and Ubuntu! You can see the full set at the OpenSUSE Builder site and download instructions for your particular platform.
An OPAM binary installation doesn't need OCaml to be installed on the system, so you can initialize a fresh, modern version of OCaml on older systems without needing it to be packaged there. On CentOS 6 for example:
cd /etc/yum.repos.d/
wget http://download.opensuse.org/repositories/home:ocaml/CentOS_6/home:ocaml.repo
yum install opam
opam init --comp=4.01.0
For this version, we focused on improving the user interface and workflow. OPAM is a complex piece of software that needs to handle complex development situations. This implies things might go wrong, which is precisely when good support and error messages are essential. OPAM 1.2 has much improved stability and error handling: fewer errors and more helpful messages plus better state backups when they happen.
In particular, a clear and meaningful explanation is extracted from the solver whenever you are attempting an impossible action (unavailable package, conflicts, etc.):
$ opam install mirage-www=0.3.0
The following dependencies couldn't be met:
- mirage-www -> cstruct < 0.6.0
- mirage-www -> mirage-fs >= 0.4.0 -> cstruct >= 0.6.0
Your request can't be satisfied:
- Conflicting version constraints for cstruct
This sets OPAM ahead of many other package managers in terms of user-friendliness. Since this is made possible using the tools from irill (which are also used for Debian), we hope that this work will find its way into other package managers. The extra analyses in the package solver interface are used to improve the health of the central package repository, via the OPAM Weather service.
And in case stuff does go wrong, we added the opam upgrade --fixup
command that will get you back to the closest clean state.
The command-line interface is also more detailed and convenient, polishing and
documenting the rough areas. Just run opam <subcommand> --help
to see the
manual page for the below features.
More expressive queries based on dependencies.
$ opam list --depends-on cow --rec
# Available packages recursively depending on cow.0.10.0 for 4.01.0:
cowabloga 0.0.7 Simple static blogging support.
iocaml 0.4.4 A webserver for iocaml-kernel and iocamljs-kernel.
mirage-www 1.2.0 Mirage website (written in Mirage)
opam2web 1.3.1 (pinned) A tool to generate a website from an OPAM repository
opium 0.9.1 Sinatra like web toolkit based on Async + Cohttp
stone 0.3.2 Simple static website generator, useful for a portfolio or documentation pages
Check on existing opam
files to base new packages from.
$ opam show cow --raw
opam-version: "1"
name: "cow"
version: "0.10.0"
[...]
Clone the source code for any OPAM package to modify or browse the interfaces.
$ opam source cow
Downloading archive of cow.0.10.0...
[...]
$ cd cow.0.10.0
We've also improved the general speed of the tool to cope with the much bigger size of the central repository, which will be of importance for people building on low-power ARM machines, and added a mechanism that will let you install newer releases of OPAM directly from OPAM if you choose so.
Packaging new libraries has been made as straight-forward as possible. Here is a quick overview, you may also want to check the OPAM 1.2 pinning post.
opam pin add <name> <sourcedir>
will generate a new package on the fly by detecting the presence of an opam
file within the source repository itself. We'll do a followup post next week
with more details of this extended opam pin
workflow.
The package description format has also been extended with some new fields:
bug-reports:
and dev-repo:
add useful URLsinstall:
allows build and install commands to be split,flags:
is an entry point for several extensions that can affect your package.Packagers can limit dependencies in scope by adding one
of the keywords build
, test
or doc
in front of their constraints:
depends: [
"ocamlfind" {build & >= 1.4.0}
"ounit" {test}
]
Here you don't specifically require ocamlfind
at runtime, so changing it
won't trigger recompilation of your package. ounit
is marked as only required
for the package's build-test:
target, i.e. when installing with
opam install -t
. This will reduce the amount of (re)compilation required
in day-to-day use.
We've also made optional dependencies more consistent by removing version
constraints from the depopts:
field: their meaning was unclear and confusing.
The conflicts
field is used to indicate versions of the optional dependencies
that are incompatible with your package to remove all ambiguity:
depopts: [ "async" {>= "109.15.00"} & "async_ssl" {>= "111.06.00"} ]
becomes:
depopts: [ "async" "async_ssl" ]
conflicts: [ "async" {< "109.15.00"}
"async_ssl" {< "111.06.00"} ]
There is an upcoming `features` field that will give more flexibility in a clearer and consistent way for such complex cases.
Efforts were made on the build of OPAM itself as well to make it as easy as possible
to compile, bootstrap or install. There is no more dependency on camlp4 (which has
been moved out of the core distribution in OCaml 4.02.0), and the build process
is more conventional (get the source, run ./configure
, make lib-ext
to get the few
internal dependencies, make
and make install
). Packagers can use make cold
to build OPAM with a locally compiled version of OCaml (useful for platforms where
it isn't packaged), and also use make download-ext
to store all the external archives
within the source tree (for automated builds which forbid external net access).
The whole documentation has been rewritten as well, to be better focused and easier to browse. Please leave any feedback or changes on the documentation on the issue tracker.
The public beta of OPAM 1.2 is just out. You're welcome to give it a try and give us feedback before we roll out the release!
We'd be most interested on feedback on how easily you can work with the new pinning features, on how the new metadata works for you... and on any errors you may trigger that aren't followed by informative messages or clean behaviour.
If you are hosting a repository, the administration scripts may help you quickly update all your packages to benefit from the new features.
We are proud to announce that OPAM 1.1.1 has just been released.
This minor release features mostly stability and UI/doc improvements over
OPAM 1.1.0, but also focuses on improving the API and tools to be a better
base for the platform (functions for opam-doc
, interface with tools like
opamfu
and opam-installer
). Lots of bigger changes are in the works, and
will be merged progressively after this release.
Installation instructions are available on the wiki.
Note that some packages may take a few days until they get out of the pipeline. If you're eager to get 1.1.1, either use our binary installer or compile from source.
The 'official' package repository is now hosted at opam.ocaml.org, synchronised with the Git repository at http://github.com/ocaml/opam-repository, where you can contribute new packages descriptions. Those are under a CC0 license, a.k.a. public domain, to ensure they will always belong to the community.
Thanks to all of you who have helped build this repository and made OPAM such a success.
From the changelog:
Fix opam-admin make <packages> -r
(#990) Explicitly prettyprint list of lists, to fix opam-admin depexts
(#997)
OpamSolver.empty_universe
for flexible universe instantiation (#1033)OpamFormula.eval_relop
and OpamFormula.check_relop
(#1042)OpamCompiler.compare
to match Pervasives.compare
(#1042)OpamCompiler.eval_relop
(#1042)OpamPackage.Name.compare
(#1046)version_constraint
and version_formula
to OpamFormula
(#1046)info
an alias for show
and added the alias
uninstall
(#944)opam init --root=<relative path>
(#1047)opam info
(#1052).install
files usable outside of opam (#1026)--resolve
option to opam-admin make
that builds just the archives you need for a specific installation (#1031)which
by a more portable call (#1061)opam config report
to help with bug reports (#1034)opam upgrade <pkg>
(#1001)opam init
to a non-empty root directory (#974)After a while staged as RC, we are proud to announce the final release of OPAM 1.1.0!
Thanks again to those who have helped testing and fixing the last few issues.
The repository format has been improved with incompatible new features; to account for this, the new repository is now hosted at opam.ocaml.org, and the legacy repository at opam.ocamlpro.com is kept to support OPAM 1.0 installations, but is unlikely to benefit from many package updates. Migration to opam.ocaml.org will be done automatically as soon as you upgrade your OPAM version.
You're still free, of course, to use any third-party repositories instead or in addition.
NOTE: When switching from 1.0, the internal state will need to be upgraded.
THIS PROCESS CANNOT BE REVERTED. We have tried hard to make it fault-
resistant, but failures might happen. In case you have precious data in your
~/.opam
folder, it is advised to backup that folder before you upgrade
to 1.1.0.
Using the binary installer:
Using the .deb packages from Anil's PPA (binaries are currently syncing): add-apt-repository ppa:avsm/ppa apt-get update sudo apt-get install opam
For OSX users, the homebrew package will be updated shortly.
or build it from sources at :
OPAM is a source-based package manager for OCaml. It supports multiple simultaneous compiler installations, flexible package constraints, and a Git-friendly development workflow. OPAM is edited and maintained by OCamlPro, with continuous support from OCamlLabs and the community at large (including its main industrial users such as Jane-Street and Citrix).
The 'official' package repository is now hosted at opam.ocaml.org, synchronised with the Git repository at http://github.com/ocaml/opam-repository, where you can contribute new packages descriptions. Those are under a CC0 license, a.k.a. public domain, to ensure they will always belong to the community.
Thanks to all of you who have helped build this repository and made OPAM such a success.
Too many to list here, see https://raw.github.com/OCamlPro/opam/1.1.0/CHANGES
For packagers, some new fields have appeared in the OPAM description format:
depexts
provides facilities for dealing with system (non ocaml) dependenciesmessages
, post-messages
can be used to notify the user eg. of licensing information,
or help her troobleshoot at package installation.available
supersedes ocaml-version
and os
constraints, and can contain
more expressive formulasAlso, we have integrated the main package repository with Travis, which will help us to improve the quality of contributions (see Anil's post).
OPAM 1.1.0 is ready, and we are shipping a release candidate for packagers and all interested to try it out.
This version features several bug-fixes over the September beta release, and quite a few stability and usability improvements. Thanks to all beta-testers who have taken the time to file reports, and helped a lot tackling the remaining issues.
This release is synchronized with the migration of the main repository from ocamlpro.com to ocaml.org. A redirection has been put in place, so that all up-to-date installation of OPAM should be redirected seamlessly. OPAM 1.0 instances will stay on the old repository, so that they won't be broken by incompatible package updates.
We are very happy to see the impressive amount of contributions to the OPAM repository, and this change, together with the licensing of all metadata under CC0 (almost pubic domain), guarantees that these efforts belong to the community.
The internal state will need to be upgraded at the first run of OPAM 1.1.0.
THIS PROCESS CANNOT BE REVERTED. We have tried hard to make it fault-
resistant, but failures might happen. In case you have precious data in your
~/.opam folder
, it is advised to backup that folder before you upgrade to 1.1.0.
Using the binary installer:
You can also get the new version either from Anil's unstable PPA: add-apt-repository ppa:avsm/ppa-testing apt-get update sudo apt-get install opam
or build it from sources at :
Too many to list here, see https://raw.github.com/OCamlPro/opam/1.1.0-RC/CHANGES
For packagers, some new fields have appeared in the OPAM description format:
depexts
provides facilities for dealing with system (non ocaml)
dependenciesmessages
, post-messages
can be used to notify the user or help her troubleshoot at package installation.available
supersedes ocaml-version
and os
constraints, and can contain
more expressive formulasWe are very happy to announce the beta release of OPAM version 1.1.0!
OPAM is a source-based package manager for OCaml. It supports multiple simultaneous compiler installations, flexible package constraints, and a Git-friendly development workflow which. OPAM is edited and maintained by OCamlPro, with continuous support from OCamlLabs and the community at large (including its main industrial users such as Jane-Street and Citrix).
Since its first official release last March, we have fixed many bugs and added lots of new features and stability improvements. New features go from more metadata to the package and compiler descriptions, to improved package pin workflow, through a much faster update algorithm. The full changeset is included below.
We are also delighted to see the growing number of contributions from
the community to both OPAM itself (35 contributors) and to its
metadata repository (100+ contributors, 500+ unique packages, 1500+
packages). It is really great to also see alternative metadata
repositories appearing in the wild (see for instance the repositories
for Android, Windows and so on). To be sure that the
community efforts will continue to benefit to everyone and to
underline our committment to OPAM, we are rehousing it at
http://opam.ocaml.org
and switching the license to CC0 (see issue #955,
where 85 people are commenting on the thread).
The binary installer has been updated for OSX and x86_64:
You can also get the new version either from Anil's unstable PPA: add-apt-repository ppa:avsm/ppa-testing apt-get update sudo apt-get install opam
or build it from sources at :
NOTE: If you upgrade from OPAM 1.0, the first time you will run the
new opam
binary it will upgrade its internal state in an incompatible
way: THIS PROCESS CANNOT BE REVERTED. We have tried hard to make this
process fault-resistant, but failures might happen. In case you have
precious data in your ~/.opam
folder, it is advised to backup that
folder before you upgrade to 1.1.
opam update -u
equivalent to opam update && opam upgrade --yes
opam-admin
tool, bundling the features of opam-mk-repo
and
opam-repo-check
+ new 'opam-admin stats' toolavailable
: field in opam files, superseding ocaml-version
and os
fieldsopam config env
now detects the current shell and outputs a sensible default if
no override is provided.opam pin
stability and start display information about dev revisionsman
field in .install
files.install
filesstublibs
field in .install
filesopam config var VARIABLE
when variable is simple
(eg. prefix
, lib
, ...)opam list
now display only the installed packages. Use opam list -a
to get
the previous behavior.ocamlot
)--sexp
option to opam config env
to load the configuration under emacs~/.opam/log
on each invocation of OPAMversion+patches
are now handled as if this
was simply version
OpamVCS
functor to generate OPAM backendsopam update
opam search
now also searches through the tagsAPI.list
and API.SWITCH.list
messages
field--jobs
command line option and add %{jobs}%
to be used in OPAM filesopam info
command (more information, non-zero error code when no patterns match)opam reinstall
when reinstalling a package wich is a dependency of installed packagesOPAMSWITCH
to be able to call OPAM in different switchesopam-client
can now be used in a toplevel-n
now means --no-setup
and not --no-checksums
anymore../ocaml/
~/.opam/opam-init/variable.sh
after a switchI am very happy to announce the first official release of OPAM!
Many of you already know and use OPAM so I won't be long. Please read http://www.ocamlpro.com/blog/2013/01/17/opam-beta.html for a longer description.
1.0.0 fixes many bugs and add few new features to the previously announced beta-release.
The most visible new feature, which should be useful for beginners with
OCaml and OPAM, is an auto-configuration tool. This tool easily enables all
the features of OPAM (auto-completion, fix the loading of scripts for the
toplevel, opam-switch-eval alias, etc). This tool runs interactively on each
opam init
invocation. If you don't like OPAM to change your configuration
files, use opam init --no-setup
. If you trust the tool blindly, use
opam init --auto-setup
. You can later review the setup by doing
opam config setup --list
and call the tool again using opam config setup
(and you can of course manually edit your ~/.profile (or ~/.zshrc for zsh
users), ~/.ocamlinit and ~/.opam/opam-init/*).
Please report:
Packages for Debian and OSX (at least homebrew) should follow shortly and I'm looking for volunteers to create and maintain rpm packages. The binary installer is up-to-date for Linux and Darwin 64-bit architectures, the 32-bit version for Linux should arrive shortly.
If you want to build from sources, the full archive (including dependencies) is available here:
http://www.ocamlpro.com/pub/opam-full-latest.tar.gz
If you are upgrading from 0.9.* you won't have anything special to do apart
installing the new binary. You can then update your package metadata by
running opam update
. If you want to use the auto-setup feature, remove the
"eval opam config env
line you have previously added in your ~/.profile
and run opam config setup --all
.
So everything should be fine. But you never know ... so if something goes horribly wrong in the upgrade process (of if your are upgrading from an old version of OPAM) you can still trash your ~/.opam, manually remove what OPAM added in your ~/.profile (~/.zshrc for zsh users) and ~/.ocamlinit, and start again from scratch.
Great success on github. Thanks everybody for the great contributions!
https://github.com/OCamlPro/opam: +2000 commits, 26 contributors https://github.com/OCamlPro/opam-repository: +1700 commits, 75 contributors, 370+ packages
on http://opam.ocamlpro.com/ +400 unique visitor per week, 15k 'opam update' per week +1300 unique visitor per month, 55k 'opam update' per month 3815 unique visitor since the alpha release
The full change-log since the beta release in January:
1.0.0 [Mar 2013]
0.9.6 [Mar 2013]
~/.profile
when using dash (eg. in Debian/Ubuntu)0.9.5 [Mar 2013]
opam remove <pkg> --keep-build-dir
keeps the folder if a source archive is extractedopam config exec "CMD ARG1 ... ARGn" --switch=SWITCH
to execute a command in a subshellopam update
wrt. pinned packagesopam config setup
)~/.ocamlinit
when running opam init
0.9.4 [Feb 2013]
-a
type
instead of which
to detect existing commands)depends
and depopts
fields in opam info
opam info pkg.version
shows the metadata for this given package versiondoc
fields in .install
filesopam list
now only shows installable packages0.9.3 [Feb 2013]
0.9.2 [Jan 2013]
opam repo remove repo-name
opam config env
opam-foo
scripts (which can be called using opam foo
)0.9.1 [Jan 2013]