Development Environment

If you are writing code, the first task to tackle is setting up the development environment.

You will need to check out the code, and at a minimum, build the Rust binaries.

Most developers will also install Python and/or R: If you are only interested in developing a feature in one of these languages, you will not need to set up the other.

Clone the OpenDP Repo

If you want to submit PRs, but don’t have write access to the OpenDP repository, you will either need to request to join the organization or make a fork. The GitHub documentation explains forking.

Clone the repo (or your fork) and change into the opendp directory that’s created.

git clone git@github.com:opendp/opendp.git
cd opendp

If you have not set up SSH, you can clone with https instead:

git clone https://github.com/opendp/opendp.git

Rust Setup

If you have not already, install the Rust toolchain.

Make sure you are on the latest Rust version:

rustup update

Now run cargo build in the rust subdirectory of the repo:

cd rust
cargo build --all-features

This will compile a debug build of the OpenDP shared library, placing it in the directory opendp/rust/target/debug. (The specific name of the library file will vary depending on your platform.)

Substitute cargo build with cargo test to test, or cargo check to check syntax.

Note that Python and R require builds with different features. Details are in the Python Setup and R Setup sections below. Setting a feature changes how the crate compiles.

Comprehensive Rust Feature List

Name	Description
contrib	Enable to include constructors that have not passed the vetting process.
honest-but-curious	Enable to include constructors whose differential privacy (or stability) properties rely on the constructor arguments being correct. That is, if a user/adversary is ‘honest’ in specifying the constructor arguments, then even if they later become ‘curious’ and try to learn something from the measurement outputs, they will not be able to violate the differential privacy promises of the measurement.
floating-point	Enable to include transformations and measurements with floating-point vulnerabilities.
untrusted	Enables untrusted features contrib and floating-point.
ffi	Enable to include C foreign function interfaces.
derive	Enable to support code generation and links to proofs in documentation.
bindings	Enable to generate Python and R source code. Also enables the ffi and derive features.
partials	Enabled by default. When enabled, then_* functions are generated from make_* functions. Also enables the derive feature.
use-openssl	Enabled by default. Use OpenSSL for secure noise generation.

A list of features available in bindings languages (R, Python) can be found in Feature Listing.

To make the crate compile faster, FFI functions in debug builds support a reduced set of primitive types. Release-mode builds support the full set of primitive types and undergo compiler optimizations, but take longer to compile. You can compile a release build by adding the --release flag. In contrast to debug builds, release builds are located in opendp/rust/target/release. To use a release-mode binary from the Python bindings, set the environment variable OPENDP_TEST_RELEASE=1 before importing OpenDP.

For more on our Rust programming patterns:

• OpenDP Rust Initiation
  • Transformation Structure
  • Example Transformation Constructor
  • Measurement Structure
  • Developer Loop
  • Next Steps

Python Setup

First, build a debug binary that works with Python. (Note that the resulting binary will not work with R.)

cd rust
cargo build --all-features

If you only need to regenerate the Python bindings, this is sufficient:

cargo check --all-features

If you have not already, install Python version 3.9 or higher.

You can install a local Python package that uses your new OpenDP binary.

Optional Virtual Environment

We recommend setting up a virtual environment first, but this is optional:

# recommended. conda is just as valid
cd python
python3 -m venv .venv
source .venv/bin/activate

Change to the python directory, install dependencies, and then install the Python OpenDP library itself.

cd python

pip install -r requirements-dev.txt
pip install -e '.[scikit-learn,polars]'

requirement-dev.txt is compiled from requirements-dev.in: To update dependencies, follow the directions in that file.

In the second line, the -e flag is significant! It stands for “editable”, meaning you only have to run this command once. That is, you do not need to reinstall the OpenDP Python package if changes are made in the /python/src folder or to the library binary, but you should restart the Python interpreter or kernel.

At this point, you should be able import OpenDP as a locally installed package:

import opendp

Note

If you encounter the following error on import:

OSError: dlopen ... (mach-o file, but is an incompatible architecture)

You should check that the architecture from rustc -vV matches your Python architecture. This can occur if you are on a Mac M1 and have an x86_64 Python install.

Python Tests

You can test that things are working by running OpenDP’s Python test suite, using pytest. Run the tests from the python directory.

pytest -v

If everything has gone well, you’ll see a bunch of output, then a line similar to this:

================== 57 passed in 1.02s ==================

If pytest is not found, don’t forget to activate your virtual environment!

This is just a quick overview of building OpenDP.

Python Documentation

This documentation website is built with Sphinx. The source code and developer documentation is here.

R Setup

First, build a debug binary that works with R. (Note that the resulting binary will not work with Python.)

cd rust
cargo build --features untrusted,bindings

If you have not already, install R.

Then, set an environment variable to the absolute path of the OpenDP Library binary directory:

export OPENDP_LIB_DIR=`realpath target/debug`

The default R install for MacOS also includes GUI elements like Tcl/Tk, so for the smoothest development experience we suggest these additional installs:

brew install harfbuzz fribidi libgit2 xquartz

Then, install devtools in R:

install.packages(c("devtools", "RcppTOML", "lintr"))

After each edit to the R or Rust source, run the following command in R to (re)load the R package:

devtools::load_all("R/opendp/", recompile=TRUE)

To do a full package installation from local sources:

tools/r_stage.sh && Rscript -e 'devtools::install("R/opendp")'

To restore to a developer setup, run:

tools/r_stage.sh -c

R Tests

Run tests (tests are located in R/opendp/tests/):

devtools::test("R/opendp")

R also has a built-in check function that runs tests and checks for common errors:

devtools::check("R/opendp")

To run the same check manually, use:

R CMD build R/opendp
R CMD check opendp_*.tar.gz --as-cran

It is important R CMD check is run on the .tar.gz, not on R/opendp, because check depends on some of the changes build makes within the .tar.gz.

R Documentation

This script uses roxygen to generate R/opendp/man pages from #' code comments, and then uses pkgdown to render the documentation website.

tools/r_stage.sh -d

Docs Setup

The documentation build is described in the docs/README.md.

For more on proof writing patterns:

• OpenDP Proof Initiation
  • Proof Structure
  • Function Example
    • LaTex Template
    • Preconditions
    • Pseudocode
    • Postcondition
    • Proof

Release Process

Our release process uses github workflows.

Environment Variables

Name	Description
OPENDP_LIB_DIR	Overrides the directory in which the OpenDP language binding looks for the OpenDP Library binary. See example in R Setup.
OPENDP_POLARS_LIB_PATH	Each OpenDP Polars plugin contains a path to the OpenDP Library binary. When OpenDP is used as a query server, library paths in queries submitted by clients are stale (local to the client). This environment variable overrides paths in new OpenDP Polars plugins and OnceFrames. For Python, you can read this value from opendp._lib.lib_path (read-only). This is separate from OPENDP_LIB_DIR because we anticipate it diverging for R.
OPENDP_HEADLESS	Used by CI. When true, The Python opendp package will import without the presence of the OpenDP Library binary.
OPENDP_SPHINX_PORT and OPENDP_SPHINX_URI	When configured, links to proof documents hosted by Sphinx point to the URI and port. The URI defaults to localhost. Allows for a local documentation site. Start the server from docs/ with make sphinx-server.
OPENDP_RUSTDOC_PORT and OPENDP_RUSTDOC_URI	When configured, links in proof documents to Rustdocs include the URI and port. The URI defaults to localhost. Allows for a local documentation site. Start the server from docs/ with make rustdoc-server.
OPENDP_TEST_RELEASE	When true, and OPENDP_LIB_DIR is set, the library will attempt to load the release binary instead of the debug binary.

Developer Tooling

There are many development environments that work with Rust and LaTex. Here are a few:

• VS Code

• Intellij IDEA

• Sublime

Use whatever tooling you are comfortable with.

A few notes on VS Code:

• Be sure to install the rust-analyzer plugin, not the Rust plugin

• Open rust-analyzer’s extension settings, search “features” and add "untrusted", "bindings"

• Look for Problems in the bottom panel for live compilation errors as you work

• Other useful extensions are “Better Toml”, “crates” and “LaTex Workshop”

• To configure VS Code with suggested tasks and settings: cp -a .vscode-suggested .vscode

A few notes on Intellij IDEA:

• Both Intellij IDEA community edition and the CodeWithMe plugin are free

• Be sure to open the project at the root of the git repository

• Be sure to install the Python and Rust plugins for interactivity

• Be sure to “attach” the Cargo.toml in the red banner the first time you open a Rust source file

• Use run configurations to build the Rust library and run tests