Aggregation#
The most common aggregators like length, sum, mean, median and quantile are covered in Essential Statistics.
In addition to these aggregators, OpenDP also supports other variations of counting queries. A counting query tells you how many rows in a dataset meet a given condition.
>>> import polars as pl
>>> import opendp.prelude as dp
>>> dp.enable_features("contrib")
To get started, we’ll recreate the Context from Working with Tabular Data.
>>> context = dp.Context.compositor(
... data=pl.scan_csv(dp.examples.get_france_lfs_path()),
... privacy_unit=dp.unit_of(contributions=36),
... privacy_loss=dp.loss_of(epsilon=1.0),
... split_evenly_over=5,
... )
Frame Length vs Expression Length#
Frame length is not the same as expression length. These quantities can differ if the expression changes the number of rows.
>>> query_len_variations = (
... context.query()
... .group_by("SEX")
... .agg(
... [
... # total number of rows in the frame, including nulls
... dp.len(signed=True),
... # total number of rows in the HWUSUAL column (including nulls)
... pl.col.HWUSUAL.dp.len(signed=True),
... ]
... )
... # explicitly specifying keys makes the query satisfy pure-DP
... .with_keys(pl.LazyFrame({"SEX": [1, 2]}))
... )
>>> query_len_variations.summarize()
shape: (2, 4)
┌─────────┬──────────────┬─────────────────┬───────┐
│ column ┆ aggregate ┆ distribution ┆ scale │
│ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ str ┆ f64 │
╞═════════╪══════════════╪═════════════════╪═══════╡
│ len ┆ Frame Length ┆ Integer Laplace ┆ 360.0 │
│ HWUSUAL ┆ Length ┆ Integer Laplace ┆ 360.0 │
└─────────┴──────────────┴─────────────────┴───────┘
These two statistics are equivalent, but the frame length (the first) can be used to release stable grouping keys, while the column length (the second) can be preprocessed with filtering.
The OpenDP Library will still use margin descriptors that may reduce the sensitivity of the column length if it detects that the column has not been transformed in a way that changes the number of rows.
>>> query_len_variations.release().collect()
shape: (2, 3)
┌─────┬─────────┬─────────┐
│ SEX ┆ len ┆ HWUSUAL │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪═════════╪═════════╡
...
└─────┴─────────┴─────────┘
Unique Counts#
A count of the number of unique values in a column is as sensitive as the frame or column length when protecting user contributions. However, unlike the frame length, the sensitivity does not reduce to zero when protecting changed records, as a change in an individual’s answer may result in one more, or one less, unique value.
>>> query_n_unique = context.query().select(
... [
... # total number of unique elements in the HWUSUAL column (including null)
... pl.col.HWUSUAL.dp.n_unique(signed=True),
... ]
... )
>>> query_n_unique.summarize()
shape: (1, 4)
┌─────────┬───────────┬─────────────────┬───────┐
│ column ┆ aggregate ┆ distribution ┆ scale │
│ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ str ┆ f64 │
╞═════════╪═══════════╪═════════════════╪═══════╡
│ HWUSUAL ┆ N Unique ┆ Integer Laplace ┆ 180.0 │
└─────────┴───────────┴─────────────────┴───────┘
>>> query_n_unique.release().collect()
shape: (1, 1)
┌─────────┐
│ HWUSUAL │
│ --- │
│ i64 │
╞═════════╡
...
└─────────┘
These examples use signed=True to preserve negative noisy counts as
Int64 values. With signed=False, negative noisy counts will be
returned as zero.
This release tells us that the number of null values is relatively small.
Signed Counts#
Pass signed=True to return counts as an Int64,
which allows negative noisy outputs that are not clamped to zero.
This applies to all counting queries (dp.len, expression
len, count, null_count and n_unique).
For now, omitting signed (or passing None) retains unsigned output
and emits a warning. The default will change to True in a future release.
For example:
>>> query_signed = context.query().select(
... [
... dp.len(signed=True),
... pl.col.HWUSUAL.dp.len(signed=True).alias(
... "expr_len"
... ),
... pl.col.HWUSUAL.dp.count(signed=True).alias("count"),
... pl.col.HWUSUAL.dp.null_count(signed=True).alias(
... "null_count"
... ),
... pl.col.HWUSUAL.dp.n_unique(signed=True).alias(
... "n_unique"
... ),
... ]
... )
>>> query_signed.release().collect()
shape: (1, 5)
┌─────┬──────────┬───────┬────────────┬──────────┐
│ len ┆ expr_len ┆ count ┆ null_count ┆ n_unique │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪══════════╪═══════╪════════════╪══════════╡
│ ... ┆ ... ┆ ... ┆ ... ┆ ... │
└─────┴──────────┴───────┴────────────┴──────────┘
Null and Non-Null Counts#
You can release a count of the number of null or non-null records, respectively, as follows:
>>> query_counts = context.query().select(
... [
... # total number of non-null elements in the HWUSUAL column
... pl.col.HWUSUAL.dp.count(signed=True),
... # total number of null elements in the HWUSUAL column
... pl.col.HWUSUAL.dp.null_count(signed=True),
... ]
... )
>>> query_counts.summarize()
shape: (2, 4)
┌─────────┬────────────┬─────────────────┬───────┐
│ column ┆ aggregate ┆ distribution ┆ scale │
│ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ str ┆ f64 │
╞═════════╪════════════╪═════════════════╪═══════╡
│ HWUSUAL ┆ Count ┆ Integer Laplace ┆ 360.0 │
│ HWUSUAL ┆ Null Count ┆ Integer Laplace ┆ 360.0 │
└─────────┴────────────┴─────────────────┴───────┘
Notice that the count and null_count are complementary: you
could instead release len for HWUSUAL grouped by whether the
value is null.
You can take advantage of this to estimate both statistics with the same privacy loss, but with half as much noise.
>>> query_counts_via_grouping = (
... context.query()
... .with_columns(
... pl.col("HWUSUAL").is_null().alias("HWUSUAL_is_null")
... )
... .group_by("HWUSUAL_is_null")
... .agg(dp.len(signed=True))
... # we're grouping on a bool column, so the groups are:
... .with_keys(
... pl.LazyFrame({"HWUSUAL_is_null": [True, False]})
... )
... )
>>> query_counts_via_grouping.summarize()
shape: (1, 4)
┌────────┬──────────────┬─────────────────┬───────┐
│ column ┆ aggregate ┆ distribution ┆ scale │
│ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ str ┆ f64 │
╞════════╪══════════════╪═════════════════╪═══════╡
│ len ┆ Frame Length ┆ Integer Laplace ┆ 180.0 │
└────────┴──────────────┴─────────────────┴───────┘
The noise scale dropped from 360 to 180…
>>> query_counts_via_grouping.release().collect()
shape: (2, 2)
┌─────────────────┬─────────┐
│ HWUSUAL_is_null ┆ len │
│ --- ┆ --- │
│ bool ┆ i64 │
╞═════════════════╪═════════╡
...
└─────────────────┴─────────┘
…but we still get answers to all of the same queries!