Designing Isolated analysis

We started with the definition of the arithmetic mean:

In the description above, we calculated the mean in what looks like a different way, adding values and keeping a running total of the count. We can see that this is, in fact, the same thing if we reframe the count from :

As the count can be seen as a sum, we can make the jump from the case where all the values are together to when they are apart. As it doesn't matter what order you sum numbers, we can get local sums and then add them together to get a global value, as in the description above. This means that, for TREs:

For our purposes, we can see that both numerator and denominator have , so we can sum the local count and local sum from each TRE.

Monoids

There is a mathematical structure called a monoid. A monoid is a set (), with an operation () that needs three properties.

• Using on two elements of has to make another
• The operation has to be associative, so for and in ,
• There needs to be an identity element, where and

This looks suspiciously like the function that combines the objects as described above. Luckily for us, a lot of statistics can be computed using building blocks that can be combined with associative operations: real numbers and addition form a monoid (with 0 as an identity element), and positive real numbers and multiplication (with 1 as an identity element).

If we can express the partial results from nodes in a federation () and the operation used to combine them as a monoid, we can abstract the aggregation phase of a federated analysis as:

Then using some other function () applied to that aggregated partial result to calculate a final result, the overall function is then

For our mean example, the partial state is a tuple . The monoid operation combines them element-wise:

and our finalisation function is simply

There are other ways to federate statistics, but if you can break the calculation down into something that can be calculated from monoids that can be aggregated, you can federate it.

Do it yourself in Python

To help you understand how basic statistics can be federated, you can run through these examples. You can define three functions to do a federated analysis yourself.

1. node_function runs in each node and summarises a list into a partial result: a summary of the local data
2. aggregate_function merges two partial results into another partial result
3. finalise_function uses a merged result to calculate the final result of the desired analysis

These will be assembled into a simulation of a federated analytics pipeline. The examples run on three lists of numbers, one made visible to you so you can verify your node_function does what you intend, two hidden from you, representing datasets you can't access directly.

The code currently in the cells will calculate the global count for you. If you change the node_function, the later stages might throw errors. Your job is to fix these errors and calculate the final result. Good luck!

Computing partial results at the nodes

The node_function is what runs in each node to get out the data necessary to calculate your final result. In reality, you're unlikely to be operating on a list of numbers; this is just so you can think about the partial results that needs to come out of each node.

Write your definition of node_function in the box below. If you change the function names, the evaluation will not run properly.

Give me a hint

The first node's data is , and your function gives the partial result:

Aggregating partial results

Next, you need to combine your partial results into something that describes the whole dataset across nodes. To do this, the aggregator has to have some function that combines two partial results, which can then be applied to all of the partial results like this:

result = PartialResult()

for partial in partial_results:
    result = aggregate_function(result, partial)

Many programming languages, including python, provide a "reduce" function that performs the same operation.

Define your aggregate_function below.

Give me a hint

The aggregate result is:

After the partial results from nodes have been aggregated, all but the simplest statistics need some further processing before they give their result. To do this, we need a function definition to apply to the aggregated result. For the simple statistics, returning the value is sufficient.

Give me a hint

The final result is:

.

The partialstats module uses this approach to provide functions for aggregating common statistics and a scaffold for making your own functions.

Below, there is a demonstrator to help you get a feel for how an isolated analysis can work.

Overall, this analysis

Running an example

Here are some text boxes. It has some example data in it that will work for the kinds of analysis that take a single number from each row of a dataset. You can put your own numbers in, just separate them with commas.

This page will pretend that this is a dataset held across different nodes.

Functions

Local function

The way it does this is by applying a local function to each of the datasets. The local function, "",

Aggregation function

The analysis takes the output of for each dataset and applies an aggregation function to these intermediate values. The aggregation function, "",