# Workflow Overview

This document provides an overview of the workflows and processes in the ABM Framework.

! (./Untitled - ABM Framework Overview.jpg)

## Layers
The ABM Framework has 4 layers:
- **Layer 4, Workflow and Analysis,** the highest level of the model;
- **Layer 3, Batch,** which runs multiple simulation runs in parallel;
- **Layer 2, Simulation,** which handles an individual simulation run;
- **Layer 1, Core,** which handles all the internal operations of a simulation.

The basic workflow progression has four stages, all handled in Layer 4:
1. **Parameter Sweep,** where a large number of parameter combinations are explored;
2. **Calibration,** where the results of the parameter sweep are compared to calibration targets, and realistic parameter combinations are selected;
3. **Experiment,** where the calibrated parameter combinations are plugged back into the model, typically under a range of scenarios;
4. **Analysis,** where the results of the counterfactual scenarios are compared and visualized.

## Here's a walkthrough, with examples from a gonorrhea transmission model.

### 1. Parameter Sweep
The *ParameterSweep* is meant to explore the parameter space of the model and return results for the purpose of calibration.
The user defines a parameter set, prior distributions for each parameter, the size the parameter sweep (number of replicates), and the stopping condition(s).
For example, we might have a parameter **SymptomProportion** for how likely each gonorrhea infection is to develop symptoms,
and we might want this probability to vary uniformly from 0.1 to 0.8 between different simulations we run.
We might want a sweep size of 5 and each simulation to stop after 52 weeks.
The *ParameterSweep* uses this input to generate a set of parameter configurations (e.g., a **List of Dictionaries,** in Python) to be tested in the model.
In our example, this would look like:

```python
paramConfigList = [
    {"scenario" : "sweep", "runnumber" : 1, "SymptomProportion": 0.25, "endtime": 52},
    {"scenario" : "sweep", "runnumber" : 2, "SymptomProportion": 0.50, "endtime": 52},
    {"scenario" : "sweep", "runnumber" : 3, "SymptomProportion": 0.11, "endtime": 52},
    {"scenario" : "sweep", "runnumber" : 4, "SymptomProportion": 0.68, "endtime": 52},
    {"scenario" : "sweep", "runnumber" : 5, "SymptomProportion": 0.33, "endtime": 52},
]
```

The *ParameterSweep* now creates a *Batch* and passes it the set of parameter configurations.

Hypothetically, the *ParameterSweep* could divide the paramConfigList into separate batches, and then distribute these 
across machines in a cluster. However, for the simplest local workflow, the *ParameterSweep* can use just one *Batch*
for testing all the parameter configurations in paramConfigList.

#### 1.1 Batch
The *Batch* takes a set of parameter configurations (e.g., List of Dictionaries) and assigns each to a single *Simulation*. 
Each *Simulation* is distributed to a thread so that many *Simulations* within a *Batch* can run in parallel.

#### 1.2 Simulation
The *Simulation* receives just a single Dictionary of parameter values (i.e., one parameter configuration) from the 
*Batch*, called *ParamDict*.
The *Simulation* uses *ParamDict* to create the starting conditions for a new simulation using the *WorldBuilder*.
The *WorldBuilder* accesses the **Core Layer** in order to create the major entities and managers for the simulation. 
Then, the *WorldBuilder* passes the initialized *Simulation* to *SingleRunner*, who executes the simulation until the 
stopping condition(s) are met.

#### 1.3 Core
The *SingleRunner* runs the simulation by asking the *Schedule* to execute actions until the stopping condition(s) are met. 
The *Schedule* has agents perform actions (e.g., infect another agent with gonorrhea), 
which then lead to new *Events* being added to the *Schedule* for future execution (e.g., develop gonorrhea symptoms -> visit STI clinic).
The *Observer* gathers data from the simulated population and saves it in an efficient manner (.csv). 

#### 1.4 Simulation 
When the *Simulation* detects that it is complete, it passes the final data from that simulation (e.g., address of .csv 
file) up to the *Batch*.

#### 1.5 Batch
When the *Batch* detects that all the *Simulations* have been completed, it pools the data from all of the *Simulations* 
in the Batch (e.g., into one .csv file for the whole *Batch*).
This pooled data is then passed to the *Calibrator*.

### 2. Calibrator (Bayesian Melding)
The *Calibrator* performs calibration using the data generated by a *ParameterSweep*. 
The *ParameterSweep* output is compared to calibration targets defined by the user. 
For example, the prevalence of gonorrhea in the model (part of the output generated by the *ParameterSweep*) is compared 
to empirical data where 93 individuals tested positive for gonorrhea out of 2075 individuals tested (Johnson Jones et al. 2019).
Then, the *Simulation* data is compared to this target data to ask:

If this Simulation were the underlying population, how likely is it that we would get 93 positive tests out of 2075 trials?

These likelihoods are then used to calculate likelihood_weights for each *Simulation*, so that the most-likely 
(i.e., most realistic) *Simulations* have higher likelihood_weights. Then, these weights are used to resample with 
replacement to select the *CalibratedTrajectories*. How many times an individual trajectory is resampled is its 
resample_weight, to be used later in visualization and analysis.
From each *CalibratedTrajectory*, the *ParamDict* is extracted and used to create a new List of Dictionaries, 
representing the calibrated set of parameter configurations to be plugged back into the model by the *Experiment*.

In our example, we might be left with 3 out of our original 5 trajectories.
```python
paramConfigList = [
    {"runnumber" : 2, "SymptomProportion": 0.50},
    {"runnumber" : 4, "SymptomProportion": 0.68},
    {"runnumber" : 5, "SymptomProportion": 0.33},
]
```

The *Calibrator* has some visualization functionality, so that the *CalibratedTrajectories* might be approved by the user
before continuing with the *Experiment*. If the *CalibratedTrajectories* are not realistic enough, or there are not a 
sufficient number of resampled trajectories, it may be appropriate to adjust the prior distributions of at least some
parameters and re-run the *ParameterSweep*.

### 3. Experiment
The *Experiment* runs multiple different *Scenarios* using the same set of parameter configurations, as passed to it by the *Calibrator*. 
The *Scenarios* are defined by the user. 
For example, we might compare two *Scenarios*, one where asymptomatic individuals are tested on average every month ("Scenario A"), 
vs one where asymptomatic individuals are never tested ("Scenario B"). Both *Scenarios* will be run using the same parameter configuration.

However, each *Scenario* will have its own *Batch*. The scenario information (i.e., whether to test every month or never)
will be passed as a parameter (String) through the *Batch* and *Simulation* down to the mechanism in the Core Layer
(e.g., a Screener which causes the Agents to be screened at the appropriate frequency for the scenario being run).

Hypothetically, the *Experiment* could distribute the *Scenarios* to different machines in a cluster. 
However, for a local workflow, the *Scenarios*/*Batches* will be executed separately, in sequence.

#### 3.1 Batch (Scenario A)
For Scenario A, our parameter configuration is this List of Dictionaries:

```python
paramConfigList = [
    {"scenario" : "A", "runnumber" : 2, "SymptomProportion": 0.50},
    {"scenario" : "A", "runnumber" : 4, "SymptomProportion": 0.68},
    {"scenario" : "A", "runnumber" : 5, "SymptomProportion": 0.33},
]
```

The *Batch* assigns each Dictionary to a single *Simulation*, and distributes these across threads to be run in parallel.
Within each *Simulation*, the process is identical except that the *Scenario* parameter will now cause screening
to happen every month for agents without symptoms (e.g., through a Screener class which puts an event in the *Schedule*).

When the *Batch* detects that all the *Simulations* have been completed, it pools the data from all of the *Simulations* 
in the Batch (e.g., into one .csv file for the whole *Batch*; "19nov2025_screeningABM_scenarioA.csv"). 
The *Experiment* then moves on to the next *Scenario*.

#### 3.2 Batch (Scenario B)
The process repeats, with only the *scenario* changing in our List of Dictionaries: 

```python
paramConfigList = [
    {"scenario" : "B", "runnumber" : 2, "SymptomProportion": 0.50},
    {"scenario" : "B", "runnumber" : 4, "SymptomProportion": 0.68},
    {"scenario" : "B", "runnumber" : 5, "SymptomProportion": 0.33},
]
```

Again, the *Batch* assigns each Dictionary to a single *Simulation*, and these are run on separate threads. 
In this example, the scenario parameter causes the Screener to be turned off, so that asymptomatic agents are never 
screened. All the data from this *Batch* will be stored as its own .csv (e.g., "19nov2025_screeningABM_scenarioB.csv").

Once every *Scenario*'s *Batch* has been completed, we move onto the final stage.

### 4. Analysis
The *Analyzer* takes the data output from the *Experiment* and uses it to test the research question or hypothesis. 
E.g., monthly gonorrhea screening will decrease gonorrhea incidence and decrease healthcare costs long-term. 

We have separate .csv files from each *Scenario*, which we can now combine (provided each row is labeled with the 
Scenario information) into one pandas DataFrame for analysis and visualization. We also have the resampled_weights 
from the *Calibrator*, which we can use to weight the results.

To compare the *Scenarios*, we can
- plot the incidence of the trajectories, with side-by-side panels for Scenario A and Scenario B (line thickness determined by resampled_weights)
- plot the annual healthcare costs associated with gonorrhea over time, in both scenarios side-by-side (line thickness determined by resampled_weights)
- calculate the cumulative healthcare costs of each scenario, given 3% discounting year-over-year (weighted using resampled_weights)
