# ABM Framework - Design

## 1. Overview
This is a framework for building sophisticated social agent-based models with particular application to public health.
It should be suited for representing diverse populations and complex interactions among individuals. 

**Key Features:**
- support for heterogeneous agents
- Networks
- designed for medium-scale (1000 to 1 million agents)
- Built in Python

**Target Users:**
academic researchers in epidemiology and public health

## 2. Core Concepts
- What is an agent?
  - an agent is an object representing single entity
  - the most basic agent is an individual human or animal
  - the individuals should have several attributes; by default, including age, sex, gender
  - there might also be agents representing other entities, such as a disease vector, an infection, an institution, etc
  - 
- Network representation and topology
  - individuals should be able to form edges between themselves
  - these edges can be represented as an agent themselves
  - subclasses of edges represent different types of relationships (i.e., friendship, classmate, student-teacher)
  - the edge agent is stored in a Graph() (NetworkX) for that particular edge type
- Environment as a dynamic network
  - there is no explicit representation of space
  - agents may have some coarse spatial attribute (e.g., county of residence, city of residence, neighborhood)
- Time stepping and simulation progression
  - time should be discrete
  - events should be able to be scheduled for a future point in time
  - 
- Key abstractions and their definitions
  - **Agent**: Represents any entity in the simulation (individual, disease, institution, etc.). Has properties/attributes that define its state and methods that define its behaviors.
  - **Edge**: Represents a connection between two agents. Different types (Friendship, Work, Household, DrugExposure, etc.) each with type-specific properties and behaviors.
  - **Network**: Represents all relationships of one type. Contains agents as nodes and relationships as edges. Separate networks exist for each relationship type.
  - **Event**: Something scheduled to happen at a specific time. Can affect agents, relationships, or the overall state of the simulation.
  - **Schedule**: Discrete time steps in the simulation. Agents and events are scheduled relative to time.
  - **Simulation**: Orchestrates the entire model for a single parameter configuration. Manages agents, networks, time, and events. Coordinates stepping forward in time and executing scheduled events.
  - **Batch**: A collection of simulations with a set of parameter configurations, executed in parallel across multiple threads/cores. 
  - **Experiment**: A sequence of at least two batches with a calibration step in between. Runs an initial batch, calibrates parameters based on observed data, then runs subsequent batches with updated parameter configurations.

## 3. Architecture & Design
### High-level system design
The framework follows a layered architecture with clear separation of concerns:

1. **Core Layer**: Defines entities and manages collections of entities
   - **Entity APIs**: Define the core abstractions
     - **Agent**: Base class for all entities in the simulation
     - **Edge**: Base class for all relationships between agents
   - **Management APIs**: Manage entities
     - **Observer**: Base class for monitoring and data collection during simulation
     - **Collective**: Manages Agent objects and provides query/filtering capabilities
     - **NetworkManager**: Manages separate NetworkX graphs for each edge type (friendship_network, work_network, etc.)
     - **Schedule**: Manages events and time stepping

2. **Simulation Layer**: Coordinates the core layer to run simulations
   - **WorldBuilder**: Configures collectives, networks, and initial state for a simulation run
   - **Simulation**: Orchestrates agents, networks, time, and event scheduling for a single run

3. **Batch Layer**: Runs multiple simulations with a set of parameter configurations in parallel
   - **Batch**: Executes simulations with a set of parameter configurations, each in a separate thread/core, collects and aggregates results

4. **Workflow Layer**: High-level orchestration of complex multi-step analyses
   - **Experiment**: Iteratively explores parameter space by running batches with calibration steps in between
   - **Calibrator**: Performs statistical inference to estimate model parameters from batch results
   - **Analysis**: Provides utilities for post-simulation analysis, visualization, and metrics computation
   - **Workflow**: Coordinates complex workflows (parameter sweep → calibration → experiment → analysis), manages data flow between steps

### Core components
- **Agent**: Base class for all entities. User subclasses it to define specific agent types (Person, Disease, Institution, etc.)
- **Edge**: Base class for all relationships. User subclasses it to define edge types (FriendshipEdge, WorkEdge, HouseholdEdge, etc.)
- **Observer**: Base class for data collection and monitoring. Can track agents, networks, events, and custom metrics during simulation
- **Collective**: Base class for managing groups of agents
  - Holds and organizes a collection of Agent objects
  - Provides methods for querying agents (by type, properties, location, etc.)
  - Handles agent addition and removal
  - Can be subclassed to create domain-specific groupings with membership criteria
- **Population**: Default collective that includes all agents
  - Simple subclass of Collective with no filtering
- **NetworkManager**: Container class for managing all Graph objects
  - Maintains separate graphs for each edge type (friendship_network, work_network, etc.)
  - Supports pluggable graph backends (NetworkX default, igraph optional, extensible to others)
  - Provides methods to add/remove edges between agents
  - Provides query methods for relationships (get neighbors, get edges of specific type, etc.)
  - Handles graph operations across multiple relationship types
  - Abstracts away direct manipulation of underlying graph library objects
- **Networks**: Collection of Graph objects, one for each edge type
  - Nodes in each graph are Agent objects
  - Edges in each graph are Edge objects (stored as edge attributes)
  - Graph backend can be selected per network type based on expected density and performance needs
- **Scheduler**: Manages time stepping and event execution
  - Maintains a queue of scheduled events
  - Advances discrete time steps
  - Executes events at their scheduled times
  - Provides API for agents to schedule events
  - Handles event ordering when multiple events occur at the same time
- **Simulation**: Main orchestrator class that manages:
  - Agent population and lifecycle
  - All networks and their updates
  - Parameter set (configuration for this specific simulation run)
  - Data collection during simulation (agent states, network metrics, custom data)
  - Delegates time stepping and event execution to the Scheduler
- **Experiment**: Manages parameter space exploration, batch simulation execution, and data collection
  - Defines parameter space (ranges, discrete values, lists)
  - Generates parameter permutations (Cartesian product or sampling strategies)
  - Runs multiple simulations with different parameter configurations
  - Executes simulations in parallel across multiple worker processes (configurable parallelism)
  - Collects data from each simulation run (agent states, network metrics, etc.)
  - Aggregates results across multiple runs
  - Supports load balancing across worker processes
- **Calibrator**: Performs statistical inference to estimate model parameters
  - Imports experiment results and observed data
  - Implements inference methods (e.g., Bayesian calibration, ABC, other statistical approaches)
  - Generates posterior parameter distributions or point estimates
  - Selects new parameter configurations based on inference results
  - Outputs new parameter configurations for subsequent experiment runs
  - Supports iterative calibration workflows (experiment → calibration → experiment)

### Design patterns used
- **Inheritance**: Agent and Edge have subclasses for different types (polymorphism)
- **Composition**: Simulation contains Networks, which contain Agents and Edges
- **Separation of concerns**: Each network type is independent; users analyze one at a time (following NetLogo's approach)
- **Graph abstraction**: NetworkX handles graph operations; users work with high-level abstractions

### Integration with graph libraries and Python ecosystem
- **Graph backend abstraction**: Framework supports pluggable graph backends (NetworkX, igraph, others)
  - Default backend is NetworkX for flexibility and ecosystem compatibility
  - igraph backend available as option for dense networks (better performance for dense graphs)
  - Users can specify backend per network type based on expected density and use case
- Graph algorithms (centrality, clustering, etc.) can be directly applied to each network via the chosen backend
- Compatible with NumPy, SciPy, Pandas for statistical analysis
- Leverages Python's object model for extensibility and ease of use

## 4. API & Usage

### Core Layer APIs

#### Entity APIs

##### Agent API
Users define custom agent types by subclassing the Agent base class. The base class provides:
- Initialization of agent properties (custom attributes like age, health status, location)
- Storage of internal references to the scheduler (set by the Simulation when the agent is added)
- Agents are event-driven: behavior is defined through events scheduled on them, not through step() methods

##### Edge API
Users define custom edge types by subclassing the Edge base class. The base class provides:
- Initialization of relationship properties (strength, type, metadata)
- Methods to modify relationship state over time

#### Management APIs

##### Observer API


##### Population API
The Population class provides methods for:
- Adding and removing agents
- Querying agents by type or properties
- Counting total agents and agent demographics
- Iterating over agents

##### NetworkManager API
The NetworkManager class provides methods for:
- Creating relationships between agents (specifying edge type and edge object)
- Removing relationships
- Querying neighbors by relationship type
- Accessing the underlying NetworkX graphs for analysis
- Computing network metrics for specific relationship types

##### Scheduler API
The Scheduler class provides methods for:
- Scheduling events at future time steps
- Advancing time step by step
- Querying current simulation time
- Retrieving and executing pending events

### Simulation Layer APIs

#### WorldBuilder API
The WorldBuilder class provides methods for:
- Creating collectives of agents with custom attributes
- Creating networks for different relationship types
- Configuring initial relationships between agents
- Performing complete setup of a simulation

### API Specifications

#### Parameters API
The Parameters class provides methods for:
- Storing and accessing simulation parameters as key-value pairs
- Getting parameter values with optional defaults
- Setting parameter values
- Validating parameters against a schema
- Exporting parameters to a dictionary

#### Simulation API
The Simulation class provides methods for:
- Initializing a simulation with a parameter set
- Adding agents and relationships to collectives and networks
- Scheduling events
- Stepping the simulation forward
- Accessing collected data after simulation completes
- Querying collective and network state at any time

### Batch Layer APIs

#### Batch API
The Batch class provides methods for:
- Accepting a set of parameter configurations
- Executing simulations for each parameter configuration in parallel across threads/cores
- Collecting results from each simulation
- Aggregating results across all simulations in the batch

### Workflow Layer APIs

#### ParameterSweep API
The ParameterSweep class provides methods for:
- Defining the parameter space (which parameters to vary and their value ranges)
- Generating parameter configurations using Cartesian product or sampling strategies
- Running a batch with the generated parameter configurations
- Collecting and aggregating results across all parameter configurations

#### Experiment API
The Experiment class provides methods for:
- Running batches with parameter configurations derived from calibration results
- Comparing different scenarios or counterfactuals (e.g., baseline vs. intervention)
- Taking calibrated parameter configurations as input
- Running simulations across multiple scenarios with the same calibrated parameters and random seeds (for fair comparison)
- Controlling randomness to ensure comparable results across scenarios
- Aggregating and comparing results across scenarios

#### Calibrator API
The Calibrator class provides methods for:
- Loading batch results and observed data
- Specifying which model outputs to compare with observations
- Selecting or implementing an inference method (Bayesian calibration, ABC, etc.)
- Running inference to estimate posterior parameter distributions
- Sampling new parameter configurations from the posterior
- Outputting calibrated parameter sets for subsequent experiment runs

#### Analysis API
The Analysis class provides methods for:
- Computing and aggregating metrics across multiple simulations
- Generating summary statistics of simulation results
- Performing statistical tests and comparisons
- Creating visualizations of results
- Exporting results to standard formats (CSV, JSON)

#### Workflow API
The Workflow class provides a high-level interface for coordinating complex multi-step analyses:
- Defining a sequence of steps (parameter sweep → calibration → experiment → analysis)
- Automatically passing results from one step to the next
- Managing data flow between ParameterSweep, Calibrator, Experiment, and Analysis components
- Tracking the full workflow history and results
- Supporting branching workflows (e.g., multiple calibration runs, multiple counterfactuals)
- Saving and loading workflow state for reproducibility
- Distributing workflow execution across HPC or distributed systems (supporting batch job submission, job schedulers like SLURM)

### Configuration and Setup
Models are configured through a parameter dictionary that specifies:
- scenario/counterfactual
- Population size and initial demographics
- Network initialization parameters (size, type, density)
- Agent behavior parameters (rates, probabilities, thresholds)
- Model-specific parameters defined by the user

### Data Collection and Metrics
Data collection happens automatically during simulation execution. Users can:
- Specify what agent-level data to collect
- Specify what network-level metrics to collect
- specify what population-level metrics to collect
- Define custom collectors for domain-specific data
- Access collected data as structured results after simulation

Available metrics include:
- Agent counts and demographics
- Network size and density by relationship type
- Network centrality measures (degree, betweenness, eigenvector, etc.)
- Custom user-defined metrics

### Visualization and Export
The framework provides support for:
- Exporting simulation results to standard data formats (CSV, JSON)
- Exporting network structures for external visualization
- Built-in network visualization tools
- Time-series visualization of collected metrics
- Integration with standard scientific Python tools (Pandas, NumPy, SciPy) for analysis

## 5. Implementation Details

### Technology Stack
- **Language**: Python 3.9+
- **Graph library**: NetworkX 3.0+
- **Data structures**: Built-in Python collections for agent and event management
- **Optional dependencies**: NumPy, SciPy, Pandas for advanced statistical analysis
- **Visualization**: Matplotlib, NetworkX built-in visualization tools
- **Parallel execution**: Python's multiprocessing for running multiple simulations in parallel

### Core Implementation Strategy
- **Agent/Edge inheritance**: Users subclass base Agent and Edge classes to define custom types
- **Graph-based networks**: NetworkX graphs store agent-to-agent relationships, with Edge objects as edge attributes
- **Event scheduling**: Priority queue (heap) for efficient event scheduling and retrieval
- **Data collection**: Centralized collector that records metrics at each time step and aggregates across runs
- **Parallel execution**: Python's multiprocessing module for running multiple simulations in parallel with configurable worker processes and load balancing

### Performance Considerations for Medium-Scale Networks (1K-1M agents)
- **Memory efficiency**: Agents and edges are Python objects; network structure uses NetworkX's adjacency dict representation
- **Time stepping**: Efficient iteration through scheduled events at each time step
- **Network analysis**: NetworkX algorithms are optimized for sparse networks; dense networks may require sampling strategies
- **Scaling strategy**: For 1M+ agents, consider:
  - Using network sampling or subgraph analysis
  - Distributed execution across multiple machines
  - Periodic network simplification (removing inactive nodes/edges)

### Key Algorithms and Optimizations
- **Event scheduling**: Binary heap (priority queue) for O(log n) insertion and retrieval of events
- **Agent iteration**: Efficient access patterns to population for step execution
- **Network queries**: Leveraging NetworkX's optimized neighbor queries and graph algorithms
- **Data aggregation**: Vectorized operations using NumPy/Pandas for result aggregation across runs

### Key Design Decisions
- **No explicit spatial representation**: Agents have optional location attributes but no enforcement of spatial constraints; relationships are purely network-based
- **Separate networks by type**: Each relationship type has its own NetworkX graph, avoiding complexity of MultiGraph while maintaining separation of concerns
- **Event-driven simulation**: Agents don't all step synchronously; events can be scheduled for specific times
- **Immutable parameter sets**: Each simulation run receives a fixed parameter set from the Experiment; parameters do not change during a run

### Example Use Cases
- Epidemic modeling on social contact networks (disease spread, intervention effects)
- Opinion dynamics and social influence on networks (polarization, consensus)
- Public health interventions (vaccination strategies, resource allocation)
- Network formation and evolution (how relationships change over time)
- Multi-agent economic models (trading, cooperation on networks)

### Limitations and Constraints
- **Network size**: Maximum practical size is 1M+ agents; beyond that requires distributed computing or approximation techniques
- **Dense networks**: Framework is optimized for sparse networks; very dense networks (>10M edges) may have memory/performance issues
- **Scalability**: Framework supports local multi-core parallelization via multiprocessing. For very large-scale parameter sweeps, workflows can be distributed to HPC systems with SLURM/job schedulers
- **Spatial modeling**: Not designed for continuous spatial simulations; use specialized spatial ABM frameworks for those use cases
- **Real-time constraints**: Framework is for offline research; not designed for interactive real-time simulations
- **Stochasticity**: Full control over randomness is the user's responsibility; framework does not provide built-in stochasticity management beyond event scheduling

### Distributed Computing and HPC Support
- **Local parallelization**: Built-in multiprocessing for single-machine multi-core execution
- **HPC deployment**: Workflows can be submitted to HPC clusters using SLURM or similar job schedulers
- **Batch job generation**: Framework can generate batch scripts for distributed experiment execution
- **Result aggregation**: Distributed results are automatically collected and aggregated post-execution

