{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# cadCAD Tutorials: The Robot and the Marbles, part 2\n",
    "In [Part 1](../robot-marbles-part-1/robot-marbles-part-1.ipynb) we introduced the 'language' in which a system must be described in order for it to be interpretable by cadCAD and some of the basic concepts of the library:\n",
    "* State Variables\n",
    "* Timestep\n",
    "* State Update Functions\n",
    "* Partial State Update Blocks\n",
    "* Simulation Configuration Parameters\n",
    "\n",
    "This article will introduce the concept of __Policies__. But first let's copy the base configuration from Part 1. As a reminder, here's the description of the simple system we are using for illustration purposes.\n",
    "\n",
    "__The robot and the marbles__  \n",
    "* Picture a box (`box_A`) with ten marbles in it; an empty box (`box_B`) next to the first one; and a robot arm capable of taking a marble from any one of the boxes and dropping it into the other one. \n",
    "* The robot is programmed to take one marble at a time from the box containing the largest number of marbles and drop it in the other box. It repeats that process until the boxes contain an equal number of marbles.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "\n",
    "# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # \n",
    "# List of all the state variables in the system and their initial values\n",
    "initial_conditions = {\n",
    "    'box_A': 10, # as per the description of the example, box_A starts out with 10 marbles in it\n",
    "    'box_B': 0 # as per the description of the example, box_B starts out empty\n",
    "}\n",
    "# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # \n",
    "\n",
    "def update_A(params, step, sL, s, _input):\n",
    "    y = 'box_A'\n",
    "    add_to_A = 0\n",
    "    if (s['box_A'] > s['box_B']):\n",
    "        add_to_A = -1\n",
    "    elif (s['box_A'] < s['box_B']):\n",
    "        add_to_A = 1\n",
    "    x = s['box_A'] + add_to_A\n",
    "    return (y, x)\n",
    "\n",
    "def update_B(params, step, sL, s, _input):\n",
    "    y = 'box_B'\n",
    "    add_to_B = 0\n",
    "    if (s['box_B'] > s['box_A']):\n",
    "        add_to_B = -1\n",
    "    elif (s['box_B'] < s['box_A']):\n",
    "        add_to_B = 1\n",
    "    x = s['box_B'] + add_to_B\n",
    "    return (y, x)\n",
    "\n",
    "# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # \n",
    "# In the Partial State Update Blocks, the user specifies if state update functions will be run in series or in parallel\n",
    "partial_state_update_blocks = [\n",
    "    { \n",
    "        'policies': { # We'll ignore policies for now\n",
    "        },\n",
    "        'variables': { # The following state variables will be updated simultaneously\n",
    "            'box_A': update_A,\n",
    "            'box_B': update_B\n",
    "        }\n",
    "    }\n",
    "]\n",
    "# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # \n",
    "\n",
    "# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # \n",
    "# Settings of general simulation parameters, unrelated to the system itself\n",
    "# `T` is a range with the number of discrete units of time the simulation will run for;\n",
    "# `N` is the number of times the simulation will be run (Monte Carlo runs)\n",
    "# In this example, we'll run the simulation once (N=1) and its duration will be of 10 timesteps\n",
    "# We'll cover the `M` key in a future article. For now, let's leave it empty\n",
    "simulation_parameters = {\n",
    "    'T': range(10),\n",
    "    'N': 1,\n",
    "    'M': {}\n",
    "}\n",
    "# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # \n",
    "\n",
    "from cadCAD.configuration import Configuration\n",
    "\n",
    "# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # \n",
    "# The configurations above are then packaged into a `Configuration` object\n",
    "config = Configuration(initial_state=initial_conditions, #dict containing variable names and initial values\n",
    "                       partial_state_update_blocks=partial_state_update_blocks, #dict containing state update functions\n",
    "                       sim_config=simulation_parameters #dict containing simulation parameters\n",
    "                      )\n",
    "\n",
    "from cadCAD.engine import ExecutionMode, ExecutionContext, Executor\n",
    "exec_mode = ExecutionMode()\n",
    "exec_context = ExecutionContext(exec_mode.single_proc)\n",
    "executor = Executor(exec_context, [config]) # Pass the configuration object inside an array\n",
    "raw_result, tensor = executor.execute() # The `execute()` method returns a tuple; its first elements contains the raw results\n",
    "\n",
    "%matplotlib inline\n",
    "import pandas as pd\n",
    "df = pd.DataFrame(raw_result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 15376 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.plot('timestep', ['box_A', 'box_B'], grid=True, \n",
    "        colormap = 'RdYlGn',\n",
    "        xticks=list(df['timestep'].drop_duplicates()), \n",
    "        yticks=list(range(1+(df['box_A']+df['box_B']).max())));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Policies\n",
    "In part 1, we ignored the `_input` argument of state update functions. That argument is a signal passed to the state update function by another set of functions: Policy Functions.\n",
    "\n",
    "Policy Functions are most commonly used as representations of the behavior of agents that interact with the components of the system we're simulating in cadCAD. But more generally, they describe the logic of some component or mechanism of the system. It is possible to encode the functionality of a policy function in the state update functions themselves (as we did in part 1, where we had the robot's algorithm reside in the `update_A` and `update_B` functions), but as systems grow more complex this approach makes the code harder to read and maintain, and in some cases more inefficient because of unnecessary repetition of computational steps.\n",
    "\n",
    "The general structure of a policy function is:\n",
    "```python\n",
    "def policy_function(params, step, sL, s):\n",
    "    ...\n",
    "    return {'value1': value1, 'value2': value2, ...}\n",
    "```\n",
    "Just like State Update Functions, policies can read the current state of the system from argument `s`, a Python `dict` where the `dict_keys` are the __names of the variables__ and the `dict_values` are their __current values__. The Policy Function must return a dictionary, which will be passed as an argument (`_input`) to the state update functions.\n",
    "![Policy](policy.png \"Policy\")\n",
    "\n",
    "Let's update our simulation so that the robot arm's logic is encoded in a Policy instead of in the State Update Functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # \n",
    "# We specify the robot arm's logic in a Policy Function\n",
    "def robot_arm(params, step, sL, s):\n",
    "    add_to_A = 0\n",
    "    if (s['box_A'] > s['box_B']):\n",
    "        add_to_A = -1\n",
    "    elif (s['box_A'] < s['box_B']):\n",
    "        add_to_A = 1\n",
    "    return({'add_to_A': add_to_A, 'add_to_B': -add_to_A})\n",
    "    \n",
    "\n",
    "# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # \n",
    "# We make the state update functions less \"intelligent\",\n",
    "# ie. they simply add the number of marbles specified in _input \n",
    "# (which, per the policy function definition, may be negative)\n",
    "def increment_A(params, step, sL, s, _input):\n",
    "    y = 'box_A'\n",
    "    x = s['box_A'] + _input['add_to_A']\n",
    "    return (y, x)\n",
    "\n",
    "def increment_B(params, step, sL, s, _input):\n",
    "    y = 'box_B'\n",
    "    x = s['box_B'] + _input['add_to_B']\n",
    "    return (y, x)\n",
    "# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # \n",
    "\n",
    "\n",
    "# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # \n",
    "# In the Partial State Update Blocks, \n",
    "# the user specifies if state update functions will be run in series or in parallel\n",
    "# and the policy functions that will be evaluated in that block\n",
    "partial_state_update_blocks = [\n",
    "    { \n",
    "        'policies': { # The following policy functions will be evaluated and their returns will be passed to the state update functions\n",
    "            'robot_arm': robot_arm\n",
    "        },\n",
    "        'states': { # The following state variables will be updated simultaneously\n",
    "            'box_A': increment_A,\n",
    "            'box_B': increment_B\n",
    "        }\n",
    "    }\n",
    "]\n",
    "# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # \n",
    "\n",
    "# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # \n",
    "# The configurations above are then packaged into a `Configuration` object\n",
    "config = Configuration(initial_state=initial_conditions, #dict containing variable names and initial values\n",
    "                       partial_state_update_blocks=partial_state_update_blocks, #dict containing state update functions\n",
    "                       sim_config=simulation_parameters #dict containing simulation parameters\n",
    "                      )\n",
    "\n",
    "exec_mode = ExecutionMode()\n",
    "exec_context = ExecutionContext(exec_mode.single_proc)\n",
    "executor = Executor(exec_context, [config]) # Pass the configuration object inside an array\n",
    "raw_result, tensor = executor.execute() # The `execute()` method returns a tuple; its first elements contains the raw results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 15376 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.plot('timestep', ['box_A', 'box_B'], grid=True, \n",
    "        xticks=list(df['timestep'].drop_duplicates()), \n",
    "        colormap = 'RdYlGn',\n",
    "        yticks=list(range(1+(df['box_A']+df['box_B']).max())));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, the results are the same as when the robot arm logic was encoded within the state update functions.  \n",
    "\n",
    "Several policies may be evaluated within a Partial State Update Block. When that's the case, cadCAD's engine aggregates the outputs of the policies and passes them as a single signal to the state update functions. \n",
    "![Policies](policies.png \"Policies\")\n",
    "\n",
    "Aggregation of policies is defined in cadCAD as __key-wise sum (+) of the elements of the outputted `dict`s__.\n",
    "```python\n",
    ">policy_1_output = {'int': 1, 'str': 'abc', 'list': [1, 2], '1-only': 'Specific to policy 1'}\n",
    ">policy_2_output = {'int': 2, 'str': 'def', 'list': [3, 4], '2-only': 'Specific to policy 2'}\n",
    ">print(aggregate([policy_1_output, policy_2_output]))\n",
    "```\n",
    "```\n",
    "{'int': 3, 'str': 'abcdef', 'list': [1, 2, 3, 4], '1-only': 'Specific to policy 1', '2-only': 'Specific to policy 2'}\n",
    "```\n",
    "\n",
    "To illustrate, let's add to another system another robot arm identical to the first one, that acts in tandem with it. All it takes is to add a policy to the `dict` that describes the partial state update block."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # \n",
    "# In the Partial State Update Blocks, \n",
    "# the user specifies if state update functions will be run in series or in parallel\n",
    "# and the policy functions that will be evaluated in that block\n",
    "partial_state_update_blocks = [\n",
    "    { \n",
    "        'policies': { # The following policy functions will be evaluated and their returns will be passed to the state update functions\n",
    "            'robot_arm_1': robot_arm,\n",
    "            'robot_arm_2': robot_arm\n",
    "        },\n",
    "        'variables': { # The following state variables will be updated simultaneously\n",
    "            'box_A': increment_A,\n",
    "            'box_B': increment_B\n",
    "        }\n",
    "    }\n",
    "]\n",
    "# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # \n",
    "\n",
    "# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # \n",
    "# The configurations above are then packaged into a `Configuration` object\n",
    "config = Configuration(initial_state=initial_conditions, #dict containing variable names and initial values\n",
    "                       partial_state_update_blocks=partial_state_update_blocks, #dict containing state update functions\n",
    "                       sim_config=simulation_parameters #dict containing simulation parameters\n",
    "                      )\n",
    "\n",
    "exec_mode = ExecutionMode()\n",
    "exec_context = ExecutionContext(exec_mode.single_proc)\n",
    "executor = Executor(exec_context, [config]) # Pass the configuration object inside an array\n",
    "raw_result, tensor = executor.execute() # The `execute()` method returns a tuple; its first elements contains the raw results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 25460 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "df = pd.DataFrame(raw_result)\n",
    "df.plot('timestep', ['box_A', 'box_B'], grid=True, \n",
    "        xticks=list(df['timestep'].drop_duplicates()), \n",
    "        colormap = 'RdYlGn',\n",
    "        yticks=list(range(1+(df['box_A']+df['box_B']).max())));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because we have made it so that both robots read and update the state of the system at the same time, the equilibrium we had before (with 5 marbles in each box) is never reached. Instead, the system oscillates around that point.\n",
    "\n",
    "---\n",
    "\n",
    "_About BlockScience_  \n",
    "[BlockScience](http://bit.ly/github_articles_M_02) is a research and engineering firm specialized in complex adaptive systems and applying practical methodologies from engineering design, development and testing to projects in emerging technologies such as blockchain. Follow us on [Medium](http://bit.ly/bsci-medium) or [Twitter](http://bit.ly/bsci-twitter) to stay in touch."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
