Advanced usage =================== .. warning:: This section is under review, and the contents may not be up-to-date. This section covers more advanced modelx concepts and techniques that are not covered by the earlier sections. Advanced Concepts ----------------- In this section, more concepts we haven't yet covered are introduced. Some of them are demonstrated by examples in the following section. Space Members ^^^^^^^^^^^^^ Spaces can contain cells and other spaces. In fact, spaces have 3 kinds of their "members". You can get those members as if they are attributes of the containing spaces, by attribute access(``.``) expression. Cells As we have seen in the previous example, spaces contain cells, and the cells belong to spaces. One cells must belong to one ane only one space. The :py:meth:`cells ` property of Space returns a dictionary of all the cells associated with their names. (Sub)spaces As previously mentioned, spaces can be created in another space. Spaces in another space are called subspaces of the containing space. There 2 kinds of subspaces, static subspaces and dynamic subspace. Static subspaces are those that are created manually, just like those created in models. There is no difference between spaces created directly under a model and static spaces created under a space, except for their parents being different types. You can create a static subspace by calling ``new_space`` method of their parents:: model, space = new_model(), new_space() space.new_space('Subspace1') @defcells def foo(): return 123 You can get a subspace as an attribute of the parent space, or by accessing the parent space's :py:attr:`~modelx.core.space.UserSpace.spaces` property:: >>> space.spaces['Subspace1'].foo() 123 >>> space.Subspace1.foo() 123 The other kind of subspaces, is dynamic subspaces. Unlike static suspaces, dynamic subspaces can only be created in spaces, but not directly in models. Dynamic spaces are parametrized spaces that are created on-the-fly when requested through call(``()``) or subscript(``[]``) operation on their parent spaces. We'll explore more on dynamic spaces in the next section, in conjunction with space inheritance by going through an example. References Often times you want access from cells formulas in a space to other objects than cells or subspaces in the same space. References are names bound to arbitrary objects that are accessible from within the same space:: model, space = new_model(), new_space() @defcells def bar(): return 2 * n ``bar`` cells above refers to ``n``, which has not yet been defined. Without ``n`` being defined, calling ``bar`` will raise an error. To define a reference ``n``, you can simply assign a value to ``n`` attribute of ``space``:: >>> space.n = 3 >>> bar() 6 The :py:meth:`refs ` property of space returns a mapping of reference names to their objects:: >>> list(space.refs.keys()) ['__builtins__', 'n', '_self'] Be default, ``__builtins__`` and ``_self`` are defined in any space. In fact, ``__builtins__`` is defined by default as a "global" reference in the model. Global references are names accessible from any space in a model. Other than the default reference, you can define your own, by simply assigning a value as an attribute of the model:: >>> model.z = 4 >>> list(model.refs.keys()) ['z', '__builtins__'] >>> list(space.refs.keys()) ['z', '__builtins__', 'n', '_self'] ``__builtins__`` points to Python builtin module. It is defined to allow cells formulas to use builtin functions. ``_self`` points to the space itself. This allows cells formulas to get access to its parent space. As mentioned earlier, formulas of cells are evaluated in the namespace that is associated with their parent spaces. The namespace of a space is a mapping of names to the space members. As explained in the previous section, space members are either cells of the space, or subspaces of the space or references accessible from the space. The table below breaks down all the members in the namespace by its types and sub-types. +-----------+--------------------+---------------------------------------------------+ |cells |self cells |Cells defined in or overridden in the space | | +--------------------+---------------------------------------------------+ | |derived cells |Cells inherited from one of the base spaces | +-----------+--------------------+---------------------------------------------------+ |spaces |self spaces |Subspace defined in or overridden in the space | | +--------------------+---------------------------------------------------+ | |derived spaces |Subspace inherited from one of the base spaces | +-----------+--------------------+---------------------------------------------------+ |references |self references |References defined in or overridden in the space | |(refs) +--------------------+---------------------------------------------------+ | |derived references |References inherited from one of the base spaces | | +--------------------+---------------------------------------------------+ | |global references |Global references defined in the parent model | | +--------------------+---------------------------------------------------+ | |local references |Only ``_self`` that refers to the space itself | | +--------------------+---------------------------------------------------+ | |parameters |(Only in dynamic spaces) Space parameters | +-----------+--------------------+---------------------------------------------------+ Each type of the members has "self" members and "derived" members. Those distinctions stem from space inheritance explained in the next section. Space Inheritance ^^^^^^^^^^^^^^^^^ Inheritance in modelx is a feature analogous to class inheritance in object-oriented programming languages, such as Python. By making full use of space inheritance, you can organize multiple spaces sharing similar features into an inheritance tree of spaces, minimizing duplicated formula definitions, keeping your model organized and transparent while maintaining model integrity. Inheritance lets one space use(inherit) other spaces, as base spaces. The inheriting space is called a derived space of the base spaces. The cells in the base spaces are copied automatically in the derived space. In the derived space, formulas of cells from base spaces can be overridden. You can also add cells to the derived space, that do not exist in any of the base spaces. A space can have multiple base spaces. This is called multiple inheritance. The base spaces can have their base spaces, and derived-base relationships between spaces make up a directional graph of dependency. In case of multiple inheritance, we need a way to order base spaces to determine the priority of base spaces. modelx uses the same algorithm as Python for ordering bases, which is, C3 superclass linearization algorithm (a.k.a C3 Method Resolution Order or MRO). The links below are provided in case you want to know more about C3 MRO. * https://en.wikipedia.org/wiki/C3_linearization * https://www.python.org/download/releases/2.3/mro/ More complex example -------------------- Let's see how inheritance works by a simple code of pricing life insurance policies. First, we are goint to create a very simple life model as a space and name it ``Life``. Then we'll populate the space with cells that calculate the number of death and remaining lives by age. Then to price a term life policy, we will derive a ``TermLife`` space from the ``Life`` space, and add some cells to calculate death benefits paid to the insured, and their present value. Next, we want to model an endowment policy. Since the endowment policy pays out a maturity benefit in addition to the death benefits covered by the term life policy, we derive a ``Endowment`` space from ``TermLife``, and make a residual change to the ``benefits`` formula. Creating the Life space ^^^^^^^^^^^^^^^^^^^^^^^ Below is a mathematical representation of the life model we'll build as a ``Life`` space. .. math:: &l(x) = l(x - 1) - d(x - 1)\\ &d(x) = l(x) * q where, :math:`l(x)` denotes the number of lives at age x, :math:`d(x)` denotes the number of death occurring between the age x and age x + 1, :math:`q` denotes the annual mortality rate (for simplicity, we'll assume a constant mortality rate of 0.003 for all ages for the moment.) One letter names like l, d, q would be too short for real world practices, but we use them here just for simplicity, as they often appear in classic actuarial textbooks. Yet another simplification is, we set the starting age of x at 50, just to get output shorter. As long as we use a constant mortality age, it shouldn't affect the results whether the starting age is 0 or 50. Below the modelx code for this life model: .. literalinclude:: /samples/sample_inheritance.py :lines: 6-21 The second to last line of the code above has the same effect as putting ``@defcells`` decorator on top of each of the 3 function definitions. This line creates 3 new cells from the 3 functions in the ``Life`` space, and rebind names ``l``, ``d``, ``q`` to the 3 cells in the current scope. You must have noticed that ``l(x)`` formula is referring to the name ``x0``, which is not defined yet. The last line is for defining ``x0`` as the issue age in the ``Life`` model and assigning a value to it. To examine the space, we can check values of the cells in ``Life`` as below:: >>> l(60) 97040.17769489168 >>> life.frame l d q x 50.0 100000.000000 300.000000 NaN 51.0 99700.000000 299.100000 NaN 52.0 99400.900000 298.202700 NaN 53.0 99102.697300 297.308092 NaN 54.0 98805.389208 296.416168 NaN 55.0 98508.973040 295.526919 NaN 56.0 98213.446121 294.640338 NaN 57.0 97918.805783 293.756417 NaN 58.0 97625.049366 292.875148 NaN 59.0 97332.174218 291.996523 NaN 60.0 97040.177695 NaN NaN NaN NaN NaN 0.003 Deriving the Term Life space ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Next, we'll see how we can extend this space to represent a term life policy. To simplify things, here we focus on one policy with the sum assured of 1 (in whatever unit of currency). With this assumption, if we define ``benefits(x)`` as the expected value at issue of benefits paid between the age x and x + 1, then it should equate to the probability of death between age x and x + 1, of the insured at the point of issue. In a math expression, this should be written: .. math:: benefits(x) = d(x) / l(x0) where :math:`l(x)` and :math:`d(x)` are the same definition from the preceding example, and :math:`x0` denotes the issue age of the policy. And further we define the present value of benefits at age x as: .. math:: pv\_benefits(x) = \sum_{x'=x}^{x0+n}benefits(x')/(1+disc\_rate)^{x'-x} ``n`` denotes the policy term in years, and ``disc_rate`` denotes the discounting rate for the present value calculation. Continued from the previous code, we are going to derive the ``TermLife`` space from the ``Life`` space, to add the benefits and present value calculations. .. literalinclude:: /samples/sample_inheritance.py :lines: 25-41 The first line in the sample above creates ``TermLife`` space derived from the ``Life`` space, by passing the ``Life`` space as ``bases`` parameter to the ``new_space`` method of the model. The ``TermLife`` space at this point has the same cells as its sole base space ``Life`` space. The following 2 cells definitions (2 function definitions with ``defcells`` decorators), are for adding the cells that did not exist in ``Life`` space. The formulas are referring to the names that are not defined yet. Those are ``n``, ``disc_rate``. We need to define those in the ``TermLife`` space. The reference ``x0`` is inherited from the ``Life`` space. .. literalinclude:: /samples/sample_inheritance.py :lines: 46-47 You get the following results by examining the ``TermLife`` space (The order of the columns in the DataFrame may be different on your screen).:: >>> term_life.pv_benefits(50) 0.02959822305108317 >>> term_life.frame d q l pv_benefits benefits x 50.0 300.000000 NaN 100000.000000 0.029598 0.003000 51.0 299.100000 NaN 99700.000000 0.026598 0.002991 52.0 298.202700 NaN 99400.900000 0.023607 0.002982 53.0 297.308092 NaN 99102.697300 0.020625 0.002973 54.0 296.416168 NaN 98805.389208 0.017652 0.002964 55.0 295.526919 NaN 98508.973040 0.014688 0.002955 56.0 294.640338 NaN 98213.446121 0.011733 0.002946 57.0 293.756417 NaN 97918.805783 0.008786 0.002938 58.0 292.875148 NaN 97625.049366 0.005849 0.002929 59.0 291.996523 NaN 97332.174218 0.002920 0.002920 60.0 NaN NaN NaN 0.000000 0.000000 61.0 NaN NaN NaN 0.000000 NaN NaN NaN 0.003 NaN NaN NaN You can see that the values of ``l``, ``d``, ``q`` cells are the same as those in ``Life`` space, as ``Life`` and ``LifeTerm`` have exactly the same formulas for those cells, but be aware that those cells are not shared between the base and derived spaces. Unlike class inheritance in OOP languages, space inheritance is in terms of space instances(or objects), not classes, so cells are copied from the base spaces to derived space upon creating the derived space. Deriving the Endowment space ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ We're going to create another space to test overriding inherited cells. We will derive ``Endowment`` space from ``LifeTerm`` space. The diagram below shows the relationships of the 3 spaces considered here. A space from which an arrow originates is derived from the space the arrow points to. .. figure:: /images/Inheritance1.png :align: center Life, TermLife and Endowment The endowment policy pays out the maturity benefit of 1 at the end of its policy term. We have defined ``benefits`` cells as the expected value of benefits, so in addition to the death benefits considered in ``LifeTerm`` space, we'll add the maturity benefit by overriding the ``benefits`` definition in ``Endowment`` space. In reality, the insured will not get both death and maturity benefits, but here we are considering an probabilistic model, so the benefits would be the sum of expected value of death and maturity benefits: .. literalinclude:: /samples/sample_inheritance.py :lines: 53-62 And the same operations on the ``Endowment`` space produces the following results:: >>> endowment.pv_benefits(50) 1.0 >>> endowment.frame pv_benefits benefits l q d x 50.0 1.000000 0.003000 100000.000000 NaN 300.000000 51.0 0.997000 0.002991 99700.000000 NaN 299.100000 52.0 0.994009 0.002982 99400.900000 NaN 298.202700 53.0 0.991027 0.002973 99102.697300 NaN 297.308092 54.0 0.988054 0.002964 98805.389208 NaN 296.416168 55.0 0.985090 0.002955 98508.973040 NaN 295.526919 56.0 0.982134 0.002946 98213.446121 NaN 294.640338 57.0 0.979188 0.002938 97918.805783 NaN 293.756417 58.0 0.976250 0.002929 97625.049366 NaN 292.875148 59.0 0.973322 0.002920 97332.174218 NaN 291.996523 60.0 0.970402 0.970402 97040.177695 NaN NaN 61.0 0.000000 NaN NaN NaN NaN NaN NaN NaN NaN 0.003 NaN You can see ``pv_benefits`` for all ages and ``benefits`` for age 60 show values different from ``TermLife`` as we overrode ``benefits``. ``pv_benefits(50)`` being 1 is not surprising. The ``disc_rate`` set to 1 in ``TermLife`` space is also inherited to the ``Endowment`` space. The discounting rate of benefits being 1 means by taking the present value of the benefits, we are simply taking the sum of all expected values of future benefits, which must equates to 1, because the insured gets 1 by 100% chance. Dynamic spaces ^^^^^^^^^^^^^^ In many situations, you want to apply a set of calculations in a space, or a tree of spaces, to different data sets. You can achieve that by applying the space inheritance on dynamic spaces. Dynamic spaces are parametrized spaces that are created on-the-fly when requested through call(``()``) or subscript(``[]``) operation on their parent spaces. To define dynamic spaces in a parent space, you create the space with a parameter function whose signature is used to define space parameters. The parameter function should return, if any, a mapping of parameter names to their arguments, to be pass on to the ``new_space`` method, when the dynamic spaces are created. To see how this works, let's continue with the previous example. In the last example, we manually set the issue age ``x0`` of the policy to 50, and the policy term ``n`` to 10. We'll extend this example and create policies as dynamic spaces with with different policy attributes. Assume we have 3 term life polices with the following attributes: ========= ========= =========== Policy ID Issue Age Policy Term ========= ========= =========== 1 50 10 2 60 15 3 70 5 ========= ========= =========== We'll create this sample data as a nested list:: data = [[1, 50, 10], [2, 60, 15], [3, 70, 5]] The diagram shows the design of the model we are going to create. The lines with a filled diamond shape on one end indicates that ``Policy`` model is the parent space of the 3 dynamic spaces, ``Policy1``, ``Policy2``, ``Policy3``, each of which represents each of the 3 policies above. While ``Policy`` is the parent space of the 3 dynamic space, it is also the base space of them. ``Policy`` space inherits its members from ``Term`` model, and in turn ``Policy`` is inherited by the 3 dynamic spaces. This inheritance is represented by the unfilled arrowhead next the filled diamond. .. figure:: /images/Inheritance2.png Below is a script to extend the model as we designed above. .. literalinclude:: /samples/sample_inheritance.py :lines: 72-86 The ``params`` function is passed to the constructor of the ``Policy`` space as the argument of ``formula`` parameter. The signature of ``params`` func is used to determine the parameter of the dynamic spaces, and the returned dictionary is passed to the ``new_space`` as arguments when the dynamic spaces are created. ``params`` is called when you create the dynamic subspaces of ``Policy``, by calling the n-the element of ``Policy``. ``params`` is evaluated in the ``Policy``'s namespace. ``_self`` is a spacial reference that points to ``Policy``. The parameter ``policy_id`` becomes available within the namespace of each dynamic space. In each of the dynamic spaces, the values of ``x0`` and ``n`` are taken from ``data`` for each policy:: >>> policy(1).pv_benefits(50) 0.02959822305108317 >>> policy(2).pv_benefits(60) 0.04406717516109439 >>> policy(3).pv_benefits(70) 0.014910269595243001 >>> policy(3).frame n x0 d benefits l pv_benefits q x NaN 5.0 70.0 NaN NaN NaN NaN 0.003 70.0 NaN NaN 300.000000 0.003000 100000.000000 0.014910 NaN 71.0 NaN NaN 299.100000 0.002991 99700.000000 0.011910 NaN 72.0 NaN NaN 298.202700 0.002982 99400.900000 0.008919 NaN 73.0 NaN NaN 297.308092 0.002973 99102.697300 0.005937 NaN 74.0 NaN NaN 296.416168 0.002964 98805.389208 0.002964 NaN 75.0 NaN NaN NaN 0.000000 NaN 0.000000 NaN 76.0 NaN NaN NaN NaN NaN 0.000000 NaN >>> policy.spaces {'Policy1': , 'Policy2': , 'Policy3': } Dynamic spaces of a base space are not passed on to the derived spaces. When a space is derived from a base space that has dynamically created subspaces, those dynamically created subspaces themselves are not passed on to the derived spaces. Instead, the parameter function of the base space is inherited, so subspaces of the derived space are created upon call(using ``()``) or subscript (using ``[]``) operators the derived space with arguments. Reading Excel files ------------------- You can read data stored in an Excel file into newly created cells. Space has two methods ``new_cells_from_excel`` and ``new_space_from_excel``. ``new_space_from_excel`` is also available on Model. You need to have Openpyxl package available in your Python environment to use these methods. ``new_cells_from_excel`` method reads values from a range in an Excel file, creates cells and populates them with the values in the range. ``new_space_from_excel`` methods reads values from a range in an Excel file, creates a space, and in that space, creates dynamic spaces using one or some of the index rows and/or columns as space parameters, and creates cells in the dynamics spaces populating them with the values in the range. Refer to the modelx reference for concrete description of those methods. Exporting to Pandas objects --------------------------- If you have Pandas installed in your Python environment, you can export values of cells to Pandas' DataFrame or Series objects. Spaces have ``frame`` property, which generates a DataFrame object whose columns are cells names, and whose indexes are cells parameters. Multiple cells in a space may have different sets of parameters. Generated