Technical Manual of the Ceylan-Myriad Layer

Why Providing Two Different Build/Deploy/Run Systems

We felt that OTP build tools and Emakefiles were not expressive enough for our needs: as mentioned in Building Myriad, a full, rather complete/complex/powerful build system based on GNU make is used by Ceylan-Myriad natively instead, and has been fully satisfactory for years (simple, lightweight, reliable, controllable, flexible, fast, etc.).

It allows to introduce all the generic rules we wanted, to define many conditional settings, to walk through an arbitrarily nested source tree, to integrate within a layered stack (notably alongside some other Ceylan-* libraries that depend on Ceylan-Myriad) and to perform a multi-stage build to accommodate the compilation and use of parse-transforms, with their own set of prerequisites.

More precisely we routinely (see WOOPER or Seaplus) rely on layers built on top of Myriad, which define their own parse transforms that are themselves parse-transformed by Myriad's one - and it works great.

However, to better integrate with other Erlang developments (which are mostly OTP-compliant), we added the (optional) possibility of generating a Myriad OTP library application out of the build tree, ready to be integrated into an (OTP) release and to be available as an Hex package. For that we rely on rebar3, relx and hex.

So currently all our Erlang-based developments can also be built and tested through rebar3, and this support is checked at each commit thanks to continuous integration.

We use less frequently releases (we rely on a basic deployment procedure of our own) and even less hex, yet they were supported once, so we believe that their integration should be at least fairly close to be operational (if not, patches welcome!).

Relying on Rebar3

Despite the kind support of the rebar3 authors and much time spent on its integration, sometimes our build based on it (for Myriad and the projects built on top of it) has encountered issues or has been lagging behind our native one.

Now we believe that all pending issues have been solved (rebar3 is a neat tool), yet being able to switch back to another lighter, ad-hoc, more controlled build system is sometimes a relief - at least a welcome security. Anyway the user can choose between these two (native vs rebar3) build machineries. As for us, we still prefer our native build system, even if it leaves to the developer the task of installing the needed prerequisites by him/herself.

OTP Application

Myriad is not an active OTP application, and as such does not rely on, or provides, services running in the background; so no supervision tree or gen_server is involved here, just a library application ready for OTP integration [7].

$ cd ~/Software && git clone https://github.com/erlang/rebar3.git
    && cd rebar3 && ./bootstrap

$ git pull && ./bootstrap

$ make rebar3-application

$ cd src/utils
$ make myriad_otp_application_run
       Running unitary test myriad_otp_application_run (third form) from
          myriad_otp_application_test

--> Testing module myriad_otp_application_test.

Starting the Myriad application.
Myriad version: {1,0,11}.
Current user name: 'stallone'.
Stopping the Myriad application.
Successful end of test of the Myriad application.
=INFO REPORT==== 18-Jul-2019::22:37:24.779037 ===
   application: myriad
   exited: stopped
   type: temporary

--> Successful end of test.

(test finished, interpreter halted)

$ erl -pz _build/default/lib/myriad/ebin/
Erlang/OTP 22 [erts-10.4] [source] [64-bit] [smp:8:8] [...]

Eshell V10.4  (abort with ^G)
1> application:start(myriad).
ok
2> text_utils:format( "Hello ~s", [ world ] ).
"Hello world"
3> application:stop(myriad).
=INFO REPORT==== 18-Jul-2019::22:47:36.429804 ===
   application: myriad
   exited: stopped
   type: temporary

-include_lib("myriad/include/spawn_utils.hrl").

-include("spawn_utils.hrl").

OTP Release

Quite similarly, to obtain a Ceylan-Myriad OTP release (relx being used in the background), possibly for a given profile like default (development mode) or prod (production mode) - refer to REBAR_PROFILE in GNUmakevars.inc, one just has to run, from the root of Myriad:

$ make rebar3-release

Hex Package

The hex package manager relies on mix, which is commonly installed with Elixir (another language built on top of the Erlang VM).

Thanks to the rebar3 integration with the rebar3_hex plugin specified in Myriad's (generated) rebar.config, hex will be automatically installed and set up.

By following the publishing guidelines ([1], [2]), we were able to publish Hex packages for Myriad that can be freely used. And there was much rejoicing!

One just has to specify for example {deps,[myriad]}. in one's rebar.config, and that's it.

For more details, one may have a look at:

• rebar.config.template, the general rebar configuration file used when generating the Myriad OTP application and release
• rebar-for-hex.config.template, to generate a corresponding Hex package for Myriad (whose structure and conventions is quite different from the previous OTP elements)
• rebar-for-testing.config.template, the simplest test of the previous Hex package: an empty rebar project having for sole dependency that Hex package

Other OTP-related Make Targets of Interest

To populate/update the OTP build tree (by default, from the GIT root, for example _build/default/lib/myriad/ for Myriad) of the current Ceylan layer, one may use:

$ make rebar3-compile

(this is especially useful in order to be able to use directly, from an OTP application, changes just performed in a Ceylan-based layer)

To update both the OTP build tree and the local ebin directory of each Ceylan layer on which the current layer depends, use:

$ make rebar3-local-update

(note this will be a no-op from Myriad, as it does not depend on any Ceylan layer)

To publish an Hex package (once the proper version number has been set in GNUmakevars.inc, see MYRIAD_VERSION):

$ make rebar3-hex-publish

To test such a package:

$ make test-hex-package

To populate directly the OTP local build tree with the Ceylan dependencies located alongside the current install (not useful for Myriad - which depends on none, but useful for upper layers) rather than fetching them through Hex (otherwise may more Hex packages would have to be published for testing during development):

$ make rebar3-local-update

Many more targets are defined in GNUmakerules-explicit.inc.

Defining a token

A token (a compilation-time symbol) may or may not defined.

To define my_token, simply ensure that a -Dmy_token command-line option is specified to the compiler (ex: refer to ERLANG_COMPILER_TOKEN_OPT, in GNUmakevars.inc, for an example of definition for these flags).

To define my_token and set it to the integer value 127, use the -Dmy_token=127 command-line option. Values can also be floats (ex: -Dmy_token=3.14) or atoms (ex: -Dmy_token=some_atom).

A special token is myriad_debug_mode; if it is defined at all (and possibly associated to any value), the debug mode of Myriad is enabled.

We recommend that layers built on top of Myriad define their own token for debug mode (ex: foobar_debug_mode), to be able to finely select appropriate debug modes (of course all kinds of modes and configuration settings can be considered as well).

Defining the code to inject

Based on the defined tokens, code may be injected; this code can be any Erlang expression, and the value to which it will evaluate (at runtime) can be used as any other value in the program.

Injecting a single expression (i.e. not multiple ones) is not a limitation: not only this single expression can be a function call (thus corresponding to arbitrarily many expressions), but more significantly a series of expressions can be nested in a begin / end block, making them a single expression [9].

Using tokens to enable code injection

Various primitives for code injection are available in the cond_utils (mostly pseudo-) module [10].

There is first if_debug/1, to be used as:

cond_utils:if_debug(EXPR_IF_IN_DEBUG_MODE)

Like in:

A = "Joe",
cond_utils:if_debug(io:format("Hello ~s!",[A]))

or, to illustrate expression blocks:

cond_utils:if_debug(begin
                        C=B+1,
                        io:format("Goodbye ~p",[C])
                      end)

These constructs will be replaced by the expression they specify for injection, at their location in the program, iff the myriad_debug_mode token has been defined, otherwise they will be replaced by nothing at all (hence with exactly no runtime penalty; and the result of the evaluation of if_debug/1 is then not an expression).

Similarly, if_defined/2, used as:

cond_utils:if_defined(TOKEN, EXPR_IF_DEFINED)

will inject EXPR_IF_DEFINED if TOKEN has been defined (regardless of any value associated to this token), otherwise the if_defined/2 call will be removed as a whole [11].

As for if_defined/3, it supports two expressions:

cond_utils:if_defined(TOKEN, EXPR_IF_DEFINED, EXPR_OTHERWISE)

For example:

% Older versions being less secure:
TLSSupportedVersions = cond_utils:if_defined(us_web_relaxed_security,
             ['tlsv1.3', 'tlsv1.2', 'tlsv1.1', 'tlsv1'],
             ['tlsv1.3'])

If us_web_relaxed_security has been defined, the first list will be injected, otherwise the second will.

Note that a call to if_defined/3 results thus in an expression.

Finally, with if_set_to/{3,4}, the injection will depend not only of a token being defined or not, but also onto the value (if any) to which it is set.

For if_set_to/3:

cond_utils:if_defined(TOKEN, VALUE, EXPR_IF_SET_TO_THIS_VALUE)

will inject EXPR_IF_SET_TO_THIS_VALUE iff TOKEN has been defined and set to VALUE. As a result, the specified expression will not be injected if some_token has been set to another value, or not been defined at all.

Usage example, -Dsome_token=42 having possibly been defined beforehand:

cond_utils:if_set_to(some_token,42, SomePid ! hello)])

As for if_set_to/4, in:

cond_utils:if_set_to(TOKEN, VALUE, EXPR_IF_SET_TO_THIS_VALUE, EXPR_OTHERWISE)

EXPR_IF_SET_TO_THIS_VALUE will be injected iff TOKEN has been defined and set to VALUE, otherwise (not set or set to a different value) EXPR_OTHERWISE will be.

Example:

Level = cond_utils:if_set_to(my_token, foobar_enabled, 1.0, 0.0) + 4.5

Finally, the switch_set_to/{2,3} primitives allow to generalise these if-like constructs, with one among any number of code branches selected based on the build-time value of a token, possibly with defaults (should the token not be defined at all, or defined to a value that is not among the ones associated to a code branch).

For that we specify a list of pairs, each made of a value and of the corresponding expression to be injected if the actual token matches that value, like in:

cond_utils:switch_set_to(TOKEN, [
       {VALUE_1, EXPR_1},
       {VALUE_2, EXPR_2},
       % [...]
       {VALUE_N, EXPR_N}])

For example:

cond_utils:switch_set_to(my_token, [
     {my_first_value, io:format("Hello!")},
     {my_second_value, begin f(), g(X,debug), h() end},
     {some_third_value, a(X,Y)}])

A compilation-time error will be raised if my_token is not set, or if it is set to none of the declared values (i.e. not in [my_first_value, my_second_value, some_third_value]).

A variation of this primitive exists that applies a default token value if none was, or if the token was set to a value that is not listed among any of the ones designating a code branch, like in:

cond_utils:switch_set_to(TOKEN,
                           [ {VALUE_1, EXPR_1},
                             {VALUE_2, EXPR_2},
                             % [...]
                             {VALUE_N, EXPR_N}],
                           DEFAULT_VALUE)

As always with primitives that define a default, alternate branch, they always inject an expression and thus can be considered as such.

For example:

ModuleFilename = atom_to_list( cond_utils:switch_set_to(some_token,
              [{1, foo}, {14, bar}, {20, hello}], 14) ++ ".erl"

Here, if some_token is not defined, or defined to a value that is neither 1, 14 or 20, then the 14 default value applies, and thus ModuleFilename is set to "bar.erl".

Refer to cond_utils_test.erl for further usage examples.

Controlling assertions

It may be convenient that, depending on a compile-time token (ex: in debug mode, typically triggered thanks to the -Dmyriad_debug_mode compilation flag), assertions (expressions expected to evaluate to the atom true) are enabled, whereas they shall be dismissed as a whole should that token not be defined.

To define an assertion enabled in debug mode, use assert/1, like in:

cond_utils:assert(foo(A,B)=:=10)

Should at runtime the expression specified to assert/1 be evaluated to a value V that is different from the atom true, a {assertion_failed,V} exception will be thrown.

More generally, an assertion may be enabled by any token (not only myriad_debug_mode) being defined, like in:

cond_utils:assert(my_token,bar(C))

Finally, an assertion may be enabled iff a token (here, some_token) has been defined and set to a given value (here, 42), like in:

cond_utils:assert(some_token,42,not baz() andalso A)

This may be useful for example to control, on a per-theme basis, the level of checking performed, like in:

cond_utils:assert(debug_gui,1,basic_testing()),
  cond_utils:assert(debug_gui,2,more_involved_testing()),
  cond_utils:assert(debug_gui,3,paranoid_testing()),

Note that, in this case, a given level of checking should include the one just below it (ex: more_involved_testing() should call basic_testing()).

Usage Hints

For tokens, at least currently they must be defined as immediate values (atoms); even using a mute variable, like for the _Default=my_token expression, or a variable, is not supported (at least yet).

Note that, for primitives that may not inject code at all (ex: if_debug/1), if their conditions are not fulfilled, the specified conditional code is dismissed as a whole, it is not even replaced for example by an ok atom; this may matter if this conditional is the only expression in a case clause for example, in which case a compilation failure like "internal error in core; crash reason: function_clause in function v3_core:cexprs/3 called as v3_core:cexprs[...]" will be reported (the compiler sees unexpectedly a clause not having even a single expression).

A related issue may happen when switching conditional flags: it will select/deselect in-code expressions at compile time, and may lead functions and/or variables to become unused, and thus may trigger at least warnings [12].

For functions that could become unused due to the conditional setting of a token, the compiler could certainly be silenced by exporting them; yet a better approach is surely to use:

-compile({nowarn_unused_function,my_func/3}).

or:

-compile({nowarn_unused_function,[my_func/3, my_other_func/0]}).

As for variables, should A, B or C be reported as unused if some_token was not set, then the basic_utils:ignore_unused/1 function (mostly a no-op) could be of use:

[...]
cond_utils:if_defined(some_token,
                      f(A, B, C),
                      basic_utils:ignore_unused([A, B, C])),
[...]

Alternatively, nowarn_unused_vars could be used instead, at least in some modules.

For more information

Refer for usage and stubs to the cond_utils module (defined in myriad/src/meta), knowing that it is actually implemented thanks to the Myriad parse transform.

For examples and testing, see the cond_utils_test module.

Various Flavours of User Interfaces

Such a user interface may be:

• either text-only, within a console, relying either on the very basic text_ui (for raw text) or its more advanced term_ui counterpart (for terminal-based outputs)
• or graphical, with gui

Text-based user interfaces are quite useful, as they are lightweight, incur few dependencies (if any), and can be used with headless remote servers (text_ui and term_ui work well through SSH, and require no X server nor mouse).

As for graphical-based user interfaces, they are the richest, most usual, and generally the most convenient, user-friendly interfaces.

The user interfaces provided by Myriad are stateful, they rely on a state that can be:

• either explicit, in a functional way; thus having to be carried in all calls
• or implicit, using - for that very specific need only - the process dictionary (even if we try to stay away of it as much as possible)

We tested the two approaches and preferred the latter (implicit) one, which was found considerably more flexible and thus finally fully superseded the (former) explicit one.

We made our best so that a lower-level API interface (relying on a more basic backend) is strictly included in the higher-level ones (ex: term_ui adds concepts - like the one of window or box - to the line-based text_ui), in order that any program using a given user interface may use any of the next, upper ones as well (provided implicit states are used), in the following order: the text_ui API is included in the one of term_ui, which is itself included in the one of gui.

We also defined the settings table, which is a table gathering all the settings specified by the developer, which the current user interface does its best to accommodate.

Thanks to these "Matryoshka" APIs and the settings table, the definition of a more generic ui interface has been possible. It selects automatically, based on available local software dependencies, the most advanced available backend, with the most relevant settings.

For example a relevant backend will be automatically selected by:

$ cd test/user-interface/src
$ make ui_run

On the other hand, if wanting to select a specified backend:

$ make ui_run CMD_LINE_OPT="--use-ui-backend term_ui"

(see the corresponding GNUmakefile for more information)

Raw Text User Interface: text_ui

This is the most basic, line-based monochrome textual interface, directly in raw text with no cursor control.

Located in {src,test}/user-interface/textual, see text_ui.erl for its implementation, and text_ui_test.erl for an example of its use.

Terminal Text User Interface: term_ui

This is a more advanced textual interface than the previous one, with colors, dialog boxes, support of locales, etc., based on dialog (possibly whiptail could be supported as well). Such backend of course must be available on the execution host then.

For example, to secure these prerequisites:

# On Arch Linux:
$ pacman -S dialog

# On Debian-like distros:
$ apt-get install dialog

Located in {src,test}/user-interface/textual, see term_ui.erl for its implementation, and term_ui_test.erl for an example of its use.

Graphical User Interface: gui

This interface relied initially on gs (now deprecated), now on wx (a port of wxWidgets), maybe later in HTML 5 (possibly relying on the Nitrogen web framework for that). For the base dialogs, Zenity could have been on option.

The goal is to provide a small, lightweight API (including message types) that are higher-level than wx, and do not depend on any particular GUI backend (such as wx, gs, etc.) to avoid that user programs become obsolete too quickly, as backends for GUI rise and fall relatively often.

So for example the messages received by the user programs shall not mention wx, and they should take a form compliant with WOOPER message conventions, to easily enable user code to rely on WOOPER if wanted.

Located in {src,test}/user-interface/graphical, see gui.erl, gui_color.erl, gui_text.erl, gui_canvas.erl, etc., with a few tests (gui_test.erl, lorenz_test.erl).

Related information of interest:

• wxErlang: Getting started and Speeding up, by Arif Ishaq
• http://wxerlang.dougedmunds.com/

Erlang-Dedicated Scripts

More General Scripts

Script-based Apps

These shell scripts are actually user-facing shell interfaces that plug directly on some more involved Erlang programs, i.e. applications that are available here.

generate-password.sh

Generates a proper random password respecting various rules, whose simple application can be transparently checked (probably at least more easily audited than most password managers - thus maybe more trustable).

Usage: generate-password.escript [-a ALPHABET|--alphabet ALPHABET]

[-l MIN_LEN MAX_LEN|--length MIN_LEN MAX_LEN] [-h|--help]

Generates a suitable password, where:

- ALPHABET designates the set of characters to draw from (default one being 'extended'), among:

* 'base': alphanumeric letters, all cases [A-Za-z0-9]

* 'extended': 'base' + basic punctuation (i.e. '[](){}:,;-_.!?')

* 'full': 'base' + all punctuation (i.e. basic + '"'@ /&$*^%=+|')

- MIN_LEN and MAX_LEN are the respective minimum and maximum numbers of characters

(bounds included) used to generate this password [default: between 15 and 20]

merge.sh

Helps merging efficiently and reliably file trees; it is actually a rather involved text-based application that allows scanning/comparing/merging trees, typically in order to deduplicate file hierarchies that were cloned once yet may have since then diverged.

Usage: following operations can be triggered:

- 'merge.sh --input INPUT_TREE --reference REFERENCE_TREE'

- 'merge.sh --scan A_TREE'

- 'merge.sh --rescan A_TREE'

- 'merge.sh --resync A_TREE'

- 'merge.sh --uniquify A_TREE'

- 'merge.sh -h' or 'merge.sh --help'

Ensures, for the first form, that all the changes in a possibly more up-to-date, "newer" tree (INPUT_TREE) are merged back

to the reference tree (REFERENCE_TREE), from which the first tree may have derived. Once executed, only a refreshed,

complemented reference tree will exist, as the input tree will have been removed: all its original content (i.e. its content

that was not already in the reference tree) will have been transferred in the reference tree.

In the reference tree, in-tree duplicated content will be either kept as it is, or removed as a whole (to keep only one

copy thereof), or replaced by symbolic links in order to keep only a single reference version of each actual content.

At the root of the reference tree, a '.merge-tree.cache' file will be stored, in order to avoid any later recomputations of

the checksums of the files that it contains, should they have not changed. As a result, once a merge is done, the reference

tree may contain an uniquified version of the union of the two specified trees, and the input tree will not exist anymore

after the merge.

For the second form (--scan option), the specified tree will simply be inspected for duplicates, and a corresponding

'.merge-tree.cache' file will be created at its root (to be potentially reused by a later operation).

For the third form (--rescan option), an attempt to rebuild an updated '.merge-tree.cache' file will be performed,

computing only the checksum of the files that were not already referenced, or whose timestamp or size changed.

For the fourth form (--resync option), a rebuild even lighter than the previous rescan of '.merge-tree.cache' will be done,

checking only sizes (not timestamps), and updating these timestamps.

For the fifth form (--uniquify option), the specified tree will be scanned first (see the corresponding operation), and

For the fifth form (--uniquify option), the specified tree will be scanned first (see the corresponding operation), and

then the user will be offered various actions regarding found duplicates (being kept as are, or removed, or replaced with

symbolic links), and once done a corresponding up-to-date '.merge-tree.cache' file will be created at its root (to be

potentially reused by a later operation).

For the sixth form (-h or --help option), displays this help.

Note that the --base-dir A_BASE_DIR option can be specified by the user to designate the base directory of all relative

paths mentioned. When a cache file is found, it can be either ignored (and thus recreated) or re-used, either as it is or

after a weak check, where only file existence, sizes and timestamps are then verified (not checksums).

Motivation

A value of a given type (ex: a float) can actually correspond to quantities as different as meters and kilowatts per hour.

Therefore units shall preferably be specified alongside with values being processed, and a language to express, check and convert these units must be retained. Of course units are of interest as other metadata are - such as accuracy, semantics, etc.

Available Support

The Myriad layer provides such a service, in a very basic, ad hoc form (which is useful to introduce "special" non-physical, non-standard units, such as euro/year), meant to be enriched over time.

Specifying Units

1> MyInputValue="-24 mS.m^-1".
2> {Value,Unit}=unit_utils:parse_value_with_unit(MyInputValue).
3> io:format("Corresponding value: ~f.~n", [ Value ] ).
Corresponding value: -24.0.
4> io:format("Corresponding unit: ~s.~n",
   [unit_utils:unit_to_string(Unit)]).
"s^3.A^2.g^-1.m^-3, of order -6"
5> unit_utils:value_with_unit_to_string(Value,Unit).
"-2.4e-5 s^3.A^2.g^-1.m^-3"

Possible Improvements Regarding Dimensional Analysis

Some programming languages provide systems to manage dimensional information (ex: for physical quantities), generally through add-ons or libraries (rarely as a built-in feature).

A first level of support is to provide, like here, an API to manage units. Other levels can be:

1. to integrate unit management directly, seamlessly in language expressions, as if it was built-in (as opposed to having to use explicitly a third-party API for that); for example at least half a dozen different libraries provide that in Python
2. to be able to define "polymorphic units and functions", for example to declare in general that a speed is a distance divided by a duration, regardless of the possible units used for that
3. to perfom static dimensional analysis, instead of checking units at runtime

The two latter use cases can for example be at least partially covered by Haskell libraries.

About SQL

Some amount of SQL (Structured Query Language) support for relational database operations is provided by the Myriad layer.

As this support relies on an optional prerequisite, this service is disabled by default.

Database Back-end

To perform SQL operations, a corresponding software solution must be available.

The SQL back-end chosen here is the SQLite 3 library. It provides a self-contained, serverless, zero-configuration, transactional SQL database. It is an embedded SQL database engine, as opposed to server-based ones, like PostgreSQL or MariaDB.

It can be installed on Debian thanks to the sqlite3 and sqlite3-dev packages, sqlite on Arch Linux..

We require version 3.6.1 or higher (preferably: latest stable one). It can be checked thanks to sqlite3 --version.

Various related tools are very convenient in order to interact with a SQLite database, including sqlitebrowser and sqliteman.

On Arch Linux, one can thus use: pacman -Sy sqlite sqlitebrowser sqliteman.

Testing the back-end:

$ sqlite3 my_test

SQLite version 3.13.0 2016-05-18 10:57:30
Enter ".help" for usage hints.
sqlite> create table tblone(one varchar(10), two smallint);
sqlite> insert into tblone values('helloworld',20);
sqlite> insert into tblone values('my_myriad', 30);
sqlite> select * from tblone;
helloworld|20
my_myriad|30
sqlite> .quit

A file my_test, identified as SQLite 3.x database, must have been created, and can be safely removed.

Erlang SQL Binding

This database system is directly accessed thanks to an Erlang binding.

Two of them have been identified as good candidates:

• erlang-sqlite3: seems popular, with many contributors and users, actively maintained, based on a gen_server interacting with a C-node, involving only a few source files
• esqlite: based on a NIF, so more able to jeopardize the stability of the VM, yet potentially more efficient

Both are free software.

We finally preferred erlang-sqlite3.

By default we consider that this back-end has been installed in ~/Software/erlang-sqlite3. The SQLITE3_BASE variable in myriad/GNUmakevars.inc can be set to match any other install path.

Recommended installation process:

$ mkdir ~/Software
$ cd ~/Software
$ git clone https://github.com/alexeyr/erlang-sqlite3.git
Cloning into 'erlang-sqlite3'...
remote: Counting objects: 1786, done.
remote: Total 1786 (delta 0), reused 0 (delta 0), pack-reused 1786
Receiving objects: 100% (1786/1786), 3.24 MiB | 570.00 KiB/s, done.
Resolving deltas: 100% (865/865), done.
Checking connectivity... done.
$ cd erlang-sqlite3/
$ make
rm -rf deps ebin priv/*.so doc/* .eunit/* c_src/*.o config.tmp
rm -f config.tmp
echo "normal" > config.tmp
./rebar get-deps compile
==> erlang-sqlite3 (get-deps)
==> erlang-sqlite3 (compile)
Compiled src/sqlite3_lib.erl
Compiled src/sqlite3.erl
Compiling c_src/sqlite3_drv.c
[...]

Testing the binding:

make test
./rebar get-deps compile eunit
==> erlang-sqlite3 (get-deps)
==> erlang-sqlite3 (compile)
==> erlang-sqlite3 (eunit)
Compiled src/sqlite3.erl
Compiled src/sqlite3_lib.erl
Compiled test/sqlite3_test.erl
======================== EUnit ========================
module 'sqlite3_test'
  sqlite3_test: all_test_ (basic_functionality)...[0.002 s] ok
  sqlite3_test: all_test_ (table_info)...ok
  [...]
  sqlite3_lib: delete_sql_test...ok
  sqlite3_lib: drop_table_sql_test...ok
  [done in 0.024 s]
  module 'sqlite3'
=======================================================
All 30 tests passed.
Cover analysis: ~/Software/erlang-sqlite3/.eunit/index.html

Pretty reassuring.

SQL Support Provided By the Myriad Layer

To enable this support, once the corresponding back-end (see Database Back-end) and binding (see Erlang SQL Binding) have been installed, the USE_SQLITE variable should be set to true in myriad/GNUmakevars.inc and Myriad shall be rebuilt.

Then the corresponding implementation (sql_support.erl) and test (sql_support_test.erl), both in myriad/src/data-management, will be built (use make clean all from the root of Myriad) and able to be run (execute make sql_support_run for that).

Testing it:

$ cd myriad/src/data-management
$ make sql_support_run
       Compiling module sql_support.erl
       Compiling module sql_support_test.erl
       Running unitary test sql_support_run
 [...]
 --> Testing module sql_support_test.
 Starting SQL support (based on SQLite3).
 [...]
 Closing database.
 Stopping SQL support.
 --> Successful end of test.
 (test finished, interpreter halted)

Looks good.

SQL-related Troubleshooting

Text Conventions

The purpose here is to ensure a sufficient code homogeneity; for example in all source files are in such a "canonical form", analysing their differences (diff) is made simpler.

Any text editor can be used, provided that it saves source files with the UNIX, not DOS, conventions (i.e. lines terminating by the LF character, not by the CRLF characters).

The use of syntax highlighting is encouraged.

Recommended text editors are:

• emacs / xemacs
• nedit
• ErlIDE (based on Eclipse)
• gedit

Source files should be formatted for a 80-character width: no character should be present after the 79th column of a line.

Except in very specific cases, only ASCII code should be used (ex: no accentuated characters).

Tabulations should be preferred to series of spaces, and the text should be formatted according to 4-character tabulations.

All redundant whitespaces should be removed, preferably automatically (see the Emacs whitespace-cleanup command). This is why, with the emacs settings that we recommend, pressing the F8 key removes for example the yellow areas in the current buffer by replacing any series of four spaces by a corresponding tabulation.

We would prefer that contributed files (especially source ones) are "whitespace-clean" before being committed. As mentioned, such a transformation can be done directly from Emacs. If using another editor, please ensure that the fix-whitespaces.sh script has been run on the target sources (possibly automatically thanks to a VCS hook) before committing them.

All elements of documentation should be written in English, possibly translated to other languages. Spell-checking is recommended.

Coding Practices

In terms of coding style, we would like that the sources remain as uniform as possible, regarding naming, spacing, code/comments/blank line ratios.

We would like that, roughly and on average, the same ratio applies for blank lines, comments and lines of code.

For that one may use the either directly make stats from the root of the layer or the make-code-stats.sh script.

For example:

In the Erlang source code found from XXX/ceylan/myriad, we have:
+ 160 source files (*.erl), 54 header files (*.hrl)
+ a grand total of 89151 lines:
  - 27186 of which (30.4%) are blank lines
  - 28751 of which (32.2%) are comments
  - 33214 of which (37.2%) are code

As not all typos may be detected at compilation-time (ex: wrong spelling for a module), we recommend, for source code, the use of additional static checkers, as discussed in the type-checking section.

Execution Targets

Two execution target modes have been defined:

• development (the default): meant to simplify the task of developers and maintainers by reporting as much information and context as possible, even at the expense of some performances and reliability (ex: no retry in case of failure, shorter time-outs not to wait too long in case of errors, more checks, etc.)
• production: mostly the reciprocal of the development mode, whose purpose is to favor efficient, bullet-proof operations

These execution targets are compile-time modes, i.e. they are set once for all when building the layer at hand (probably based, if using OTP, on the rebar corresponding modes - respectively dev and prod).

See EXECUTION_TARGET in GNUmakevars.inc to read and/or set them.

The current execution target is of course available at runtime on a per-layer level, see basic_utils:get_execution_target/0 for more information.

This function shall be compiled once per layer to be accurate, in one of its modules. It is just a matter of adding the following include in such module:

-include_lib("myriad/utils/basic_utils.hrl").

Other Conventions

• for clarity, we tend to use longer variable names, in CamelCase
• we tend to use mute variables to clarify meanings and intents, as in _Acc=[] (beware, despite being muted, any variable in scope that bears the same name will be matched)
• as there is much list-based recursion, a variable named H means Head and T means Tail (as in [Head|Tail])

Header dependencies

Only a very basic dependency between header files (*.hrl) and implementation files (*.erl) is managed.

As expected, if X.hrl changed, X.beam will be recompiled whether or not X.erl changed. However, any Y.erl that would include X.hrl would not be automatically recompiled.

Typically, when in doubt after having modified a record in a header file, just run make rebuild from the root of that layer (build is fast anyway, as quite parallel).

About the table module

This is a pseudo module, which is not meant to exist as such (no table.erl, no table.beam).

The Myriad parse transform replaces references to the table module by references to the map_hashtable module. See table transformations for more information.