= ReCo.jl
:source-highlighter: highlight.js
:highlightjs-languages: bash, julia, latex

image:https://img.shields.io/badge/code%20style-blue-4495d1.svg[Code Style: Blue, link=https://github.com/invenia/BlueStyle]

**Re**inforcement learning of **co**llective behavior.

== Setup

The following steps from the setup have to be followed everytime before running anything in the following sections. An exception is installing the dependencies which has to be done only the first time.

=== Launch Julia

To activate the environment, navigate to the main directory `/ReCo.jl` and then run the following to launch Julia:

[source,bash]
----
cd ReCo.jl
julia --threads auto
----

`auto` automatically sets the number of threads to use. If you want to use a specific number `N` of threads, replace `auto` with `N`.

=== Activating environment

After launching Julia, the package environment has to be activated by running the follwing in the REPL:

[source,julia]
----
using Pkg
Pkg.activate(".")
----

=== Install/update dependencies

After activating the package environment, run the following to install/update the package dependencies:

[source,julia]
----
Pkg.update()
----

This step has to be done by running the line above for the first time using the package to install its dependencies. Running the line above after the first time will update the installed dependencies which is optional.

=== Import the package

You can import the package by running:

[source,julia]
----
using ReCo
----

This will export the package's methods that are intended to be used by the end user.

== Help mode

To access the documentation of the presented package methods further in this README, run `using ReCo` first. Then, enter the help mode by pressing `?` in the REPL. Now, enter the method's name followed by enter to see its documentation.

== Run a simulation

Initialize a simulation with 100 particles having a self-propulsion velocity of 40.0 and return the relative path to the simulation directory:

[source,julia]
----
sim_dir = init_sim(100, 40.0)
----

Run the simulation:

[source,julia]
----
run_sim(sim_dir, duration=20.0)
----

The values for the number of particles, self-propulsion velocity and simulation duration are used here as an example. For more information about possible values and other optional arguments, see the documentation of `init_sim` or `run_sim`.

== Simulation visualization

=== Animation

To generate an animation of a simulation, run the following:

[source,julia]
----
animate(sim_dir)
----

The method's documentation includes all possible optional arguments and where the output can be found.

=== Snapshot plot

To plot only one snapshot of a simulation, run the following:

[source,julia]
----
plot_snapshot(sim_dir)
----

This will ask for the number of the snapshot to plot out of the total number of snapshots. The method's documentation includes all possible optional arguments and where the output can be found.

== Run a reinforcement learning process

Run a reinforcement learning process and return the environment helper and the the path of the process directory relative to the directory `ReCo.jl`:
[source,julia]
----
env_helper, rl_dir = run_rl(ENVTYPE)
----

`ENVTYPE` has to be replaced by one of the environments named after the file names in the directory `ReCo.jl/RL/Envs`, for example: `LocalCOMEnv`. A description of an environment is included at the beginning of the corresponding file.
//TODO: Descriptions of envs

The documentation of `run_rl` includes all possible optional arguments.

=== Q-matrix

`env_helper` has the abstract type `EnvHelper`. To access the Q-matrix, enter the following:

[source,julia]
----
env_helper.shared.agent.policy.learner.approximator.table
----

To generate a LaTeX table with the states and actions combintation names for the Q-matrix, run the follwing:

[source,julia]
----
include("src/RL/latex_table.jl")
latex_rl_table(env_helper, FILENAME_WITHOUT_EXTENSION)
----

`FILENAME_WITHOUT_EXTENSION` has to be replaced by the wanted file name without extension of the `.tex` file. The method's documentation explains where the output is placed.

The output file can be used in a LaTeX document:

[source,latex]
----
\input{FILENAME_WITHOUT_EXTENSION}
----

=== Rewards

To access the rewards, run the following:

[source,julia]
----
env_helper.shared.hook.rewards
----

To plot the rewards, run the following:

[source,julia]
----
plot_rewards(rl_dir)
----

The method's documentation explains where the output is placed.

=== Mean kappa

To plot the mean of kappa as the ratio of the eigenvalues of the gyration tensor, run the following:

[source,julia]
----
include("analysis/mean_kappa.jl")
plot_mean_κ(; rl_dir=rl_dir, n_last_episodes=N_LAST_EPISODES)
----

`N_LAST_EPISODES` is the number of the last episodes of the learning process to average over. The method's documentation explains where the output is placed.

== Run analysis

After running the following command blocks in the REPL, the output can be found in the directory `ReCo.jl/exports/graphics`.

=== Mean squared displacement and random walk

[source,julia]
----
include("analysis/mean_squared_displacement.jl")
run_msd_analysis()
run_random_walk()
----

=== Radial distribution function

[source,julia]
----
include("analysis/radial_distribution_function/radial_distribution_function.jl")
run_radial_distribution_analysis()
----

=== Reward discount analysis

//TODO

== Graphics

//TODO