= ReCo.jl
:source-highlighter: highlight.js
:highlightjs-languages: bash, julia, latex

image:https://img.shields.io/badge/code%20style-blue-4495d1.svg[Code Style: Blue, link=https://github.com/invenia/BlueStyle]

**Re**inforcement learning of **co**llective behavior.

== Setup

The following steps from the setup have to be followed every time before running anything in the following sections. An exception is installing the dependencies which has to be done only the first time.

=== Launch Julia

To activate the environment, navigate to the main directory `/ReCo.jl` and then run the following to launch Julia:

[source,bash]
----
cd ReCo.jl
julia --threads auto
----

`auto` automatically sets the number of threads to use. If you want to use a specific number `N` of threads, replace `auto` with `N`.

=== Activating environment

After launching Julia, the package environment has to be activated by running the following in the REPL:

[source,julia]
----
using Pkg
Pkg.activate(".")
----

=== Install/update dependencies

After activating the package environment, run the following to install/update the package dependencies:

[source,julia]
----
Pkg.update()
----

This step has to be done by running the line above for the first time using the package to install its dependencies. Running the line above after the first time will update the installed dependencies which is optional.

=== Import the package

You can import the package by running:

[source,julia]
----
using ReCo
----

This will export the package's functions that are intended to be used by the end user.

== Help mode

To access the documentation of the presented package functions further in this README, run `using ReCo` first. Then, enter the help mode by pressing `?` in the REPL. Now, enter the function's name followed by enter to see its documentation.

== Run a simulation

Initialize a simulation with 100 particles having a self-propulsion velocity of 40.0 and return the relative path to the simulation directory:

[source,julia]
----
sim_dir = init_sim(100, 40.0)
----

Run the simulation:

[source,julia]
----
run_sim(sim_dir, duration=20.0)
----

The values for the number of particles, self-propulsion velocity and simulation duration are used here as an example. For more information about possible values and other optional arguments, see the documentation of `init_sim` or `run_sim`.

== Simulation visualization

=== Animation

To generate an animation of a simulation, run the following:

[source,julia]
----
animate(sim_dir)
----

The function's documentation includes all possible optional arguments and where the output can be found.

=== Snapshot plot

To plot only one snapshot of a simulation, run the following:

[source,julia]
----
plot_snapshot(sim_dir)
----

This will ask for the number of the snapshot to plot out of the total number of snapshots. The function's documentation includes all possible optional arguments and where the output can be found.

== Run a reinforcement learning process

Run a reinforcement learning process and return the environment helper and the the path of the process directory relative to the directory `ReCo.jl`:
[source,julia]
----
env_helper, rl_dir = run_rl(ENVTYPE)
----

`ENVTYPE` has to be replaced by one of the environments named after the file names in the directory `ReCo.jl/RL/Envs`, for example: `LocalCOMEnv`. A short description of an environment is included at the beginning of the corresponding file.

The documentation of `run_rl` includes all possible optional arguments.

=== Q-matrix

`env_helper` has the abstract type `EnvHelper`. To access the Q-matrix, enter the following:

[source,julia]
----
env_helper.shared.agent.policy.learner.approximator.table
----

To generate a LaTeX table with the states and actions combintation names for the Q-matrix, run the following:

[source,julia]
----
include("src/RL/latex_table.jl")
latex_rl_table(env_helper, FILENAME_WITHOUT_EXTENSION)
----

`FILENAME_WITHOUT_EXTENSION` has to be replaced by the wanted file name without extension of the `.tex` file. The function's documentation explains where the output is placed.

The output file can be used in a LaTeX document:

[source,latex]
----
\input{FILENAME_WITHOUT_EXTENSION}
----

=== Rewards

To access the rewards, run the following:

[source,julia]
----
env_helper.shared.hook.rewards
----

To plot the rewards, run the following:

[source,julia]
----
plot_rewards(rl_dir)
----

The function's documentation explains where the output is placed.

=== Mean kappa

To plot the mean of kappa as the ratio of the eigenvalues of the gyration tensor, run the following:

[source,julia]
----
include("analysis/mean_kappa.jl")
plot_mean_κ(; rl_dir=rl_dir, n_last_episodes=N_LAST_EPISODES)
----

`N_LAST_EPISODES` is the number of the last episodes of the learning process to average over. The function's documentation explains where the output is placed.

== Run analysis

=== Mean squared displacement and random walk

[source,julia]
----
include("analysis/mean_squared_displacement.jl")
run_msd_analysis()
run_random_walk()
----

The output is `ReCo.jl/exports/graphics/mean_squared_displacement.pdf` and `ReCo.jl/exports/graphics/random_walk.pdf`.

=== Radial distribution function

[source,julia]
----
include("analysis/radial_distribution_function/radial_distribution_function.jl")
run_radial_distribution_analysis()
----

The output is `ReCo.jl/exports/graphics/radial_distribution.pdf` and `ReCo.jl/exports/graphics/radial_distribution_all_vs.pdf`.

=== Reward discount analysis

[source,julia]
----
include("analysis/reward_discount_analysis.jl")
run_reward_discount_analysis()
----

The output is `ReCo.jl/exports/graphics/reward_discount_analysis.pdf`.

== Graphics

The directory `ReCo.jl/graphics` has some Julia files that generate graphics related to this package. The function in every file that has to be run to generate the corresponding graphics starts with `plot_` or `gen_`. The output is placed in `ReCo.jl/exports/graphics`.