= ReCo.jl
:source-highlighter: highlight.js
:highlightjs-languages: bash, julia

image:https://img.shields.io/badge/code%20style-blue-4495d1.svg[Code Style: Blue, link=https://github.com/invenia/BlueStyle]

**Re**inforcement learning of **co**llective behavior.

== Setup

The following steps from the setup have to be followed every time before running anything in the following sections. An exception is installing the dependencies which has to be done only the first time.

=== Launch Julia

To activate the environment, navigate to the main directory `ReCo.jl` and then run the following to launch Julia:

[source,bash]
----
cd ReCo.jl
julia --threads auto
----

`auto` automatically sets the number of threads to use. If you want to use a specific number `N` of threads, replace `auto` with `N`.

=== Activating environments

After launching Julia, the package environment has to be activated by running the following in the REPL:

[source,julia]
----
using Pkg
Pkg.activate(".")
----

`"."` stands for the current path which has to be the path of the package directory `ReCo.jl`.

If specified otherwise, an environment in a subdirectory has to be activated instead of the package environment. To activate an environment of a subdirectory `SUBDIRECTORY`, run the following:

[source,julia]
----
cd("SUBDIRECTORY")
using Pkg
Pkg.activate(".")
----

=== Install/update dependencies

After activating the package environment, run the following to install/update the package dependencies:

[source,julia]
----
Pkg.update()
----

This step has to be done by running the line above for the first time using the package to install its dependencies. Running the line above after the first time will update the installed dependencies which is optional.

This step has to be done **additionally** after activating an environment in a subdirectory.

=== Import the package

You can import the package by running:

[source,julia]
----
using ReCo
----

This will export the package's functions that are intended to be used by the end user.

== Help mode

To access the documentation of the presented package functions further in this README, run `using ReCo` first. Then, enter the help mode by pressing `?` in the REPL. Now, enter the function's name followed by enter to see its documentation.

== Run a simulation

Initialize a simulation with 100 particles having a self-propulsion velocity of 40.0 and return the relative path to the simulation directory:

[source,julia]
----
sim_dir = init_sim(; n_particles=100, v₀=40.0)
----

Run the simulation:

[source,julia]
----
run_sim(sim_dir; duration=20.0)
----

The values for the number of particles, self-propulsion velocity and simulation duration are used here as an example. For more information about possible values and other optional arguments, see the documentation of `init_sim` or `run_sim`.

== Simulation visualization
For visualization, the environment of the subdirectory `visualization` has to be activated.

=== Animation

To generate an animation of a simulation, run the following:

[source,julia]
----
include("Animation.jl")
using .Animation
animate(sim_dir)
----

The function's documentation includes all possible optional arguments and where the output can be found.

=== Snapshot plot

To plot only one snapshot of a simulation, run the following:

[source,julia]
----
include("SnapshotPlot.jl")
using .SnapshotPlot
plot_snapshot(sim_dir)
----

This will ask for the number of the snapshot to plot out of the total number of snapshots. The function's documentation includes all possible optional arguments and where the output can be found.

== Run a reinforcement learning process

Run a reinforcement learning process and return the environment helper and the the relative path of the process directory:

[source,julia]
----
env_helper, rl_dir = run_rl(ENVTYPE)
----

`ENVTYPE` has to be replaced by one of the environments named after the file names in the directory `RL/Envs`, for example: `LocalCOMEnv`. A short description of an environment is included at the beginning of the corresponding file.

The documentation of `run_rl` includes all possible optional arguments.

=== Q-matrix

`env_helper` has the abstract type `EnvHelper`. To access the Q-matrix, enter the following:

[source,julia]
----
env_helper.shared.agent.policy.learner.approximator.table
----

=== Rewards

To access the rewards, run the following:

[source,julia]
----
env_helper.shared.hook.rewards
----

To plot the rewards, activate the environment of the subdirectory `visualization` and run the following:

[source,julia]
----
include("RewardsPlot.jl")
using .RewardsPlot
plot_rewards(rl_dir)
----

The function's documentation explains where the output is placed.

=== Mean kappa

To plot the mean of kappa as the ratio of the eigenvalues of the gyration tensor, activate the environment of the subdirectory `analysis` and run the following:

[source,julia]
----
include("mean_kappa.jl")
plot_mean_κ(; rl_dir=rl_dir, n_last_episodes=N_LAST_EPISODES)
----

`N_LAST_EPISODES` is the number of the last episodes of the learning process to average over. The function's documentation explains where the output is placed.

== Run analysis

First, activate the environment of the subdirectory `analysis`.

=== Mean squared displacement and random walk

[source,julia]
----
include("mean_squared_displacement.jl")
run_msd_analysis()
run_random_walk()
----

The output is `exports/graphics/mean_squared_displacement.pdf` and `exports/graphics/random_walk.pdf`.

=== Radial distribution function

[source,julia]
----
include("radial_distribution_function/radial_distribution_function.jl")
run_radial_distribution_analysis()
----

The output is `exports/graphics/radial_distribution.pdf` and `exports/graphics/radial_distribution_all_vs.pdf`.

=== Reward discount analysis

[source,julia]
----
include("reward_discount_analysis.jl")
run_reward_discount_analysis()
----

The output is `exports/graphics/reward_discount_analysis.pdf`.

== Graphics

The subdirectory `graphics` has some Julia files that generate graphics related to this package. The function in every file that has to be run to generate the corresponding graphics starts with `plot_` or `gen_`. The output is placed in `exports/graphics`. The subdirectory of a file has an environment that has to be activated first.