Browse Source

update README.org file.

master
Craig Oates 1 year ago
parent
commit
cc66f7359f
  1. 193
      README.org

193
README.org

@ -71,7 +71,7 @@ Ritherdon via the server (/AWS/).
The (AWS) server stored every reading taken in a SQLite database and this
project pokes and prods at the data -- to plot charts/graphs.
** Design Notes and Trade-off Decisions
** Design Notes and Trade-Off Decisions
1. Essentially, this project is about taking the data from
=data/lm1-exhibiton-all.csv= and =data/lm2-exhibiton-all.csv= and producing
@ -85,19 +85,184 @@ project pokes and prods at the data -- to plot charts/graphs.
2. You will need to split the .csv files up yourself after you have cloned
the repository, using the scripts mentioned in point 2.
3. *The database containing the actual data is not included with this repository.*
4. The database used for the /No Gaps in the Line/ exhibition is approximately
500MB and I thought it was unreasonable to expect people to download and work
with a repository of that size -- for a repository of this nature.
5. The data exported from the database contains the data between 2021-06-13
1. The database used for the /No Gaps in the Line/ exhibition is approximately
500MB and I thought it was unreasonable to expect people to download and work
with a repository of that size -- for a repository of this nature.
4. The data exported from the database contains the data between 2021-06-13
(13^th June, 2021) and 2021-08-01 (1^st August, 2021) for both Light Meters
(the length of the exhibition).
6. Chose to work with .csv files out of convenience more than anything else --
easiest format to export the data to from the SQlite database.
7. I used [[https://en.wikipedia.org/wiki/Bash_(Unix_shell)][Bash]], [[https://en.wikipedia.org/wiki/AWK][Awk]] and [[https://github.com/BurntSushi/ripgrep][Ripgrep]] (/rg/), also, out of convenience, they were
(the length of the exhibition): There is (not much) more (test) data in the
database but the data selected/exported seemed the most appropriate decision.
5. I chose to work with .csv files out of convenience more than anything else --
it is the easiest format to export the data to from the SQlite database.
6. I used [[https://en.wikipedia.org/wiki/Bash_(Unix_shell)][Bash]], [[https://en.wikipedia.org/wiki/AWK][Awk]] and [[https://github.com/BurntSushi/ripgrep][Ripgrep]] (/rg/), also, out of convenience, they were
already on my computer.
8. I used [[https://bokeh.org/][Bokeh]] because I have already used it and it is the only thing I know
7. I used [[https://bokeh.org/][Bokeh]] because I have already used it and it is the only thing I know
which can create interactive charts as individual HTML files, which I can
just email to someone who is not comfortable with computers.
9. I used [[https://www.python.org/][Python]] because of Bokeh.
10. Overall, Nicola wants to work with the charts this data produces so
any decisions made should be in service to that end.
share with anyone not comfortable with computers.
8. I used [[https://www.python.org/][Python]] because of Bokeh.
9. Overall, Nicola wants to work with the charts this data produces so
any decisions made should be in service to that end.
10. I have taken a hard-coded approach to filenames with the code because the
code is not the main objective here, the charts are; *Long-term flexibility
and maintenance is not a concern here.*
** Set-Up and Using the Code
Open your terminal, making sure you are in the directory you want the repository
cloned to.
*The Bash (/.sh/) scripts need calling before the Python (/.py/) ones.* You need to
process the =lm1-exhibition-all.csv= and =lm2-exhibition-all.csv= file first because
the Python (/.py/) scripts assumes certain files are already in the =/data=
directory.
#+begin_src shell
cd <INSERT YOUR PATH OF CHOICE HERE...>
git clone https://git.abbether.net/return-to-ritherdon/ritherdon-charts.git
cd ritherdon-charts
# You need to split the two files in /data first...
./separator.sh
./totalilator.sh
# You should have new files and directories in /data now...
# Then create a Python virtual-environment (venv) to make the charts...
# I've named the one here 'venv' (the second one) and stored it in the root of
# this project's directory.
python3 -m venv venv
# Activate the virtual-environment...
source venv/bin/activate
# You should see '(venv)' in your (terminal) promt, for example,
# (venv) yourname@yourpc:~/local-dev/ritherdon-charts$
# Install Python dependencies/packages via pip...
pip install -r requirements.txt
#+end_src
When the packages have finished installed (via ~pip~), you should be ready to
go. From there, you can simply call the Python (/.py/) scripts (from terminal with
the venv. activated). For example,
#+begin_src shell
# Make sure you are in the terminal with the virtual-environment activated...
python lm1-hourly-totals.py
# Output from script...
python daily-totals.py
# Output from script...
#+end_src
When you have finished, you will need to deactivate your Python
virtual-environment. You can do that by entering ~disable~ in your terminal. You
should see the ~(venv)~ part of your prompt removed.
#+begin_src shell
# Before you disable your Python virtual-environment (venv)...
# (venv) yourname@yourpc:~/local-dev/ritherdon-charts$
# Disable your Python venv.
disable
# After you have disabled your Python virtual-environment (venv)...
# yourname@yourpc:~/local-dev/ritherdon-charts$
#+end_src
** Working with the Files/Data Produced After Running the Project's Scripts
When you clone the repository, you will find the =/data= directory will have the
following layout,
#+begin_src shell :results code
tree -L 1 data
#+end_src
#+RESULTS:
#+begin_src shell
data
├── lm1-exhibiton-all.csv # Approx. 60MB
└── lm2-exhibiton-all.csv # Approx. 96MB
0 directories, 2 files
#+end_src
*The =/output= directory not exist before you run the scripts.*
After you run the Bash scripts (~seperator.sh~ and ~totalilator.sh~), you should see
something like the following in the =/data= directory,
#+begin_src shell :results code
tree -L 2 data
#+end_src
#+RESULTS:
#+begin_src shell
data
├── light-meter-1
   ├── 2021-06-13 # Directory of readings taken per hour taken on 13/06/2021.
   ├── 2021-06-13.csv # File containing all the reading taken on 13th June 2021.
   ├── 2021-06-14 # Directory of readings taken per hour taken on 14/06/2021.
   ├── 2021-06-14.csv # File containing all the reading taken on 14th June 2021.
# More files and folder here...
   ├── 2021-07-30 # Directory of readings taken per hour taken on 30/07/2021
   └── 2021-07-30.csv # File containing all the reading taken on 30th July 2021.
├── light-meter-1-daily-totals.csv # Total number of readings recorded for each day.
├── light-meter-1-hourly-totals
   ├── 2021-06 # Directory containing files with hourly totals (per day) for June.
   └── 2021-07 # Directory containing files with hourly totals (per day) for July.
├── light-meter-2
   ├── 2021-06-13 # Directory of readings taken per hour taken on 13/06/2021.
   ├── 2021-06-13.csv # File containing all the reading taken on 13th June 2021.
   ├── 2021-06-14 # Directory of readings taken per hour taken on 14/06/2021.
   ├── 2021-06-14.csv # File containing all the reading taken on 14th June 2021.
# More files and folders here...
   ├── 2021-07-30 # Directory of readings taken per hour taken on 30/07/2021.
   └── 2021-07-30.csv # File containing all the reading taken on 30th July 2021.
├── light-meter-2-daily-totals.csv # Total number of readings recorded for each day.
├── light-meter-2-hourly-totals
   ├── 2021-06 # Directory containing files with hourly totals (per day) for June.
   └── 2021-07 # Directory containing files with hourly totals (per day) for July.
├── lm1-exhibiton-all.csv # Original file.
└── lm2-exhibiton-all.csv # Original file.
104 directories, 100 files
#+end_src
*The =/output= directory should now exist in the project's root directory -- with
nothing in it.* This is where the charts will go, after running the Python (/.py/)
scripts. The ~separator.sh~ script created it. The project's root directory should
now look like the following,
#+begin_src shell :results code
tree -L 1
#+end_src
#+RESULTS:
#+begin_src shell
.
├── assets
├── dailies-overlayed.py
├── dailies-side-by-side.py
├── daily-breakdowns.py
├── daily-totals.py
├── data # This should have a load of new files and directories...
├── day-to-day-comparisons.py
├── LICENSE
├── lm1-hourly-totals.py
├── lm2-hourly-totals.py
├── output # This should now exist and charts go in here...
├── README.org
├── requirements.txt
├── separator.sh
├── sql-statements.sql
├── totalilator.sh
└── venv # See 'Set-Up and Using the Code' section if you do not see this...
4 directories, 13 files
#+end_src

Loading…
Cancel
Save