Browse Source

update README.org file.

master
Craig Oates 1 year ago
parent
commit
cc66f7359f
  1. 187
      README.org

187
README.org

@ -71,7 +71,7 @@ Ritherdon via the server (/AWS/).
The (AWS) server stored every reading taken in a SQLite database and this
project pokes and prods at the data -- to plot charts/graphs.
** Design Notes and Trade-off Decisions
** Design Notes and Trade-Off Decisions
1. Essentially, this project is about taking the data from
=data/lm1-exhibiton-all.csv= and =data/lm2-exhibiton-all.csv= and producing
@ -85,19 +85,184 @@ project pokes and prods at the data -- to plot charts/graphs.
2. You will need to split the .csv files up yourself after you have cloned
the repository, using the scripts mentioned in point 2.
3. *The database containing the actual data is not included with this repository.*
4. The database used for the /No Gaps in the Line/ exhibition is approximately
1. The database used for the /No Gaps in the Line/ exhibition is approximately
500MB and I thought it was unreasonable to expect people to download and work
with a repository of that size -- for a repository of this nature.
5. The data exported from the database contains the data between 2021-06-13
4. The data exported from the database contains the data between 2021-06-13
(13^th June, 2021) and 2021-08-01 (1^st August, 2021) for both Light Meters
(the length of the exhibition).
6. Chose to work with .csv files out of convenience more than anything else --
easiest format to export the data to from the SQlite database.
7. I used [[https://en.wikipedia.org/wiki/Bash_(Unix_shell)][Bash]], [[https://en.wikipedia.org/wiki/AWK][Awk]] and [[https://github.com/BurntSushi/ripgrep][Ripgrep]] (/rg/), also, out of convenience, they were
(the length of the exhibition): There is (not much) more (test) data in the
database but the data selected/exported seemed the most appropriate decision.
5. I chose to work with .csv files out of convenience more than anything else --
it is the easiest format to export the data to from the SQlite database.
6. I used [[https://en.wikipedia.org/wiki/Bash_(Unix_shell)][Bash]], [[https://en.wikipedia.org/wiki/AWK][Awk]] and [[https://github.com/BurntSushi/ripgrep][Ripgrep]] (/rg/), also, out of convenience, they were
already on my computer.
8. I used [[https://bokeh.org/][Bokeh]] because I have already used it and it is the only thing I know
7. I used [[https://bokeh.org/][Bokeh]] because I have already used it and it is the only thing I know
which can create interactive charts as individual HTML files, which I can
just email to someone who is not comfortable with computers.
9. I used [[https://www.python.org/][Python]] because of Bokeh.
10. Overall, Nicola wants to work with the charts this data produces so
share with anyone not comfortable with computers.
8. I used [[https://www.python.org/][Python]] because of Bokeh.
9. Overall, Nicola wants to work with the charts this data produces so
any decisions made should be in service to that end.
10. I have taken a hard-coded approach to filenames with the code because the
code is not the main objective here, the charts are; *Long-term flexibility
and maintenance is not a concern here.*
** Set-Up and Using the Code
Open your terminal, making sure you are in the directory you want the repository
cloned to.
*The Bash (/.sh/) scripts need calling before the Python (/.py/) ones.* You need to
process the =lm1-exhibition-all.csv= and =lm2-exhibition-all.csv= file first because
the Python (/.py/) scripts assumes certain files are already in the =/data=
directory.
#+begin_src shell
cd <INSERT YOUR PATH OF CHOICE HERE...>
git clone https://git.abbether.net/return-to-ritherdon/ritherdon-charts.git
cd ritherdon-charts
# You need to split the two files in /data first...
./separator.sh
./totalilator.sh
# You should have new files and directories in /data now...
# Then create a Python virtual-environment (venv) to make the charts...
# I've named the one here 'venv' (the second one) and stored it in the root of
# this project's directory.
python3 -m venv venv
# Activate the virtual-environment...
source venv/bin/activate
# You should see '(venv)' in your (terminal) promt, for example,
# (venv) yourname@yourpc:~/local-dev/ritherdon-charts$
# Install Python dependencies/packages via pip...
pip install -r requirements.txt
#+end_src
When the packages have finished installed (via ~pip~), you should be ready to
go. From there, you can simply call the Python (/.py/) scripts (from terminal with
the venv. activated). For example,
#+begin_src shell
# Make sure you are in the terminal with the virtual-environment activated...
python lm1-hourly-totals.py
# Output from script...
python daily-totals.py
# Output from script...
#+end_src
When you have finished, you will need to deactivate your Python
virtual-environment. You can do that by entering ~disable~ in your terminal. You
should see the ~(venv)~ part of your prompt removed.
#+begin_src shell
# Before you disable your Python virtual-environment (venv)...
# (venv) yourname@yourpc:~/local-dev/ritherdon-charts$
# Disable your Python venv.
disable
# After you have disabled your Python virtual-environment (venv)...
# yourname@yourpc:~/local-dev/ritherdon-charts$
#+end_src
** Working with the Files/Data Produced After Running the Project's Scripts
When you clone the repository, you will find the =/data= directory will have the
following layout,
#+begin_src shell :results code
tree -L 1 data
#+end_src
#+RESULTS:
#+begin_src shell
data
├── lm1-exhibiton-all.csv # Approx. 60MB
└── lm2-exhibiton-all.csv # Approx. 96MB
0 directories, 2 files
#+end_src
*The =/output= directory not exist before you run the scripts.*
After you run the Bash scripts (~seperator.sh~ and ~totalilator.sh~), you should see
something like the following in the =/data= directory,
#+begin_src shell :results code
tree -L 2 data
#+end_src
#+RESULTS:
#+begin_src shell
data
├── light-meter-1
   ├── 2021-06-13 # Directory of readings taken per hour taken on 13/06/2021.
   ├── 2021-06-13.csv # File containing all the reading taken on 13th June 2021.
   ├── 2021-06-14 # Directory of readings taken per hour taken on 14/06/2021.
   ├── 2021-06-14.csv # File containing all the reading taken on 14th June 2021.
# More files and folder here...
   ├── 2021-07-30 # Directory of readings taken per hour taken on 30/07/2021
   └── 2021-07-30.csv # File containing all the reading taken on 30th July 2021.
├── light-meter-1-daily-totals.csv # Total number of readings recorded for each day.
├── light-meter-1-hourly-totals
   ├── 2021-06 # Directory containing files with hourly totals (per day) for June.
   └── 2021-07 # Directory containing files with hourly totals (per day) for July.
├── light-meter-2
   ├── 2021-06-13 # Directory of readings taken per hour taken on 13/06/2021.
   ├── 2021-06-13.csv # File containing all the reading taken on 13th June 2021.
   ├── 2021-06-14 # Directory of readings taken per hour taken on 14/06/2021.
   ├── 2021-06-14.csv # File containing all the reading taken on 14th June 2021.
# More files and folders here...
   ├── 2021-07-30 # Directory of readings taken per hour taken on 30/07/2021.
   └── 2021-07-30.csv # File containing all the reading taken on 30th July 2021.
├── light-meter-2-daily-totals.csv # Total number of readings recorded for each day.
├── light-meter-2-hourly-totals
   ├── 2021-06 # Directory containing files with hourly totals (per day) for June.
   └── 2021-07 # Directory containing files with hourly totals (per day) for July.
├── lm1-exhibiton-all.csv # Original file.
└── lm2-exhibiton-all.csv # Original file.
104 directories, 100 files
#+end_src
*The =/output= directory should now exist in the project's root directory -- with
nothing in it.* This is where the charts will go, after running the Python (/.py/)
scripts. The ~separator.sh~ script created it. The project's root directory should
now look like the following,
#+begin_src shell :results code
tree -L 1
#+end_src
#+RESULTS:
#+begin_src shell
.
├── assets
├── dailies-overlayed.py
├── dailies-side-by-side.py
├── daily-breakdowns.py
├── daily-totals.py
├── data # This should have a load of new files and directories...
├── day-to-day-comparisons.py
├── LICENSE
├── lm1-hourly-totals.py
├── lm2-hourly-totals.py
├── output # This should now exist and charts go in here...
├── README.org
├── requirements.txt
├── separator.sh
├── sql-statements.sql
├── totalilator.sh
└── venv # See 'Set-Up and Using the Code' section if you do not see this...
4 directories, 13 files
#+end_src

Loading…
Cancel
Save