From cc66f7359fdb4bda36bb44e7283173f2ab016ca3 Mon Sep 17 00:00:00 2001 From: Craig Oates Date: Thu, 23 Mar 2023 04:20:16 +0000 Subject: [PATCH] update README.org file. --- README.org | 193 +++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 179 insertions(+), 14 deletions(-) diff --git a/README.org b/README.org index 07f2777..2fcd06a 100644 --- a/README.org +++ b/README.org @@ -71,7 +71,7 @@ Ritherdon via the server (/AWS/). The (AWS) server stored every reading taken in a SQLite database and this project pokes and prods at the data -- to plot charts/graphs. -** Design Notes and Trade-off Decisions +** Design Notes and Trade-Off Decisions 1. Essentially, this project is about taking the data from =data/lm1-exhibiton-all.csv= and =data/lm2-exhibiton-all.csv= and producing @@ -85,19 +85,184 @@ project pokes and prods at the data -- to plot charts/graphs. 2. You will need to split the .csv files up yourself after you have cloned the repository, using the scripts mentioned in point 2. 3. *The database containing the actual data is not included with this repository.* -4. The database used for the /No Gaps in the Line/ exhibition is approximately - 500MB and I thought it was unreasonable to expect people to download and work - with a repository of that size -- for a repository of this nature. -5. The data exported from the database contains the data between 2021-06-13 + 1. The database used for the /No Gaps in the Line/ exhibition is approximately + 500MB and I thought it was unreasonable to expect people to download and work + with a repository of that size -- for a repository of this nature. +4. The data exported from the database contains the data between 2021-06-13 (13^th June, 2021) and 2021-08-01 (1^st August, 2021) for both Light Meters - (the length of the exhibition). -6. Chose to work with .csv files out of convenience more than anything else -- - easiest format to export the data to from the SQlite database. -7. I used [[https://en.wikipedia.org/wiki/Bash_(Unix_shell)][Bash]], [[https://en.wikipedia.org/wiki/AWK][Awk]] and [[https://github.com/BurntSushi/ripgrep][Ripgrep]] (/rg/), also, out of convenience, they were + (the length of the exhibition): There is (not much) more (test) data in the + database but the data selected/exported seemed the most appropriate decision. +5. I chose to work with .csv files out of convenience more than anything else -- + it is the easiest format to export the data to from the SQlite database. +6. I used [[https://en.wikipedia.org/wiki/Bash_(Unix_shell)][Bash]], [[https://en.wikipedia.org/wiki/AWK][Awk]] and [[https://github.com/BurntSushi/ripgrep][Ripgrep]] (/rg/), also, out of convenience, they were already on my computer. -8. I used [[https://bokeh.org/][Bokeh]] because I have already used it and it is the only thing I know +7. I used [[https://bokeh.org/][Bokeh]] because I have already used it and it is the only thing I know which can create interactive charts as individual HTML files, which I can - just email to someone who is not comfortable with computers. -9. I used [[https://www.python.org/][Python]] because of Bokeh. -10. Overall, Nicola wants to work with the charts this data produces so - any decisions made should be in service to that end. + share with anyone not comfortable with computers. +8. I used [[https://www.python.org/][Python]] because of Bokeh. +9. Overall, Nicola wants to work with the charts this data produces so + any decisions made should be in service to that end. +10. I have taken a hard-coded approach to filenames with the code because the + code is not the main objective here, the charts are; *Long-term flexibility + and maintenance is not a concern here.* + +** Set-Up and Using the Code + +Open your terminal, making sure you are in the directory you want the repository +cloned to. + +*The Bash (/.sh/) scripts need calling before the Python (/.py/) ones.* You need to +process the =lm1-exhibition-all.csv= and =lm2-exhibition-all.csv= file first because +the Python (/.py/) scripts assumes certain files are already in the =/data= +directory. + +#+begin_src shell + cd + git clone https://git.abbether.net/return-to-ritherdon/ritherdon-charts.git + cd ritherdon-charts + + # You need to split the two files in /data first... + ./separator.sh + ./totalilator.sh + + # You should have new files and directories in /data now... + + # Then create a Python virtual-environment (venv) to make the charts... + # I've named the one here 'venv' (the second one) and stored it in the root of + # this project's directory. + python3 -m venv venv + + # Activate the virtual-environment... + source venv/bin/activate + + # You should see '(venv)' in your (terminal) promt, for example, + # (venv) yourname@yourpc:~/local-dev/ritherdon-charts$ + + # Install Python dependencies/packages via pip... + pip install -r requirements.txt +#+end_src + +When the packages have finished installed (via ~pip~), you should be ready to +go. From there, you can simply call the Python (/.py/) scripts (from terminal with +the venv. activated). For example, + +#+begin_src shell + # Make sure you are in the terminal with the virtual-environment activated... + python lm1-hourly-totals.py + # Output from script... + + python daily-totals.py + # Output from script... +#+end_src + +When you have finished, you will need to deactivate your Python +virtual-environment. You can do that by entering ~disable~ in your terminal. You +should see the ~(venv)~ part of your prompt removed. + +#+begin_src shell + # Before you disable your Python virtual-environment (venv)... + # (venv) yourname@yourpc:~/local-dev/ritherdon-charts$ + + # Disable your Python venv. + disable + + # After you have disabled your Python virtual-environment (venv)... + # yourname@yourpc:~/local-dev/ritherdon-charts$ +#+end_src + +** Working with the Files/Data Produced After Running the Project's Scripts + +When you clone the repository, you will find the =/data= directory will have the +following layout, + +#+begin_src shell :results code + tree -L 1 data +#+end_src + +#+RESULTS: +#+begin_src shell +data +├── lm1-exhibiton-all.csv # Approx. 60MB +└── lm2-exhibiton-all.csv # Approx. 96MB + +0 directories, 2 files +#+end_src + +*The =/output= directory not exist before you run the scripts.* + +After you run the Bash scripts (~seperator.sh~ and ~totalilator.sh~), you should see +something like the following in the =/data= directory, + +#+begin_src shell :results code + tree -L 2 data +#+end_src + +#+RESULTS: +#+begin_src shell + data + ├── light-meter-1 + │   ├── 2021-06-13 # Directory of readings taken per hour taken on 13/06/2021. + │   ├── 2021-06-13.csv # File containing all the reading taken on 13th June 2021. + │   ├── 2021-06-14 # Directory of readings taken per hour taken on 14/06/2021. + │   ├── 2021-06-14.csv # File containing all the reading taken on 14th June 2021. + # More files and folder here... + │   ├── 2021-07-30 # Directory of readings taken per hour taken on 30/07/2021 + │   └── 2021-07-30.csv # File containing all the reading taken on 30th July 2021. + │ + ├── light-meter-1-daily-totals.csv # Total number of readings recorded for each day. + ├── light-meter-1-hourly-totals + │   ├── 2021-06 # Directory containing files with hourly totals (per day) for June. + │   └── 2021-07 # Directory containing files with hourly totals (per day) for July. + │ + ├── light-meter-2 + │   ├── 2021-06-13 # Directory of readings taken per hour taken on 13/06/2021. + │   ├── 2021-06-13.csv # File containing all the reading taken on 13th June 2021. + │   ├── 2021-06-14 # Directory of readings taken per hour taken on 14/06/2021. + │   ├── 2021-06-14.csv # File containing all the reading taken on 14th June 2021. + # More files and folders here... + │   ├── 2021-07-30 # Directory of readings taken per hour taken on 30/07/2021. + │   └── 2021-07-30.csv # File containing all the reading taken on 30th July 2021. + │ + ├── light-meter-2-daily-totals.csv # Total number of readings recorded for each day. + ├── light-meter-2-hourly-totals + │   ├── 2021-06 # Directory containing files with hourly totals (per day) for June. + │   └── 2021-07 # Directory containing files with hourly totals (per day) for July. + │ + ├── lm1-exhibiton-all.csv # Original file. + └── lm2-exhibiton-all.csv # Original file. + + 104 directories, 100 files +#+end_src + +*The =/output= directory should now exist in the project's root directory -- with +nothing in it.* This is where the charts will go, after running the Python (/.py/) +scripts. The ~separator.sh~ script created it. The project's root directory should +now look like the following, + +#+begin_src shell :results code + tree -L 1 +#+end_src + +#+RESULTS: +#+begin_src shell +. +├── assets +├── dailies-overlayed.py +├── dailies-side-by-side.py +├── daily-breakdowns.py +├── daily-totals.py +├── data # This should have a load of new files and directories... +├── day-to-day-comparisons.py +├── LICENSE +├── lm1-hourly-totals.py +├── lm2-hourly-totals.py +├── output # This should now exist and charts go in here... +├── README.org +├── requirements.txt +├── separator.sh +├── sql-statements.sql +├── totalilator.sh +└── venv # See 'Set-Up and Using the Code' section if you do not see this... + +4 directories, 13 files +#+end_src