From df2bac4ab45bc731e12e4c12e210954fd031d96c Mon Sep 17 00:00:00 2001 From: Craig Oates Date: Thu, 23 Mar 2023 05:05:56 +0000 Subject: [PATCH] update README.org file. --- README.org | 113 ++++++++++++++++++++++++++++++++--------------------- 1 file changed, 69 insertions(+), 44 deletions(-) diff --git a/README.org b/README.org index 2fcd06a..3003586 100644 --- a/README.org +++ b/README.org @@ -18,13 +18,14 @@ Gaps in the Line]] exhibition by [[http://www.nicolaellis.com][Nicola Ellis]], h Manchester, U.K. This project ties into a larger collection of software projects related to the -/Personal Flash in Real-Time/ artworks. Which, in turn, is a small piece of the -much larger [[https://www.nicolaellisandritherdon.com/][Return to Ritherdon]] project (devised and completed by Nicola -Ellis). For more information on the software-side of the /Personal Flash in -Real-Time/ artworks, please use the links below: +/Personal Flash in Real-Time/ artworks. And, those artworks are a small piece of +the much larger [[https://www.nicolaellisandritherdon.com/][Return to Ritherdon]] project (devised and completed by Nicola +Ellis). For more information on the artworks and where they sit in the larger +project, please use the links below: - [[https://git.abbether.net/return-to-ritherdon/rtr-docs][rtr-docs]] (The documentation repository for all the /Person Flash in Real-Time/ - software projects) + software projects, with an overview of how they tie into the Return to + Ritherdon project) - [[https://git.abbether.net/return-to-ritherdon][Return to Ritherdon Org. Page]] (The 'home page' for the /Return to Ritherdon/ project on this site, containing a list of all the publicly available repositories) @@ -35,16 +36,16 @@ manufacturing electrical enclosures and other related products. So, if you have spent any time in the U.K. and seen one of those green electrical boxes lurking on a street corner, there is a good chance these folks made it. -*NOTE: This project does not contain documentation in the [[https://git.abbether.net/return-to-ritherdon/rtr-docs][rtr-docs]] repository.* It -is a self-contained mini-project which is not directly related to the /Personal -Flash in Real-Time/ artworks. +*NOTE: This project does not contain documentation in the [[https://git.abbether.net/return-to-ritherdon/rtr-docs][rtr-docs]] repository.* +This is a self-contained mini-project which is not /directly related/ to the +/Personal Flash in Real-Time/ artworks. *** Examples/Screenshots At the time of writing, the scripts in this repository produce over one-hundred charts/files. So, here are a selection of the types of charts produced after -processing the data in =data/lm1-exhibiton-all.csv= and -=/data/lm2-exhibition-all.csv=. +processing the data in the =data/lm1-exhibiton-all.csv= and +=/data/lm2-exhibition-all.csv= files. [[file:assets/daily-totals.png]] @@ -59,14 +60,14 @@ processing the data in =data/lm1-exhibiton-all.csv= and ** Overview of the /Personal Flash in Real-Time/ Artworks /Personal Flash in Real-Time/ consists of two artworks, named /Personal Flash in -Real-Time (Andy)/ and /Personal Flash in Real-Time (Tony)/. Each one measured the -light in the welding booths in the Ritherdon Factory and forwarded those +Real-Time (Andy)/ and /Personal Flash in Real-Time (Tony)/. Each one measured +the light in the welding booths in the Ritherdon Factory and forwarded those readings on to a server running in Amazon's 'cloud' -- see [[https://aws.amazon.com/][Amazon Web Services]] -(/AWS/) for more information. While this was happening, two sets of lights -residing in Castlefield Gallery would turn on and off whenever the system +(AWS) for more information. While this was happening, two sets of lights, +residing in Castlefield Gallery, would turn on and off whenever the system detected someone welding in Ritherdon. The would happen because the Relays, controlling the lights would receive the latest Light Meter readings taken in -Ritherdon via the server (/AWS/). +Ritherdon via the server (AWS). The (AWS) server stored every reading taken in a SQLite database and this project pokes and prods at the data -- to plot charts/graphs. @@ -79,10 +80,10 @@ project pokes and prods at the data -- to plot charts/graphs. 2. The =separator.sh= and =totalilator.sh= scripts split the .csv files, mentioned above, into smaller files in an attempt to make them (.csv files) easier to work with on average hardware. - 1. On top of that,I have only committed the .csv files mentioned in point 1 + 1. On top of that,I have only committed the CSV files mentioned in point 1 to the repository as a means to reduce the clutter in the repositories Git commit history. - 2. You will need to split the .csv files up yourself after you have cloned + 2. You will need to split the CSV files up yourself after you have cloned the repository, using the scripts mentioned in point 2. 3. *The database containing the actual data is not included with this repository.* 1. The database used for the /No Gaps in the Line/ exhibition is approximately @@ -92,28 +93,28 @@ project pokes and prods at the data -- to plot charts/graphs. (13^th June, 2021) and 2021-08-01 (1^st August, 2021) for both Light Meters (the length of the exhibition): There is (not much) more (test) data in the database but the data selected/exported seemed the most appropriate decision. -5. I chose to work with .csv files out of convenience more than anything else -- +5. I chose to work with CSV files out of convenience more than anything else -- it is the easiest format to export the data to from the SQlite database. -6. I used [[https://en.wikipedia.org/wiki/Bash_(Unix_shell)][Bash]], [[https://en.wikipedia.org/wiki/AWK][Awk]] and [[https://github.com/BurntSushi/ripgrep][Ripgrep]] (/rg/), also, out of convenience, they were +6. I used [[https://en.wikipedia.org/wiki/Bash_(Unix_shell)][Bash]], [[https://en.wikipedia.org/wiki/AWK][Awk]] and [[https://github.com/BurntSushi/ripgrep][Ripgrep]] (rg), also, out of convenience, they were already on my computer. 7. I used [[https://bokeh.org/][Bokeh]] because I have already used it and it is the only thing I know which can create interactive charts as individual HTML files, which I can share with anyone not comfortable with computers. 8. I used [[https://www.python.org/][Python]] because of Bokeh. -9. Overall, Nicola wants to work with the charts this data produces so - any decisions made should be in service to that end. +9. Overall, Nicola wants to work with the charts produced by this data so any + decisions made should be in service to that end. 10. I have taken a hard-coded approach to filenames with the code because the - code is not the main objective here, the charts are; *Long-term flexibility - and maintenance is not a concern here.* + code is not the main objective here, the charts are; In other words, + *long-term flexibility and maintenance is not a concern here.* ** Set-Up and Using the Code Open your terminal, making sure you are in the directory you want the repository cloned to. -*The Bash (/.sh/) scripts need calling before the Python (/.py/) ones.* You need to -process the =lm1-exhibition-all.csv= and =lm2-exhibition-all.csv= file first because -the Python (/.py/) scripts assumes certain files are already in the =/data= +*The Bash (.sh) scripts need calling before the Python (.py) ones.* You need to +process the =lm1-exhibition-all.csv= and =lm2-exhibition-all.csv= files first because +the Python (.py) scripts assumes certain files are already in the =/data= directory. #+begin_src shell @@ -143,8 +144,8 @@ directory. #+end_src When the packages have finished installed (via ~pip~), you should be ready to -go. From there, you can simply call the Python (/.py/) scripts (from terminal with -the venv. activated). For example, +go. From there, you can simply call the Python (.py) scripts (from terminal with +the =venv= activated). For example, #+begin_src shell # Make sure you are in the terminal with the virtual-environment activated... @@ -155,22 +156,30 @@ the venv. activated). For example, # Output from script... #+end_src -When you have finished, you will need to deactivate your Python -virtual-environment. You can do that by entering ~disable~ in your terminal. You -should see the ~(venv)~ part of your prompt removed. +When you have finished, you will need to deactivate the virtual-environment. You +can do that by entering ~disable~ in your terminal. You should see the ~(venv)~ +part of your prompt removed. #+begin_src shell - # Before you disable your Python virtual-environment (venv)... + # Before you disable your Python's virtual-environment (venv)... # (venv) yourname@yourpc:~/local-dev/ritherdon-charts$ # Disable your Python venv. disable - # After you have disabled your Python virtual-environment (venv)... + # After you have disabled your Python's virtual-environment (venv)... # yourname@yourpc:~/local-dev/ritherdon-charts$ #+end_src -** Working with the Files/Data Produced After Running the Project's Scripts +From here, you can either write you own scripts to form new charts or just play +with the CSV files in something like Microsoft Excel or Libre Office Calc. + +** Working with the Files/Data Produced After Running the Project's Bash Scripts + +#+begin_quote +=/data= stores the CSV files and =/output= stores the charts. Run ~./separator.sh~ +to get started. +#+end_quote When you clone the repository, you will find the =/data= directory will have the following layout, @@ -188,9 +197,13 @@ data 0 directories, 2 files #+end_src -*The =/output= directory not exist before you run the scripts.* +This =/data= directory is responsible for storing the /raw/ data (I.E. the CSV +files). The charts, created via the Python (.py) scripts, reside in the +=/output= directory. + +*The =/output= directory should not exist until you run ~./separator.sh~.* -After you run the Bash scripts (~seperator.sh~ and ~totalilator.sh~), you should see +After you run the Bash scripts (~./seperator.sh~ and ~./totalilator.sh~), you should see something like the following in the =/data= directory, #+begin_src shell :results code @@ -201,12 +214,12 @@ something like the following in the =/data= directory, #+begin_src shell data ├── light-meter-1 - │   ├── 2021-06-13 # Directory of readings taken per hour taken on 13/06/2021. + │   ├── 2021-06-13 # Directory of readings taken per hour taken on 13/06/2021. │   ├── 2021-06-13.csv # File containing all the reading taken on 13th June 2021. - │   ├── 2021-06-14 # Directory of readings taken per hour taken on 14/06/2021. + │   ├── 2021-06-14 # Directory of readings taken per hour taken on 14/06/2021. │   ├── 2021-06-14.csv # File containing all the reading taken on 14th June 2021. # More files and folder here... - │   ├── 2021-07-30 # Directory of readings taken per hour taken on 30/07/2021 + │   ├── 2021-07-30 # Directory of readings taken per hour taken on 30/07/2021 │   └── 2021-07-30.csv # File containing all the reading taken on 30th July 2021. │ ├── light-meter-1-daily-totals.csv # Total number of readings recorded for each day. @@ -215,12 +228,12 @@ something like the following in the =/data= directory, │   └── 2021-07 # Directory containing files with hourly totals (per day) for July. │ ├── light-meter-2 - │   ├── 2021-06-13 # Directory of readings taken per hour taken on 13/06/2021. + │   ├── 2021-06-13 # Directory of readings taken per hour taken on 13/06/2021. │   ├── 2021-06-13.csv # File containing all the reading taken on 13th June 2021. - │   ├── 2021-06-14 # Directory of readings taken per hour taken on 14/06/2021. + │   ├── 2021-06-14 # Directory of readings taken per hour taken on 14/06/2021. │   ├── 2021-06-14.csv # File containing all the reading taken on 14th June 2021. # More files and folders here... - │   ├── 2021-07-30 # Directory of readings taken per hour taken on 30/07/2021. + │   ├── 2021-07-30 # Directory of readings taken per hour taken on 30/07/2021. │   └── 2021-07-30.csv # File containing all the reading taken on 30th July 2021. │ ├── light-meter-2-daily-totals.csv # Total number of readings recorded for each day. @@ -235,7 +248,7 @@ something like the following in the =/data= directory, #+end_src *The =/output= directory should now exist in the project's root directory -- with -nothing in it.* This is where the charts will go, after running the Python (/.py/) +nothing in it.* This is where the charts will go, after running the Python (.py) scripts. The ~separator.sh~ script created it. The project's root directory should now look like the following, @@ -256,7 +269,7 @@ now look like the following, ├── LICENSE ├── lm1-hourly-totals.py ├── lm2-hourly-totals.py -├── output # This should now exist and charts go in here... +├── output # *NEW* This should now exist and charts go in here... ├── README.org ├── requirements.txt ├── separator.sh @@ -266,3 +279,15 @@ now look like the following, 4 directories, 13 files #+end_src + +From here, you can either create some charts with the Python (.py) scripts and +take a look at them in =/output= or open the CSV files in =/data= and inspect the +data. + +*** Overview of the CSV files in =/data= + +I need to write this... + +*** Overview of Scripts and Charts + +I need to write this...